How I Structure New Projects In Golang

Summary

The video discusses setting up a scalable project structure for a Go web application using Echo as the framework. It advocates for a pragmatic approach focused on simplicity, avoiding over-engineering. The proposed structure includes folders for binaries, commands, database logic, handlers, business logic packages, shared types, and utilities. It demonstrates this structure in an app called Leue - a marketplace to sell recurring revenue for upfront cash. The video argues developers should be full stack and balance tradeoffs themselves rather than follow dogmatic paradigms.

Takeaways

• 💻 The video discusses transitioning a company's project, named Laue, from Next.js and TypeScript to Go and Svelte Kit due to a significant increase in funding and project scope.
• 🔨 The presenter emphasizes there's no perfect project structure in programming; instead, it's about choosing what works best for the team and being willing to adapt.
• 💡 Encourages a result-driven development mindset over stressing about tools, packages, and organization methods.
• 🛠️ Explains the Go project structure, including the use of a 'bin' folder for binaries, a 'cmd' folder for main package entry points, and other organizational strategies.
• 📰 Highlights the flexibility of Go frameworks for the backend, stating that the choice of framework (Echo, Gin, etc.) doesn't significantly impact the end result.
• 💥 Discusses performance concerns, suggesting that for most projects, performance won't be a limiting factor until a very large user base is reached.
• 📖 Offers insights into database interaction, the use of global variables for ease of use, and the pros and cons of different approaches to handling database queries.
• 🏆 Shares a pragmatic approach to coding, advocating for simplicity and focusing on getting results rather than adhering strictly to certain programming paradigms.
• 📈 Talks about the importance of integrating business logic and data handling correctly within the project's structure to ensure scalability and maintainability.
• 📚 Advises on the significance of having a balanced view on technologies and methodologies, emphasizing that what works well for one team or project may not for another.

Q & A

What is Leue and what was the original technology stack used to build it?

Leue is a company that the speaker created 3 years ago. The original project was built using Next.js for the front-end and TypeScript for the back-end.

Why did the speaker decide to rewrite Leue and change the technology stack?

The scope of the project changed dramatically after the company received a large amount of funding. The speaker wanted to scale up the project, so they decided to rewrite the back-end in Go and use SpellKit for the front-end.

What is the speaker's view on choosing frameworks and architectural patterns?

The speaker believes there is no one "perfect" choice. Developers should use what feels right for their needs instead of forcing specific paradigms. Performance differences rarely matter until an application reaches very large scale.

How does the speaker structure Go projects and why?

The speaker uses a simple and modular structure with folders for handlers, packages, data, types, etc. This makes it easy to understand and scale up the project over time as needed.

Why does the speaker use a global database connection variable?

For ease of use across different packages. The speaker argues this is okay if used carefully and may not matter much for testing since database access should be integration tested.

What is the purpose of the CMD folder and how does it work?

The CMD folder contains main.go files that bootstrap different entry points into the application, like API servers. The binaries are then built into the bin folder for execution.

Where is business logic implemented in this structure?

Most business logic is contained in the packages folder, separate from the database access in the data folder and HTTP handler functions.

How are endpoints and handlers organized?

Handlers are organized by HTTP method into handler functions. Endpoints are grouped by concepts like buy/sell side using middleware.

How does the front-end connect to the Go back-end?

The front-end uses SpellKit and calls API endpoints exposed from the Echo server in the main CMD API bootstrap.

What is the speaker's view on specialization between front-end and back-end development?

The speaker argues developers should have full stack capabilities instead of identifying as a front-end or back-end specialist, since it's all just writing code.

Outlines

00:00

😊 Introducing the leue project and its transition to version 2.0

The paragraph introduces leue, a company created 3 years ago that is transitioning from a Next.js and TypeScript stack to Golang and SvelteKit. The scope and funding have increased dramatically, necessitating scaling up the backend in Golang. The video will focus on setting up a Golang project structure.

05:02

🤔 Using simple scripts over CLI packages for database management

The paragraph discusses using simple scripts to handle things like dropping databases and running migrations instead of CLI packages. These scripts are run through Make commands for convenience. A makefile is simple and commonly installed on Linux.

10:02

😉 Embracing Bun as the ORM over sqlc despite drawbacks

The paragraph analyzes the tradeoffs between using sqlc for type-safe queries versus Bun for more dynamic queries. Despite drawbacks like losing type safety, Bun is ultimately embraced for this project.

15:03

🧐 Separating code by responsibility into different packages

The paragraph explains the structure of separating code into different packages based on responsibility - data layer for db access, handlers for HTTP handling, package for business logic, types for shared types, and util for helpers. This structure scales well.

20:04

💪 Being a well-rounded developer to create full-stack applications

The paragraph advocates for being a full-stack developer comfortable on both frontend and backend instead of limiting yourself. Any language or framework has tradeoffs - focus on weighing pros and cons based on your own needs.

Keywords

💡project structure

This refers to how the code and files of a software project are organized on disk. The video discusses principles and tradeoffs in structuring Go projects, emphasizing flexibility based on the project context versus rigid adherence to architectural patterns.

💡code organization

Closely related to project structure, this refers to logical grouping and arrangement of code - things like separation of concerns into handlers, business logic packages, data access components etc. The video advocates starting simple and evolving structure based on emerging needs.

💡framework

Used in context of web application frameworks like Echo, Gorilla etc. The speaker makes the point these are interchangeable and what matters is meeting project needs over notion of one being better.

💡global variables

The video shows an example of using a global database connection variable for simplicity. It discusses tradeoffs of this approach and notes it can work depending on context - not always a strict no-no.

💡data layer

This refers to components dealing with storage and retrieval of data, like databases, ORM mappers etc. The video uses the term "data package" and houses queries, DB interaction code under a top-level /data folder.

💡handlers

Used to denote web request handlers - functions called to handle specific HTTP calls like GET, POST etc. The video uses /handlers folder to group dedicated handler functions for different routes.

💡business logic

Domain-specific application logic beyond simple data access is referred to as business logic. The video advocates putting this in /package folder - purpose-specific packages for cohesive functionality.

💡shared code

The video uses /types and /util folders for shared code like custom types and helper functions used across the app, avoiding duplication.

💡architectural patterns

Several structured approaches like clean architecture and hexagonal architecture are mentioned that provide opinionated guidelines on project structure. Speaker notes these have tradeoffs and ideal approach depends on specific project.

💡simplicity

An overriding theme in the video is to start simple and not overengineer, allowing the structure to evolve organically based on emerging needs rather than enforced top-down patterns.

Highlights

There's no such thing as a perfect project structure, do what feels right and change it along the way

Focus on results instead of how you program things

Performance doesn't matter until you have a large user base, then it's a luxury problem

Framework doesn't matter, they all get the same results

Use simple scripts instead of complex CLI packages

Group database logic in a data folder, keep it separate

Test business logic with units tests and database interaction at integration level

Handler functions handle the HTTP requests and extract JSON

Business logic goes in the packages folder if needed

Types folder stores shared types between packages

Util folder has helper functions

Start simple then evolve architecture over time

Don't be a backend/frontend developer, be a problem solver

Weight pros and cons to see if something works for you

Project structure can scale as big as needed

Transcripts

so we're recently starting to rewrite

leue into the 2.0 version for the people

I don't know what laue is it's a company

I created three years ago and the

original project was basically written

in next GS everything in typescript

which worked very well but right now we

received a big amount of funding and the

scope is changed dramatically so it's

time to scale up and we decided to write

the back end in goaling and use spelt

kit as the front end right so uh and in

this video I'm going to show you how we

set up our gooling project structure

because it's still a question that comes

up a lot like Hey how do you structure

gooling projects and before we continue

if you like the videos that I'm

providing to you please consider

subscribing give me a thumbs up leave

some questions in the comments and jump

into the 4,000 people big Discord server

where we learn from each other each and

every single day and for the people that

want to level up I also have the

full-time GF course which is 50% off

right now for a limited amount of time

and I also have my patreon page so check

these things out if you want to learn

more about goang so before we actually

continue with the folder structure here

you can already see it on the left side

of my screen there is no such thing as

the perfect project structure or the

good project structure or something that

is good something that is wrong I think

that doesn't exist in programming uh

space I think a lot of people are

forcing paradigms to you because they

created that Paradigm and they want to

Market it so we cannot blame them right

if you make something you're probably

going to Advocate that it's going to be

the best thing because it sols a lot of

problems and maybe it will right but

whatever you do in life there's always

some it's always going to give you

benefits but it always also going to

give you some uh negative Parts also

that's something you need to keep in

mind right um so if you have the if you

have the possibility to start from

scratch you actually never know what

what's going to be the best thing you

can have something in your mind like

okay if we do it like this and and and

these are the Futures we need this is

probably something we're going to

utilize in the future maybe we should do

it this way most of the time it's going

to change dramatically over time right

it's going it's never going to stick

like you like you originally planned it

right so that's something you need to

take in consideration and that's why I

think everybody needs to have a more

result driven development mindset which

is basically instead of being confused

about oh the tools and packages and ways

to organize stuff I think you just need

to go and Chase your good feeling what

feels right what feels sus what feels

good you know what I mean and change it

along the way I think and just focus on

the results instead of on um the way you

program stuff right all right cool uh so

basically how do we do these things well

it's very simple uh we have um goang

project here

and first of all we have this bin folder

here and this bin folder is basically

just uh a folder where we are dumping in

our binaries right uh and these binaries

they are coming from the CMD folder

right you can see there is no main. go

in the rout right that's because we

using this pattern it's it's a very

common thing where you have these um

separate packages right and each of

these packages they have a main. go file

here you can see that right and in this

main.go we basically bootstrap um or API

which is an

echo project right uh it's using the

echo framework so what framework do you

need to use actually guys it does not

really matter right you can use gen

fiber Echo gorilla you can use the

standard Library it doesn't really

matter right people say yeah you don't

need to use that or you need to use this

it doesn't matter it really doesn't

matter they all do the same things and

they will all get you the same result

right that's that's that's the reality

right uh and you could say performance

performance doesn't matter right again

people are going to be very very

confused and angry about the comment I

say like performance doesn't matter like

if per maybe performance will matter at

a certain point of time and if

performance is a

bottleneck then I'm going to say

congratulations to you because you

actually makes made something that has

some traction that that's a big traction

because before you go to be in a

performance bottleneck you're going to

have a very very very very large user

base and that's a luxury problem and

that's going to be solved at the time

it's going to be there right uh which in

your case most of the time will never be

right so performance is not something uh

especially in goling with echoin you

will you cannot go wrong it it it has a

good performance don't worry about that

guys don't worry about about about these

paradigms uh about that fugazi right so

we have this main thing which basically

boots up um our API server and all that

stuff right then we're going to have

some other things so what I like to do

um which is a little bit weird a lot of

people use some kind of um CLI package I

think I don't know maybe Viper or

something I have no clue I never use

these things and with the CLI package

you can write all these commands and

then you can execute them based on the

user input but what I like to do is very

lazy and very simple is just make um for

example this drop thingy here right

which basically drops all our accounts

and why do I have this because um in our

migration sometimes you can do make down

for example which will um migrate will

will

will use all the down migration you know

what I mean delete these tables but

sometimes you're going to end up in a

dirty State and then it it basically is

going to refuse to uh drop your tables

right make down is not going to work

it's going to say yo your database is in

a dirty state of course in production

that's a good thing that that happens

right uh but in development it's

sometimes it's nasty because you have in

a dirty State and you need to force do

that stuff uh you need to force drop and

sometimes you need to manually delete

some tables and it's just a pain and

yeah so I made this very dirty simple

script it basically just deletes the

whole shebang even the

migrations uh and then we can just do a

makeup again

um and it's all good right that's this

drop and the same thing with migrate you

can see that basic um with this other

thing here for for example the seat it

basically go it's a dirty script that

seeds some stuff into the database um

just for yeah testing purposes right if

you want to test your front end uh you

just dump some some some seats uh into

into your database um so you can see

this all very simple scripts and the way

we actually trigger them is in our make

file you can see uh for example make

down it just runs these main. go files

right this the same thing here with up

with uh drop and all that stuff it it

uses these these uh separate programs

right very simple um you could use

something like a CLI package like I said

what's good what's bad it doesn't really

matter this was just easy peasy without

a lot of squeezy you know what I mean uh

so that's that so the next step here uh

like you let let me show up this make

file right so I'm using a make file

people say you should drop that in in

favor for other

stuff like I said man if make file works

for me it works for me right maybe it

doesn't work for you maybe you want to

use something else that's perfectly fine

right um it all depends what you want to

use it doesn't really matter right uh

it's

fine all right because you have these

goal tools I don't know go make or

something you have these these cool

stuff U where you can write goang things

and but like I said a make file

everybody knows it it's simple it's most

of the time already installed on Linux

things so it's fine right uh so the next

step is a very important step is our DB

I'm going to uh start with this DB

because this is a little bit of

controversial right so the DB package

has just one file and what this does is

basically creates

um it it basically uh bootstraps or

database right and as you can see we are

using bun here but we and it's a global

variable so there's a lot of things we

need to discuss here right um so I

started out we started out with using

sqlc right sqlc is something that

generates um goang codes and structures

models based on U your schema your SQL

schema uh and you can write your SQL

queries and it basically generates code

for that which is actually very very

good right

but need to drink some coffee for this

one but the problem with sqlc is it's

it's getting very tedious something we

had a ton of queries already and I

thought man this is just so nasty all

these queries and then we have a schema

for our migrations we have a schema for

sqlc and you need to sync these things

and then some people are working uh like

me and Teddy we working on that it's a

two people project right now and we are

doing work for for 16 uh but the thing

is that it's just it gives conflict it

was was

nasty for current operations I'll really

like it but we need to have some

more dynamically stuff going on some

more genetic stuff and I used BN for a

long time um in in other projects and I

really like it right of course something

that's the downside like I said for

every positive thing also comes

something that's some negative stuff and

the cons for bun is that your queries

are not going to be type safe right if

you make a mistake

that's bad for you nobody's going to

it's going to it's going to give you an

error or something or or user that not

being found uh but hey it is what it is

right like I said pros and cons what do

you want it's it's a a weight scale that

needs to be in Balan for you right not

for somebody else on the internet it

needs to be in balanced for you and for

your team and is everybody thinks that

the scale is in balanc and it's fine

right everybody know the consequences

here um but also everybody can use the

positive side Ides on that right so

that's what we do and it's a global

variable why is it a global variable

just for the ye of use and of course

this is very this could be dangerous but

like I said a global at the right time

at the right place is is is okay it's

it's it's no big of a deal because we

are using this data

folder and this data folder is most of

the time is going to be used models for

a lot of people but I think models is so

weird because I think models they belong

on the catwalk you know what I mean um

it's I think everything that involves

storing data retrieving data structures

which is just

data I think it it can be in a data

folder it makes a lot of sense right so

everything uh that's

involving data

structures queries for inserting or

retrieving or updating whatever all

database related logic is housing into

this data folder right just like models

most of the time

and what this does is basically it uses

it uses queries um it the only thing it

does is basically interacting with your

database right that's what it's doing

you got have a query you're going to do

some joints or some some separate

queries you're going to aggregate that

data and return it that's basically what

this data folder is this data package is

going to do right and nothing else and

nothing nothing less and nothing

more for tests it's very important

because the global variables uh can be a

problem in your tests but I don't like

to test the data or your database into

the unit form right I don't want to use

unit tests for data I think interacting

with your database should be on the

integration Port again this is something

that can be discussed it has its pros

and it has its cons it really doesn't

matter right um I think if you test your

business logic based on on unit tests

that's pretty good and if you use your

data everything that needs to be

interacting with your database if you do

that on an integration

level it doesn't really matter that your

database is

global the only thing you need to do is

instantiate the correct database right

if you have if you want to do some test

you use a test database um and if you

yeah whatever right and if you deploy

the production you're basically going to

init initialize your database with the

production credentials right so that's

the data thing right the next step we

have is very simple is our handlers and

our handlers are just plain functions

right there are no uh you can see for

example here uh some middleware where we

validate gbt we have um some error stuff

for centralized error

handling some lock middleware it's very

simple uses the S from gooling the new

stuff pretty amazing let me show you

something here can I show some something

diff uh something interesting uh for

example handle get User it's just very

simple handle get user we always prefix

this with handle and then we use get and

that's not for example not for getting

the user it's the get request right

handle get user handle post user handle

update user right um just a convenience

method is that the way to go I have no

clue it works for me

uh you're you're like I said you're the

ball pros of your painting you can do

whatever you want right of course if you

join a team um that already has these

kind of things set out laid out yeah you

need to follow them right but if you can

choose from the start you can do

whatever you want right as long as it

makes sense for you um so what do we do

with these hand handlers well uh like I

said before we have the CMD API Maino

file and then we have these very simple

um uh yeah like like these these these

route handlers right this these routing

stuff so for example uh what we're going

to do here because in leue we have the

concept of sside accounts and buy side

accounts because there's a double-sided

marketplace where you're going to sell

or you're going you're going going to

auction your uh recurring Revenue in

return for upfront cash from

institutional investors right so we have

sell site and buy side and of course the

sell side they cannot they can only do

specific certain actions and the same

for the buy side so what we do here is

we make this a group thingy which

basically is most of the time in every

single web framework you're going to

have something like that where you're

going to group your handlers with some

um how do you call it with some

middleware right and uh it basically

some some authentication authorization

middleware right he's going to see oh

yeah this user is this a cide user yes

then he can do these things if it's a

buy side user then you cannot do these

things right then uh you are basically

not authorized to access these things

right so that's what we do here handlers

and then we're going to say handle

cellsite demo metrics in this case we

don't follow this stuff it's probably my

bad handlers handle post funding request

handlers handle get funding request you

know what I mean very simple um no big

of a deal here right so that's basically

our handlers here then we have package

um which is

something where more of the business

logic houses because most of the time

you're going to have something like a

transport right you're going to have

your Json transport your Json HTTP

Handler so what it's going to do is it's

going to get some Json data it's going

to uh Marshall that or decode that into

a structure right and then you're going

to have that structure and then you're

going to decide do we need to do some

business business logic onto that data

or can we basically just use our data uh

package and just store that user into a

database right or store a funding

request into a database some some simple

crit doesn't really need to have some um

business logic because it's very simple

it's just receiving data decoding data

validating data and storing data that

doesn't actually is some kind of a

business logic in my opinion but if

there is some business Logic for example

you need to fetch some transaction from

the the bank you need to aggregate them

do some calculations and all that stuff

uh maybe some third party things are

going on some third party API calls or

need to be um executed then we put

everything inside of a

package folder here and in this case

it's going to be salt and then client y

yada y you can see the project is very

very early in the stage we already have

some project running but that's the next

GS this is still in development right so

that's going into package uh that's

basically a goaling em to to put it in

in package you could use it lip you can

you can call it whatever you want but I

think package is something that is very

common inside of the goang community

right then we have some types here uh

and types is very important

because you could put these types into

Data but these types are basically

shared you can use you can make you can

you can call this common is is is I can

I see that a lot of times in other

projects in other repositories on getup

people call it common people call it

shared in this case just types because U

the only thing we have is some kind of

enums quote quote I need to be very

careful for the YouTube police right

because they are actually no enims but

they act like enims right um constants I

would say all the constants are here all

the types are here that are shared

between uh packages right that's in

types um and you could basically split

that out right you could make it a a

file for each type but then you're going

to have a file with three lines of

sometimes that's who cares right you

just dump everything into the types at

go and if your types are getting too big

then you can start splitting things out

and group Things based on on domain or

something right but that's for later on

that's a problem we don't have yet right

and then of course the UIL right util

utils shared whatever you're going to

call this is a very simple stuff where

we going to have uh like nothing inside

right is Pro app M and all that stuff

some simple helper functions that are

going to help you with uh simple simple

things in your application right crypto

is

this not quite sure what this is

probably T did some stuff right anyway I

think it's for encrypting um some kind

of API tokens we don't want to uh every

API token from from a third party

library for our user we don't want to

expose that uh with plain text we want

to basically encrypt that stuff into the

database so if it's get hacked they are

the hacker right not the client

uh that's basically it of course get

ignore go mod some make read me that's

all that stuff that's basically it guys

and with this project structure you can

scale it as as far as you want right you

because every data structure you're

going to have or every data logic you're

going to have is going to be in the data

every new Handler is going to be in the

handlers every new business logic you

can put that into package and every new

type you put it into type and you can

scale it as big as you want is this

going to change over time most likely it

will right it's like I said it's

something that works for me for a very

long time this this structure it's very

simple um don't make it too complex with

all that clean architecture and

hexagonal architecture and they will all

have his benefits don't get me wrong

like these are basically invented by

very smart people but I always say start

very simple right start simple make some

uh diesel driven stuff and I'm going to

show you how it looks right so you can

see this is basically the sells side

dashboard right um which again another

question is like people say how do you

make this beautiful wise well guys it's

very simple uh don't be a single-sided

developer very important people want to

be a backend developer people want to be

a front-end developer I don't think

there is something such as a backend and

a front end developer in my opinion you

are a programmer or you're not you are

an engineer or you're not you can solve

problems or you're not because by the

end of the day the only thing you do is

write code right you're pressing buttons

If it's JavaScript typescript goang

python rust it doesn't really matter you

can write it in every every single

language there are some benefits and

there are some cons there are some pros

there are some Cons with everything in

life and the only thing that matters is

if your weight scale of pros and cons is

inbalance or may basically if your Pros

are a little bit

um uh if if your Pros weigh a little bit

more actually weit a little bit less

right if the balance is a little bit

more in the favor of the pros instead of

the Conant it's a win for you right but

that does not mean that my balance is

the same my weight balance is going to

be balanced the same as yours or my

neighbors or whatever right or other

produ um engineering teams it's all

subject to the people are using it right

that's something you need to uh take in

consideration um that's basically the

thing of course we are we are doing some

stuff it's it's it's still in progress

here we we we have some look at this

it's it's we have some stuff but we're

still uh working on that right that's

basically it and it's calling basically

spell kit in the back end and all that

stuff maybe I'm going to make it another

video because we're already 20 minutes

in and I'm not even done it's crazy guys

I really want to provide 10minute videos

but I can because the things I need to

say uh take some time right you know

what I mean it takes some time so that's

basically on B it if you have more

questions feel free to dump it in the

comments like I said uh is not for

everyone right people going to going to

dislike it people going to like it it is

what it is it's the balance again right

it's again the weight balance as long as

the people like it or in favor of the

people that dislike it it's all good for

me Isn't it so thank everyone for

watching this video and I'm looking

forward to see you in the next video or

live stream peace stay safe and have fun