4 projects that helped me escape web dev

Summary

In this video, the speaker shares their journey of transitioning from web development to systems programming. They discuss the challenges of switching fields and how personal projects in C, C++, Go, and networking concepts helped them build a stronger skill set. Notable projects include a Go-based web crawler, a peer-to-peer note-sharing platform, and a custom HTTP server in C, all of which expanded their understanding of low-level programming. The speaker also emphasizes the value of exploring operating systems and kernel programming through hands-on experiences, ultimately opening up new career opportunities beyond web development.

Takeaways

• 😀 The speaker transitioned from web development to systems programming due to a growing interest in computer organization and architecture.
• 😀 Web development became less challenging, with many technologies abstracted away from developers, leading to a desire for deeper technical engagement.
• 😀 The main challenge in switching to systems programming was the lack of direct experience, but the speaker overcame this by doing self-driven projects.
• 😀 Key systems programming concepts and technologies learned included C, C++, Go, networking, and database systems.
• 😀 The speaker's first major project was a web crawler written in Go, which fetched and parsed web pages and stored content in MongoDB for keyword searching.
• 😀 The web crawler used concurrency with Go routines to improve performance by handling different tasks in parallel.
• 😀 Another project, Peerotes, was a peer-to-peer file sharing platform, focusing on network programming and utilizing TCP connections for file transfers between peers.
• 😀 Building an HTTP server from scratch in C exposed the speaker to low-level network programming concepts, like signals and inter-process communication.
• 😀 The speaker gained hands-on experience with system tools like CMake, Make, and Docker while working on projects involving C programming.
• 😀 Working with XV6, a toy operating system, helped the speaker understand kernel programming and OS concepts by building and extending the OS with custom features.
• 😀 The speaker's project portfolio, showcasing experience in systems programming, helped them secure an internship as a production engineer at Meta, combining software and systems engineering roles.

Q & A

What motivated the speaker to shift from web development to systems programming?

The speaker became more interested in computer organization and architecture after taking related courses in college. They found web development too abstract, as it didn't allow them to explore how technologies worked at a low level, which led them to seek new challenges in systems programming.

What were the main technologies the speaker focused on to transition into systems programming?

The speaker focused on learning C, C++, Go, networking concepts, and database technologies. These were key components for the systems programming roles they were targeting.

How did the speaker gain experience in systems programming despite having a background in web development?

The speaker worked on personal projects that incorporated systems programming concepts, such as a web crawler, a peer-to-peer file sharing platform, and an HTTP server. These projects allowed them to develop the necessary skills and showcase them on their resume.

What was the main function of the web crawler the speaker built in Go?

The web crawler was designed to fetch web pages, parse their content, and build an inverted index stored in MongoDB. This index allowed for keyword-based searching of URLs, similar to how search engines like Google work.

How did the speaker enhance the performance of the web crawler project?

The speaker used Go routines to implement concurrency and parallelism, allowing different stages of the crawler to run in parallel. This helped improve the performance of the program.

What role did MongoDB play in the web crawler project?

MongoDB was used to store the inverted index generated by the web crawler. This allowed for fast keyword-based search functionality, enabling users to find URLs containing specific keywords.

What is the unique feature of the peer-to-peer file sharing platform (Peerotes) the speaker developed?

The platform allowed users to share files directly with each other (peer-to-peer) while using a central server to store metadata, such as file filters and owner's IP addresses. This approach minimized the reliance on central servers for file storage.

What did the speaker learn by building an HTTP server in C?

The speaker gained experience with low-level programming concepts, including handling networking, interprocess communication, and process forking. They also learned to use tools like CMake and Docker to manage development environments and compile code.

What challenges did the speaker face when building the HTTP server in C?

The speaker encountered challenges such as dealing with signals (used for interprocess communication), managing resources after process forking, and handling network-level operations that were not abstracted by higher-level languages.

What did the speaker learn from working with XV6, the toy operating system?

The speaker learned the differences between user space and kernel space programming by modifying and extending the XV6 operating system. This project gave them a deeper understanding of low-level kernel programming concepts.