build this to impress your quant trading interviewer (as a software engineer)

Summary

In this video, the creator discusses a market data dissemination simulator project built using gRPC, aimed at helping junior developers understand key concepts in client-server communication, middleware, and distributed systems. The project consists of a server and client application where the server reads configuration files, generates order book updates, and sends snapshots and incremental updates to clients. The creator emphasizes the importance of understanding scalable, asynchronous systems and encourages viewers to improve the project with additional features like a front-end visualization or a database persistence layer. The goal is to impress potential employers and justify architectural decisions.

Takeaways

• 😀 Understanding middleware and client-server communication in distributed systems is a key skill that many junior developers miss.
• 😀 Many developers know how to produce programs that output data, but lack the knowledge of how to stream that data between different services or applications.
• 😀 The project presented is a market data dissemination simulator that uses gRPC, which is a scalable framework for building cross-service APIs.
• 😀 The project consists of two main applications: a server that handles market data and a client that subscribes to updates and processes them.
• 😀 The server is responsible for reading configuration files, managing an order book, and disseminating snapshots and incremental updates to clients.
• 😀 gRPC is used in the project to implement a bidirectional stream, allowing clients to request subscriptions or unsubscriptions and receive real-time updates.
• 😀 Efficiency is emphasized by sending only incremental updates to clients instead of full snapshots of the order book each time.
• 😀 The server also handles specific events, like clearing incremental updates when a new snapshot is received, ensuring clients maintain the most current state.
• 😀 One suggestion for extending the project is to build a front-end that visually displays the order book data instead of using a console application.
• 😀 Another suggestion is to add a second server application responsible for persisting the received updates into a database, providing better data management and scalability.
• 😀 The video emphasizes the importance of making architectural decisions and being able to justify them during an interview, especially when discussing choices of databases or technologies.

Q & A

What is the main purpose of the project described in the video?

The main purpose of the project is to teach junior or intern-level quantitative developers or software engineers how to build a market data dissemination simulator using client-server communication, gRPC, and microservice architecture.

Why is understanding client-server communication important for junior developers?

Understanding client-server communication is important because it is a fundamental concept in distributed systems, especially in microservice architectures. Junior developers often know how to produce data in applications, but they may not understand how to communicate that data across different systems efficiently.

What does the server in this project do?

The server reads a configuration file that specifies instruments and their contract details, generates order book updates, and disseminates both snapshots and incremental updates to connected clients based on subscription requests.

What is the role of gRPC in this project?

gRPC is used to handle the bidirectional streaming communication between the client and server. It ensures efficient and scalable real-time data transfer, allowing clients to subscribe to data, receive snapshots, and get incremental updates.

What does the subscription mechanism do in this project?

The subscription mechanism allows clients to connect to the server, specify an instrument ID, receive an initial snapshot of the order book, and then continuously receive incremental updates. The client can also unsubscribe when needed.

Why are snapshots sent instead of the entire order book state every time?

Snapshots are sent instead of the entire order book state to improve efficiency. This reduces the amount of data transmitted by only sending the updated state when necessary, instead of resending the full order book with every change.

What happens when a client unsubscribes from an instrument?

When a client unsubscribes, the server sends an empty snapshot to signify that the client should clear its cached data for that instrument. This prevents unnecessary data from being stored or processed after the client has unsubscribed.

How can this project be extended for more complexity?

The project can be extended by building a frontend to visualize the order book, or by adding another server that persists data into a database (e.g., MySQL, Cassandra) to create a more complex and useful system.

Why is it important to justify technical decisions in an interview?

Justifying technical decisions in an interview is important because it shows a deep understanding of the technologies and architectures used. Employers want to know that you can make informed choices based on the project's needs and explain the reasoning behind them.

What additional skills does this project aim to teach beyond coding?

This project aims to teach the ability to build scalable, distributed systems and communicate effectively about architectural decisions. It also emphasizes understanding middleware, cross-application communication, and how to impress employers with complex, well-thought-out projects.