Design Youtube - System Design Interview

Summary

The video script discusses designing a high-level architecture for a YouTube-like application, focusing on the core functionalities of video uploading and viewing. It highlights the complexity of implementing these features at YouTube's scale, emphasizing the importance of reliability, availability, and minimizing latency. The speaker describes a potential infrastructure involving load balancers, application servers, object storage, and NoSQL databases for metadata. The video also touches on the challenges of video encoding, the use of CDN for optimized video delivery, and the trade-offs between different database systems, sharing YouTube's historical approach to scaling MySQL with the introduction of Vitess.

Takeaways

• 🎯 The core functionalities of YouTube include video uploading and viewing, which require a scalable and reliable architecture.
• 🔄 Dealing with the scale of YouTube involves handling 50 million uploads per day and billions of video views, necessitating a robust infrastructure.
• 🛡️ Reliability is crucial; videos must be stored without risk of corruption or deletion, leveraging object storage solutions like AWS S3 or Google Cloud Storage.
• 🌐 Availability is favored over consistency, meaning it's better to serve slightly stale data than to risk unavailable service.
• 🚀 Video encoding is an asynchronous task that requires a large number of workers to handle the daily upload volume efficiently.
• 💡 Using a CDN (Content Delivery Network) ensures videos are streamed quickly and geographically close to viewers, improving latency.
• 📚 Metadata and user information are stored in a NoSQL database, such as MongoDB, to allow for fast reads and flexible data storage.
• 🔄 Denormalization in NoSQL can improve performance by avoiding joins, but updates to user information may require propagating changes across multiple documents.
• 🚦 Rate limiting may be necessary to prevent abuse of the system, such as uploading an excessive number of videos.
• 🔍 Additional services for recommendations and search would likely be built on top of the core metadata storage, incorporating user history and preferences.
• 🛠️ YouTube's initial use of MySQL and the development of Vitess show that with the right engineering solutions, even relational databases can scale to meet massive demands.

Q & A

What are the core functionalities of YouTube that the design proposal focuses on?

The design proposal focuses on two main functionalities: uploading videos from a user's perspective and watching videos from a user's perspective.

What is the estimated scale of daily uploads for YouTube?

The estimated scale of daily uploads for YouTube is 50 million videos per day.

How does YouTube handle the reliability of video storage?

YouTube uses object storage, such as AWS S3 or Google Cloud Storage, which handles replication and ensures that videos are reliably stored and not subject to deletion or corruption.

What is the read-to-write ratio for YouTube users?

For every one user uploading a video, there are a hundred users watching videos. This means that for every five users watching a video per day, there are five billion videos being watched per day.

What does YouTube prioritize in terms of data management: availability or consistency?

YouTube prioritizes availability over consistency. It is more important for the platform to respond correctly and quickly to user requests, even if it means occasionally serving slightly outdated data.

How does YouTube address the latency issue for video playback?

YouTube addresses latency by using a Content Delivery Network (CDN) to distribute video content geographically close to end users and by streaming videos in small chunks to start playback quickly, even before the entire video is loaded.

What type of database does YouTube initially use for storing video metadata and user information?

YouTube initially uses a relational database, specifically MySQL, for storing video metadata and user information.

How did YouTube scale their MySQL database to handle a large amount of read traffic?

YouTube scaled their MySQL database by adding read-only replicas and implementing sharding. They also developed an engine called Vitess to decouple the application layer from the database layer, handling sharding and request routing logic.

What is the role of a message queue in the video uploading process on YouTube?

The message queue is used to manage the video encoding process, which is an asynchronous task. Videos are added to the queue and then sent to encoding services, which can handle the encoding in parallel.

Why is denormalization acceptable in the context of YouTube's NoSQL database design?

Denormalization is acceptable because it improves performance by eliminating the need for joins. It allows for duplicate information to be stored, which speeds up read operations, as seen with user profile pictures being stored with each video document.

What protocol does YouTube use for video streaming and why?

YouTube uses HTTP requests built on top of TCP for video streaming. TCP is favored for its reliability, ensuring that the entire video is received without any missing gaps, which is important for delivering a smooth viewing experience.