Facebook and memcached - Tech Talk

Summary

The video is abnormal, and we are working hard to fix it.
Please replace the link and try again.

Takeaways

• 😀 mcash proxy was used to resolve synchronization issues by running it on the SQL server, ensuring better syncing across the country and more efficient scaling.
• 😀 Modifying SQL queries to include mcash commands helped maintain low connection counts while ensuring efficient replication of data.
• 😀 By optimizing memory allocation from powers of two to a power of 1.3, memory usage efficiency increased to 90%, saving substantial resources.
• 😀 Changing system calls to a more efficient version reduced CPU time by 50%, significantly lowering operational costs and improving performance.
• 😀 A simple string parsing optimization, by storing string lengths in a variable, improved CPU usage by a factor of three, saving millions in costs.
• 😀 The company operates at a large scale, supporting 100 million users, and aims for 150 million by the end of the year, driving the need for constant optimization.
• 😀 Optimizations in user space and kernel-level changes, including modifying network driver handling for multi-core processors, helped improve overall system performance.
• 😀 The original server design, which used a single core, was outdated, necessitating network driver optimizations to better handle modern multi-core server architectures.
• 😀 Through a combination of client, server, and kernel-level optimizations, the system was able to handle increasing data traffic while keeping costs low.
• 😀 Despite appearing to be small fixes, these optimizations at scale had massive effects, reducing the number of servers needed and saving millions of dollars.

Q & A

What was the initial problem that led to the development of the Mcash proxy solution?

The initial problem was the issue of synchronizing Mcash tiers across different servers and regions, particularly the challenge of keeping the connection count low while ensuring consistency and performance at scale.

Why did the team decide to run the Mcash proxy on SQL servers instead of on web servers?

Running the Mcash proxy on SQL servers allowed them to execute Mcash commands directly via SQL queries, keeping the connection count low and ensuring synchronization of Mcash across regions without overloading the web servers.

How did the team modify the SQL query to support Mcash proxy effectively?

They modified the SQL query to include a Mcash command, which was sent to the Mcash proxy for execution on the cache tier, allowing for efficient synchronization and memory cache management across multiple locations.

What was the main objective behind optimizing Mcash to reduce the number of connections?

The main objective was to reduce the connection count between clients and servers, which would improve system efficiency, minimize resource consumption, and reduce operational costs at scale.

What was the issue with the slab allocation for memory in the original Mcash design?

The slab allocation used powers of two (e.g., 256k, 512k, 1MB) for chunk sizes, which resulted in significant memory wastage because small data sizes would still occupy entire chunks, leading to inefficient memory usage and about 40% wasted space.

How did the team improve the memory allocation in Mcash?

By experimenting with different chunk sizes, the team optimized the slab allocation to use a 1.3 power rather than powers of two, which improved memory usage efficiency to around 90%, significantly reducing memory wastage.

What CPU optimization did the team implement in Mcash to improve system performance?

The team rearchitected system calls, replacing multiple individual calls with more efficient ones, which reduced CPU usage by 50%, leading to a significant performance boost and saving millions of dollars in operational costs.

How did the team address string parsing inefficiencies in Mcash?

They optimized string parsing by storing the length of the string in a variable, which eliminated the need to repeatedly call the `strlen` function, reducing CPU usage by a factor of three and improving overall system performance.

Why did the team consider modifying network drivers in the kernel for further optimization?

At the scale Mcash was operating, optimizing network drivers in the kernel was necessary to ensure efficient distribution of network interrupts across multiple CPUs, particularly as servers evolved from single-core to multi-core systems.

What were the key takeaways from the Mcash optimizations in terms of impact?

The key takeaways were that small optimizations, when applied at scale, could lead to substantial improvements in efficiency, saving millions of dollars in infrastructure costs, reducing memory wastage, and enhancing CPU performance, all crucial for supporting millions of users.