The Consistent Hash Exchange: Making RabbitMQ a better broker - Jack Vanlightly

Summary

This presentation explores strategies for scaling out message consumers and maintaining message order in RabbitMQ using consistent hash exchange and partitioning techniques. It discusses the challenges of scaling with a single queue and introduces consistent hashing as a method to ensure that related messages are processed in order by the same consumer. The talk also covers issues like weakened processing order guarantees and large queue problems, presenting solutions such as using routing keys for causal ordering and leveraging data locality for efficiency. The presenter shares insights on message ordering safety levels and demonstrates the impact of prefetch settings and consumer behavior on message order through practical demos. Additionally, the potential of quorum queues and consumer group functionality for improving deployment and management is highlighted.

Takeaways

• 📚 Consistent Hash Exchange and Partitioning: The presentation discusses techniques for scaling out consumers in message brokers like RabbitMQ using consistent hash exchange and partitioning a single queue into multiple queues to avoid issues with scaling and maintaining message order.
• 🔒 Safety and Ordering Guarantees: Scaling out with multiple consumers can lead to weakened processing ordering guarantees. The talk emphasizes the importance of maintaining causal ordering to ensure that related messages are processed in the correct sequence.
• 🚀 Scaling with Consumer Groups: Using consumer groups and partitioning messages by a hash of a routing key (like booking ID or client ID) can help distribute the message load evenly while ensuring that all related messages go to the same consumer for processing.
• 🔑 Routing Key Importance: The choice of routing key is crucial for maintaining message order. Using a unique identifier like a booking ID as the routing key helps in achieving causal ordering and ensures related messages are processed by the same consumer.
• 🔄 Challenges with Competing Consumers: The talk highlights problems that can arise with competing consumers, such as the weakening of message ordering guarantees and difficulties with large queues and synchronization.
• 🛠️ Solutions for Processing Order: The presentation suggests using a consistent hashing exchange to ensure a guaranteed processing order of dependent messages by leveraging a well-chosen routing key.
• 🌐 Data Locality Patterns: Partitioning messages can lead to data locality patterns, which allow for in-memory processing and reduced reliance on caches, thus improving efficiency.
• 🔍 Message Ordering Safety Scale: The script introduces a safety scale for message ordering, explaining different levels of safety from independent messages to those requiring strict ordering for related events.
• 👨‍💻 Consumer Group Assignment: The talk mentions the difficulty of consumer group assignment in RabbitMQ and introduces a project called 'rebalancer' to help with dynamic queue assignment and rebalancing.
• 🔧 Improvements in RabbitMQ: The presentation concludes with a discussion about improvements in RabbitMQ, particularly in the area of consistent hashing, which has been enhanced for better distribution and load balancing.

Q & A

What is the main topic discussed in the script?

The main topic discussed in the script is consistent hash exchange and partitioning a single queue into multiple queues in RabbitMQ, focusing on maintaining message ordering and improving the safety and scalability of message brokers.

Why is scaling out consumers a simple way to handle high message velocity?

Scaling out consumers is simple because you just point your consumer at the queue and RabbitMQ will do its best to distribute messages evenly, given the prefetch settings and unit of parallelism, which is the consumer itself.

What is an alternative method to scaling out consumers mentioned in the script?

An alternative method mentioned is sharding or partitioning, where the unit of parallelism is the queue. This involves routing messages based on the hash of a routing key, such as a booking ID or client ID, to ensure all related messages go to the same queue and consumer.

What does using a consistent hash exchange provide in terms of message ordering?

Using a consistent hash exchange provides causal ordering. Although it does not provide a totally ordered set of messages, it maintains the important ordering guarantees, ensuring that messages related to the same entity are processed in order.

What problems can arise with competing consumers that the script discusses?

The script discusses that competing consumers can lead to the weakening of processing ordering guarantees. When a single sequence is processed in parallel, the processing order does not match the stored sequence in the queue, which can cause issues.

How does the script suggest avoiding issues with large queues in RabbitMQ?

The script suggests using consistent hashing exchange to ensure guaranteed processing order of dependent messages and to avoid issues related to large queues, such as problematic blocking synchronization and the need for replication repair in the broker.

What is the benefit of using a booking ID or client ID as a routing key in the context of message partitioning?

Using a booking ID or client ID as a routing key ensures that all messages related to a given object or entity always go to the same queue and therefore always to the same consumer, maintaining causal ordering and avoiding out-of-order processing.

What is data locality and how does partitioning messages by routing keys contribute to it?

Data locality refers to the pattern where all related data or messages are processed by the same consumer, allowing for in-memory processing and reduced reliance on external caches or databases. Partitioning messages by routing keys contributes to data locality by ensuring that all related events are consumed by the same consumer.

What is the impact of prefetch settings on message ordering and safety?

Prefetch settings impact message ordering and safety by determining how many messages a consumer will hold at one time. A prefetch of one ensures safer, more sequential processing, while a larger prefetch can lead to more disorder and less predictable message ordering.

How does the script address the issue of deduplication in message processing?

The script addresses deduplication by suggesting the use of in-memory data structures, such as dictionaries, to store and check for duplicate message IDs or routing keys when all dependent messages are routed to the same consumer, allowing for efficient and reliable deduplication.

What is the significance of the quorum keys mentioned in the script?

Quorum keys, as mentioned in the script, are significant because they offer a solution to the problem of synchronizing an empty mirror, which can make queues unavailable. Quorum keys ensure that data is not lost and provide a more robust mechanism for maintaining message redundancy and availability.

What improvements to the consistent hash exchange plugin are suggested in the script?

The script suggests improvements such as consumer group functionality on the client side, which would allow for easier setup and management of consumer groups and their queue assignments. Additionally, it mentions the need for better balance and distribution of routing keys for optimal performance.