Dapr Community Call - May 29th 2024 (#104)

Summary

In the Dapper Community call, Youron Schneider, one of Dapper's original founders, introduces the outbox pattern and payload projections in the upcoming 1.4 release. The outbox pattern allows single-transaction commits across databases and message brokers, crucial for system reliability. Schneider demonstrates the new feature enabling different payloads for database storage and message publishing, showcasing Dapper's language, cloud, and hosting agnosticism, and its ability to simplify complex distributed system challenges.

Takeaways

• 😀 The Dapper Community Call is a platform for discussing updates and developments within Dapper as a product and community.
• 🌟 The community call originally featured Ryan Winter discussing Azure IoT and Dapper pluggable components, but was replaced by a talk by Dapper's co-founder, Youron, on the outbox pattern and payload projections in Dapper 1.4.
• 🔍 Youron Schneider is a core maintainer of the Dapper project and a stream commed member from Drid, focusing on open-source and serverless container platforms.
• 📈 The outbox pattern, introduced in Dapper 1.1.2, allows committing a single transaction across a database and a message broker, solving the dual write problem in distributed systems.
• 🔧 Dapper 1.4 introduces payload projections, enabling developers to decouple the state saved in the database from the payload published to downstream subscribers.
• 🎯 The outbox pattern is highly requested by the community for its reliability and consistency in event-driven architectures.
• 🛠️ Dapper abstracts the complexities of distributed systems, allowing developers to use different databases and message brokers interchangeably.
• 📚 Youron mentioned writing a blog post to demystify the technical details of how Dapper implements the outbox pattern and payload projections.
• 💻 A live demo was given, showcasing how to use Dapper's transaction API to save state in Redis and publish a different payload to Kafka with a single HTTP call.
• 🔄 Dapper's resiliency features can be applied to both database operations and pub/sub messaging, enhancing the robustness of applications.
• 📝 The Dapper community is encouraged to try out preview features, provide feedback, and contribute to the project through documentation, blog posts, and sample code.

Q & A

What is the purpose of the Dapper Community calls?

The purpose of the Dapper Community calls is to gather everyone together to discuss new and exciting developments within the Dapper product and community, including talks, presentations, and updates.

Who was originally scheduled to talk about Azure IoT and Dapper pluggable components, and what happened?

Ryan Winter was originally scheduled to talk about Azure IoT and Dapper pluggable components, but due to a series of unfortunate events, he could not make it, and Yaron Schneider stepped in to talk about the outbox pattern and payload projections.

What is Yaron Schneider's role in the Dapper project?

Yaron Schneider is one of the original founders of the Dapper project, a core maintainer on the DAP runtime, and a stream commed member from Drid, a company he founded after working at Microsoft.

What is the outbox pattern, and why is it difficult to implement?

The outbox pattern, also known as the transaction outbox pattern, is the ability to commit a single transaction across a database and a message broker. It is difficult to implement because it requires solving the dual-write problem in distributed systems, which involves coordinating the same data or payload being saved to two separate systems transactionally.

What is the dual-write problem in the context of the outbox pattern?

The dual-write problem refers to the challenge of needing to save the same data or payload to two separate systems, such as a database and a message broker, transactionally, which is difficult due to potential failures in the systems that can leave the data inconsistent.

What is the 'anticipator pattern' mentioned by Yaron Schneider?

The 'anticipator pattern' is a term used by Yaron Schneider to describe the method used in Dapper to solve the dual-write problem. It allows Dapper to handle the complexities of committing a single transaction across different state stores and pub/sub brokers.

What is the outbox projections feature in Dapper 1.14, and how does it address a limitation of the outbox pattern?

The outbox projections feature in Dapper 1.14 allows developers to decouple the state saved in the database from the payload published to the downstream subscriber. This addresses the limitation of the outbox pattern where previously the same state had to be used for both the database and the published message.

How does Dapper handle resiliency in the context of the outbox pattern?

Dapper allows the application of resiliency policies to the database operation and the pub/sub layer independently. This means that if the entire operation fails, Dapper can retry the operation in an exponential backoff manner, and if a message cannot be delivered to the underlying subscriber, it can be sent to a dead letter.

What is the process for trying out new preview features in Dapper?

To try out new preview features in Dapper, users should look at the documentation and follow the instructions provided. Some preview features might require a flag to be turned on in the Dapper configuration, explicitly enabling the feature.

How can community members contribute to Dapper and what are the benefits?

Community members can contribute to Dapper by writing blog posts, submitting issues, creating samples, and participating in conferences or workshops. They can also earn badges for their contributions, such as presenter badges for doing presentations or writer badges for updating documentation.