Is Django async yet? - Riccardo Magliocchetti

Summary

Ricardo Malti explores the challenges and opportunities of transitioning Django's synchronous API ecosystem to asynchronous operations. He discusses the benefits of async processing for better resource utilization, particularly in high-traffic applications, and demonstrates the current limitations in Django's async support. While key features like view sets and permissions are not fully async-compatible, Malti emphasizes the potential for community-driven development in improving Django’s async framework. His talk also touches on performance measurements using OpenTelemetry, highlighting the overhead of switching between sync and async contexts, and the importance of contributing to open-source solutions for a more robust Django ecosystem.

Takeaways

• 😀 The speaker, Ricardo Malti, explores the challenges and current state of asynchronous (async) support within the Django framework, particularly for API development.
• 😀 One of the key challenges faced is the lack of full asynchronous support for model view sets in Django's asynchronous framework, requiring custom solutions.
• 😀 The speaker emphasizes that the transition to async is not aimed at improving speed or reducing latency but at optimizing resource usage, particularly memory consumption.
• 😀 Django's asynchronous framework still lacks full functionality, including asynchronous model view sets, HTML rendering, and related field handling in serializers.
• 😀 OpenTelemetry was used to track async transitions and help identify areas where code switches between sync and async, although this is not a performance benchmarking tool.
• 😀 Despite async code offering potential resource savings, the lack of full async database support in Django limits its overall performance benefits in production environments.
• 😀 The absence of built-in permissions in Django's asynchronous REST framework led to the manual removal of authentication for the demo, which was a workaround.
• 😀 The speaker highlights that asynchronous support is still evolving in Django and encourages the community to contribute to the development of missing features.
• 😀 Ricardo used an example from Django’s official tutorial to demonstrate the process of moving from synchronous to asynchronous views, showcasing some of the key issues along the way.
• 😀 While transitioning to async may introduce some overhead, the main concern at this stage is not performance but the overall stability and completeness of async support in Django's ecosystem.

Q & A

What was the main objective of Ricardo Malti's talk?

The main objective of Ricardo Malti's talk was to explore the asynchronous (async) capabilities of the Django framework, particularly in the context of Django's API ecosystem, and to highlight the challenges and gaps in its current async support.

What issues did the speaker face when working with async Django?

The speaker encountered several issues, including the lack of async support for model view sets, missing functionality for related fields in serializers, and problems with HTML rendering and permissions handling in async Django. He also had to manually implement certain features that would normally be provided by Django.

Why did the speaker focus on transitioning from synchronous to asynchronous services?

The speaker's primary goal was not to improve performance or reduce latency, but to optimize the use of system resources, particularly memory. He aimed to create a more efficient backend that consumed less memory while still handling high traffic, without rewriting the code or APIs from scratch.

How did the speaker address the lack of async support in Django views?

The speaker manually implemented the necessary view functionality, creating custom views and mixing them with Django's standard view set functionalities. He had to work around missing async features by manually handling things like pagination and authentication, and by using sync-to-async helpers.

What was the result of the speaker's efforts to implement a fully asynchronous Django backend?

The speaker was unable to achieve a fully asynchronous Django backend at the time of the talk, due to the incomplete async support in Django. Despite several fixes and attempts, the backend was still not ready for production, but the speaker encouraged others to contribute to the project to improve it.

What role did OpenTelemetry play in the demo presented by the speaker?

OpenTelemetry was used to track asynchronous operations in the Django framework. The speaker implemented an instrumentation that recorded when the application switched between synchronous and asynchronous code paths, providing visibility into these transitions and the related stack traces.

How does the speaker suggest measuring the effectiveness of async Django services?

The speaker suggests focusing on end-to-end performance metrics from the user's point of view, such as latency, memory savings, and the ability to serve more requests with the same resources. He emphasized that while overhead from sync-to-async transitions is present, the primary goal should be optimizing resource usage.

What were some of the technical challenges the speaker faced during his transition to async Django?

Some of the technical challenges included handling missing async implementations for certain features like model view sets and HTML rendering, dealing with the absence of default permissions, and addressing the issue of related fields not being properly handled in async serializers.

What is the current state of async support in the Django framework, according to the speaker?

According to the speaker, async support in Django is still incomplete. While there are external packages and some basic async features available, there are significant gaps in functionality, especially in areas like view sets and database interaction, which make it unsuitable for production use in its current state.

How did the speaker address missing functionality in Django's async API?

The speaker manually implemented the missing functionalities, such as view sets, by writing custom code and using existing sync-to-async helpers. He also removed certain features, like authentication, that were not implemented for async views in the Django framework at the time.