LitmusChaos Contributors Meeting - October 24th 2024 | Chaos Engineering

Summary

In this community meeting, key updates and contributions to the Litmus ecosystem were shared. Highlights included a new password update feature in Litmus ETL, improvements to pod log retrieval, and enhanced annotation support for Helm deployments. Contributions from Dong Yong Kim and Calvin Audier focused on improving user experience and simplifying configuration management. Additionally, Sho presented optimizations to the Chaos Exporter, significantly reducing unnecessary load on the Kube API server. The meeting also featured an announcement about the upcoming Kos Carnival event and a call for papers. Overall, it was a concise session with a focus on technical enhancements and community engagement.

Takeaways

• 😀 The meeting began with a warm welcome to the community members and an announcement about the upcoming Kos Carnival, scheduled for January 22-23, 2024, with CFPs open until November 4th (Indian Time).
• 😀 Balik Shan presented updates on password upgrades and the ability to delete or rerun experiments in G-State within Litmus Chaos Engineering.
• 😀 A new command (`CTL update password`) allows users to update their passwords directly through Litmus C, with an additional PR to prompt users to update their password upon initial login.
• 😀 Dong Yung Kim discussed two main issues: making pod log retrieval more intuitive and fixing project creation errors related to default password handling when using Litmus CTL.
• 😀 The first issue involved improving the ARL fetching process to provide more informative logs when experiments fail to fetch, helping users better understand the state of their tasks.
• 😀 The second issue addressed a project creation error in which the default password was not updated, despite the project being created. The fix involved separating condition handling and improving readability.
• 😀 Calvin Audier introduced support for P annotations in Litmus Helm, particularly for adding global pod annotations, which reduces redundancy when deploying multiple pods.
• 😀 The P annotation support helps manage required custom annotations for tracking and reporting, particularly for enterprise-level users.
• 😀 Sho highlighted optimizations in the Chaos Exporter, focusing on reducing unnecessary API server load by transitioning from direct polling to using shared Informers.
• 😀 The change to the Chaos Exporter reduces load on the Kubernetes API server and improves reconciliation efficiency by using event-based triggers instead of a constant one-second polling loop.
• 😀 Sho mentioned that benchmarking is in progress to assess the performance impact of the optimizations, with planned stress tests to verify improvements under high load conditions.

Q & A

What is the purpose of the Kos Carnival mentioned in the meeting?

The Kos Carnival is a virtual event scheduled for January 22-23, focusing on two main tracks: Chaos Engineering and Incident Management. The event aims to bring together community members to share knowledge and expertise, with CFP submissions open until November 4, 2024.

What is the new feature introduced by Balik Shan regarding G State experiments?

Balik Shan introduced a feature that allows users to delete or stop G State experiments once they are initiated. This provides more flexibility in managing experiments in the Litmus platform.

How does the new password update feature in Litmus CLI work?

The new password update feature allows users to update their password directly from the Litmus CLI using the command `ctl update password`. This makes the password update process more seamless and user-friendly.

What was the issue Dong Yung Kim addressed related to pod logs?

Dong Yung Kim worked on improving the clarity and intuitiveness of pod logs. The issue was that the logs returned the same error message without providing an explanation, making it difficult for users to understand the cause of failure. The improvement ensures that the logs now display more meaningful messages based on the state of the experiment.

What specific problem did Dong Yung Kim address regarding project creation?

Dong Yung Kim fixed an issue where creating a new project after logging in with an un-updated default password caused an error, even though the project was created successfully. The fix improved error handling and ensured that the creation process is more reliable.

What contribution did Calvin Audier make regarding Litmus Helm?

Calvin Audier added support for pod annotations in Litmus Helm, particularly introducing a global pod annotation option. This improvement reduces the need for repetitive annotation configurations across different pods, making the deployment process more efficient.

How does the P annotation support improve the Litmus Helm deployment?

The P annotation support in Litmus Helm enables users to apply a global annotation to all pods with a single configuration, rather than applying it individually to each component. This reduces redundancy and simplifies management of pod annotations.

What was the problem Sho addressed with the Chaos Exporter, and how did he solve it?

Sho addressed performance issues with the Chaos Exporter, which was causing unnecessary load on the API server by polling for updates every second. He optimized the process by switching to an event-driven approach using shared informers, reducing the load on the server and improving the efficiency of metric updates.

What is the impact of switching to shared informers for Chaos Exporter?

Switching to shared informers reduces the strain on the API server by eliminating constant polling. Instead, the system now updates metrics only when there is a change in the state of a chaos engine or result. This results in reduced API load and more efficient reconciliation loops.

What future plans are mentioned regarding performance benchmarking for the Chaos Exporter?

Sho mentioned plans to conduct a stress test with Shubam to properly benchmark the new optimizations for the Chaos Exporter. Although preliminary tests showed some improvement, a full-scale benchmark is needed to measure the impact under heavier loads.