An Odyssey with ArgoCD. From Git to Helm | Farah Adbib & Antonio Alferez

Summary

This presentation outlines a company’s journey of implementing Argo CD and Helm to streamline Kubernetes deployments. The speakers, Anton and Farah, describe their initial decisions around infrastructure, including security concerns, cluster management, and deployment processes. They explore the challenges faced with Git repositories, Helm charts, and CI/CD pipelines, as well as the lessons learned from the pitfalls of their earlier approach. By shifting to Helm rendered manifests and refining their pipeline structure, they significantly improved deployment efficiency, making the process more developer-friendly while ensuring better version control and faster feedback loops. Key takeaways focus on the importance of aligning tools with company-specific needs.

Takeaways

• 😀 The ecosystem and requirements of your company play a crucial role in shaping decisions about tools and approaches, as what works for one company may not work for another.
• 😀 Security and isolation needs drove the decision to have a separate Argo CD instance per Kubernetes cluster, enhancing control over clusters with differing requirements.
• 😀 Developers must be able to easily deploy applications, so any deployment mechanism should be user-friendly for engineering teams.
• 😀 Argo CD was initially integrated with Git as the source repository, but this approach led to the complexity of managing G mirrors and repositories for each cluster.
• 😀 Separating application code from Helm charts and splitting them into different repositories created challenges in version control and synchronization, leading to confusion among developers.
• 😀 The initial configuration required multiple repositories, pipelines, and versions, which became difficult to manage as the system evolved.
• 😀 The concept of rendered Helm manifests, where Helm charts are pre-rendered in the CI/CD pipeline before being pushed to Argo CD, significantly improved the deployment process.
• 😀 Shifting the rendering logic to the CI/CD pipeline reduced the load on Argo CD, speeding up feedback loops and simplifying deployment.
• 😀 By unifying the application code, Helm charts, and Docker files into one repository, the team simplified versioning, deployment, and maintenance processes.
• 😀 The new system introduces versioning compatibility by using the same versioning scheme for both Docker images and Helm charts, making it easier to track what’s deployed in each environment.
• 😀 Introducing features like deployment from a branch is now much simpler, thanks to the new versioning system, streamlining the deployment process and enabling smoother operational changes.

Q & A

Why did the team choose to use Argo CD with Helm for their deployment process?

The team chose Argo CD with Helm due to the need for a centralized deployment system that integrates with Kubernetes clusters. They wanted to automate and streamline the application deployment across 55 clusters with varying environments while maintaining control over versions and ensuring the process was easy for developers.

What security concerns influenced the team’s decision to deploy Argo CD with Helm?

The team’s decision was influenced by the need for security isolation between different data centers. They opted for a setup where each Kubernetes cluster had its own instance of Argo CD to maintain isolated environments and control over which clusters could access certain artifacts.

How did the team manage different Kubernetes clusters and environments?

The team managed the clusters by setting up different configurations for each cluster and environment. They used separate repositories for application code, Helm charts, and values to ensure easy deployment and clear separation between clusters and environments.

What challenges did the team face with their initial implementation of Argo CD and Helm?

The team faced several challenges including the complexity of managing multiple repositories, the need for G mirrors, confusion over development and operational branches, and difficulties with version control. These challenges slowed down the deployment process and led to misalignments in application versions.

What changes did the team make to overcome these challenges?

The team moved rendering logic out of Argo CD and into their CI/CD pipeline, eliminated the need for G mirrors, unified repositories, and streamlined version management by using a single version for both the application and Helm chart. They also introduced a separate repository for values override to make changes more efficient.

How did the switch to rendering Helm charts in the CI/CD pipeline improve the deployment process?

By moving the rendering of Helm charts into the CI/CD pipeline, the team was able to catch issues earlier in the process, avoid rendering delays in Argo CD, and simplify the deployment process. It allowed them to push pre-rendered manifest files into Git, reducing the complexity of Helm templating within Argo CD.

What is the benefit of using a single version for both the application code and Helm charts?

Using a single version for both the application code and Helm charts ensures consistency and simplifies the management process. It allows the team to track specific versions easily, as they are aligned, making deployments and troubleshooting more straightforward.

What was the impact of introducing the concept of 'Helm rendered manifests'?

The introduction of 'Helm rendered manifests' allowed the team to generate and store the final, rendered manifest files, eliminating the need for Argo CD to handle rendering and templating. This approach reduced complexity, sped up feedback loops, and made it easier to track changes through Git commits.

How did the team ensure that they could quickly identify the specific versions deployed in their clusters?

The team used Git tags and commit hashes to track specific versions deployed in each cluster. By tagging commits and using Git notes, they could easily associate a version of the application with a particular environment and track any changes made.

What key takeaway did the team learn about introducing new infrastructure and processes?

The key takeaway was that ecosystem-specific requirements are crucial when introducing new infrastructure. It’s important to understand your own environment and adjust solutions accordingly. Additionally, the team learned that developer experience should be a primary consideration, and not to fall into the sunk cost fallacy when solutions are not working as expected.