How to Propagate Secrets Everywhere with External Secrets Operator (ESO) and Crossplane

Summary

The video discusses managing secrets in a Kubernetes environment, emphasizing the importance of secure and efficient handling. It introduces the use of external Secrets operator for pulling and pushing secrets between different secret managers and clusters. The speaker shares their experience in setting up a control plane cluster and using Crossplane to create new clusters with necessary secrets automatically propagated. The video provides a detailed walkthrough of the process, including the use of policies for access control, and demonstrates how to handle secrets for both cluster creation and database access, ensuring security and transparency for end-users.

Takeaways

• 🔐 The video discusses the management of secrets in a Kubernetes environment, emphasizing the importance of secure and efficient handling.
• 📚 The speaker aims to answer less commonly asked but important questions about making secrets easy for users, propagation, generation, and safe usage across different locations.
• 🖥️ The speaker manages multiple resources like clusters, applications, and databases, some of which generate or consume secrets, and highlights the challenges in handling them.
• 🔄 The issue with manually added secrets in secret stores and Kubernetes secrets that are not stored in these stores is addressed as a problematic situation.
• 🤖 The use of external Secrets operator is mentioned as a tool to pull and push secrets from and to secret managers like Google Secret Manager, AWS Secret Manager, and Azure Key Vault.
• 🚀 The speaker's requirement for a control plane cluster is to have secrets pulled from a manager and automatically loaded with necessary credentials and secrets.
• 🔄 The concept of copying secrets from one place to another is discussed, with a preference for pushing and pulling secrets to and from a secret manager for safety.
• 🛠️ The video provides a solution using Crossplane and external Secrets operator to automate the process of creating clusters pre-loaded with necessary secrets and operators.
• 🔑 The importance of policies in ensuring that only authorized users can create cluster claims and have access to certain secrets is emphasized for security.
• 📈 The video includes a practical demonstration of the setup using YAML files and the Crossplane trace command to show the creation and management of resources and secrets.
• 🎯 The ultimate goal is to make the handling of secrets transparent and effortless for end users, allowing them to focus on what matters while maintaining security and safety.

Q & A

What is the main focus of the video regarding Secrets management?

The main focus of the video is to discuss and address less commonly asked but important questions about managing Secrets in a way that makes them easy and irrelevant to users, propagates them safely, and allows their generation and use in different places without exposure.

What problem does the speaker face with managing multiple resources and Secrets?

The speaker faces the problem of managing a variety of resources like clusters, applications, and databases that generate or consume Secrets. Some Secrets are manually added to secret stores, while others are generated as Kubernetes secrets and not stored in any secret manager, leading to a potentially insecure situation.

How does the speaker propose to solve the problem of making Secrets management automatic and less burdensome?

The speaker proposes using tools like the external Secrets operator to pull and push secrets from and to secret managers, and automating the process of creating clusters pre-loaded with necessary Secrets and credentials, making the whole process transparent and seamless for end users.

What is the role of the external Secrets operator in the speaker's solution?

The external Secrets operator plays a crucial role by enabling the pulling of secrets from a secret manager to a Kubernetes cluster and pushing secrets to a secret manager from one place to another. It helps in authenticating with the secret manager and facilitates the secure propagation of Secrets.

How does the speaker handle the creation and management of Secrets for a new database in one place and an application consuming that database in another?

The speaker uses a combination of Crossplane and external Secrets operator to create a secret in the control plane cluster, push it to the secret manager, and then pull it into the cluster where the application is running. This ensures that the secret is available where it is needed without being manually copied or moved.

What is the speaker's approach to ensuring security and transparency in Secrets management?

The speaker emphasizes the importance of keeping Secrets secure by storing them in a secret manager and using tools like the external Secrets operator to manage them. The goal is to make the process transparent and invisible to end users, allowing them to focus on what matters to them while ensuring their safety and security.

How does the speaker demonstrate the application of the discussed concepts in a practical scenario?

The speaker provides a detailed walkthrough of creating a Kubernetes cluster with Crossplane, managing AWS credentials, and using the external Secrets operator to pull and push secrets as needed. They also show how to handle the creation and propagation of secrets for a database and an application running in different clusters.

What is the significance of the 'creds' entry in the cluster claim?

The 'creds' entry in the cluster claim is significant because it instructs Crossplane to take the AWS credentials secret and put it into the Crossplane system namespace. This allows the new cluster created to have access to the necessary credentials for interacting with external services.

How does the speaker address the eventual consistency in the context of Secrets management?

The speaker acknowledges the concept of eventual consistency, especially when resources are being created and managed across different clusters. They explain that some resources might initially show errors, but eventually, everything will become consistent once all the resources are up and running.

What is the role of policies in ensuring the security of the entire setup?

Policies play a crucial role in defining who can do what within the system, specifically who can create cluster claims and what values are allowed. By setting up appropriate policies, the speaker ensures that the creation and management of clusters and Secrets are restricted to authorized users, enhancing the overall security of the setup.

What is the speaker's final takeaway from the video?

The speaker's final takeaway is the demonstration of how Crossplane, external Secrets, and other tools can be combined to effectively manage Secrets and resources in a way that is secure, automated, and transparent to end users. The speaker shows that with the right tools and configurations, complex tasks like propagating secrets between clusters can be made simple and seamless.