How to Propagate Secrets Everywhere with External Secrets Operator (ESO) and Crossplane
Summary
TLDRThe video discusses managing secrets in a Kubernetes environment, emphasizing the importance of secure and efficient handling. It introduces the use of external Secrets operator for pulling and pushing secrets between different secret managers and clusters. The speaker shares their experience in setting up a control plane cluster and using Crossplane to create new clusters with necessary secrets automatically propagated. The video provides a detailed walkthrough of the process, including the use of policies for access control, and demonstrates how to handle secrets for both cluster creation and database access, ensuring security and transparency for end-users.
Takeaways
- π The video discusses the management of secrets in a Kubernetes environment, emphasizing the importance of secure and efficient handling.
- π The speaker aims to answer less commonly asked but important questions about making secrets easy for users, propagation, generation, and safe usage across different locations.
- π₯οΈ The speaker manages multiple resources like clusters, applications, and databases, some of which generate or consume secrets, and highlights the challenges in handling them.
- π The issue with manually added secrets in secret stores and Kubernetes secrets that are not stored in these stores is addressed as a problematic situation.
- π€ The use of external Secrets operator is mentioned as a tool to pull and push secrets from and to secret managers like Google Secret Manager, AWS Secret Manager, and Azure Key Vault.
- π The speaker's requirement for a control plane cluster is to have secrets pulled from a manager and automatically loaded with necessary credentials and secrets.
- π The concept of copying secrets from one place to another is discussed, with a preference for pushing and pulling secrets to and from a secret manager for safety.
- π οΈ The video provides a solution using Crossplane and external Secrets operator to automate the process of creating clusters pre-loaded with necessary secrets and operators.
- π The importance of policies in ensuring that only authorized users can create cluster claims and have access to certain secrets is emphasized for security.
- π The video includes a practical demonstration of the setup using YAML files and the Crossplane trace command to show the creation and management of resources and secrets.
- π― The ultimate goal is to make the handling of secrets transparent and effortless for end users, allowing them to focus on what matters while maintaining security and safety.
Q & A
What is the main focus of the video regarding Secrets management?
-The main focus of the video is to discuss and address less commonly asked but important questions about managing Secrets in a way that makes them easy and irrelevant to users, propagates them safely, and allows their generation and use in different places without exposure.
What problem does the speaker face with managing multiple resources and Secrets?
-The speaker faces the problem of managing a variety of resources like clusters, applications, and databases that generate or consume Secrets. Some Secrets are manually added to secret stores, while others are generated as Kubernetes secrets and not stored in any secret manager, leading to a potentially insecure situation.
How does the speaker propose to solve the problem of making Secrets management automatic and less burdensome?
-The speaker proposes using tools like the external Secrets operator to pull and push secrets from and to secret managers, and automating the process of creating clusters pre-loaded with necessary Secrets and credentials, making the whole process transparent and seamless for end users.
What is the role of the external Secrets operator in the speaker's solution?
-The external Secrets operator plays a crucial role by enabling the pulling of secrets from a secret manager to a Kubernetes cluster and pushing secrets to a secret manager from one place to another. It helps in authenticating with the secret manager and facilitates the secure propagation of Secrets.
How does the speaker handle the creation and management of Secrets for a new database in one place and an application consuming that database in another?
-The speaker uses a combination of Crossplane and external Secrets operator to create a secret in the control plane cluster, push it to the secret manager, and then pull it into the cluster where the application is running. This ensures that the secret is available where it is needed without being manually copied or moved.
What is the speaker's approach to ensuring security and transparency in Secrets management?
-The speaker emphasizes the importance of keeping Secrets secure by storing them in a secret manager and using tools like the external Secrets operator to manage them. The goal is to make the process transparent and invisible to end users, allowing them to focus on what matters to them while ensuring their safety and security.
How does the speaker demonstrate the application of the discussed concepts in a practical scenario?
-The speaker provides a detailed walkthrough of creating a Kubernetes cluster with Crossplane, managing AWS credentials, and using the external Secrets operator to pull and push secrets as needed. They also show how to handle the creation and propagation of secrets for a database and an application running in different clusters.
What is the significance of the 'creds' entry in the cluster claim?
-The 'creds' entry in the cluster claim is significant because it instructs Crossplane to take the AWS credentials secret and put it into the Crossplane system namespace. This allows the new cluster created to have access to the necessary credentials for interacting with external services.
How does the speaker address the eventual consistency in the context of Secrets management?
-The speaker acknowledges the concept of eventual consistency, especially when resources are being created and managed across different clusters. They explain that some resources might initially show errors, but eventually, everything will become consistent once all the resources are up and running.
What is the role of policies in ensuring the security of the entire setup?
-Policies play a crucial role in defining who can do what within the system, specifically who can create cluster claims and what values are allowed. By setting up appropriate policies, the speaker ensures that the creation and management of clusters and Secrets are restricted to authorized users, enhancing the overall security of the setup.
What is the speaker's final takeaway from the video?
-The speaker's final takeaway is the demonstration of how Crossplane, external Secrets, and other tools can be combined to effectively manage Secrets and resources in a way that is secure, automated, and transparent to end users. The speaker shows that with the right tools and configurations, complex tasks like propagating secrets between clusters can be made simple and seamless.
Outlines
π Introduction to Secrets Management
The paragraph introduces the topic of secrets management and outlines the challenges faced when dealing with secrets in a multi-resource environment. It highlights the need for making secrets easy and irrelevant to users, propagating them securely, and generating and using them in different locations without exposing them. The speaker shares their experience in managing resources like clusters and databases that generate or consume secrets and the issues they encountered with manual secret management and the use of secret stores like Google Secret Manager, AWS Secret Manager, and Azure Key Vault. The aim is to find a solution that doesn't complicate life and makes the process as seamless as possible.
π€ The Role of External Secrets Operator
This paragraph discusses the role of the External Secrets Operator in solving some of the problems faced with secrets management. It explains how the operator can pull secrets from various secret managers and also push secrets to them. The speaker shares their need for a control plane cluster that can create, manage, and remove Kubernetes clusters with the required secrets for external services, such as container image registries. The challenge is to have these secrets automatically available from the start and not require multiple steps or manual intervention, aiming for a seamless and magical experience for users.
π Streamlining Secret Propagation and Access
The speaker presents a solution to the challenges discussed earlier by using Crossplane and the External Secrets Operator. They demonstrate how to deal with secrets without causing discomfort to users. The process involves using a cluster claim to create a Kubernetes cluster in AWS with all the necessary secrets for operation. The AWS credentials secret is manually created in the control plane cluster and used by both Crossplane and the External Secrets Operator. The operator is configured to pull secrets from the secret manager, and the end-user's experience is simplified by allowing them to write a few lines of YAML and interact with the Kubernetes API. The speaker emphasizes the importance of eventual consistency in this process and the safety of secrets stored in the secret manager.
π Moving Secrets Between Clusters
The paragraph delves into the specifics of moving secrets between clusters, particularly for databases that may be running in a different location from the applications that consume them. The speaker explains the process of creating a database using Crossplane, which generates a Kubernetes secret with authentication details in the control plane cluster. To make this secret available to the application running in a different cluster, the speaker describes a composition that pushes the secret to the secret manager and then pulls it into the target cluster. This ensures that the secret is not just copied but managed and propagated securely.
π Conclusion and Demonstration
The speaker concludes by summarizing the steps taken to manage secrets effectively using Crossplane, External Secrets, and other tools. They demonstrate the creation of an AWS RDS database server and the associated Kubernetes secrets through a composite claim. The process involves creating a root password secret in the control plane cluster, pushing it to the secret manager, and then pulling it into the target cluster where the application runs. The speaker confirms the successful creation and synchronization of secrets by listing them in the respective namespaces and emphasizes the ease and security of this approach for both developers and end-users.
Mindmap
Keywords
π‘Secrets
π‘External Secrets Operator
π‘Kubernetes
π‘Crossplane
π‘Control Plane Cluster
π‘Secret Propagation
π‘Container Image Registry
π‘Database Authentication
π‘Terraform
π‘Open Function
π‘Eventual Consistency
Highlights
The speaker aims to address less commonly asked but important questions about managing secrets in a secure and user-friendly manner.
The challenge of making secrets easy and irrelevant to users is discussed, focusing on user experience and security.
The issue of propagating secrets without exposure is explored, emphasizing the balance between accessibility and protection.
The generation and safe usage of secrets in different locations is a key topic, highlighting the complexity of modern IT infrastructure.
The speaker shares personal experiences with managing resources, clusters, applications, and databases that generate or consume secrets.
The use of secret stores like Google Secret Manager, AWS Secret Manager, and Azure Key Vault is mentioned for managing secrets.
The speaker discusses the limitations of Kubernetes secrets not being stored in secret stores, pointing out a potential security risk.
The external Secrets operator is introduced as a tool to pull and push secrets between different secret managers.
The speaker's need for a control plane cluster that can create, manage, and remove Kubernetes clusters from a central point is explained.
The requirement for clusters to have secrets pulled from a manager upon creation for interaction with outside services is detailed.
The speaker emphasizes the desire for a streamlined process where clusters are pre-loaded with necessary credentials and secrets.
The concept of moving secrets between clusters is discussed, with a preference for copying or pushing secrets to secret managers instead.
The goal of making security transparent and effortless for users is articulated, aiming to create a 'magic'θ¬η experience.
The speaker presents a solution involving Crossplane and external Secrets operator to manage AWS credentials and create Kubernetes clusters.
The process of copying the AWS credentials secret into new clusters and the use of the creds entry in Crossplane compositions are explained.
The importance of policies in ensuring secure access and usage of secrets is highlighted, discussing CNO policies and their role.
The speaker demonstrates the application of the discussed concepts through a practical example involving a new Kubernetes cluster creation.
The use of external Secrets operator for pulling secrets into a cluster and its integration with Crossplane is showcased.
The speaker addresses the eventual consistency of Kubernetes resources and how it relates to the secret management process.
A comprehensive example involving the creation of an AWS RDS database server and the propagation of its secrets to another cluster is provided.
The speaker illustrates the process of pushing and pulling secrets between secret managers and clusters for a database server use case.
The end goal of creating a secure yet transparent secret management experience for users is reiterated, emphasizing user safety and simplicity.
Transcripts
[Music]
today I want to talk about Secrets but
don't go away don't go away just yet I
know that for many Secrets is either
boring or you think you already know
everything there is to know about
managing Secrets I will not talk about
obvious secret stuff instead I want to
try to answer some less often asked yet
important questions how do we make
Secrets easy and irrelevant to users how
do we propagate Secrets without making
them exposed how do we generate secrets
in one place but use them in another
safely and a few others today's video
will be different than what you might
expect from Secrets today I want to
explore and solve a few problems I was
facing not long ago and that I believe I
solved without making anyone suffer let
me explain the problem before I show you
a possible
solution I tend to manage a number of
resources there are clusters
applications databases and so on and so
forth most of those generate or consume
Secrets some of them do both some
secrets were added manually to a Secret
store like a Google secret manager AWS
secret manager Azure key Vault or
whatever else you you are using or I'm
using others however are generated as
kubernetes secrets and not stored in any
of those secret stores which by itself
is a bad situation which I will explain
in a moment so far you probably do not
see a problem you might be inclined to
say that a tool like external Secrets
operator can solve those issues it can
pull secrets from almost any secret
manager if you're already using it you
might even be aware that it can also
push secrets to a secret manager choice
if you're not familiar with external
secrets you might want to watch that
video just don't do it right now finish
this one first if that's what you're
thinking you're right external Secrets
operator or any other similar tool
solves some of my problems but not all
let me explain what I need I have a
control plane cluster in my case that's
cross plane and in yours that might be
something else it's could be a simple
server from which you run terraform or a
pipeline or anything else from where you
do stuff it does not matter for now what
does matter is that kubernetes clusters
are created managed and removed from
that control plane now those clusters
need to have a number of secrets that
enable them to interact with outside
Services a good example would be
credentials that must be used to pull or
even push images from or to a container
image registry almost almost every
cluster needs some secrets to operate
correctly now those Secrets should be
pulled from whichever Secrets manager I
might be using and that means that they
all have to have external Secrets
operator or whichever other similar tool
we might be using however that operator
needs a secret with authentication so
that it can fetch secrets from the
secrets manager to be honest I know that
there are many many ways to accomplish
all that but to tend to complicate My
Life by treating all those requirements
as Baseline requirements as table Stakes
when I do create a cluster all those
need to be there from the start and I
don't want to have to perform many steps
to get them ideally I should say create
a cluster and apart from a cluster being
created it should come pre-loaded with
the external Secrets operator
credentials it needs and secrets pulled
I don't want to work to get those they
just need to get their automatically
there's more though sometimes I will set
up a new database in one place but run
an application that consumes that
database in another I might for example
instruct the control plane to create and
manage a database and that will among
other things result in a secret with
database authentication being created in
the control plane cluster I don't need
that secret there I need it somewhere
else I need it in the cluster where the
application that consumes assumes the
database is running hence that secret
needs to be moved from one place to
another actually that's not correct it
should not be moved but rather copied
however I cannot just copy the secret
that would be silly since I'm supposed
to keep secrets in the secret manager
hence I need to push that secret to the
secret manager from one place and pull
it from the secret manager to another
there's still more though I don't want
those things to be a burden to others
creating and managing databases or
applications or whatever they're
managing I want all that to be
transparent no invisible I want people
to be secure yet not to have to learn
all the details or worry about those
things it should be like magic people
should be in control of things that
matter to them while those that don't
should just happen Secrets or even
Security in general fall into the I do
not care that I am safe category
normally I have quite a few other
requirements but those I discussed so
far are enough to give you an idea what
I'm trying to accomplish that was enough
talk let me show you how I solved those
issues let me show you how to deal with
Secrets without making anyone
suffer before we proceed I should tell
you that all the commands I will execute
in this video are available in aist and
the link to it is in the description
please use it if you would like to
reproduce what I'm showcasing right now
my cube cutle is pointing to the control
plane cluster over there if you take a
look at the crossplane system Nam space
we can see that aw crets is available I
created that secret manually and
hopefully hopefully that's the only
secret I will manage myself that secret
contains AWS credentials which for
example are used by crossplane to
communicate with my AWS account to
create kubernetes clusters databases and
whatever else I need at the same time
that secret is also used by external
Secrets operator to pull Secrets I
instructed to pull by the way I made an
assumption that you're familiar with
crossplane if you're not please watch
those videos or even better watch the
crossplay tutorial I published uh as a
set of videos short file ago the link to
the playlist is in the description
alternatively you can explore the
composition yourself the link to the
repo is in the description as well the
important note within the context of
today's subject is that I need that
secret to be available in clusters I
will create from this control plane so
let's take a look at one example and
this is the part that makes everything
simple for the end users it is a cluster
claim that creates everything required
to run a kubernetes cluster in AWS if
you followed my work you already saw a
variation of such crossplay composition
so I will not go into details the
interesting part is the creds entry that
font tells crossplane to take the AWS
creds secret and put it into the
crossplane system namespace that secret
in that cluster should contain the creds
access key ID and secret access key keys
as a result crossplane will create a
cluster and everything it needs and it
will copy the AWS cret secret into that
new cluster now to be completely clear
that's not the best way to propagate
secrets from one place to another a
better way would be to use external
Secrets operator and instruct it to pull
a secret from a secret manager however
to do that we need external secrets to
be able to authenticate with the secrets
manager and that's the primary purpose
of coping AWS creds secret an important
note is that the end user has freedom to
choose which secret to copy to which
namespace to place it and which keys to
include on the other hand the end user
does not need to worry about the details
there's no need to know how it was done
and what's the magic behind the process
further on we have the open function
field which installs open function in
that cluster that one is not directly
related to Secrets however among other
things open function builds container
images and stores them in a registry for
that to work it needs credentials stored
in a kubernetes secret by the way if
you're not familiar with open function
please check out that video uh since I
already explained it over there I will
not go into details of how it works and
what it does in in this one in this
video to accommodate the need for a
secret with container registry
authentication as well as other secrets
we might need that is the external
Secrets section in it we have the
enabled field that will ensure that
external Secrets operator is deployed to
the cluster and the store which will
create cluster Secrets store resource
that's the one that needs a secret that
allows it to connect to the secret
manager that secret is retrieved from
the aw crets we explored earlier and all
we have to do is specify the keys
finally with external Secrets operator
fully operational and capable of
fetching secrets from the secrets
manager we are listing all the secrets
we would like to pull into that cluster
in this specific case there is one that
provides authentication to The Container
image registry needed for oper function
now now you might be thinking that the
whole setup is not secure and you would
be right secrets are stored safely in
the secrets manager but anyone could
create a cluster with any or all of the
secrets that's where policies would come
in typically I would set up CNO policies
that would define who can do what and
more specifically who can create those
cluster claims and which values are
allowed that combined with the fact that
secrets are safely pulled from a Secrets
manager makes it pretty safe except the
fact that kubernetes Secrets themselves
are not always safe but that would be a
subject for a separate video the good
news is that the end users do not need
to worry about any of the things we
discussed all they have to do is write
those 30s something lines of yam and
send it to the kubernetes API so let's
do just that now we can observe what's
happening with the crossplane trace
command quite a few resources were
created in the control plane cluster
some of them will create and manage a WS
resources others will create stuff in
the control plane cluster while some
will set up things in the new cluster
once it's operational you will notice
that quite a few resources are having
errors that's okay some of them like for
example Nam spaces cannot be created yet
since the cluster where they should be
applied is not yet operational
everything will be eventually consistent
and while waiting for that to happen we
can focus on the few of those resources
related to the secrets we discussed by
the way I explored eventual consistency
both from the theoretical and from the
Practical perspectives in those videos
over there uh so please check them out
if you would like to understand better
what I meant by uh all this by eventual
consistency we have the ATM cluster
credits resources that is copying the
hyperscaler credentials from the control
plane cluster to the new cluster that's
the one external Secrets needs to
authenticate with the secrets manager
which in my case is in AWS then we have
a cluster Secret store that will create
external Secret store that uses those
credentials finally there is the push
secret that will be pulled from the
secrets manager and which will be used
by open function to push those images
that's it that was yet another
explanation that matters to you the
person that builds who builds services
like the one we are exploring today but
not to the end users they do not need to
care yet they will be safe all that's
left to do is to wait until all the
resources are up and running so let's
fast forward and Trace the claim one
more time there might be a problem with
open function it is still in creating
state or to be more precise crossplay
things that it is creating but it's
probably up and running it's a massive
massive Helm chart that takes ages to
deploy as and it's probably timing out I
I'll fix that later it's that's not
relevant for this story what does matter
is that we can now retrieve Cube config
and retrieve secrets from the crossplane
system namespace it's there the AWS cred
secret was automatically created in this
new cluster we can also list external
Secret store and observe that that it
was created correctly the fact that the
status is valid means that it was able
to connect to the secrets manager in AWS
in this case we can also list external
Secrets resources and observe that push
secret was synced that means that it
succeeded in pulling the register secret
from the secret manager we can confirm
that as well by retrieving kubernetes
secrets from the production Nam space
the push secret is there that's the one
that contains the authentication to The
Container image registry open function
needs what did we do let me paint it for
you let me show you a diagram of what we
did the control plane cluster contained
crossplane and a secret with the AWS
authentication from there on we applied
a crossplane composite claim that
created a new kubernetes cluster in my
case eks with everything that it
requires the secret from the control
plane was copied to the new cluster and
external Secrets operator was installed
and configured using that secret to
communicate with the aw Secrets manager
finally external Secrets operator pulled
the secret with credentials from the
container image registry which will be
used by open function that was also
installed and configured inside that new
cluster but here's a big part that's not
enough I have a potentially more
interesting example to
demonstrate databases are special they
often do not run in the same place where
applications are running for example I
might have an awsrds database and then
applic running in a kubernetes cluster
if RDS is created with crossplane
crossplane will create a kubernetes
secret with the authentication details
however that secret will be created in
the control plane cluster the one where
cross plane is running while the
application that should use it might be
running in a different cluster that
means that we might need to safely
propagate the secret from one cluster to
another ideally we should not just copy
the secret but push it to the secrets
manager and pull it from there wherever
we need it that's exactly what we will
do next and I already prepared a
composition that will do just that so
let's take a look at the Manifest that
will claim it that manifest will create
a poql server and everything else needed
for it to run in AWS a database inside
that server and quite a few other things
those are not important right now what
matters is the secrets section by the
way please let me know through comments
in this video if it would be interesting
to explore database as a service in one
of the upcoming videos that claim will
create a secret in the control plane
cluster the one that runs cross plane
but as I already mentioned we need it
elsewhere that elsewhere could be the
new cluster we created earlier it
already contains open function which is
a great way to define build and deploy
applications which might need to connect
to a database hence we need the secret
in that other cluster the store name
defines the external Secret store that's
the one we created earlier when we
created the cluster RDS requires a root
password so we are fetching it from the
aw SEC manager through the pull root
password key value that's similar to
what we did before the interesting part
comes next the push to store parameter
tells the composite resource to push
that secret it will create in the
control plane cluster to the secret
manager the van LS external Secrets
operator which we are propping here has
the capability not only to pull but also
to push Secrets further on we are
telling the claim to pull the secret
from the secret manager into the
production name space inside The A Team
cluster let's apply the claim and see
what happens first I will unset Cube
config so add Cube cutle points to the
control plane cluster and apply the
claim let's see what we got with the
cross PL phas command among others we
got the secret pull resource that pulls
root password from The hyperscalers
Secret store then there is the push
store resource that pushes DB secret to
The Secret store and finally the pull
cluster resource that pulls the secret
with the creds to the other cluster it
will take a while until all the
resources available so let's fast
forward and Trace the claim again all
the resources are available and we can
take a look at what we got to begin with
there should be the secret with the root
password in the control plane cluster
since that's the data crossplane itself
needs we can see that the external
secret was created which in turn created
the mightyb kubernetes secret with the
password pulled from the Clusters uh not
clusters but Secrets manager once the
database server was created crossplane
created a secret with the IP port
username and password that can be used
to access the database server since we
need that info in the other cluster the
one where the application that might use
that database should be running the
composite resource created the external
Secrets push secret which in turn pushed
it to the secrets manager we can confirm
that's indeed what happened by going to
the AWS console and there we go the
secret was indeed created finally the
secret should be pulled into the other
cluster so let's export Cube config
again and list all external secrets in
the production name space the myb
external secret is there and you know
what that means right if you L the
secrets in the production name space we
can see that it was pulled from the
secrets manager so what did I do what
did we do from the control plane cluster
the one with cross plane we created a
composite claim that created all the AWS
resources required to run uh RDS
database server and it created database
inside it among other things the server
needed a root password so the composite
resource created an external Secret
which in turn pulled the root password
from the aw Secrets manager and used it
to create a kubernetes secret once the
database was created cross created a
secret with the connection info since we
needed that info in a secret in a
different cluster it also created
external Secrets push secret which
pushed it to the secrets manager finally
it created an external secrets in the
cluster where the application should run
which in turn pulled the connection info
from the secrets manager and created
kubernetes secret from there on an
application can connect to the database
from that cluster by simply mounting the
secret that must have been overwhelming
but hopefully useful to see what can be
done by combining crossplane with
external secrets and a few other tools
thank you for watching see you in the
next one
cheers
Browse More Related Video
Day-32 | How to Manage Hundreds of Kubernetes clusters ??? | KOPS | #k8s #kubernetes #devops
SERVICIOS y LOAD BALANCERS en KUBERNETES
Functionality and Usage of Key Vault - AZ-900 Certification Course
HashiCorp Vault Read Write and Delete secrets - Part 3 | HashiCorp Vault tutorial series
Azure DevOps Workload Identity Federation with Azure Overview. NO MORE SECRETS!
Custom Networking for Amazon EKS using Rafay
5.0 / 5 (0 votes)