Introduction to HashiCorp Terraform with Armon Dadgar
Summary
TLDRThe transcript introduces Terraform, a tool for infrastructure provisioning, emphasizing its ability to handle both initial setup (day one) and ongoing evolution (day two plus). Terraform operates on an infrastructure as code approach, allowing users to define their infrastructure in config files. The core process involves three main commands: refresh, plan, and apply, which help reconcile the desired state with the current state of infrastructure. As usage scales from individuals to organizations, Terraform supports collaboration through version control and Terraform Enterprise, and modularity for broad consumption, promoting a unified workflow for infrastructure management.
Takeaways
- π Terraform is a tool for infrastructure provisioning that addresses both the initial setup (day one) and the ongoing evolution of infrastructure (day two plus).
- π Infrastructure as code is a core concept of Terraform, where infrastructure is described declaratively in human-readable configuration files.
- π Terraform manages infrastructure across various providers, including cloud providers like AWS, Azure, and GCP, as well as on-premise solutions like OpenStack and VMware.
- π οΈ The main Terraform workflow involves three commands: refresh, plan, and apply, which help in reconciling the real-world state with the desired configuration.
- π The refresh command updates Terraform's understanding of the current infrastructure state by querying the respective infrastructure providers.
- π The plan command helps in determining the actions needed to achieve the desired state from the current state, providing a roadmap for infrastructure changes.
- π The apply command executes the changes outlined in the plan, managing dependencies and sequencing to ensure a smooth transition to the desired infrastructure state.
- π Terraform's architecture consists of a monolithic core that manages the lifecycle of resources and supports numerous providers for connecting with different services.
- π Terraform Enterprise is a solution for team collaboration, providing centralized state management, sequential execution of changes, and secure handling of sensitive credentials.
- π¦ Modules in Terraform allow for encapsulation of infrastructure patterns, making complex setups accessible to a broader audience by defining input and output variables.
- π A central registry for Terraform modules exists, enabling sharing of common infrastructure patterns and facilitating self-service infrastructure management within organizations.
Q & A
What is Terraform and what problem does it solve?
-Terraform is a tool for infrastructure provisioning. It addresses the challenges of not only setting up initial infrastructure (day one) but also evolving it over time (day two and beyond). Terraform allows users to define their infrastructure in a set of configuration files, adopting an infrastructure as code approach.
What is the significance of the 'infrastructure as code' approach that Terraform uses?
-The 'infrastructure as code' approach allows users to declaratively define their infrastructure in human-readable configuration files. This makes the infrastructure reproducible, version-controlled, and easier to manage. It simplifies the process of provisioning and evolving infrastructure by treating it as a software development task.
What are the three main commands used in Terraform's workflow?
-The three main commands in Terraform's workflow are 'terraform refresh', 'terraform plan', and 'terraform apply'. 'Refresh' reconciles Terraform's view of the infrastructure with the real world, 'plan' determines the changes needed to reach the desired state, and 'apply' executes the plan against the real world to update the infrastructure accordingly.
How does Terraform handle dependencies between different infrastructure components?
-Terraform intelligently determines the correct order for provisioning resources based on their dependencies. It can identify which resources need to be created or updated sequentially and which can be done in parallel, ensuring that the infrastructure is built and managed efficiently.
What is the role of Terraform's core in its architecture?
-Terraform's core is responsible for processing the configuration files provided by the user and managing Terraform's state. It figures out the relationships between different resources, determines what needs to be created, updated, or destroyed, and manages the entire lifecycle of the infrastructure.
How does Terraform support connectivity with various infrastructure providers?
-Terraform supports many different providers that allow it to connect with a wide range of infrastructure services, including cloud providers like AWS, Azure, and GCP, on-premise infrastructure like OpenStack or VMware, and higher-level platforms and services like Heroku, Kubernetes, and Lambda functions.
What is Terraform Enterprise and how does it enhance the Terraform workflow?
-Terraform Enterprise is a version control system for Terraform configurations. It allows teams to collaborate on managing the same infrastructure by centralizing the state file and ensuring sequential application of changes. It also provides features for managing sensitive credentials securely and avoiding conflicts when multiple team members work on the same infrastructure.
What challenges arise when multiple team members collaborate on Terraform configurations?
-When multiple team members collaborate, challenges include maintaining a consistent view of the configuration, avoiding conflicts when running changes in parallel, and managing sensitive credentials securely. Terraform Enterprise addresses these issues by centralizing state management, ensuring sequential application of changes, and encrypting sensitive variables.
How do Terraform modules help in managing and sharing infrastructure configurations?
-Terraform modules encapsulate infrastructure configurations as black boxes, with input variables to customize their behavior and output variables to provide results. They simplify the management and sharing of common infrastructure patterns by hiding the complexity and making them easily consumable for a broader audience.
What is the role of the Terraform registry in the ecosystem?
-The Terraform registry is a central repository where users can find and share modules for common infrastructure patterns. It includes official modules from cloud providers and contributions from the community, making it easier for users to adopt best practices and manage infrastructure in various environments.
How can organizations use a private Terraform registry to standardize infrastructure management?
-Organizations can run a private Terraform registry within their infrastructure to standardize and manage their internal infrastructure patterns. Publishers can contribute approved patterns to the registry, allowing a larger base of consumers to self-service their infrastructure needs while adhering to organizational best practices.
Outlines
π οΈ Introduction to Terraform and Provisioning
This paragraph introduces Armon Dadgar speaking about Terraform, a tool for infrastructure provisioning. It explains the two main challenges in provisioning: the day one challenge of going from nothing to something, and the day two plus challenge of evolving existing infrastructure over time. Terraform addresses these by using an infrastructure as code approach, allowing users to declaratively define their infrastructure in configuration files. This approach enables the description of infrastructure topology in a simple, human-readable way, covering aspects like VPCs, security groups, virtual machines, and load balancers. The paragraph outlines the Terraform workflow, which includes commands like refresh, plan, and apply, to manage the creation, modification, and destruction of infrastructure resources.
π Terraform Workflow and Evolution
The second paragraph delves into the Terraform workflow, emphasizing the consistency between day one and day two experiences. It describes how Terraform's refresh command updates its view of the infrastructure, the plan command figures out the necessary changes to reach the desired configuration, and the apply command executes these changes. The paragraph also touches on the concept of decommissioning infrastructure using Terraform's destroy command. The advantages of this workflow include the ability to manage infrastructure evolution and the efficient cleanup of resources at the end of their lifecycle. The paragraph then discusses Terraform's architecture, highlighting its core component responsible for lifecycle management and the providers that connect Terraform to various services and resources.
π Scaling Terraform in Teams and Organizations
This paragraph explores how Terraform is used in team and organizational settings, starting with individual practitioners and scaling up to larger groups. It addresses the challenges of collaboration, such as maintaining a consistent view of configurations and avoiding conflicts when multiple team members make changes simultaneously. The paragraph introduces Terraform Enterprise as a solution for centralizing state management and ensuring sequential application of changes. It also discusses the management of sensitive credentials and the transition from early operators to a broader development audience. The concept of Terraform modules is introduced as a way to encapsulate complexity and make infrastructure management more accessible to a wider audience.
π Sharing and Managing Terraform Modules
The final paragraph focuses on the management and sharing of Terraform modules, both through a global public registry and private registries within organizations. It explains how modules published by cloud providers and internal teams can simplify infrastructure management by abstracting away complexity and presenting a standardized approach to provisioning. The paragraph discusses the benefits of using a private registry within an organization's Terraform Enterprise setup, allowing for the publication of approved infrastructure patterns and self-service access for internal developers. It outlines different ways of consuming modules, from direct Terraform code usage to a more graphical, WYSIWYG approach for those less familiar with Terraform. The paragraph concludes by emphasizing Terraform's role in enabling a wide range of users to self-serve infrastructure while adhering to organizational standards and not overburdening the providers with custom infrastructure generation.
Mindmap
Keywords
π‘Terraform
π‘Provisioning
π‘Infrastructure as Code (IaC)
π‘Configuration Files
π‘Workflow
π‘State Management
π‘Providers
π‘Terraform Enterprise
π‘Modules
π‘Registry
Highlights
Terraform is a tool for infrastructure provisioning.
Provisioning involves two main challenges: day one and day two plus.
Day one challenge is going from running nothing to running something.
Day two plus challenge is evolving existing infrastructure over time.
Terraform uses an infrastructure as code approach.
Infrastructure is defined in Terraform config files.
Terraform allows for the declarative definition of infrastructure topology.
The Terraform workflow involves three main commands: refresh, plan, and apply.
Terraform refresh reconciles Terraform's view with the real world.
Terraform plan reconciles the real world with the desired configuration.
Terraform apply executes the plan against the real world.
Terraform's architecture consists of a monolithic core and multiple providers.
Providers are how Terraform connects to different infrastructure and services.
Terraform can manage a wide range of resources, from IaaS to SaaS.
Terraform is an open source project with over 100 providers and thousands of contributors.
Terraform Enterprise is a tool for team collaboration and infrastructure management.
Terraform modules encapsulate infrastructure complexity and can be shared through a registry.
A private registry can be run within an organization for internal module sharing.
Terraform enables self-service infrastructure for a broad audience while maintaining compliance.
Transcripts
- Hi, my name is Armon Dadgar,
and I wanna talk a little bit about Terraform today.
So Terraform, if you've never heard of it,
is our tool for doing provisioning.
So when we talk about provisioning,
there's really two different problems
we're talking about, right?
So as we provision, there is our day one challenge
where we haven't actually started running anything yet.
And so how do we go
from running nothing to running something?
And then there's actually our more challenging day two plus,
which is: great, day one, we had nothing and we provisioned
our initial set of infrastructure.
And then day two, we have an existing set of infrastructure,
but we're trying to evolve it.
We're changing it over time.
We're adding new services, removing services,
and generally evolving the way our infrastructure looks.
So how does Terraform really solve this problem?
We fundamentally believe in taking
an infrastructure as code approach.
And so what I mean by that is we allow you
to declaratively define in a set
of Terraform config files
what you want your infrastructure to look like.
So in something that's very simple and human readable,
you describe your overall topology.
And so what I mean by that is when we talk about
infrastructure as many moving pieces,
I might have a VPC or my core network.
On top of that, I might provision my security group rules,
the way my network security is set up.
And then within that environment,
I define my set of virtual machines.
I might have some VMs that I wanna provision.
And then overlaid on top of that, I have a load balancer.
And so you can imagine how this graph extends over time.
I add different pieces,
different amounts of a complexity to it,
and I incrementally evolve my infrastructure
day over day, week over week.
And so while this is sort of graphical in nature,
the way we capture it in a Terraform configuration
is very lightweight in describing
sort of high-level resources.
We describe, you know, the VPC resource
and the fields that are required
to create this element of our infrastructure.
And then as we define these other things,
we can reference the other components of our infrastructure.
And so what Terraform does is gives us a workflow
for creating and managing this.
And ultimately, this is sort of three different
main commands that we run.
The first command is Terraform refresh.
And what Terraform refresh does is basically reconcile
what Terraform thinks the world looks like,
Terraform's view of our infrastructure,
with the real world.
And so how this works is Terraform goes out
and talks to the real world.
It'll query our different infrastructure providers,
whether that's VMware, or Amazon, or Azure,
and ask them, what's actually running?
And so in this way, Terraform gets an up-to-date view
of what our infrastructure actually is.
Then we run a plan.
A plan is where Terraform
figures out what it needs to do.
And so you can think about a plan as reconciling
sort of the real world,
what things are actually running,
with what we want to be running,
our sort of desired configuration.
So when we talk about Terraform's configuration,
what we're putting into this graph,
this is our desired state of the world.
This might not actually be the world as it is.
So as an example, on day one, when we execute this,
nothing is yet running.
So when Terraform runs the plan,
it realizes that there is nothing
yet running in the real world.
So to get to this desired configuration,
Terraform needs to boot all four of these resources.
And so this becomes the plan.
Terraform will give us an output that says:
to bring the world to look like what you want it to be,
I'm gonna go boot these four things.
So then the last part is actually doing this.
This is the apply phase of Terraform's execution.
So when we apply Terraform's execution plan,
it starts with our plan of what we need to do,
and it goes out and executes it against the real world.
And so in doing this, Terraform will go out and figure out,
what is the right order in which these need to be done?
So there's a natural dependency here.
The network must be defined first
before we can define the security groups around it.
Once those are defined, we can boot all of our VMs,
whether we're booting one or booting,
you know, hundreds of these in parallel.
And Terraform will figure out where it has
that opportunity to create things in parallel
and where it needs to be sequential.
And once these VMs are available,
then Terraform can lastly create
our load balancer and say, all done.
So this gives us a way to do our day one infrastructure.
So day one, we described
what we want our universe to look like.
We ran Terraform refresh,
it said there was nothing existing yet.
Plan said all of these things are gonna be created,
and then apply goes out and builds it.
What now happens over day two
is we start evolving our infrastructure.
So where this was our infrastructure day one.
Day two we might come in and say,
I really want a DNS record that points to my load balancer.
I want to have, you know,
a content distribution network
like Fastly sitting in front of that.
And I want my VMs to be connected to a monitoring system,
so I might monitor these different VMs.
And so now we change our Terraform configuration,
we enhance it by adding, you know,
the definition of these other elements to our TF config,
and we rerun the exact same workflow.
So when we run refresh,
Terraform will realize these first four resources exist,
but these new ones do not.
And so when we run our plan, it'll tell us
that nothing needs to be done with these four.
They already exist and they're in the desired state,
but that we must add three new elements
to get to where we need to be.
And so now when we run Terraform apply,
it goes ahead and talks to the real world
and adds these three new things
and brings the world into looking like our desired state.
The advantage of this workflow is a few fold.
One, our day one experience is identical to our day two.
And this really is important
because day two is sort of forever:
this is something we're gonna keep doing.
And then the other advantage of this
is something we actually skipped,
which is day N when we decide we might need
to decommission this.
We don't need this service anymore.
We don't want this infrastructure.
Maybe this was just a staging environment
or a test environment.
We then have the option to come in and destroy.
And when we do a destroy, it's basically an unwinding
of everything that's been done, right?
In some sense, it's a specialized version of apply,
which really looks at, creates a special destroy plan,
and then goes out and talks to the real world
to destroy all the elements that exist.
And so now we can efficiently, at the end of our lifecycle,
go from nothing to something, evolving it day over day.
And the final day, when we need to decommission it,
Terraform knows how to clean up all the resources
associated with that application.
So, so far, everything we've talked about
is really about the workflow.
This is not specific to any technology.
So how does Terraform actually make any of this stuff work?
This is where Terraform's architecture becomes important.
So when we talk about the way Terraform actually works,
there's sort of two major components.
One is Terraform's core.
So the monolithic core is responsible for a few things.
It takes the Terraform configuration,
which is being provided by the user,
and then it takes Terraform's view of the world
or Terraform's state.
This is managed by Terraform itself.
And so these get fed into the core,
and the core is responsible for figuring out,
what is that graph of our different resources?
How do these different pieces relate to each other?
What needs to be created, what needs to be updated,
what needs to be destroyed?
And so it does all of the essential lifecycle management.
On the backside, Terraform supports
many different providers.
And providers are how Terraform
connects out to the rest of the world.
So these can be things like cloud providers,
things like AWS, or Azure, or GCP.
They could be on-premise infrastructure.
So things like OpenStack or VMware,
but it's not restricted to just infrastructure as a service.
Terraform can also manage higher-level things,
platforms as a service such as Heroku, or Kubernetes,
or things like Lambda functions.
So these are higher level
than what we would traditionally consider
infrastructure as a service,
more in the realm of a platform as a service.
The other types of things Terraform can manage
are pure software as a service.
Very high-level things, maybe monitoring like Datadog,
or CDNs like Fastly,
or even things like managing GitHub teams.
So these are what we'd consider pure software as a service.
These are externally managed services,
but the view that we take as Terraform authors
is that all of these pieces are connected.
When I talk about a modern infrastructure,
I might be composing all of these different resources.
So I might have some set of my infrastructure
that is infrastructure as a service,
and then I might compose that and say, you know what?
I'm gonna provision a Kubernetes cluster,
and on top of my Kubernetes cluster,
I'm going to define a namespace,
and I'm going to define services.
And so these now exist
on top of infrastructure as a service,
and I might compose even these
with higher-level things like DNS and CDNs.
And so all of these
are important pieces of our infrastructure.
We can't deliver our application without the IaaS,
without the platform as a service,
and without the higher-level software as a service.
They're all part of our logical end-to-end delivery.
And what we wanna get to with Terraform
is a single, unified workflow.
We don't want it to be: you use one tool to manage this,
and now you have to figure out, you know,
how you get past this disjointed experience
to manage the next section,
and then another disjointed experience.
And you have to piece together all these things.
Instead, anything that really has an API
and has a lifecycle associated with it,
how can we enable that to be a provider
such that Terraform can help us
solve this problem end to end?
And this is really where providers come in.
Today, we have over 100 providers,
and those individually can manage
over 1,000 different resources,
where a resource can be, you know, an AWS EC2 VM,
or an Azure Block store, right?
These different types of resources or anything that a cloud
or even these higher-level things expose.
And that list is constantly growing.
Terraform is an absolutely massive open source project
with thousands of contributors.
And so every day, there's new providers and new resources.
So how does this really start getting used
at different levels within teams?
So this becomes an interesting workflow question
around how you actually manage Terraform.
So initially, this goes back
to that core workflow we talked about.
So we start with a single individual practitioner
who's using Terraform locally,
and their workflow is to start
by writing some Terraform configuration.
Then they run Terraform plan.
Plan tells them, what is this configuration gonna do?
What does Terraform think needs to be done
to apply these changes?
And if this looks good,
the practitioner will apply the changes.
And then this cycle continues.
They continue to evolve the infrastructure.
This might look very similar to software development,
where we're writing some code, we're applying unit tests,
we're committing those changes,
and then we're continuing our cycle.
So what happens is we go from a single individual,
to now we add multiple team members.
We have other people we wanna collaborate
on managing the same infrastructure.
Well, there's a few challenges,
much like with software writing, which is,
how do we make sure we have a consistent view
of what the configuration actually is?
And how do we make sure we don't step on each other's toes
and run multiple changes in parallel?
So if we both try and add five new VMs,
how do we avoid booting ten new VMs instead?
So this problem, in some sense, ends up being very similar
to the problem of using Git locally
versus using Git as a team.
So as we go to a team, we use a system like GitHub
to provide that central collaboration.
And so our equivalent of that
is what we call Terraform Enterprise.
So the Terraform Enterprise workflow
augments this a little bit.
So what we would do instead of now running it locally
and running these coordination risks, as an individual,
we're still writing our Terraform and planning locally,
but now we're pushing that into a version control system.
So this could be GitHub, could be Bitbucket,
could really be anything.
But the idea is, how do we move
to having a central repository where we're managing this
and coordinating, much like we do for source control?
And then we drive off of this
our Terraform Enterprise application.
And the goal here is a few fold.
One is how do we make sure
the state management is done locally?
So like I said, Terraform keeps track of all the resources
it's provisioned so that it knows at destroy time
what it needs to tear down,
or as we incrementally make changes, what already exists.
So this state file is an important thing
to have centrally managed so it doesn't diverge
or end up with sort of different lineages where, you know,
we fork off and then independently run Terraform,
and now the state files conflict with one another.
So by managing that centrally,
we avoid those sort of conflicts.
The other important advantage here
is that we make sure we're only doing
one run at a time, right?
So we ensure there's the sequential application.
So if Alice and Bob are both working
on the same infrastructure
and they both try and make a change,
they don't step on each other's toes.
Lastly, one of the challenges is,
oftentimes, in this environment,
developers are using local variables
to populate things like AWS IAM keys
or other sensitive credentials.
And so we don't want these sensitive credentials
to be strung about on all of our developers' machines.
Instead, we want those variables
to be kept centrally and encrypted
so that we don't have to worry about these things
getting exposed or leaking out in plain text.
We can put them in a central place
and version control and access controls
who can actually modify and view these things.
So this sort of changes our workflow
and now enables a handful of people
to collaborate safely together.
Well, what starts happening as we start wanting
to add even more people, right?
So for example, we might be going from
sort of our early operators
who are very familiar with the cloud environment
to exposing it more to our development groups.
This is where Terraform's notion of a module comes in handy.
So a Terraform module
is sort of like a black box of infrastructure
where what we can define is a set of input variables
that let us toggle the behavior of this module.
And the module might emit a set of output variables.
So an example of this might be,
I might define a module for doing Java,
for deploying my Java application in AWS.
And my inputs might be, what's the path to my JAR?
And how many instances of my Java application do I want?
And now, inside this module, we can operate as a black box
and deploy either VMs using EC2,
we can deploy Lambda functions,
we could deploy containers with ECS.
All of these become a detail hidden within this module
that a consumer doesn't need to know about.
And their output can just be, what's the address
of the load balancer that hits my application?
So in this way, we encapsulate that complexity
and make it more accessible to a broader audience.
So how do we actually manage and share these modules?
In general, we publish these things to a central registry.
So there's a central registry
that exists today for Terraform.
If you go to registry.terraform.io,
that's our global public registry.
And there you'll find hundreds of modules
that cover common forms of infrastructure,
common architectural patterns.
And many of these are actually published
by the cloud providers themselves.
So you can log in and see Google's official recommendation
for how to provision their network,
or Azure's official recommendation
for how to help manage a virtual network inside of Azure.
And so these modules make it much more consumable for us
to operate in these environments,
because we don't have to know every detail of it.
We can just go and say,
I wanna network in Amazon, or Google, or Azure,
and fill in a few of the inputs that matter to us,
and let the module take care of the rest
of the complexity, right?
And so the same idea can be extended
to different applications,
different types of infrastructure.
One way we can apply this same pattern internally
to our organization is by running a private registry.
So instead of using the public registry
that's available sort of globally,
we can run a private registry within our organization.
And this is a capability
integrated into Terraform Enterprise.
And so what this really lets us start to do
is make this split between the set of people
internally who are publishers
and are familiar with our internal infrastructure
and how it should be managed and cloud environments.
And they can push in to basically
a catalog or a module registry,
all of the different key patterns that are needed.
So we might publish, here's how we do a Java app,
here's how we do a C# app, here's how we do a database,
and expose this out to our internal developer community.
And then our much larger base of consumers
can simply go into the catalog
and pull out the things they need.
And so these consumers can know much less
about how cloud works, but still get these approved modules
that are sort of the best practice internally, right?
And there's sort of a few ways of consuming this.
For people that are more familiar with Terraform,
they can consume it just like a normal Terraform module
and do an infrastructure as code approach.
For folks that are less familiar,
they might wanna take more of a WYSIWYG approach,
and they can do that with Terraform Enterprise
by selecting their modules,
filling in the different variables,
and basically letting the system automatically generate
the equivalent Terraform for them.
So they can either use that equivalent Terraform
as a starting point and customize it further from there.
Or if they have really no interest in learning
and using Terraform directly,
they can just use the auto-generated Terraform
and self-service their infrastructure.
And so our goal really becomes,
how do we enable this large audience of consumers
to get to self-service on infrastructure in a way
that is still going to be compliant
with how the business wants to do it
and not create this immense workload on publishers
to constantly generate custom infrastructure
and custom definitions for the vast body
of consumers within an organization.
So at a snapshot, this is Terraform,
this is how it applies an infrastructure
as code approach to provisioning,
really looking at both the day one challenge,
and more importantly and more challengingly,
the day two-plus challenge.
And then the natural journey
as we go from a single individual using Terraform
to a small team, to kind of an entire organization
who's trying to leverage Terraform
for infrastructure management.
If you're interested in learning more about Terraform
or Terraform Enterprise,
I recommend checking out our online resources.
Thank you.
Browse More Related Video
Mastering Terraform: Scenario-Based Interview Questions & Solutions | Terraform Interview Mastery
DE Zoomcamp 1.3.1 - Introduction to Terraform Concepts & GCP Pre-Requisites
Mastering Terraform Interview Questions: 15 Essential Questions & Answers | Demystifying Terraform
Terraform Interview Questions | Terraform Scenario Questions | DevOps Interview Series | Terraform
Implementing Infrastructure as Code with Terraform | AWS Cloud Resume Challenge - Part 6
What is DevOps? Understanding DevOps terms and Tools
5.0 / 5 (0 votes)