DE Zoomcamp 1.3.1 - Introduction to Terraform Concepts & GCP Pre-Requisites

Summary

This session introduces Terraform, an open-source tool by HashiCorp, for provisioning infrastructure with declarative configuration files. It emphasizes Infrastructure as Code (IaC) for managing infrastructure safely and consistently. The tutorial covers setting up a GCP account, creating a project, and generating a service account with necessary permissions. It also guides through installing the Terraform client and Google Cloud SDK, authenticating with GCP, and enabling APIs for Terraform to interact with GCP resources like Cloud Storage and BigQuery.

Takeaways

• 😀 Terraform is an open-source tool by HashiCorp used for provisioning infrastructure resources with declarative configuration files.
• 🛠️ Terraform supports Infrastructure as Code (IaC), enabling DevOps best practices for change management and version control of infrastructure.
• 💾 It allows you to manage infrastructure lifecycle with stack-based deployment, enabling the creation and destruction of resource clusters.
• 🔗 The Terraform state is crucial for tracking resource changes throughout deployments.
• 💻 To use Terraform, you need the Terraform client and a GCP account, with the free tier offering up to 300 euros in credits.
• 🔑 A service account in GCP is created for services to interact with cloud resources, with restricted permissions for security.
• 🔗 Service accounts are used to grant specific permissions to interact with GCP resources like Cloud Storage and BigQuery.
• 🔗 The Google Cloud SDK (gcloud) is necessary for local interaction with GCP services, and it can be authenticated using OAuth.
• 🛠️ For the tutorial, permissions are kept simple with admin roles, but in production, custom roles with specific permissions are recommended.
• 🔄 APIs need to be enabled in GCP for services like IAM and BigQuery to interact with the cloud resources through the local environment.

Q & A

What is Terraform and what does it enable?

Terraform is an open-source tool by Hashicorp that allows you to provision infrastructure resources with declarative configuration files. It supports Infrastructure as Code (IaC), enabling DevOps best practices for change management.

What are the types of resources Terraform can manage?

Terraform can manage various types of resources including virtual machines, containers, storage, and networking resources.

What is Infrastructure as Code (IaC) and how does it benefit infrastructure management?

Infrastructure as Code (IaC) is a framework that allows you to build, change, and manage infrastructure in a safe, consistent, and repeatable way by defining resource configurations that can be version-controlled, reused, and shared.

How does Terraform's state feature help in managing infrastructure?

Terraform's state feature allows you to track resource changes throughout your deployments, enabling you to manage the infrastructure lifecycle and collaborate safely on your infrastructure.

What are the prerequisites for setting up Terraform with GCP?

The prerequisites include the Terraform client, which can be downloaded from the official site, and a GCP account, which can be a free account with a certain amount of credit depending on the region.

Why is a service account important in GCP when using Terraform?

A service account in GCP is important as it provides credentials for services to interact with GCP resources. It has restricted permissions, allowing specific services to perform necessary actions without requiring the owner's account or admin account.

How can you create a service account in GCP?

You can create a service account in GCP by navigating to the 'Service accounts' section in the IAM & Admin panel, providing a name and description, and assigning it a role, such as 'Viewer' to start with.

What is the purpose of generating and downloading a key for a service account in GCP?

Generating and downloading a key for a service account in GCP provides credentials that the service account can use to authenticate and interact with GCP resources, allowing the service to perform actions on behalf of the account.

What is the Google Cloud SDK and why is it necessary?

The Google Cloud SDK is a CLI tool that allows you to interact with your cloud services, list them, and authenticate your Google application credentials or service account key to interact with the cloud from your local machine.

How do you authenticate your local setup with GCP using the service account key?

You authenticate your local setup with GCP by setting the 'GOOGLE_APPLICATION_CREDENTIALS' environment variable to the path of the downloaded service account key and then using the 'gcloud auth' command to authenticate.

What are the two resources that will be created in the GCP environment during the exercise?

The two resources that will be created are Google Cloud Storage, which is a bucket for storing data, and BigQuery, which is Google's equivalent of a data warehouse.

Why is it necessary to enable APIs in GCP for Terraform to work?

Enabling APIs in GCP is necessary because they are the enablers of communication between your local environment and the cloud resources. They allow Terraform to interact with services like IAM and manage credentials.

Outlines

00:00

🛠️ Introduction to Terraform and GCP Infrastructure Setup

The speaker begins by introducing the session's focus on Terraform, an open-source tool by Hashicorp, which is used for provisioning infrastructure resources through declarative configuration files. These resources can range from virtual machines to storage and networking. Terraform supports Infrastructure as Code (IaC), aligning with DevOps best practices for change management. It allows for version control of infrastructure, enabling the management of resources through configuration files rather than a GUI. The session aims to cover Terraform basics and setting up GCP infrastructure using Terraform. The speaker shares their screen and proceeds to demonstrate the setup process, starting with downloading the Terraform client and accessing a GCP account, which can be obtained for free and includes a 300-euro credit for the first 90 days.

05:00

🔑 Setting Up Service Accounts and Keys in GCP

The speaker demonstrates how to create a service account in GCP, explaining that it is an account designated for services, allowing them to interact with GCP resources with restricted permissions. A service account is created with a unique name and description, and the 'Viewer' role is assigned initially. The speaker then guides through generating a JSON key for the service account, which is essential for authenticating the service account with GCP. The Google Cloud SDK (gcloud) is also introduced as a CLI tool for interacting with cloud services, and instructions are provided for checking its installation and installing it if necessary. The process involves setting an environment variable for the service account key and authenticating it, which allows the local setup to interact with the cloud environment.

10:02

🔗 Authenticating Local Setup with Cloud Environment

The speaker discusses the process of authenticating the local setup with the cloud environment using the Google Cloud SDK. They mention that the authentication process involves setting an environment variable for the service account key and using a command to authenticate it. The speaker also touches upon different authentication methods, including OAuth, which may be used in different scenarios. They provide a practical demonstration by setting the environment variable and authenticating it, which successfully links the local setup with the cloud, allowing for interaction between the two environments.

15:03

📦 Creating Resources in GCP Using Terraform

The speaker outlines the next steps in the tutorial, which involve creating two resources in the Google Cloud Platform: Cloud Storage and BigQuery. They explain that Cloud Storage is akin to a data lake for storing raw data, while BigQuery serves as a data warehouse for structured data storage. The speaker emphasizes the importance of adding permissions to the service account to enable Terraform to create these resources. They guide through adding 'Storage Admin' and 'BigQuery Admin' roles to the service account and mention the need to enable APIs for communication between the local environment and cloud resources. The speaker also advises on the importance of selecting the correct project when enabling APIs and reassures the audience that the provided credits are sufficient for the course's exercises.

20:04

🚀 Starting the Terraform Setup

The final paragraph sees the speaker preparing to begin the Terraform setup. They recap the steps taken so far, including enabling APIs and ensuring that the service account has the necessary permissions. The speaker also acknowledges that the APIs and permissions setup is complete and moves on to the actual Terraform configuration. They provide a link for downloading Terraform based on the user's operating system and express readiness to proceed with the Terraform part of the tutorial.

Keywords

💡Terraform

Terraform is an open-source infrastructure as code (IaC) tool developed by HashiCorp. It allows users to define and provision infrastructure resources such as virtual machines, containers, storage, and networking using declarative configuration files. In the video, Terraform is introduced as a tool to set up GCP (Google Cloud Platform) infrastructure, highlighting its ability to manage infrastructure lifecycle and track resource changes through its state functionality.

💡Infrastructure as Code (IaC)

Infrastructure as Code (IaC) is a methodology that involves managing and provisioning computer data centers through machine-readable definition files, rather than physical hardware configuration or interactive configuration tools. The video emphasizes IaC as a framework that enables the safe, consistent, and repeatable management of infrastructure through version-controlled configuration files, akin to using Git for infrastructure.

💡HashiCorp

HashiCorp is a software company that provides DevOps tools, including Terraform, which is the main focus of the video. The company is known for its suite of tools that address various aspects of infrastructure management, and Terraform is one of its flagship products for IaC.

💡Google Cloud Platform (GCP)

Google Cloud Platform (GCP) is a suite of cloud computing services offered by Google. It provides a variety of services including computing, data storage, and machine learning. In the context of the video, GCP is used as the cloud environment where Terraform provisions and manages infrastructure resources.

💡Service Account

A service account in GCP is a special type of account used by applications or services, rather than humans. It has restricted permissions and is used to authenticate interactions between the local Terraform client and GCP resources. The video describes the process of creating a service account and generating a key for secure API access.

💡Version Control

Version control is a system that records changes to a file or set of files over time so that specific versions can be recalled later. In the video, version control is mentioned in the context of managing Terraform configuration files, which allows for collaboration and tracking changes to infrastructure in a controlled manner.

💡Google Cloud Storage

Google Cloud Storage is a service provided by Google as part of its cloud platform that offers online file storage and access through the Internet. In the video, it is mentioned as one of the resources that Terraform can provision, where data can be stored in a manner similar to a local file system.

💡BigQuery

BigQuery is Google's fully-managed, serverless data warehouse that enables scalable analysis over petabytes of data. The video introduces BigQuery as a resource that Terraform can set up, which is used for structured data storage and complex querying capabilities.

💡OAuth

OAuth is an open standard for access delegation, commonly used as a way for Internet users to grant websites or applications access to their information on other websites without giving them the passwords. In the video, OAuth is discussed as a method for authenticating the local setup with the cloud environment using a service account key.

💡APIs

APIs, or Application Programming Interfaces, are sets of rules and protocols for building and interacting with software applications. In the context of the video, APIs are mentioned as enablers for communication between the local environment and GCP resources, such as IAM and service account credentials.

💡Data Lake

A data lake is a storage repository that holds a vast amount of raw data in its native format until it is needed. In the video, the concept of a data lake is introduced as a place to store raw data in an organized fashion within Google Cloud Storage, which is partitioned and compressed for efficient use.

Highlights

Introduction to Terraform, an open-source tool by Hashicorp for infrastructure provisioning.

Terraform uses declarative configuration files to manage infrastructure resources.

Explanation of Infrastructure as Code (IaC) and its benefits for DevOps.

Advantages of managing infrastructure lifecycle with Terraform.

How to download and install the Terraform client.

Requirements for setting up GCP infrastructure with Terraform.

Details on creating a GCP account and accessing free credits.

Step-by-step guide to creating a GCP project.

Understanding the role of service accounts in GCP.

Process of creating a service account and assigning roles in GCP.

Importance of managing keys for service accounts in GCP.

Installation and setup of the Google Cloud SDK.

Authentication of local setup with GCP using OAuth.

How to set environment variables for Google Application Credentials.

Explanation of the difference between admin roles and custom roles in GCP.

Enabling APIs for services like IAM and BigQuery.

Introduction to Google Cloud Storage and BigQuery as resources in GCP.

Concept of a data lake and its role in organizing raw data.

Terraform's role in managing resources like cloud storage and data warehouses.

Transcripts

hi everyone

so now we we are going to cover a

terraform and say we'll do that so

yeah let's start

hey everyone how's it going

so yeah today's session is uh the part

of the week one session where we are

we'll be covering the concepts of

terraform at an introductory level and

also how to set up your gcp

infrastructure using terraform so let's

get started

and

i'll just share my screen now

so terraform is actually an open source

tool

by hashicorp and

it lets you provision infrastructure

resources with declarative configuration

files these resources can be virtual

machines or containers or storage or

even networking resources

and apart from this terraform also uses

an iac style approach which supports

devops best practices for change

management which lets you manage

terraform configuration files and source

control to maintain an ideal provision

state for testing and production

environments

now what really is ise ise is

infrastructure as code

this allows

you to build it's a framework which

allows you to build change and manage

your infrastructure in a safe consistent

and repeatable way by defining resource

configurations that you can version

reuse and share so think of this as a

git version control but for

infrastructure

and this also allows you to manage the

infrastructure with configuration files

alone rather than through a graphical

user interface

so mainly the advantages are

it helps you manage your

infrastructure life cycle

in a very stack based deployment style

so basically you can deploy an entire

cluster of resources together

like create your entire set of resources

together and destroy it at any point in

time while all of these resources within

a cluster are connected with each other

you can also commit your configurations

to version control to safely collaborate

on your infrastructure

and the most important part is the

terraforms state

which allows you to track resource

changes throughout your deployments

so i'll explain more about that a little

later

so what we need is only two things here

one is the terraform client

which you can download from this link

based on whatever is your os type

and follow the instructions through and

i already have it installed so

i

won't exactly show you the steps but

it's just a two step two to three step

process and thread it will be

straightforward

and the other thing that you would need

is your gcp account

uh basically with your google

id you can also get a free account if

you have not

set up an account previously on that id

and the free version provides up to 300

euro credits uh this i mean euro in

terms of the region that we are

currently in which is germany uh based

on the region that you are set in you'll

be getting some free amount of credits

which would be

mostly more than enough for all of the

exercises that we'll be performing

throughout our course

uh this is also with a 90-day

life cycle

so you do not really need to worry if

you're going to like you know meet the

nd 20 so

so

as you can see here

i already have an account created and

i already have like a

first project set up for child but i can

walk you through creating the first

project and whatever things you'll be

needing in

terms of coordinating in terms of your

local client coordinating with the cloud

resources which is a service account and

im access and so on and so forth so

let's begin

so once you already have your

gcp account set up what you do is create

a project

i already have it but i will create a

new one here

and

let's call this something like

dtcde

whatever you feel like putting

um

my project id is dt dtcde

but i'm going to edit this and put this

to

something more unique which generally

happens so let me refresh this and okay

now it's set some unique project id for

myself

uh you can always customize it but

please make sure that i'm a unique

project id um across the entire gcp

environment

otherwise it won't

allow you to do that

all right so now

we create this

and we have a notification here that

it's creating a project

perfect now let's

wait so

now i'm switching to the project that we

just created

and

what we will do now is

go to

im admin

the service accounts

okay let's go ahead and create our

service account um so alexi do you know

what a sales account is

not really but

so i think the way i understood it

is that a service account is an account

that you create for a service as the

name suggests and service could be

anything

usually it's a service so it can be your

data pipeline or your

i don't know web service or something

like this but this is a specific project

that you work on

um like specific service

and everything that this service needs

will be configured here in this service

account

so like if a service needs to go to

cloud storage we will grant access to

this cloud storage in this service

account

and what it lets us

do is it will give us a

credentials some credentials and these

credentials will belong to this server

and then this server will get access to

all the things from gcp that we need

and

does it sound right

that is perfect yeah so in in a nutshell

services a service account is actually

an account for

as name suggests services so you

basically you won't need your owner's

account or the admin account for the

resources to interact with each other it

has restricted or limited permissions so

let's begin with

creating it so

let's um

call this service account dtcde

user or anything you would like to keep

it as

this does not need to be globally unique

because this in combination to the email

address of your project

will already make it globally unique so

you do not need to worry

um description can be anything you want

to put

maybe dtcd course something

so

let's go to next and let's let's use the

viewer um role for now but this is only

to begin with

all right and this step is not something

that we need for now but this is uh

useful if you're setting up production

environment and

want to

have multiple users associated to a

certain service account which share the

same level of permissions

okay and

let's click done

and

as you see here there is

there are no keys generated yet so what

we do is like

go here

to manage keys

and then

create a new key

let's keep it json so that in case you

ever want to break you know open the

file in the text editor and read and

retrieve your password you can always do

that

so

create it

and now

downloads it into your default default

download directory the other thing you

need is your sdk

which is a cli uh the google sdk which

is a cli tool in for you to interact

with your cloud services maybe list them

or maybe even authenticate your

google application credentials or the

service account key in order to interact

with the cloud from your local so so on

and so forth so in case you want to

check if your

gcloud is sdk is already installed then

you can do a gcloud hyphen v

and

you will see it here

um if you do not have it then i would

suggest

this link

and install it

from here

based on your os it will suggest which

which one you need to like you know

download and it's a very straightforward

installation process just follow it

through not for windows though so for

windows i prepared like a set of

instructions that we will include also

because i had

some problems doing that so i will

add a link to windows instructions there

i had to to try

multiple options before i finally had

one that worked

oh okay wow sorry to hear that

but yeah i guess

the google client is not um yet as

mature as the aws client or many

comparable providers but it's it's

getting there so hopefully we would have

uh like you know more stable client for

windows as well soon

cool but yeah thanks alexi for preparing

that um

so once you have the sdk installed

go to

go back to your instructions and set

your environment variable

google applications credentials

to the downloaded service key service

account key that you

have generated

now that we have our gcloud installed

we'll go look for our

service account key that we downloaded

so in because i'm using a mac os it by

default downloads into the download

directory in my home directory

and

my project is actually my project id is

global maxim

338113

so

the key should also be something related

to that let's search okay now we have it

there let me check the time stamp again

yes okay that we just generated

a few months ago so i am going to

set the google applications credentials

uh environment variable which is a

system um specific or let's say uh

google cloud specific um environment

variable for it to for for the for the

invite basically for the system or

google to understand that this is your

application key or the authentication

key and this is uh with which you have

to authenticate uh your other resources

so let me

set that

[Music]

and provide this key as the path

all right

and now

i use this command

[Music]

to to authenticate it so that it picks

up the key from this environment

variable

so

okay

so what's what what it is saying it what

it is saying is um

it was using and uh you know an earlier

uh default credentials which i had

already set up for myself uh mostly with

the owner account and now it's detected

a new set of keys

associated with a new project or a new

service account so

do i want to replace that so i will

set yes

and the next step is i have to

authenticate it so this will redirect me

to a browser link

i just choose my email

allow it

and now it says my cloud sdk is

authenticated with the

basically my local setup is

authenticated with the cloud environment

so

if you get this too congratulations your

local is now um able to interact with

the cloud deployment

i think this way of authenticating is

called o

auth way of authenticating that's not

the only one i think there is a

different one i think when i was playing

with this i came across with

across another one so i'll also have

this in the notes so you can check it

out

which doesn't involve a browser

but i think it's uh it's very similar

okay nice does it refresh the token on

your yeah i think it doesn't like let's

say when you do this on

some instance and you don't have access

to your browser you cannot actually

authenticate using all auth

then there is a different way of doing

this so i'll include this in notes

sounds like your sso and aws

which

which you refresh the token every time

in order to log in yeah

okay nice nice to know that

so

okay now we press now that you already

have your local setup ready

we proceed to the actual part

um

also just to show you this is the

download link for terraform and based on

your kind of os you can install it via

whichever package manager you're using

for example brew

or

choco or anything else

all right so

let's move on

i just want to add that

installing terraform on windows was

surprisingly smooth

so i just needed to unpack the

executable to a folder in my path

and that was it so it picked it up and

it just worked

so i was surprised

so usually on windows i need to

do a lot of googling to figure out how

to actually make things work

but this one was very smooth

nice

awesome

so

all right so what we're going to do with

this exercise now is create two

resources in the google environment one

is the cloud storage and one is uh the

bigquery warehouse

and we'll be explaining uh the concept

of like you know

those resources in the data sections but

for now uh

a cloud storage is actually a bucket in

in your gcp environment uh where you can

store the data in a manner of flat files

so if you've uh seen any local file

storage with directories and like with

csvs or jsons in it it it basically is

just like that

uh we also refer to it in our course as

a data lake uh which you will understand

as we proceed in the course but

a data lake is where we are going to

store all the raw data in a more

organized fashion uh partitioned by uh

more sensible directories and also

compressed with uh

like you know compressed with certain

file types not only with csvs and jsons

but also parquet files

and bigquery is actually

your google equivalent of data warehouse

where the data is like uh modeled into a

more structured format with fact and

dimension tables and like you know the

classical data warehouse concepts that

you may already be familiar with

all right but we will definitely explain

more about that in the upcoming lectures

but let's move on to the rest of the

setup

so now that your local is already

authenticated with the cloud environment

let's go and add more permissions for

your service account

um

this is the one that we just created

so

what we are going to do is uh we will

try to keep it simple for this tutorial

but it is not

advisable for a production environment

but we will

add

in order to create uh the google cloud

storage we'd add a storage admin

[Music]

do you know what this role actually does

i think it allows

to

create

buckets and then

create files there right and this is

what we need so our terraform job

needs to

go there to google cloud and create a

bucket there so right now we're granting

our service permissions to do that

that's correct right right right

so we are granting two kinds of storage

rules one is uh for

the bucket itself and the other is for

the objects within the bucket

and uh by admin it has it means it has

all the ownership level positions that

is create update um

delete uh write read and

yeah you know grant access across and so

on and so forth

um and in real production we would

restrict it to a particular

uh bucket for example right that's

that's right in in real production you

would not generally use the like you

know the self the google provided roles

the uh what you see here is actually the

predefined roads created by uh gcp

themselves in production what you would

do is actually custom create custom

roles

and associate specific permissions

uh associated to certain resources

but

for just for the simplicity of our

course we will keep it for uh to this

for now

yeah i guess it will make sense for it

to create maybe as

one service account for terraform which

will have all the admin roles and then

for another service for a different

service for our data pipeline it would

be a separate service with its own

permissions right

yes

yes that that's actually uh what uh like

you know the ideal production style is

uh with most companies so good point

cool and finally we are also going to

add a big query

oops a big query admin

because we also want to interact with

[Music]

bigquery we also need to enable apis so

what really happens is when the cloud

when the local environment interacts

with the cloud environment it does not

interact directly with the resource it

basically these apis are the enablers of

communication

so one is for the iam itself

and the other one is for im credentials

possibly i may have already

authenticated it but let's see

and i think it's important to to note

here that when you open this page you

have a

drop down list so for this one yeah so

at the top you have like a drop down

list with projects so you need to make

sure that you select the project for

which you want to authenticate like this

this one exactly because sometimes you

may have multiple projects

yes

good point

um another thing that i wanted to

mention is um

uh don't be like you know uh

worried about this uh currently i think

you should see only 300 credits in my

case it's 265 because i already have

multiple projects and multiple resources

created in each project so

i've already used some credits but

only some of it has been depleted so far

in like the last two weeks so i think we

have a lot more left to go

for you guys to run your experiments on

so okay once we have the apis enabled um

this is something that you've already

done

this is also something you've already

done but if your session has expired

then you can try it again

and now we begin with the terraform part