Introduction to DevOps - CI / CD
Summary
TLDRThis lecture introduces continuous integration (CI) and continuous delivery (CD) as key DevOps practices, emphasizing their role in streamlining software development and deployment. CI involves frequent merging of developer updates into the main source code, verified by automated builds and testing. CD takes it further, ensuring software is always ready for production release. The lecture discusses building automated pipelines, necessary tools, and illustrates a CI/CD pipeline example. It also explores how CI/CD aligns with DevOps, bridging the gap between development and operations teams.
Takeaways
- π Continuous Integration (CI) is a practice where developers integrate their work frequently, often daily, into a shared repository to facilitate smoother merging and early detection of integration issues.
- π΅ CI practices are crucial for automating the integration process, ensuring each integration is verified by an automated build and tests to maintain software reliability.
- π² Continuous Delivery (CD) extends CI by ensuring that software can be released to production at any time, focusing on automating the deployment of software to end users with confidence.
- π΅ Testing is a fundamental aspect of both CI and CD, ensuring that updates work as intended and that no broken versions are deployed to production.
- π² CI/CD pipelines are automated processes that encompass stages like building, testing, and deploying software, aiming to streamline development and operations workflows.
- π΅ DevOps is a cultural shift that aims to unify development and operations teams by adopting practices like CI/CD to enhance collaboration and streamline software delivery.
- π² A CI/CD pipeline typically involves three stages: build, test, and deploy, which can be triggered automatically by events in version control systems like GitLab or GitHub.
- π΅ Tools like GitHub Actions, GitLab CI/CD, CircleCI, and Jenkins are integral for setting up CI/CD pipelines, with some offering direct integrations with version control systems for seamless communication.
- π² Effective CI/CD implementation requires the pipeline to interact with both version control systems to detect changes and runtime environments to deploy the software.
- π΅ A practical CI/CD implementation example involves using GitLab for version control, GitLab CI/CD for the pipeline, and Microsoft Azure as the production environment, with options to deploy directly or using containerization.
Q & A
What is the primary focus of continuous integration?
-Continuous integration is a software development practice where team members integrate their work frequently, usually daily, into the main software source code. It emphasizes the automation of the integration process and includes testing to ensure updates work as intended.
How does continuous integration relate to DevOps?
-Continuous integration is a concrete way to practice the DevOps culture by making it easier to change and publish software, thus solving potential conflicts between development and operations teams.
What is the main goal of continuous delivery?
-Continuous delivery is a software development discipline where the goal is to build software in such a way that it can be released to production at any time, ensuring that the software works correctly when it is automatically published to users.
What are the key stages of a CI/CD pipeline?
-The key stages of a CI/CD pipeline include building the software, testing to ensure no broken changes are made, and deploying the application to the desired environment.
Why is automation important in continuous integration?
-Automation in continuous integration is crucial to handle the frequent integrations that occur daily. It ensures that the integration process is efficient, reliable, and reduces the likelihood of errors.
How does testing play a role in continuous integration and delivery?
-Testing is a key concept in both continuous integration and delivery. It ensures that updates to the software work as intended and that no broken versions of the software are deployed to production environments.
What is the deployment pipeline mentioned by Martin Fowler?
-The deployment pipeline is an automatic process that performs the stages of continuous integration and delivery, making the development process easier for developers.
How does the traditional conflict between development and operations teams lead to the emergence of DevOps?
-DevOps emerged from the conflict between development and operations teams, where development wanted frequent changes while operations preferred stability. DevOps aims to unify these needs through practices like CI/CD.
What are some tools that can be used to develop CI/CD pipelines?
-Some tools that can be used to develop CI/CD pipelines include GitHub Actions, GitLab CI/CD, CircleCI, Jenkins, and others. These tools help automate the building, testing, and deployment processes.
How can a CI/CD pipeline be triggered?
-A CI/CD pipeline can be triggered automatically when a certain event occurs in the version control, such as a push or merge to the main branch, or it can be triggered manually by a developer.
What are the two options for deploying a Node.js application to Microsoft Azure as presented in the script?
-The two options for deploying a Node.js application to Microsoft Azure are: Option A, deploying the application with the Web App Service using the Node.js runtime environment, and Option B, containerizing the application into a Docker container and deploying it to the Docker runtime on the Web App Service.
Outlines
π Introduction to CI/CD in DevOps
The lecture begins by outlining the course's agenda, focusing on continuous integration (CI) and continuous delivery (CD) within the DevOps framework. It emphasizes the importance of CI, defined by Martin Fowler as a practice where developers frequently integrate their work, typically daily, into the main source code. The lecture highlights the significance of automation in CI to ensure reliable integration and the crucial role of testing to prevent software breakages. The benefits of CI in reducing integration problems and easing the development process are also discussed.
π Continuous Delivery Explained
This section delves into continuous delivery, which is an extension of CI. It is defined as a software development discipline that allows for the software to be released at any time. Unlike CI, which focuses on integrating updates into the mainline, continuous delivery aims to publish changes to end-users. The necessity for robust testing is underscored to ensure that software updates are reliable and bug-free. The concept of pushing executables to production environments to validate their functionality in a real-world setting is also introduced.
π€ Bridging the Gap with DevOps
The paragraph discusses the conflict between development and operations teams in traditional software environments and how DevOps emerged to bridge this gap. It explains that while development teams desire frequent software updates, operations teams prefer stability to facilitate maintenance. DevOps practices, including CI/CD, aim to unify these conflicting goals by making software changes and deployments easier for both teams, thus fostering a collaborative culture.
π οΈ The CI/CD Pipeline Process
This segment describes the typical CI/CD pipeline process, which includes building, testing, and deploying stages. The build stage focuses on creating an executable version of the software, while the testing stage ensures that no broken changes are introduced. The deploy stage, specific to continuous delivery, aims to deploy the application to various environments, including production. The paragraph also touches on automated pipeline triggers, such as updates to the main branch in version control systems.
π Interacting with Version Control and Runtime Environments
The lecture explains the importance of the CI/CD pipeline's interaction with version control systems and runtime environments. It discusses the need for the pipeline to be informed of events in version control to trigger builds and the necessity to deploy software to the desired runtime environment. The paragraph also mentions tools and methods for establishing this communication, such as webhooks and direct integrations.
π οΈ Tools and Integrations for CI/CD Pipelines
This section lists various CI/CD pipeline tools, such as GitHub Actions, GitLab CI/CD, CircleCI, and Jenkins. It highlights the benefits of direct integrations between version control systems and CI/CD tools, simplifying the configuration process. The paragraph also discusses the implementation of a CI/CD pipeline for a Node.js application on GitLab, with deployment options to Microsoft Azure, either directly or via Docker containers.
π Visualizing CI/CD Pipeline Options
The final paragraph provides a visual overview of the CI/CD pipeline process for a Node.js application, considering different deployment options on Azure. It outlines the steps from the initial change event in version control to building, testing, and deploying the application. The discussion includes the decision between deploying directly to a Node.js runtime or containerizing the application and deploying it to a Docker runtime environment within Azure.
π Conclusion and References
The lecture concludes with references to Martin Fowler's definitions and various pipeline tool and cloud provider websites used in the material. It encourages attendees to contact the course staff with any questions, emphasizing the availability of resources for further learning.
Mindmap
Keywords
π‘Continuous Integration (CI)
π‘Continuous Delivery (CD)
π‘DevOps
π‘Pipeline
π‘Version Control
π‘Automated Build
π‘Testing
π‘Deployment
π‘Integration Problems
π‘GitLab CI/CD
π‘Microsoft Azure
Highlights
Introduction to Continuous Integration and Continuous Delivery in the context of DevOps.
Continuous Integration (CI) is a practice where team members integrate their work frequently, often daily.
Integration in CI refers to merging developer updates into the main software source code.
Automated builds verify each integration to ensure code relevance and successful software integration.
Testing is a critical component of CI to ensure updates work as intended and prevent software breakage.
Continuous Delivery (CD) aims to build software that can be released to production at any time.
CD is a continuation of CI, focusing on publishing software changes to end users.
A robust testing process is essential for CD to prevent bugs from reaching end users.
Executables in CD are pushed into production environments to ensure software functionality.
CI/CD practices can significantly reduce integration problems and streamline software development.
DevOps is a set of practices that enhance cooperation between development and operations teams.
CI/CD is a concrete way to practice the DevOps culture by automating software changes and deployments.
A CI/CD pipeline is an automated process that performs the stages of CI and CD.
A typical CI/CD pipeline consists of build, test, and deploy stages.
Pipelines can be triggered automatically by events in version control systems like GitLab or GitHub.
Version control systems and CI/CD pipelines need to communicate effectively, often through webhooks or direct integrations.
CI/CD pipeline tools include GitHub Actions, GitLab CI/CD, CircleCI, and Jenkins.
GitLab and GitHub's CI/CD tools offer direct integration with their version control systems for seamless communication.
A CI/CD pipeline needs to interact with the runtime environment to deploy software.
An example CI/CD implementation involves deploying a Node.js application from GitLab to Microsoft Azure.
There are two main deployment options on Azure: using the Web App Service or the Web App for Containers service.
The decision between deploying directly from source code or containerizing the application affects CI/CD pipeline design.
It's recommended to start with a simple, minimum viable pipeline and improve it over time.
Transcripts
Unknown Speaker 0:01 Welcome to this theory lecture about continuous
integration and continuous delivery. This is the final lecture of the course introduction
to DevOps. Here is the contents of this lecture set on the agenda. So, first of all, we will
discuss what is continuous integration and delivery, then we will move on to see how
it relates to DevOps as a concept as a whole. Then we will see how to build a automatic
pipeline that performs the continuous integration and delivery practices. After that, I will
present some tools and integrations that can be used to develop some pipelines, for example.
And finally, I will visualize an example of a C) / CD pipeline, which will give you a
clear understanding of the CI / CD process. So, first of all, let's start with continuous
integration. So, according to Martin Fowler, continuous integration is a software development
practice where members of a team integrate their work frequently. And usually each developer
integrates at least daily. So now, the term integration means to essentially merge the
updates that the developer has been doing into the main software source code. So if
we are thinking in terms of version control, it means that the developer will update his
or her changes to the version control repository often (eg. daily). And it can even happen
more frequently, and like Fowler says, leading to multiple integrations per day (if there
are multiple developers in a team, which is usually the case). If then all of the developers
will integrate their work daily, which will lead to a lot of integrations. Now, because
this integration happens a lot during the day, it is really important to be automated.
So that is the idea of the automatisation in the continuous integration. And well, the
definition by Fowler, continues that each integration is verified by an automated build.
So this refers to an automated process that will check that the code is relevant and can
be integrated into the existing source code (of the software). And testing is a really
important part of continuous integration - testing will make sure that the updates to the software
will work as intended. So without testing the updates, it would be really unreliable
to automatically update the software: it could break really easily if the developer has made
some kind of a mistake that leads to some problems on the software. So testing is really
key concept in continuous integration, software must be tested in order to automatize to process.
And finally, Fowler states that many teams find that this approach leads to significantly
reduced integration problems. So it refers to the testing and also the continuous integration
as a whole: when the process is such that it tests the updates and then integrates them
into the software automatically, it really makes the work of the software developers
more easy. They don't have to manually see that everything works and try to solve the
all kinds of possible situations - the pipeline or the integration that happens automatically
really makes the development work more easy
Unknown Speaker 5:04 and effective. So next here is the definition
of continuous delivery by Martin Fowler. So he defines that the continuous delivery is
a software development discipline, where you build software in such a way that the software
can be released to production at any time. So compared to continuous integration, where
the goal was just to integrate the updates of the source code into the main line of development,
for example, the master branch on version control, in continuous delivery, the goal
is to actually publish the changes of the software to the end users of the software.
So this is kind of a continuation to the CI. And usually, Continuous Delivery isn't done
without a continuous integration practice. It is even more important to verify that the
software works correctly, when it is automatically published to the users - there is no room
for mistakes. If there were to be any bugs in the software, they would be apparent to
the users and the software might break and things like that. So it is really important
to have a good testing process in order to practice this continuous delivery. Like Fowler
defines, continuous delivery is achieved by continuously integrating the software. So
having the previously mentioned continuous integration done is one stage of continuous
delivery. Other stages of continuous delivery can include, for example, building the executable
of the software and running automated tests on those executables. So the testing is present
on both continuous integration and deployment and software testing as a whole is a really
big concept - it cannot be placed strictly on some place; it needs to be present as much
as possible in order to make sure that the software works. Finally, Fowler states that
the executables are pushed into production environments. And the idea is to ensure that
the software will work in production. This statement doesn't necessarily mean that the
end users will be using the software, but the "ultimate" way of doing Continuous Delivery
would be that the software is actually published to the environment where the customers and
the users of the software will be using it. But it is also like Fowler states, it is possible
to have continuous delivery in such a way that it isn't necessarily accessible by the
end users of the software. It can only be accessisble by just the developers - however,
it still makes their development work easier. But the most efficient way to do continuous
delivery would be to automatically publish the software to the end users. Finally, Fowler
mentions the deployment pipeline, which essentially is an automatic process that will make and
perform the stages of continuous integration and continuous delivery. So let's next talk
about continuous integration and delivery and how they relate to DevOps. DevOps is the
key term on this course, since this is introduction to DevOps and it means the set of practices
that enhance the cooperation between the development and operations of a software product. So
Unknown Speaker 9:38 usually, traditionally, in big software systems,
there is a separate theme for the software development and a separate team for the operations
and maintenance of the software. DevOps has emerged from the conflict that happens between
the development and operations team as they have a different kind of needs and desires
regarding the software: the development team usually wants to publish and change the software
as much as possible in order to fix bugs and produce new features and enhance the software
in whatever ways while the operations team, on the other hand wants to have the software
stay as static as possible; they don't want to change the software since it can mean that
it would be harder to maintain it: it can mean that the software needs to be updated
to some customers. It essentially means more work to the operations team. So the development
team wants to change the software as much as possible, while the operations team wants
to have the software not change - they don't want any changes to the software if they are
asked. So this is where the concept of DevOps has come come to life. The idea is that the
wants and needs of the development and operations teams are tried to be unified. DevOps can
be seen as a culture that tries to unify these teams (the development and operations teams)
by having a set of practices that make the work of both of the teams easier. So now when
we think about continuous integration and continuous delivery, which essentially make
it easier to first of all, change the software, and also to publish the software. So it kind
of solves a problem for both of the teams and makes the life of the developers more
easy and also makes the life of the operations team more easy. So CI / CD therefore, is a
concrete way to practice the DevOps culture. And finally, like it was mentioned before,
the continuous integration and deployment can be implemented with pipelines. So the
term pipeline is used for a process that performs the continuous integration and delivery. So
it might be hard to grasp the concept right now, but it will come more clear once I show
you some visual graphics regarding CI / CD pipelines. So here, we have an example of
pipeline process. And this is probably the most common CI / CD pipeline process, which
has three different stages. So first phase, or stage is this build stage.
Unknown Speaker 13:26 The goal here is to build software - depending
on the software technology, there might be different kind of processes that need to be
executed in order to have the software be ready to run. For example in Java Spring applications,
the build process might include making the executable Jar file and therefore the build
process consists of building of the Jar file that can be executed in the Java runtime environment.
The second stage, commonly is the testing stage. And as I mentioned before, testing
is really a key concept in both continuous integration and continuous delivery. The goal
is to ensure that no broken changes get made the software and that is the goal in continuous
integration. In continuous delivery, the goal is to make sure that no broken version of
the software gets deployed to the production environment. So testing is therefore a really
key phase in the CI CD pipeline. And finally, the deploys stage, it relates to the continuous
delivery solely. The goal there is to deploy the application to the environment that it
is wanted to be deployed into. So, it can be some kind of a testing or staging environment
or even the production environment, where the customers will be able to access it. So,
depends on the situation, but the goal of the deploy stage is to have the built and
tested software deployed into some kind of an environment. Now, it is very common to
have this pipeline run automatically when a certain event occurs in the version control.
The pipeline also could be run manually, usually, but the most effective way is that the pipeline
is automatically triggered when for example, a software developer pushes or merges his
or her changes to the main branch of the software. So, that is the most effective way to run
the pipeline. Also, it is very possible to have the process be manual, so that when the
developer merges his or her code, there is some kind of a button that triggers the pipeline,
but the best option is to have the pipeline run automatically. And nowadays, the version
control systems like for example, Gitlab.com have a really good CI CD support. So, the
version control system and the CI CD pipeline can communicate with each other and the version
control system can basically tell the CI CD pipeline when some kind of an event has happened
in the source control (the version control). We will discuss this more during the next
slides. Okay, and when you start to implement a CI CD pipeline, so essentially an automatic
process that will build, test and deploy the application (the software) you need to have
the pipeline interact with two different entities: first of all, the version control system.
This is important because the CI CD pipeline needs to know when a certain event has been
made the version control. For example, if the pipeline is wanted to be run when the
main branch of the version control is updated, then the pipeline needs to know when a push
or a merge event has happened to the main branch. This communication can be done with
various ways - for example, with web hooks, or even ready made integrations: certain tools
integrate very well with existing version control providers and have this communication
set up automatically.
Unknown Speaker 18:28 The second thing the pipeline needs to be
able to interact with is the runtime environment - the environment where the software is wanted
to be deployed. This is because in the deploy stage the goal is that the pipeline will deploy
the software. So therefore there needs to be some kind of a way for the pipeline to
essentially deploy the software, which can for example mean putting the newest executable
of the software into the production environment. For example, in the case of a Java spring
application, the pipeline can bring the newest Jar file into the Java runtime environment
that is in the production server. So these two lines of communication are very important
when implementing a pipeline in practice. Now a little bit about the tools and integrations
that can be used when developing these pipelines. So first of all, to list some CI CD pipeline
tools: there are GitHub Actions, GitLab CI / CD, Circle CI, Jenkins and many more like
Buddy CI, for example. The first two are especially interesting because as we know GitLab is a
version control provider and GitHub is as well. So these two CI CD pipeline tools essentially
are an extension to the version control system that these providers are offering. And this
means that there is a direct integration between the, for example, the GitLab version control
system and CI CD pipeline system. As I mentioned in the previous slide, the communication between
the version control system and the pipeline is essential in order to have an automatic
CI CD process. And in the case of GitLab, for example, the communication is handled
with a direct integration. So there is very little manual work to be done, in order to
have of these two entities communicate with each other. This will be demonstrated in the
practical section in this course. Like I mentioned, even when direct integrations are not available,
there are different ways to communicate: for example via web hooks and different kinds
of API calls, where the version control system can inform the pipeline when a certain event
has happened. Or maybe their pipeline can also sort of poll the version control system
and ask it periodically if certain things have happened. So there are many ways to handle
this communication. But it is good to know that these direct integrations that I mentioned,
make the configuration of the pipeline more easy and efficient. Finally, let's talk about
a concrete CI CD implementation. Let's use the technologies that are used during this
course and are available to you. So let's assume that we have a Node JS based web application
on GitLab. Let's try to deploy it to Microsoft Azure. So now we have a GitLab.com version
control, and since there is the direct integration between the GitLab version control and the
GitLab CI CD pipeline tool, we can use the ready made by pipeline tool by GitLab. So
we have GitLab.com as the version control system, we have GitLab CI CD as the pipeline
tool, and we have Microsoft Azure as the
Unknown Speaker 23:15 production environment. Now, the Azure and
GitLab aren't directly integrated in the same way that the GitLab CI CD and the GitLab.com
are. But still, it is very possible to deploy the software into Azure and there are some
kind of integrations available and we are able to deploy the software there. And there
essentially afe two options (in Azure) in order to deploy it (the software) (when we
take a look at the offerings by Microsoft Azure): option A would be to deploy the application
with the Web App Service and selecting the Node JS runtime environment on the Web App
Service. So in this case, the Azure Web App Service would take the source code repository
from GitLab.com and then the service will deploy the application based on the source
code. Option B would be to use the Azure Web App for Containers service, which is essentially
in the same Web App Service but as a different option in the configuration there. So, another
option would be to containerize the application into a Docker container, then provide that
container to Azure (Web App Service), which will then deploy it into a Docker runtime
environment. There are also for this option B some further options: We can use different
container registries where the Azure (Web App Service) will pull the container from
but that is also something to think about. But first, it is the most important to decide
between the option A and B. So let's next see this in a more visual way.
Unknown Speaker 25:26 So here is a visualization of the possible
pipeline processes, which have to do with the presented case, on the earlier slide and
the two options that I gave. So, first of all, the pipeline will start when some kind
of a change event happens in the GitLab version control. For example, if the master branch
gets updated in a push or a merge event. Then the application can be built, and as this
was a Node.js application - the build process will be done according to the software technology.
After that is the testing phase. So now, the goal of that phase is to make sure that the
application works as intended - essentially that it is ready to be deployed into the environment
- the final environment. So then there was the two options (to deploy the application):
either deploy it to the Node JS runtime environment on Azure or deploy it to the Docker runtime
on the same Azure Service (the Web App Service). But before we are ready to deploy it into
the Docker runtime, we need to first of all build the Docker image from the application,
then push it to some Container Registry. I presented the two further options (for the
container registry): either GitLab or Azure. So, this is up to the situation, which one
would be better. When it (the container registry) is decided, then the goal is to deploy it
finally into the Web App Service.
Unknown Speaker 27:20 So, there are some options here that I presented
and there are even more options - for example: if more cloud providers would be considered,
also regarding the testing, it can be done in multiple multiple stages: for example:
unit testing, integration testing, system testing, UI testing: if they want to be separated
(to own stages) in the pipeline, it is possible. It is important to know that there are lots
of different possibilities when implementing the pipeline, but the key takeaway is to:
first design a simple pipeline that does the process as well as needed. And then maybe
on the future, improve it further but, it is important to first, implement a sort of
a minimum viable pipeline and then improve on that. It's important to find some kind
of a balance between the pipeline development and other development. So, this is a visualization
of some of the possibilities regarding the CI CD pipeline.
Unknown Speaker 28:49 Finally, here are the references that I used
during this lecture. So the Martin Fowler's definition is were the main references and
also the pipeline tools' websites and Microsoft Azure's website has been used when making
this material. Thank you for your time and don't hesitate to contact the course staff
if you have any questions regarding this topic.
Transcribed by https://otter.ai
Browse More Related Video
#1 What is DevOps ,Agile and CICD
GitLab: DevSecOps: Part 1/12: What is GitLab? The fundamental concepts of a DevSecOps pipeline.
Day-18 | What is CICD ? | Introduction to CICD | How CICD works ? | #devops #abhishekveeramalla
CI/CD In 5 Minutes | Is It Worth The Hassle: Crash Course System Design #2
ISTQB FOUNDATION 4.0 | Tutorial 11 | TDD, BDD, ATDD | DevOps and Testing | CTFL | TM SQUARE
Software Development Life Cycle: Explained
5.0 / 5 (0 votes)