Set up a CI/CD Pipeline with Cloud-Native Tools

Arwin Lashawn  on January 21, 2021

The adoption of cloud-based solutions has become increasingly common. The proof for this is evident – according to Gartner, Inc., the worldwide public cloud services market is expected to grow by 6.3% in 2020, up to a staggering $257.9 billion in value. 

The Flexera 2020 State of the Cloud Report, released on April 28, 2020, states that more than 90% of respondents have adopted cloud computing, with the top three cloud service providers being – AWS, Azure, and Google Cloud Platform. Additionally, many other companies are already in the experimental stage of transitioning to the cloud.

Among all the companies that have already leveraged the cloud for their operations, what is their top cloud initiative? To optimize their existing cloud usage. Or in other words, to reduce cost. Pretty obvious, right? 

Minimizing cost is one thing, but a more appreciated aspect is the growth gained from going the cloud way. It is much more than just jumping on the cloud train; these companies are experiencing growth faster than ever, thanks to the scalability and pay-as-you-go payment options that cloud services offer.

With cloud computing as a service being made available in a very accessible and affordable way, all of a sudden, you do not have to worry about things like server uptime and maintenance. You can simply focus on optimizing your cloud usage for your specific needs and utilize these services from virtually anywhere in the world.

Thanks to the emergence of cloud technology, many developers can focus their efforts on developing specific solutions that take full advantage of the cloud. Thus, we are starting to see more and more cloud-native or cloud-based solutions that improve developer productivity, especially through cloud-native CI/CD pipelines.

Widely-accessible cloud resources coupled with the automation of the whole CI/CD pipeline is what has given you the option to make your code live by simply pushing your changes to your code repository (provided it passes all quality checks). 

In this post, we will look at what CI/CD and cloud-native essentially mean. We will also explore several cloud-native tools that you can use to set up a fully automated CI/CD pipeline for your application. 

Feel free to use these links to navigate the guide:

What is a CI/CD Pipeline? 

For starters, we need to know what CI and CD stand for. CI stands for continuous integration, while CD stands for continuous delivery or continuous deployment. Let us now take a closer look at what each term means, its importance, and how they come together to form the CI/CD pipeline.

In sequence, continuous integration, continuous delivery, or continuous deployment are all different stages of the CI/CD pipeline.

Continuous integration is the first stage of the CI/CD pipeline. It is an automated process that ensures that code integration between multiple developers can happen seamlessly without causing the infamous merge hell. This is achieved by running code changes through multiple tests and builds that must be passed for the merge to be allowed.

The merge hell is a situation dreaded by developers. It happens when your final code breaks because of previous changes, and tracing the root of the issue causes the expenditure of a significant amount of resources. With proper continuous integration in place, frequent checks, including minor ones like syntax checks (check out Git Hooks), can go a long way in preventing issues like merge hell. 

Continuous delivery is the second stage of the CI/CD pipeline. It is an automated process that runs another set of tests before releasing the final code to a repository. In this case, the final code is always ensured to be in a releasable state for production, whether it is on the main branch or some other important feature branch.

The goal of continuous delivery is to ensure that the continuous integration of code is immediately followed up by converting the tested, bug-free final code to a releasable state. In its releasable state, the board of approval or any relevant personnel can then choose to run further QA checks on it before approving it for production.

Continuous deployment is the third stage of the CI/CD pipeline. Continuous deployment is an automated process that deploys the final changes in a code repository to production. The inclusion of this stage is preferred if there is no need for manual approval at the end of the continuous delivery stage.

One objective of continuous deployment is to deliver bug fixes to customers as soon as possible after the solution has been identified. In other words, all you have to do is push your code changes and the rest of the pipeline will be automated until production. As a result, your customers will soon be able to see the corresponding fixes without you having to do anything else.

You may still be a little confused about the differences between continuous delivery and deployment, but that’s okay. Don’t worry too much about the semantics – you can simply visualize the CI/CD pipeline as a completely automated process from the point of making code changes, all the way to production. Below is an excellent figure (by Red Hat) to depict the CI/CD pipeline as a whole. 

undefined

The gist here is the power of automation offered by a well-integrated CI/CD pipeline. It empowers you and your organization to save resources while being able to focus on writing good code and improving the overall customer experience.

Why Cloud-Native?

We came across two terms a little while ago - cloud-native and cloud-based. You might be wondering whether both refer to the same, but they don't. While both refer to development involving scalability and availability, cloud-native takes things a bit further. In the article titled Cloud-Based Vs Cloud-Native Application Development: An Important Distinction, Al Gillen did a great job in explaining the differences between the two.  He writes, “Cloud-based development refers to application development executed by means of a browser that points to a cloud-based infrastructure, cloud-native development refers more specifically to application development grounded in containers, microservices, and dynamic orchestration”.

If you are still unsure of the differences, that’s okay. The point here is to emphasize the difference between the terms cloud-native and cloud-based, as we're sure that many may assume them to be interchangeable when they really aren’t. Now, let’s jump to the whys of cloud-native!

There are many reasons for adopting cloud-native solutions. However, the primary goal of adopting cloud-native strategies is to accelerate the building of new applications and rapidly improve them based on user feedback. This paradigm then enables even the smallest of companies to compete with giants when it comes to comparing customer satisfaction. 

But you may be thinking – with speed comes loopholes like security issues and even code failure cases, resulting in a bad customer experience. The thing is, cloud-native development helps you tackle just that – it provides an avenue to not only provide blazing fast updates to users but to also run bug-free and responsive apps virtually anywhere in the world. Additionally, you can take advantage of a wide range of tools with the freedom to work with public, private, and hybrid clouds.

Now let’s take a look at some of the most popular cloud-based tools you can use to build your CI/CD pipeline. Different combinations of tools enable you to harness different development potentials so be sure to choose wisely.

Kubernetes

It is only right to start with discussing Kubernetes. Kubernetes is a platform that enables us to build and manage cloud-native apps efficiently. 

Developed in 2014, Kubernetes’ popularity has only seen a steep rise since then, now being up there with Docker and Linux as the top three most favored platforms according to the 2020 Developer Survey by Stack Overflow. 

Initially designed by Google, Kubernetes is an open-source container orchestration platform now maintained by the Cloud Native Computing Foundation. According to the Kubernetes official documentation, Kubernetes combines more than 15 years of Google's experience running production workloads at scale with best-of-breed ideas and practices from the community, so you know you can bank on what it has to offer.

So now that we know that Kubernetes is a container orchestration platform, what does it actually do? Its power lies in its automation of various manual processes involved in managing containerized applications. 

Say you want to design a system that has 99.99% server uptime that is fully automated. If ever one of your containers goes down, you would like another container to be fired up automatically and efficiently. Kubernetes is your answer. And this is only one of the many examples that Kubernetes is exceedingly beneficial for.

These are some of the things that Kubernetes is great at:

Despite all the great stuff Kubernetes can do for you, its true potential can only be unleashed when combined with other tools. So how does this then relate to creating a cloud-native CI/CD pipeline?

It’s now time to explore Tekton, a Kubernetes-native open-source framework that facilitates setting up CI/CD systems.

Tekton

Another section, another tool created by Google.

Tekton can be found under the Developer Tools tab of Google Cloud with a short and sweet description – “Kubernetes-native CI/CD building blocks”.

According to the official documentation of Tekton, Tekton consists of seven components, with the main ones being Tekton Pipelines, Tekton Triggers, and Tekton CLI. Their functions are listed below:

As you may have figured out already, developers and platform engineers use Tekton to develop well-integrated CI/CD pipelines in their organization. Considering the many available tools out there to build a CI/CD pipeline, what makes Tekton special?

Tekton is powerful yet fully customizable, giving developers a lot of freedom to define the design of their CI/CD pipeline. This then means there is a capability to handle many special cases and scenarios experienced throughout development. 

Another benefit of Tekton is its standardized nature. It installs and runs as an extension on your Kubernetes cluster, ensuring that the well-established Kubernetes resource model is used. 

Docker

Remember when I mentioned that Docker is ranked among the top three most favored platforms by developers in 2020? Like Kubernetes, Docker also receives a lot of love when it comes to setting up a cloud-native CI/CD pipeline.

What is Docker? It is a platform for developing, deploying, and running applications by taking advantage of containerization.

According to Docker, those who work at Docker see the CI/CD process as two distinct loops – the inner and outer loops. The inner loop consists of coding, building, executing, and testing, while the outer loop is a sequence starting from pushing code changes, CI building, CI testing, and finally deployment. Below is a figure depicting the process.

undefined

The two loops we discussed above are closely related, and it is important to understand how they come together to implement a CI/CD pipeline with Docker.

You will typically need to integrate a CI/CD tool with Docker such as Jenkins or Travis CI. A good step-by-step example of how to set up a CI/CD pipeline using Docker with Travis CI can be found here.

You may have noticed that Docker and Kubernetes look similar. But are they really? Their fundamental difference is that Kubernetes is meant to run on a cluster, while Docker runs on a single node. 

However, since they are dissimilar to begin with, comparing them side-by-side does not serve a good purpose. Remember when we said the full potential of a tool only shows when used with other tools?

Kubernetes and Docker work really well together to assist you in setting up a CI/CD pipeline. One of its advantages is a more robust infrastructure that is highly available and error-free, thanks to features provided by both tools. Read our guide for a more in-depth comparison of Kubernetes and Docker.

AWS

No, I haven’t forgotten about the cloud powerhouse, Amazon Web Services (AWS).

Amazon was the first company to offer cloud computing, through the Amazon Elastic Compute Cloud service back in 2006. Since then, Amazon has expanded so rapidly that a major fraction of its revenue has come from AWS alone. According to CNBC, Amazon Web Services revenue in Q4 of 2019 grew 34.0% in the quarter from a year earlier. The revenue growth was said to be faster than what analysts had expected.

So how does one fully use AWS to create a reliable cloud-native CI/CD system? For starters, here are some of Amazon’s cloud services that you might want to use.

  1. AWS CodePipeline
  2. AWS CodeCommit
  3. AWS CodeDeploy 
  4. Amazon EC2

The figure below is a simplified representation of a CI/CD pipeline you can implement on AWS. 

undefined

The main steps involved in setting up a CI/CD pipeline on AWS are:

  1. Setting in place a release pipeline for your app using AWS CodePipeline.
  2. Connecting a source code repository using AWS CodeCommit.
  3. Automating code deployments by connecting your pipeline with AWS CodeDeploy.
  4. Integrating a build-service, such as Jenkins (optional).

If you are interested in this setup, feel free to get started on the AWS CI/CD pipeline implementation guide by clicking here.

Deploy with Confidence (Conclusion)

By now, you should have a clear understanding of why using cloud-native solutions, specifically setting up a cloud-native CI/CD system is the way to go today. You should also have a good idea of what tools you can use to set up your CI/CD pipeline, as well as some of the possible tool combinations that can be considered. 

Yes, the CI/CD pipeline setup phase may be a headache, but we're sure that you will later thank yourself for going through all that trouble. Before we close, we'd advise you to not narrow down your options to a single tool when developing your ideal cloud-native CI/CD pipeline. Do tons of research before settling for a specific stack to work with, and from there, never stop learning and exploring its full potential. That’s it for this post - thank you for reading.

Keep learning, keep building - happy coding!