Why env0
Solutions
Infrastructure as Code (IaC) Automation Teams and GovernanceManaged Self-Service env0 for the Enterprise
DocumentationPricing
Resources
Case StudiesIn the NewsVideosWebinars
Blog
Login
Get a Demo
FREE TRIALContact us
Aug 10, 2022

Deploying Prometheus to a Kubernetes Cluster using env0

Yaron Yarimi
Software Engineer

Deploying a Kubernetes cluster is a complex task. Even when we have the possibility to use a managed cluster, there is still the need to deal with the different requirements and flavors of each cloud provider.

To streamline the cluster deployment process, a proficient engineer will use the modern approach — infrastructure as code tools. Using IaC tools is the easiest way to configure a well-defined and structured setup we can re-use whenever needed. This way we can be confident we’ll get the same cluster setup every time.

So this all sounds wonderful, right? Well, almost. When dealing with setting up a cluster, there is a need to use many IaC tools. Today we have different tools for deploying a cluster (Terraform, Pulumi, CloudFormation) and for configuring it (Kubernetes manifests) so if we want to streamline the entire process, we’ll need to do it in distinct steps using the different tools.

In env0, we support many IaC tools as well as defining a single workflow that helps unify and streamline Kubernetes cluster deployments. Let’s go through a common example where we’ll be using Terraform to deploy an AWS EKS cluster, and then use kubectl to install and configure Prometheus monitoring on our cluster.

Let’s start by defining the moving parts of our workflow. For this, we need to define three env0 templates: 

  • A Terraform template for the AWS EKS cluster and its prerequisites
  • Kubernetes template for Prometheus operator prerequisites (according to the guide on repository) 
  • Kubernetes template for the actual Prometheus operator installation
  • Workflow template that aggregates both

‍Terraform template - EKS cluster‍

  1. We first need to create the EKS cluster using Terraform. In the env0 UI, we’ll create a new template and select Terraform as its type. 
  1. After that, we’ll integrate the env0/k8s-modules GitHub repository with env0. This repository includes some Terraform module presets that will assist us in deploying the EKS cluster with ease.

‍

  1. We’ll hit next once, skipping the Variables step, which allows us to require or predefine environment variables for every environment created from this template (we’ll soon get to the setting required variables)
  1. We’ll hit next once again and assign the template to our relevant env0 project! Yay, we’re done! How easy was that?!

‍

Kubernetes templates - Prometheus operator and its prerequisites‍

  1. First, we must attach an `env0.yml` file that will allow us to execute the required authentication commands in order to connect to our EKS cluster before running any kubectl commands against it. Since we want to use the prometheus-operator/kube-prometheus repository, we must fork it and push an env0.yml file to its root folder. Here’s how the file will look like:

‍

version: 1
deploy:
  steps:
    setupVariables:
      after: &login-k8s
  • aws eks update-kubeconfig --name $CLUSTER_NAME
destroy:
  steps:
    setupVariables:
      after: *login-k8s

‍

This will make env0 execute the aws eks CLI commands after the step of setting up the required variables and right before executing the kubectl diff and kubectl apply/delete commands. This will take place when an environment gets deployed as well as when it gets destroyed. I’ve also used the CLUSTER_NAME placeholder, which will be populated later on using env0 as an environment variable.

  1. We now need to create the Prometheus operator using the kubectl CLI. For that, we’ll create two different Kubernetes templates, according to the required steps of the operator’s installation guide:

a. We’ll start with the prerequisites template. First, we’ll choose the Kubernetes template type:

b. Then we’ll set the VCS details of the template to use the env0/kube-prometheus Github repository we just forked, and set the Kubrenetes folder to point to the manifests/setup folder which includes some CRDs that are required by the prometheus operator.

In the same way we did with the previous template, we’ll skip the variables step and assign the template to our env0 project.

  1. Now we’ll need to repeat the same steps for the Prometheus operator template, based on the installation instructions of the repository. 

We’ll configure another Kuberentes template called Prometheus Operator - Installation with the ‌same settings except for the Kuberentes folder configuration, which will now point to the manifests folder instead (and not the manifests/setup folder).

After we’ve created all the required templates for the EKS cluster and the Prometheus Operator installation steps, we can define the workflow to establish the dependencies between all the templates.

‍

Workflow template‍

As described, we need to also create a template that streamlines both templates into a single workflow, for that we’ll need to do two things:

  1. Push an `env0.workflow.yml` file to a new GitHub repository, defining the required workflow:
environments:
  eks:
    name: eks
    templateName: 'EKS Cluster'
  prometheus-prereq:
    name: prometheus-prereq
    templateName: 'Prometheus Operator - Prerequisites'
    needs:
      - eks
  prometheus-install:
    name: prometheus-install
    templateName: 'Prometheus Operator - Installation'
    needs:
      - prometheus-prereq

This yaml file describes the dependency between the Prometheus operator environment (with its installation steps in place) and the EKS cluster environment. This dependency means that the EKS cluster template will be deployed first, followed by the Prometheus prerequisites template first, and then the Prometheus operator template.

  1. Add a workflow template that points to this yaml file:
  1. After hitting next, we’ll get to the Variables step. This step will allow us to define some required environment variables, forcing everyone who uses this template to supply them before deploying. As you probably already guessed, I’m talking about AWS credentials - these are needed both for our EKS cluster setup and for the aws eks CLI authentication command. Besides that, we also need to supply a CLUSTER_NAME variable, which will create the EKS cluster with a unique name and use it in the env0.yml file we created for the Kubernetes authentication.

‍

Deploying it all‍

After creating all the required templates, I’ll go ahead and deploy the workflow environment. Here’s how the deployment graph looks:

Right after deployment is done, I can click each of the environments to see their status and deployment logs. Here’s how the Prometheus Operator Prerequisites environment looks:

Environment status and resources of the Prometheus Operator Prerequisites environment.

‍

Output of the kubectl apply command

‍

‍

Summary

Using templates and a workflow, we deployed an EKS cluster and installed our required basic setup on it. In this quick example, the setup only included a Prometheus Operator, but we could easily add any other Kubernetes project by configuring it as a template in env0. These templates could be deployed alongside the Prometheus operator or after it, allowing us to structure a clear dependency tree of different stacks using different IaC tools (env0 also supports Pulumi, Terragrunt and CloudFormation!)

Lastly, one thing you might notice from the steps here is that we’ve coded in the Kubernetes cluster authentication. Ordinarily, authentication is something you’d expect to be in the UI. For this initial release of Kubernetes support we opted for the fastest way to get the bulk of the functionality out for you to make use of. We have plans to integrate the authentication in the same way all our supported platforms have authentication support. So, we’d like your feedback! If you’re testing Kubernetes support here in env0 and would like to see that authentication rolled up into the UI, please drop us a line. You can simply ping us on Twitter if that’s easiest, or drop us an email to hello@env0.com

‍

Interested in learning more about env0?
Request a Demo
SHARE
You may also like
Recommendations for Migrating from Terraform Cloud
Infrastructure as Code is a Creative Job
How to integrate Azure DevOps with env0
Go back to blog
CNCF Member Badge
Company
About UsIn the NewsPress ReleasesCase StudiesAdditional ResourcesCareers
Developer and DevOps
APITerraform ProviderTerratag Open Source
Terraform Cloud AlternativeDIY AlternativeAtlantis Alternative
FREE TRIAL
Follow Us
Terms of ServicePrivacy PolicySecuritySystem Status
© Copyright env0 2023
This website uses cookies. We use cookies to ensure that we give you the best experience on our website. Learn More
PreferencesDenyAccept
Privacy Preference Center
When you visit websites, they may store or retrieve data in your browser. This storage is often necessary for the basic functionality of the website. The storage may be used for marketing, analytics, and personalization of the site, such as storing your preferences. Privacy is important to us, so you have the option of disabling certain types of storage that may not be necessary for the basic functioning of the website. Blocking categories may impact your experience on the website.
Reject all cookiesAllow all cookies
Manage Consent Preferences by Category
Essential
Always Active
These items are required to enable basic website functionality.
Marketing
These items are used to deliver advertising that is more relevant to you and your interests. They may also be used to limit the number of times you see an advertisement and measure the effectiveness of advertising campaigns. Advertising networks usually place them with the website operator’s permission.
Personalization
These items allow the website to remember choices you make (such as your user name, language, or the region you are in) and provide enhanced, more personal features. For example, a website may provide you with local weather reports or traffic news by storing data about your current location.
Analytics
These items help the website operator understand how its website performs, how visitors interact with the site, and whether there may be technical issues. This storage type usually doesn’t collect information that identifies a visitor.
Confirm my preferences and close