Aug 10, 2022
Deploying Prometheus to a Kubernetes Cluster using env0
Deploying a Kubernetes cluster is a complex task. Even when we have the possibility to use a managed cluster, there is still the need to deal with the different requirements and flavors of each cloud provider.
To streamline the cluster deployment process, a proficient engineer will use the modern approach — infrastructure as code tools. Using IaC tools is the easiest way to configure a well-defined and structured setup we can re-use whenever needed. This way we can be confident we’ll get the same cluster setup every time.
So this all sounds wonderful, right? Well, almost. When dealing with setting up a cluster, there is a need to use many IaC tools. Today we have different tools for deploying a cluster (Terraform, Pulumi, CloudFormation) and for configuring it (Kubernetes manifests) so if we want to streamline the entire process, we’ll need to do it in distinct steps using the different tools.
In env0, we support many IaC tools as well as defining a single workflow that helps unify and streamline Kubernetes cluster deployments. Let’s go through a common example where we’ll be using Terraform to deploy an AWS EKS cluster, and then use kubectl to install and configure Prometheus monitoring on our cluster.
Let’s start by defining the moving parts of our workflow. For this, we need to define three env0 templates:
This will make env0 execute the aws eks CLI commands after the step of setting up the required variables and right before executing the kubectl diff and kubectl apply/delete commands. This will take place when an environment gets deployed as well as when it gets destroyed. I’ve also used the CLUSTER_NAME placeholder, which will be populated later on using env0 as an environment variable.
a. We’ll start with the prerequisites template. First, we’ll choose the Kubernetes template type:
b. Then we’ll set the VCS details of the template to use the env0/kube-prometheus Github repository we just forked, and set the Kubrenetes folder to point to the manifests/setup folder which includes some CRDs that are required by the prometheus operator.
In the same way we did with the previous template, we’ll skip the variables step and assign the template to our env0 project.
We’ll configure another Kuberentes template called Prometheus Operator - Installation with the same settings except for the Kuberentes folder configuration, which will now point to the manifests folder instead (and not the manifests/setup folder).
After we’ve created all the required templates for the EKS cluster and the Prometheus Operator installation steps, we can define the workflow to establish the dependencies between all the templates.
As described, we need to also create a template that streamlines both templates into a single workflow, for that we’ll need to do two things:
This yaml file describes the dependency between the Prometheus operator environment (with its installation steps in place) and the EKS cluster environment. This dependency means that the EKS cluster template will be deployed first, followed by the Prometheus prerequisites template first, and then the Prometheus operator template.
After creating all the required templates, I’ll go ahead and deploy the workflow environment. Here’s how the deployment graph looks:
Right after deployment is done, I can click each of the environments to see their status and deployment logs. Here’s how the Prometheus Operator Prerequisites environment looks:
Using templates and a workflow, we deployed an EKS cluster and installed our required basic setup on it. In this quick example, the setup only included a Prometheus Operator, but we could easily add any other Kubernetes project by configuring it as a template in env0. These templates could be deployed alongside the Prometheus operator or after it, allowing us to structure a clear dependency tree of different stacks using different IaC tools (env0 also supports Pulumi, Terragrunt and CloudFormation!)
Lastly, one thing you might notice from the steps here is that we’ve coded in the Kubernetes cluster authentication. Ordinarily, authentication is something you’d expect to be in the UI. For this initial release of Kubernetes support we opted for the fastest way to get the bulk of the functionality out for you to make use of. We have plans to integrate the authentication in the same way all our supported platforms have authentication support. So, we’d like your feedback! If you’re testing Kubernetes support here in env0 and would like to see that authentication rolled up into the UI, please drop us a line. You can simply ping us on Twitter if that’s easiest, or drop us an email to hello@env0.com
