What’s changed with Workflows?

We’ve enhanced Workflows to make it simpler to deploy complex environments with dependencies.

A production app’s infrastructure comprises many resources. As those resources grow, managing and deploying them requires more thought and effort. 

These days we ‌look at managing infrastructure as we do application code — split a large whole into smaller, more manageable pieces where each piece is cohesive and loosely coupled.

In theory, this textbook solution sounds great because each resource is independent and can be deployed at any time.

But in the real world, we have dependencies which make deploying those resources simultaneously impractical.

Example

A "Network and VPC" manages the ‌network configuration.

"DB" manages a database that multiple services use.

"EKS" manages the Elastic Kubernetes Service where pods of services will be deployed.

"Billing Service"", "Configuration Service" and "Notification Service" all manage the deployments of services on EKS. Those services need access to the database.

 For this use case:

"DB" depends on the VPC from the "Network and VPC" IaC stack."EKS" also depends on the VPC from the "Network and VPC" IaC stack.All service environments depend on DB, and EKS

As can be seen, the dependencies make it impossible for all the Environments to be deployed simultaneously. Additionally, these are Kubernetes based deployments with multiple frameworks that have traditionally required complex integrations between the workflows.

As the resources and environments grow, the complexity of the dependencies will also exponentially increase.

Current State

Splitting a large infrastructure set is encouraged as a best practice to reap the benefits of a microservices architecture.

We see many DevOps teams implement this technique in IaC by splitting up the application into either a) Layers or b) Phases/Dependencies.

Layers

Sometimes cloud resources are ‌too expensive or slow to provision, or have strict, low and hard limits on instances per account.

If a resource or group of resources have one or several of these characteristics, it is often extracted to a separate stack that has to be deployed once per cloud account and reused by multiple other stacks (for example: think of a typical CloudFront distribution which usually takes 15-30 minutes to provision).

Although a common use case, most CD pipeline solutions do not have a first-class solution for it. 

Phases/Dependencies

Another common pattern is one where a different prerequisite stack instance is required for each environment.

This case is much more commonly supported by most CD pipelines as a common workflow/pipeline.

A group of resources or pipeline with such a workflow is often considered an environment - a group of services dedicated to this environment with some inter-dependency between them (for example: an S3 bucket with an SQS topic or perhaps some Lambda functions).

There are two common approaches here:

Dependencies

A pretty common approach by most CI/CD providers is to define which “job”, “run” or “stack” depends on other resources.

This allows us to create a fine grain dependency workflow.

Phases

Alternatively, in a simpler take on dependencies, jobs/runs/stacks are grouped by phases and deployed sequentially, but simultaneously inside each phase. Each phase starts after the previous phase completes.

How env0 Solves this

To set up an env0 Workflow follow this simple guide:

env0 relies on a file named env0.workflow.yml which describes the dependencies and configuration of the sub environments in your workflow. Create this file with your specific definitions, where each environment should define:

1. name: This will be displayed in the workflow graph.
2. templateName: A name of a pre-defined Template to deploy;
3. needs (optional): An array of sub environments which all must be successfully deployed before this sub environment can start deploying.

environments:

vpc:

name: 'VPC and Network'

templateName: 'VPC'

db:

name: DB

templateName: 'DB'

needs:

- vpc

eks:

name: EKS

templateName: 'EKS'

needs:

- vpc

service1:

name: 'Billing Service'

templateName: 'Billing Service'

needs:

- db

- eks

service2:

name: 'Configuration Service'

templateName: 'Configuration Service'

needs:

- db

- eks

service3:

name: 'Notification Service'

templateName: 'Notification Service'

needs:

- db

- eks

‍

2. Create a new Template and select env0 Workflow as the Template Type

In the VCS step fill in your VCS details and fill in the directory that contains your env0.workflow.yml file

3. Create an Environment based on the Workflow template

a. Enable Drift Detection if so desired, using a cron expression based on the interval you would like to check the drift. For example, to check for drift twice a day at 8AM and at 8PM enter 00 08,20 * * *.

b. Select a Time-to-Live (TTL), with the ‌options‌:

Use “Unlimited” for a long running environment, “Destroy after x hours” for ephemeral environments, or “Specific time limit” for environments that have a predefined schedule.

c. Variables in env0 are inherited hierarchically from Organization -> Project -> Template -> Environment. This allows us to securely and easily reuse common variables, such as VPC IDs, API Keys, etc – while making it easy to have granular control over deployment specific variables.

4. Deploy

As the resources are created, we can view our Workflow progress in the GRAPH tab.

Each sub-environment can be deployed individually. Workflows will also trickle down the rest of the dependencies.

By using env0 to manage workflows with dependencies, you’ll also get CI/CD, TTL/ephemeral environments, scheduling, drift detection, and deployment history out of the box.

Conclusion

Whether you’re using multiple infrastructure as code frameworks in your DevOps practice (Terraform, Terragrunt, Pulumi, CloudFormation, Kubernetes, etc) or asking “How do I manage the dependencies between infrastructure resources?”, env0 Workflows enable deploying many env0 Environments with complex dependencies as a ‌unit, using a simple YAML file.