Terratest vs Terraform/OpenTofu Test: In-Depth Comparison

The release of Terraform and OpenTofu 1.6 brought with it the general availability of the Terraform Test Framework. This framework allows you to write unit and integration tests for your Terraform modules and configurations.

Why should you test your IaC? And how does the Terraform Test Framework compare to Terratest? That's what we're going to explore in this article.

Introduction - Comparing Terraform Testing Frameworks

A key part of any software development process is testing. It's how you ensure that the code you wrote actually does what it is supposed to do. We write automated tests to ensure that we get the correct output for a given input, that our code handles errors gracefully, and that it works as expected when integrated with other parts of the system.

Infrastructure as Code is still code and should be treated as such, but testing infrastructure code is a little different than traditional software testing. Terraform code isn't composed of functions or methods for you to write unit tests against. 

Terraform code doesn't produce an API that needs to be validated against a specification, and it doesn't integrate with other application components in a traditional sense. Instead, Terraform code describes the desired state of your infrastructure, and relies on the core binary and provider plugins to make that state a reality.

But there's still testing to do! In my mind there's three categories of testing for Terraform.

First, there's basic validity. Is your code syntactically correct and does it parse logically? Have you specified any arguments that don't actually exist? Forgotten to close a curly brace? Or provided an argument value that's of the wrong data type?

Next, there's testing individual components of your code, aka unit testing. This includes objects like input variables, resource blocks, and outputs.

Unit testing is especially critical when you've added conditional logic, functions, and validation into your code. If you get bad input is it handled correctly? If you make a resource creation conditional, can you verify that it works correctly?

The final category is integration testing, which verifies the deployed infrastructure works as expected. If you're deploying a network with complex routing and firewall rules, can you verify that it's configured correctly to pass good traffic and block the bad stuff? When you update a module in your larger configuration, does everything continue to function properly?

For basic validity, the built-in [.code]terraform validate[.code] command along with third-party tools like tflint can help. For unit and integration testing, there was no native solution before the Terraform Testing Framework, so third-party testing frameworks were developed.

The two most popular testing solutions are Terratest by Gruntworks (also the creators of Terragrunt) and kitchen-terraform based on Chef's InSpec. Both follow a similar pattern, and for this article, we will look at Terratest specifically.

What is Terratest

Terratest is a testing framework for Terraform developed by the good folks at Gruntworks. It uses the Golang testing library along with Go modules developed by Gruntworks for unit and integration testing with Terraform code.

For example, you might be performing variable validation with the following code:

variable "vm_size" {
    description = "Size of VM to use. Must be D or E series and between 4 and 12 CPUs."
    type = string
    
    validation {
        condition = can(regex("^Standard_{DE}{4-12}*.$", var.vm_size))
        error = "Variable vm_size must be D or E series with 4-12 CPUs. Value provided was ${var.vm_size}"
    }
    
}

As part of your unit test, you want to make sure that valid values are accepted and invalid values are rejected. Since regular expressions are notoriously tricky, you might have a unit test that tries several correct and incorrect values. This verifies that your initial expression is correct, and provides a way to test the expression later if the requirements change.

Using Terratest, you would have the testing framework attempt to generate a valid plan with the range of values you'd like to test. The code might look something like this:

package tests

import (
	"testing"

	"github.com/gruntwork-io/terratest/modules/terraform"
	"github.com/stretchr/testify/assert"
)

var tests = []struct {
    vmSize string
    valid bool
}{
    {"Standard_E4s_v5", true},
    {"Standard_G4s_v5", false},
}

func TestVmSizeValidation(t *testing.T) {
    for _, tt := range tests {
        t.Run(tt.vmSize, func(t *testing.T) {
            terraformOptions := &terraform.Options{
                TerraformDir: "../",
                Vars: map[string]interface{}{
                    "vm_size": tt.vmSize,
                },
            }
            _, err := terraform.InitAndPlanE(t, terraformOptions)
            if tt.valid {
                assert.NoError(t, err)
            } else {
                assert.Error(t, err)
            }
        })
    }
}

Each value in [.code]tests[.code] is checked to see if it produces a valid plan, with the expected result expressed by the [.code]valid[.code] boolean value.

Here is the truncated output from a test run:

> go test
TestVmSizeValidation/Standard_E4s_v5 2024-03-19T13:48:39-04:00 retry.go:91: terraform [init -upgrade=false]
TestVmSizeValidation/Standard_E4s_v5 2024-03-19T13:48:39-04:00 logger.go:66: Running command terraform with args [init -upgrade=false]
...
PASS
ok      validation/tests        1.365s

Terratest can also perform integration tests to validate infrastructure is deployed and functioning correctly. 

For example, consider a Terraform configuration that creates a static website in Azure using an Azure Storage account. Terrastest could stand up an instance of the website and verify that it produces a "200" response code when queries. When the test is over, Terratest can destroy the infrastructure.

An example snippet might look like this:

func TestTerraformHttpExample(t *testing.T) {
	t.Parallel()

	// A unique ID to use for the website name
	uniqueID := rand.Intn(100) + 100

	// Create website name value
	websiteName := fmt.Sprintf("testingsite%d", uniqueID)

	// Construct the terraform options
	terraformOptions := terraform.WithDefaultRetryableErrors(t, &terraform.Options{
		// The path to where our Terraform code is located
		TerraformDir: "../",

		// Variables to pass to our Terraform code using -var options
		Vars: map[string]interface{}{
			"website_name":    websiteName,
		},
	})

	// At the end of the test, run `terraform destroy` to clean up any resources that were created
	defer terraform.Destroy(t, terraformOptions)

	// This will run `terraform init` and `terraform apply` and fail the test if there are any errors
	terraform.InitAndApply(t, terraformOptions)

	// Run `terraform output` to get the value of an output variable
	instanceURL := terraform.Output(t, terraformOptions, "homepage_url")

	// Setup a TLS configuration to submit with the helper, a blank struct is acceptable
	tlsConfig := tls.Config{}

	// Verify that we get back a 200 OK with the expected instanceText
	http_helper.HttpGet(t, instanceURL, &tlsConfig)
}

Terratest generates a random integer to append to the website name. Then it creates the infrastructure using Terraform and grabs the output [.code]homepage_url[.code]. 

Next, Terratest sends an HTTP GET request against the website, and the test passes if the http request is successful. Finally, Terratest destroys the temporary infrastructure when the test is complete.

Here is the truncated output of a test run:

> go test -v
TestTerraformHttpExample 2024-03-19T13:46:06-04:00 retry.go:91: terraform [init -upgrade=false]
TestTerraformHttpExample 2024-03-19T13:46:06-04:00 logger.go:66: Running command terraform with args [init -upgrade=false]
...
TestTerraformHttpExample 2024-03-19T13:47:25-04:00 logger.go:66: Destroy complete! Resources: 4 destroyed.
TestTerraformHttpExample 2024-03-19T13:47:25-04:00 logger.go:66: 
PASS
ok      testing-framework/tests 79.843s

The test passed and we know that the configuration successfully deploys a website.

What is Terraform/OpenTofu Test

The Terraform Testing Framework solves the same challenges as Terratest, but through a native command in the Terraform/OpenTofu binary and using tests that are written in HCL.

 As opposed to having to learn another language, like Go or Ruby, you can write your Terraform tests in the same declarative language you've already grown comfortable with.

The tests are written in a declarative manner, since Terraform already understands how to process configurations and manage infrastructure. You can focus on writing useful tests and not the mechanics of performing IaC testing. For unit and integration testing, the Terraform Testing Framework is a great solution.

Tests are placed in files ending in either tftest.hcl or tftest.json (yes, you can write your tests in JSON too, or have another language produce them programmatically.) The core construct for testing is the newly introduced run block. Each step of a testing sequence is defined using run blocks with the following general syntax:

run "<run name>" {
    command = "<plan or apply>

    variables {
        <input variable name> = <value for run>
    }

    assert {
        condition         = <condition to test, evaluating to true or false>
        error_message     = <message if condition is false>
    }
}

Within the run block you can set input variables values and provider configurations to use for that particular run.

You can also set input variable and provider configurations at the beginning of the file, which will be used by all runs unless overridden inside the run block.

Each run can either use the [.code]plan[.code] or [.code]apply[.code] command to execute the run.

Planning runs are great for unit testing when all values will be known before infrastructure deployment. Meanwhile, apply runs are better suited for integration testing when you want to deploy the actual infrastructure and test it.

Unit Testing

Going back to our unit testing example with Terratest, how could we accomplish the same test with the Testing Framework? Since we don't need to deploy actual infrastructure to check our input variable validation, we can use a [.code]plan[.code] command type. We also want to go through a list of variable values and test each one.

One of the run blocks would look something like this:

run "test_e_series" {
    command = plan

    variables {
        vm_size = "Standard_E4s_v5"
    }

}

Since our goal is to produce a valid plan, we don't need to put an assertion. The valid plan is proof that our value was accepted by the input variable validation block.

If we want to test an invalid value, the code would look a little different:

run "test_g_series" {
    command = plan

    variables {
        vm_size = "Standard_G4s_v5"
    }

    expect_failures = [
        var.vm_size
    ]

}

We are expecting that the validation block will reject the value we've passed, so we can add the [.code]expect_failures[.code] argument with a list of validations we expect to fail.

Additional run blocks can be added for each value we want to test for our [.code]vm_size[.code] input variable. At the moment the run block doesn't support the [.code]for_each[.code] meta-argument, but that support is coming soon.

To run the tests, the command terraform test or tofu test is executed from the configuration directory. By default, the test command will look for a directory named tests and execute tests defined by .tftest.hcl or .tftest.json files inside. You can also specify a different folder or specific test files to run.

The output of the [.code]terraform test[.code] command is shown below:

> terraform test
tests\variable_tests.tftest.hcl... in progress
  run "test_e_series"... pass
  run "test_g_series"... pass
tests\variable_tests.tftest.hcl... tearing down
tests\variable_tests.tftest.hcl... pass

Success! 2 passed, 0 failed.

Each test defined in the file variable_tests.tftest.hcl was executed in the order the [.code]run[.code] blocks appear in the file. That's one interesting quirk of the Terraform Testing Framework.

While Terraform in general doesn't care about the order in which blocks appear in a configuration, [.code]run[.code] blocks are executed in order. Additionally, when there are multiple testing files, they are executed in lexicographical order.

That covers unit testing, but what about integration testing?

Integration Testing

We can draw on the integration test from Terratest and see how we can reproduce it with the Terraform Testing Framework. What we need to do is generate a unique website name and deploy infrastructure defined in our Terraform module. The following code block can handle that for us:

run "execute" {
  command = apply
  
  variables {
    website_name = "Test${substr(uuid(),0,5)}"
  }
}

The [.code]command[.code] argument set to apply will tell Terraform to actually deploy the infrastructure found in the root module, and we're using the [.code]variables[.code] block to set the value for the website_name input variable.

Now how do we go about testing the website? We can use the [.code]http[.code] data source to do so, but we don't have an http data source in our configuration, nor do we need one for normal operations. The solution is to reference a module that does have the [.code]http[.code] data source and feed it the [.code]homepage_url[.code] output from the previous run block:

run "check_site" {
  command = apply

  variables {
    website_url = run.execute.homepage_url
  }

  module {
    source = "./tests/loader"
  }

  assert {
    condition     = data.http.main.status_code == 200
    error_message = "Website ${run.execute.homepage_url} returned the status code ${data.http.main.status_code}. Expected 200."
  }
}

The [.code]module[.code] block inside a run block uses the [.code]source[.code] argument to refer to either a module stored locally or on the Terraform registry. The run block will deploy the configuration defined inside the module rather than the root module.

Our loader module has the following configuration:

It takes the [.code]website_url[.code] input variable and uses the [.code]http[.code] data source to query the URL.

The objects defined in the module can be referred to directly in the [.code]run[.code] block, so we have our [.code]assert[.code] block with the following condition:

condition     = data.http.main.status_code == 200

As long as the status code is "200", the assertion passes. Let's see how our test does!

When there are multiple [.code]run[.code] blocks in a test file, Terraform maintains the state of each run until the last test is completed.

Then it tears down the tests in reverse order. Each subsequent run block has access to the outputs of the previous blocks using the syntax [.code]run.<run_name>.<output_name>[.code].

Terratest vs Terraform Test

From the examples we just ran through, you can see that the Terraform Test Framework can address many of the use cases for Terratest. That being said, Terratest has been around longer and is more fully featured and extensible.

If you've already set up a robust process using Terratest in your organization, I don't think it makes sense to try and move to the Terraform Testing Framework. Terratest is still an actively maintained project with a dedicated community.

However, if you are just getting started with testing your Terraform code, here are some points to consider.

Language Support

Terratest uses Golang and the Terraform Testing Framework uses HCL. If you're a heavy user of Terraform, then you've probably become quite proficient in HCL.

Learning a new language AND developing Terraform tests is a pretty tall order. You may find it much simpler to use the Testing Framework if you aren't already familiar with Go. I know I did!

Flexibility

In Terratest, you are responsible for writing the Go code to orchestrate the tests, giving you the freedom to integrate other Go modules and create custom workflows.

This freedom and flexibility might be a huge boon, or just another thing to manage and make sure everyone on your team is using consistently. The Terraform Testing Framework is more proscriptive in the testing workflow, which reduces flexibility, but also creates consistency.

Community Support

The Terraform Testing Framework is relatively new and still evolving rapidly. Currently there is no large repository of tests and examples to draw upon. Terratest, on the other hand, has been around for many years and has a robust library of tests, as well as many examples and blog posts on how to implement it.

The following table compares the features of Terratest versus Terraform/OpenTofu Test:

Bear in mind that this table is current as of publication, and both the Terraform and OpenTofu projects are actively updating and enhancing their implementations of the testing framework. Don't be surprised if JUnit support or [.code]for_each[.code] looping is supported in the next versions.

CI Testing and env0

While you can run your testing manually from your terminal, testing is an integral part of the CI/CD process. To that end, env0 has introduced a new CI testing feature that leverages the OpenTofu test feature into the env0 private module registry.

When you add a module to the env0 private registry, it will now include an option to enable testing when updates are pushed for the module. You can choose to run the tests on every pull request or only when a PR is merged to the tracked branch.

The results of the test are captured in the private registry and displayed as part of each module's information. You can open the module and drill down into the logs for each test to see where things are failing.

The CI testing feature is currently in Beta, and it is available to everyone, including the free tier. You can find out more by reading the introductory blog post and reading through the docs.

‍