Trying to manage the state of your cloud infrastructure without a state management tool is much like hitting your head against a wall—it's nice when you stop. There are several ways to manage the state of your resources deployed via Terraform, one of which is using Amazon S3 and Amazon DynamoDB.

What is Terraform State?

Terraform stores information about the infrastructure it has created in the Terraform state. When you first run your Terraform code through terraform apply, it creates a default file called terraform.tfstate, which holds this information. Terraform then references this file and compares it with what it has already created to get the status of those resources. Note that this is not your entire cloud estate but what Terraform has created in that project.

For example, let’s say you wanted to create an EC2 instance in AWS via Terraform with the code below:

resource "aws_instance" "my_instance" {
 ami           = "ami-1234567890"
 instance_type = "t2.micro"
}

After running terraform apply, the state file will be updated and contain an entry regarding this newly created resource. This is in JSON format, and every time you use a Terraform command to plan/create/change/destroy infrastructure, it will check the status of the resource (in this case, the EC2 instance) in AWS and compare it to what is in the Terraform state. An example of the state file is below:

{
<some metadata>
 "resources": [
   {
     "mode": "managed",
     "type": "aws_instance",
     "name": "my_instance",
     "provider": "provider.aws",
     "instances": [
       {
         "schema_version": 1,
         "attributes": {
           "ami": "ami-1234567890",
           "availability_zone": "eu-central-1c",
           "id": "i-XXX",
           "instance_state": "running",
           "instance_type": "t2.micro",
           <etc>
         }
       }
     ]
   }
 ]
}

Now, this is all good and well if you are the only developer on a project, but what if you want to share the state with another developer? Copying the file from one to another works with two people…even then, what happens if you change simultaneously? What if you have 20 people on the team?

This is where remote state management comes in, and the good news is that Terraform has built-in support for these remote backends. A backend tells Terraform where and how to store the state (the default being a local file—the .tfstate file we spoke about). Various remote backends are supported, and we will discuss how to implement the AWS version of this.

How do we implement this?

Before actually implementing the state management, there are various setup steps that we need to ensure have been completed before we start with state management.

We need a provider.tf file where we declare the AWS provider, for example:

provider "aws" {
 region = "eu-central-1"
}

At this point, I usually create a backend.tf file for the S3 and DynamoDB resources and hold them in a separate folder where I’m running Terraform commands to ensure that the state management doesn’t get deleted if I run a terraform destroy. Go ahead and create a separate folder outside of this project and a Terraform file called backend.tf. Then, we can go ahead and start creating the resources required for remote state management.

Our directory structure ends up looking something like this:

Backend
--/backend.tf
--/provider.tf
Terraform-Project
--/main.tf
--/provider.tf

Implementing S3

Then, we can use Amazon S3 since it’s managed, highly durable and scalable (among many other great things). What’s also great is that it supports versioning, allowing us to keep each version of the updated file. Using S3 as a backend also allows us to use DynamoDB (with some specific attributes) to implement locking to ensure that others don’t try to access the state at the same time and potentially corrupt it. The S3 resource can be seen below:

resource "aws_s3_bucket" "terraform_state_bucket" {
 bucket = "terraform-state-lock"
 }

In practice, you should also add versioning and encryption so that it ends up looking like:

resource "aws_s3_bucket" "terraform_state_bucket" {
 bucket = "terraform-state-lock-bucket"

 versioning {
   enabled = true
 }

 server_side_encryption_configuration {
   rule {
     apply_server_side_encryption_by_default {
       sse_algorithm = "AES256"
     }
   }
 }

}

Note that the bucket's name must be globally unique across all AWS accounts and conform to the naming rules. If you get an error when applying, try a different bucket name.

Implementing DynamoDB

Now that we have the S3 bucket ready to be created, let’s have a look at the DynamoDB table:

resource "aws_dynamodb_table" "terraform_state_lock" {
 name         = "terraform-state-lock" 
 hash_key     = "LockID"
 attribute {
   name = "LockID"
   type = "S"
 }
}

This creates a DynamoDB table named terraform-state-lock that functions as our locking mechanism to make sure that other users don’t corrupt the state whilst we are making changes to our cloud infrastructure (and vice versa).

Setting up the backend

Now, to reference these resources as our state management mechanism, we first need to create them. Execute the terraform init and terraform apply commands in the directory where the backend.tf file is located.

Then if we go back to our main project directory where we will be working on our project, we need to reference this backend in our backend.tf file with the following code:

terraform {
 backend "s3" {
   bucket = "terraform-state-lock-bucket"
   key = "terraform"
   region = "eu-central-1"
   dynamodb_table = "terraform-state-lock"
 }
}

Now that we’ve referenced the backend, we can initialise and apply our code with terraform init and then terraform apply. If you first want to see what resources are going to be created/modified/destroyed, you can run a Terraform plan command.

Summary

In this article, we examined Terraform state and how to implement it in the AWS context using Amazon S3 and Amazon DynamoDB. This allowed us to implement versioning and locking in a highly scalable and durable environment.

If you found this helpful, please feel free to buy me a coffee!