Introduction

Terraform Cloud is a Hashicorp product that can be used to automate making changes to your cloud infrastrucure. This guide will help you configure and use Terraform Cloud to implement a "Path to Production" pipeline for such changes. The pipeline is designed to avoid any manual steps and ensure changes are well tested, and are safe to make in a production environment.

Contributing

This guide is free and open source. You can find the source on GitHub. If you spot any mistakes or have any feature requests these can be posted on the GitHub issue tracker.

License

This guide is released under the Creative Commons Attribution-ShareAlike 4.0 International license

Terraform Cloud Setup

This chapter will help you to configure the following Terraform Cloud resources, and will do so using Terraform itself. By the end of this chapter you will have:

• A Terraform Cloud Organization
• A Terraform Cloud Project
• A Terraform Cloud Workspace
• Integration between the Workspace and a GitHub repository such that commits to that repository will trigger a terraform plan in Terraform Cloud

Pre-Requisites

A number of accounts, credentials and command line utilities are required in order to follow this guide.

Accounts

1. An account on HashiCorp Cloud Platform (HCP)

2. An account on Terraform Cloud; visit Terraform Cloud and click Continue with HCP Account

3. An account on GitHub

4. An account on AWS

Command Line Utilities

1. The terraform CLI. Follow Hashicorp's installation instructions for your OS of choice. Although you'll be using Terraform Cloud to ultimately deploy your infrastructure, the local CLI is used to initially bootstrap things and can be further used to execute plans and applies in Terraform Cloud

Terraform Code Working Directory

We'll need a directory to work from which will contain our initial Terraform code that will create the Terraform Cloud resources and GitHub repository. The rest of the instructions in this chapter assume you're in the top level of that directory:

mkdir tfcloud-mgmt-scratch
cd tfcloud-mgmt-scratch

Terraform Cloud API Key

In order for the terraform CLI - both your local binary and that on the Terraform Cloud servers - to interact with Terraform Cloud, you need to generate an authorization token, otherwise known as an API key.

Create the API key:

terraform login

Create a file called credentials.auto.tfvars using the following command:

tfe_token=$(jq -r '.credentials."app.terraform.io".token' ~/.terraform.d/credentials.tfrc.json)
echo "tfe_token = \"${tfe_token}\"" > credentials.auto.tfvars

GitHub Personal Access Token for Pipelines

Similar to the Terraform Cloud API key created above, because Terraform will be interacting with GitHub to detect code changes then it needs an authorization token.

GitHub's documentation shows how to create a PAT.

Applying the principle of least privilege, the PAT token only needs read access to repositories. This enables Terraform Cloud to detect commits pushed to repositories and subsequently run plans and applies based on those changes. To grant this access select Repository access -> All repositories then under Permissions -> Repository permissions select Contents -> Read-only. Terraform Cloud also needs to be able to create webhooks. To grant this access select Permissions -> Repository permissions -> Webhooks -> Read and Write. Lastly, Terraform Cloud needs to be able to post commit statuses back to the PR to indicate whether the run was successful or not. To grant this access select Permissions -> Repository permissions -> Commit statuses -> Read and Write.

Set the appropriate credentials variable:

echo 'github_ro_token = "github_pat_*****..."' >> credentials.auto.tfvars

GitHub Personal Access Token for Repository Management

The Terraform Provider for GitHub is used to enable Terraform to create GitHub repositories and manage their settings. Because of these administrative type functions, it requires a PAT with much higher privilege than the PAT created above.

To grant this access select Repository access -> All repositories then under Permissions -> Repository permissions select Administration -> Read and write. The provider also needs write access to the contents of repositories in order to manage certain settings. To grant this access select Permissions -> Repository permissions -> Contents -> Read and write

Set the appropriate credentials variable:

echo 'github_admin_token = "github_pat_*****..."' >> credentials.auto.tfvars

AWS Credentials

Follow AWS' instructions for creating an access key for your own IAM user account. We will only need these to bootstrap connectivity between Terraform Cloud and AWS; after that then we'll use an IAM role to provide temporary security credentials for the duration of Terraform Cloud's runs.

Configure Terraform for Local Runs

At this early stage, all we want to do is ensure that terraform, when run locally, can initialize itself and run successfully but not manage any resources.

Copy and paste the following Terraform code into a new file, variables.tf, which declares the input variables we set up earlier in credentials.auto.tfvars

variable "github_admin_token" {
  type      = string
  sensitive = true
}

variable "github_ro_token" {
  type      = string
  sensitive = true
}

variable "tfe_token" {
  type      = string
  sensitive = true
}

Copy and paste the following Terraform code into a new file, terraform.tf, which will configure the Terraform Enterprise provider, AWS provider, and GitHub provider. Terraform will use local state files to keep track of any resources that it is managing.

terraform {
  required_providers {
    tfe = {
      source  = "hashicorp/tfe"
      version = "~> 0.49.2"
    }
    github = {
      source  = "integrations/github"
      version = "~> 5.39.0"
    }
  }

  required_version = "~> 1.6.0"
}

provider "tfe" {
  token = var.tfe_token
}

provider "github" {
  token = var.github_admin_token
}

Copy and paste the following into a new file, locals.tf, and adjust the values to match your desired Terraform Cloud and GitHub organization names:

locals {
  tfcloud_org_name        = "your_tfcloud_org_name"
  tfcloud_org_admin_email = "your_email_address"
  github_org_name         = "your_github_org_name"
}

Initialize Terraform:

$ terraform init

Initializing the backend...

Initializing provider plugins...
- Finding integrations/github versions matching "~> 5.39.0"...
- Finding hashicorp/tfe versions matching "~> 0.49.2"...
- Installing integrations/github v5.39.0...
- Installed integrations/github v5.39.0 (signed by a HashiCorp partner, key ID 38027F80D7FD5FB2)
- Installing hashicorp/tfe v0.49.2...
- Installed hashicorp/tfe v0.49.2 (signed by HashiCorp)

Partner and community providers are signed by their developers.
If you'd like to know more about provider signing, you can read about it here:
https://www.terraform.io/docs/cli/plugins/signing.html

Terraform has created a lock file .terraform.lock.hcl to record the provider
selections it made above. Include this file in your version control repository
so that Terraform can guarantee to make the same selections by default when
you run "terraform init" in the future.

Terraform has been successfully initialized!

You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.

If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.

At this point, a terraform plan should succeed but show no resources need to change, somewhat obviously due to us not having asked it to manage any just yet:

$ terraform plan

No changes. Your infrastructure matches the configuration.

Terraform has compared your real infrastructure against your configuration and found no differences, so no changes are needed.

Create Terraform Cloud Resources

We'd like Terraform to deploy the Terraform Cloud organization, a project within that organization, and a workspace within that project. Further, by linking the GitHub repository with the workspace, we can demonstrate Terraform Cloud's ability to automatically plan and apply changes made by commits to that repository.

To start with, copy and paste the following into main.tf to create the Terraform Cloud Organization and an OAuth client so that Terraform Cloud can watch for and react to commits to GitHub repositories.

resource "tfe_organization" "example" {
  name  = local.tfcloud_org_name
  email = local.tfcloud_org_admin_email

  lifecycle {
    prevent_destroy = true
  }
}

resource "tfe_oauth_client" "example_github" {
  name             = "tfe-tfcloud-mgmt-github-oauth-client"
  organization     = tfe_organization.example.name
  api_url          = "https://api.github.com"
  http_url         = "https://github.com"
  oauth_token      = var.github_ro_token
  service_provider = "github"
}

Next, copy and paste the following into tfcloud_variables.tf. The resources below manage common variable sets that hold the various credentials needed for both Terraform Cloud and the GitHub Terraform provider to interact with their respective APIs:

resource "tfe_variable_set" "tfcloud_common_credentials" {
  organization = tfe_organization.example.id
  name         = "tfcloud-common-credentials"
  description  = "Common credentials required for all tfcloud projects/workspaces"
}

resource "tfe_variable" "tfcloud_common_tfe_token" {
  category        = "terraform"
  key             = "tfe_token"
  sensitive       = true
  description     = "Terraform Enterprise API Token"
  value           = "Manually set to avoid storing in TF state"
  variable_set_id = tfe_variable_set.tfcloud_common_credentials.id

  lifecycle {
    ignore_changes = [value]
  }
}

resource "tfe_variable" "tfcloud_mgmt_gh_ro_token" {
  category        = "terraform"
  key             = "github_ro_token"
  sensitive       = true
  description     = "GitHub PAT for triggering runs"
  value           = "Manually set to avoid storing in TF state"
  variable_set_id = tfe_variable_set.tfcloud_common_credentials.id

  lifecycle {
    ignore_changes = [value]
  }
}

resource "tfe_variable_set" "github_provider_credentials" {
  organization = tfe_organization.example.id
  name         = "github-admin-credentials"
  description  = "Credentials required for the GitHub Terraform provider"
}

resource "tfe_variable" "tfcloud_mgmt_gh_admin_token" {
  category        = "terraform"
  key             = "github_admin_token"
  sensitive       = true
  description     = "GitHub PAT for creating and deleting repositories"
  value           = "Manually set to avoid storing in TF state"
  variable_set_id = tfe_variable_set.github_provider_credentials.id

  lifecycle {
    ignore_changes = [value]
  }
}

Next, copy and paste the following into tfcloud_mgmt_project.tf to create the Terraform Cloud project and workspace along with the associated GitHub repository. This also creates a workspace-scoped "variable set" resource to hold the credentials that Terraform Cloud will need in order to interact with both the Terraform Enterprise API and GitHub API. We follow Hashicorp's recommended practice of scoping the variable sets as narrowly as possible; we don't want any old project or workspace in our organization to be able to make changes to the Terraform Cloud organization.

locals {
  tfcloud_mgmt_project_name = "tfcloud-mgmt"
}

resource "github_repository" "tfcloud_mgmt" {
  name               = local.tfcloud_mgmt_project_name
  auto_init          = true
  gitignore_template = "Terraform"
  license_template   = "mit"

  lifecycle {
    prevent_destroy = true
  }
}

resource "github_branch_default" "tfcloud_mgmt_main" {
  repository = github_repository.tfcloud_mgmt.name
  branch     = "main"
}

resource "github_branch_protection" "tfcloud_mgmt" {
  repository_id  = github_repository.tfcloud_mgmt.name
  pattern        = github_branch_default.tfcloud_mgmt_main.branch
  enforce_admins = true

  required_status_checks {
    strict = false
    contexts = [
      "Terraform Cloud/${tfe_organization.example.name}/${tfe_workspace.tfcloud_mgmt_prod.name}",
    ]
  }
}

resource "tfe_project" "tfcloud_mgmt" {
  organization = tfe_organization.example.id
  name         = local.tfcloud_mgmt_project_name
}

resource "tfe_workspace" "tfcloud_mgmt_prod" {
  name              = "${local.tfcloud_mgmt_project_name}-prod"
  organization      = tfe_organization.example.id
  project_id        = tfe_project.tfcloud_mgmt.id
  terraform_version = "~> 1.6.0"

  tag_names = [
    local.tfcloud_mgmt_project_name,
    "prod"
  ]

  vcs_repo {
    identifier     = github_repository.tfcloud_mgmt.full_name
    oauth_token_id = tfe_oauth_client.example_github.oauth_token_id
  }
}

resource "tfe_workspace_variable_set" "tfcloud_mgmt_prod_tfcloud_common_credentials" {
  workspace_id    = tfe_workspace.tfcloud_mgmt_prod.id
  variable_set_id = tfe_variable_set.tfcloud_common_credentials.id
}

resource "tfe_workspace_variable_set" "tfcloud_mgmt_prod_tfcloud_github_provider_credentials" {
  workspace_id    = tfe_workspace.tfcloud_mgmt_prod.id
  variable_set_id = tfe_variable_set.github_provider_credentials.id
}

Running terraform apply should show that 14 resources need to be created, so go ahead and confirm to get things set up!

$ terraform apply
...
Plan: 14 to add, 0 to change, 0 to destroy.
...
Apply complete! Resources: 14 added, 0 changed, 0 destroyed.

Congratulations! You now have a Terraform Cloud organization, project and workspace configured. You also have a GitHub repository that is linked up to that workspace.

Notice that in your current working directory there is a file called terraform.tfstate which holds the state of your Terraform Cloud configuration as far as your local terraform considers it. Alas, Terraform Cloud itself knows nothing of this state of affairs. Next we'll perform a state migration which is how we get your local copy of the state into Terraform Cloud.

Migrate Local State to Terraform Cloud

Edit terraform.tf, adding the following inside the existing terraform {} block:

terraform {
  ...
  cloud {
    organization = "your_tfcloud_org_name"

    workspaces {
      name = "tfcloud-mgmt-prod"
    }
  }
}

Initialize Terraform again, and confirm that you want the current (local) state to be migrated to the Terraform Cloud Workspace:

$ terraform init

Initializing Terraform Cloud...
Do you wish to proceed?
  As part of migrating to Terraform Cloud, Terraform can optionally copy your
  current workspace state to the configured Terraform Cloud workspace.
  
  Answer "yes" to copy the latest state snapshot to the configured
  Terraform Cloud workspace.
  
  Answer "no" to ignore the existing state and just activate the configured
  Terraform Cloud workspace with its existing state, if any.
  
  Should Terraform migrate your existing state?

  Enter a value: yes
...
Terraform Cloud has been successfully initialized!
...

With the state migration complete, you can safely remove the cloud{} block that was temporarily added above; this is ignored by Terraform Cloud when the workspace is being managed by a VCS integration like GitHub.

VCS Workflow Integration

With the state now managed in Terraform Cloud, and the workspace configured to watch for changes in the tfcloud-mgmt repository, we're in a position to commit our code to the GitHub repository and confirm that everything's working as expected.

Set Credentials Variables

As you'll have seen from the configuration in tfcloud_mgmt_project.tf, given their very sensitive nature, we deliberately keep the GitHub and Terraform Cloud API tokens out of the Terraform state; the configuration of the variables is tracked by Terraform but their values are not. For similar security reasons, our local copy of credentials.auto.tfvars won't be committed to the repository. So, at this point, Terraform Cloud doesn't have a copy of the API keys it needs in order to successfully interact with both Terraform Cloud itself and with GitHub.

Using the Terraform Cloud UI, update the 3 variables with the values you have stored locally in credentials.auto.tfvars

Commit Code To The tfcloud-mgmt Repository

As the tfcloud-mgmt-prod workspace has been integrated with the GitHub repository, that repo is now the single source of truth for the state of your infrastructure.

If you go to the workspace in the Terraform Cloud UI, you'll notice that a run has already been triggered and that it errored with the following:

Error: No Terraform configuration files found in working directory

That's clear evidence that it's treating the repo as that source of truth; the repo has no code in it yet. Let's fix that:

cd ../
git clone https://github.com/your-github-org-name/tfcloud-mgmt
cd tfcloud-mgmt
cp ../tfcloud-mgmt-scratch/*.tf ../tfcloud-mgmt-scratch/*.hcl .
git checkout -b tfcloud-mgmt
git add .
git commit -m "Add tfcloud-mgmt resources"
git push

remote: error: GH006: Protected branch update failed for refs/heads/main.
remote: error: Required status check "Terraform Cloud/your-tfcloud-org/tfcloud-mgmt-prod" is expected.
To https://github.com/your-github-org/tfcloud-mgmt
 ! [remote rejected] main -> main (protected branch hook declined)
error: failed to push some refs to 'https://github.com/your-github-org/tfcloud-mgmt'

In order to have Terraform Cloud start a speculative plan, open a PR from the newly created tfcloud-mgmt branch.

The GitHub check should quite quickly progress from Pending to All checks have passed and the Details link will take you directly to the relevant run in the Terraform Cloud UI. Both the GitHub and Terraform Cloud UIs should show that no changes were detected.

Merge the PR then confirm in the Terraform Cloud UI that another plan was run which similarly detected no changes.

Tidy up

With everything committed to the repo, we can clean up the scratch directory we were previously working from. Now would be a good opportunity to backup the various API tokens in your password manager of choice as they currently only exist in credentials.auto.tfvars in your local scratch directory and in Terraform Cloud as sensitive write-only variables. Once backed up, remove the scratch directory:

rm -r ../tfcloud-mgmt-scratch

Initial Pipeline

This chapter will help you to configure a simple path to production pipeline in Terraform Cloud. The pipeline will deploy the following environments:

graph LR;
Pre-Dev-->Dev;
Dev-->Test;
Test-->Prod;

By the end of this chapter you will have:

• A GitHub repository containing Terraform code to manage an AWS VPC and some subnets.
• A Terraform Cloud project and several workspaces linked to the above GitHub repository; each workspace will control each of the above path to production stages
• OIDC authentication configured so that Terraform Cloud uses temporary credentials for interacting with AWS APIs

Create Common Variables for AWS Projects

In order for Terraform Cloud to be able to authenticate with AWS using short-lived credentials, we need to configure an OIDC connection. As all AWS projects will use this same authentication method, it makes sense to make the necessary information available via a shared variable set. Copy and paste the following into aws_common_variables.tf:

resource "tfe_variable_set" "aws_common" {
  name         = "aws-common"
  description  = "Variables common to all projects that use the AWS provider"
  organization = tfe_organization.example.id
}

resource "tfe_variable" "aws_common_oidc_provider" {
  key             = "oidc_provider"
  value           = <<-EOT
  {
    url             = "https://app.terraform.io"
    site_address    = "app.terraform.io"
    client_id_list  = [
    "aws.workload.identity",
    ]
    thumbprint_list = [
      "9e99a48a9960b14926bb7f3b02e22da2b0ab7280",
    ]
  }
  EOT
  hcl             = true
  category        = "terraform"
  description     = "Terraform Cloud OIDC Provider details"
  variable_set_id = tfe_variable_set.aws_common.id
}

resource "tfe_variable" "aws_common_tfcloud_org" {
  key             = "tfcloud_org"
  value           = tfe_organization.example.name
  category        = "terraform"
  description     = "Name of the Terraform Cloud Organization"
  variable_set_id = tfe_variable_set.aws_common.id
}

resource "tfe_variable" "aws_common_provider_auth" {
  category        = "env"
  key             = "TFC_AWS_PROVIDER_AUTH"
  value           = "true"
  description     = "Tells Terraform Cloud to authenticate to AWS"
  variable_set_id = tfe_variable_set.aws_common.id
}

Create a Module for Managing Terraform Cloud AWS-based Workspaces

As we'll be creating a number of Terraform Cloud workspaces in this section, it makes sense to create a Terraform module to ensure those workspaces are created in a consistent manner. Our module will follow the standard module structure, which defines the layout of modules and their position in the filesystem relative to other code.

To start with, we'll need to declare a number of variables that will be passed to the module. Copy and paste the following into modules/tfcloud_aws_workspace/variables.tf:

variable "pipeline_environment_name" {
  description = "Name of the pipeline environment being configured"
  type        = string
}

variable "pipeline_environment_configuration" {
  description = "Configures which AWS accounts and regions each pipeline stage will be deployed to"
  type = object({
    aws_account_id = string,
    regions        = list(string),
  })
}

variable "tfe_project" {
  description = "Details of the Terraform Cloud project that the workspace belongs to"
  type = object({
    tfe_organization = string
    project_name     = string
    project_id       = string
  })
}

variable "vcs_repo_name" {
  description = "Name of the VCS repo to link all workspaces to"
  type        = string
}

variable "vcs_repo_oauth_client_token_id" {
  description = "Oauth token used to authentication workspaces with the VCS provider"
  type        = string
}

Next, we'll have the module manage some Terraform Cloud resources. Copy and paste the following into modules/tfcloud_aws_workspace/main.tf:

variable "pipeline_environment_name" {
  description = "Name of the pipeline environment being configured"
  type        = string
}

variable "pipeline_environment_configuration" {
  description = "Configures which AWS accounts and regions each pipeline stage will be deployed to"
  type = object({
    aws_account_id = string,
    regions        = list(string),
  })
}

variable "tfe_project" {
  description = "Details of the Terraform Cloud project that the workspace belongs to"
  type = object({
    tfe_organization = string
    project_name     = string
    project_id       = string
  })
}

variable "vcs_repo_name" {
  description = "Name of the VCS repo to link all workspaces to"
  type        = string
}

variable "vcs_repo_oauth_client_token_id" {
  description = "Oauth token used to authentication workspaces with the VCS provider"
  type        = string
}

As you can see, the module is relatively simple; it simply creates a Terraform Cloud Workspace and some workspace-specific variables. As this guide is opinionated, we know that we'll be asking Terraform Cloud to create resources in an AWS account and we'd like it to use an OIDC provider in order to avoid using static authentication credentials. The workspace-specific variables help support that authentication flow:

• region - as we've configured our workspaces to be region-specific, as per Hashicorp's examples, and the AWS provider needs to know what region to operate in, we store this as a Terraform variable.

• TFC_AWS_PLAN_ROLE_ARN and TFC_AWS_APPLY_ROLE_ARN environment variables. These are part of the OIDC authentication flow; Terraform Cloud will assume these roles when running plan and apply operations respectively. We will create these roles shortly.

The OIDC setup is described in detail in Terraform Cloud's documentation.

Finally, we'll want to output the workspaces that the module creates as these will be used when configuring GitHub Pull Request checks a little later on. Copy and paste the following into modules/tfcloud_aws_workspace/outputs.tf:

output "workspace_names" {
  value = [for workspace in tfe_workspace.pipeline_environment : workspace.name]
}

output "workspace_ids" {
  value = { for workspace in tfe_workspace.pipeline_environment : workspace.name => workspace.id }
}

With the module in place, the next section will make use of it to actually create our example pipeline.

Create GitHub Repo and Terraform Cloud Project

Copy and paste the following into a new file, tfcloud_pipeline_project.tf in the tfcloud-mgmt repo.

locals {
  tfcloud_pipeline_project_name = "tfcloud-pipeline"

  tfcloud_pipeline_environment_configuration = {

    "pre-dev" = {
      aws_account_id = "320797911953"
      regions = [
        "eu-west-2"
      ]
    },

    "dev" = {
      aws_account_id = "320797911953"
      regions = [
        "eu-west-2",
      ]
    },

    "test" = {
      aws_account_id = "320797911953"
      regions = [
        "eu-west-2",
      ]
    },

    "prod" = {
      aws_account_id = "320797911953"
      regions = [
        "eu-west-2",
      ]
    },
  }

  tfcloud_pipeline_workspace_names = flatten([for k, v in local.tfcloud_pipeline_environment_configuration : module.tfcloud_pipeline_workspaces[k].workspace_names])
}

resource "github_repository" "tfcloud_pipeline" {
  name               = local.tfcloud_pipeline_project_name
  auto_init          = true
  gitignore_template = "Terraform"
  license_template   = "mit"

  lifecycle {
    prevent_destroy = true
  }
}

resource "github_branch_default" "tfcloud_pipeline_main" {
  repository = github_repository.tfcloud_pipeline.name
  branch     = "main"
}

resource "github_branch_protection" "tfcloud_pipeline_main" {
  repository_id  = github_repository.tfcloud_pipeline.name
  pattern        = github_branch_default.tfcloud_pipeline_main.branch
  enforce_admins = true

  required_status_checks {
    strict   = false
    contexts = formatlist("Terraform Cloud/%s/%s", tfe_organization.example.name, local.tfcloud_pipeline_workspace_names)
  }
}

resource "tfe_project" "tfcloud_pipeline" {
  organization = tfe_organization.example.id
  name         = local.tfcloud_pipeline_project_name
}

resource "tfe_project_variable_set" "tfcloud_pipeline_aws_common" {
  project_id      = tfe_project.tfcloud_pipeline.id
  variable_set_id = tfe_variable_set.aws_common.id
}

resource "tfe_variable_set" "tfcloud_pipeline_common" {
  name         = "${local.tfcloud_pipeline_project_name}-common"
  description  = "Variables common to all workspaces within the ${local.tfcloud_pipeline_project_name} project"
  organization = tfe_organization.example.id
}

resource "tfe_variable" "tfcloud_pipeline_common_tfcloud_project" {
  category        = "terraform"
  key             = "tfcloud_project"
  value           = local.tfcloud_pipeline_project_name
  description     = "Name of the Terraform Cloud Project"
  variable_set_id = tfe_variable_set.tfcloud_pipeline_common.id
}

resource "tfe_project_variable_set" "tfcloud_pipeline_common" {
  project_id      = tfe_project.tfcloud_pipeline.id
  variable_set_id = tfe_variable_set.tfcloud_pipeline_common.id
}

module "tfcloud_pipeline_workspaces" {
  for_each = local.tfcloud_pipeline_environment_configuration
  source   = "./modules/tfcloud_aws_workspace"

  tfe_project = {
    tfe_organization = tfe_project.tfcloud_pipeline.organization
    project_name     = tfe_project.tfcloud_pipeline.name
    project_id       = tfe_project.tfcloud_pipeline.id
  }

  pipeline_environment_name          = each.key
  pipeline_environment_configuration = each.value
  vcs_repo_name                      = github_repository.tfcloud_pipeline.full_name
  vcs_repo_oauth_client_token_id     = tfe_oauth_client.example_github.oauth_token_id
}

resource "tfe_run_trigger" "tfcloud_pipeline_dev_eu_west_2" {
  workspace_id  = module.tfcloud_pipeline_workspaces["dev"].workspace_ids["tfcloud-pipeline-dev-eu-west-2"]
  sourceable_id = module.tfcloud_pipeline_workspaces["pre-dev"].workspace_ids["tfcloud-pipeline-pre-dev-eu-west-2"]
}

resource "tfe_run_trigger" "tfcloud_pipeline_test_eu_west_2" {
  workspace_id  = module.tfcloud_pipeline_workspaces["test"].workspace_ids["tfcloud-pipeline-test-eu-west-2"]
  sourceable_id = module.tfcloud_pipeline_workspaces["dev"].workspace_ids["tfcloud-pipeline-dev-eu-west-2"]
}

resource "tfe_run_trigger" "tfcloud_pipeline_prod_eu_west_2" {
  workspace_id  = module.tfcloud_pipeline_workspaces["prod"].workspace_ids["tfcloud-pipeline-prod-eu-west-2"]
  sourceable_id = module.tfcloud_pipeline_workspaces["test"].workspace_ids["tfcloud-pipeline-test-eu-west-2"]
}

Adjust the aws_account_id values to match your AWS account setup. Whilst the pre-requisites only strictly need us to have a single AWS account, it's strongly recommended to maintain separate accounts for your different path to production environments. AWS has some guidance on this topic if you wish to explore it further but the main reasons for account separation are:

• Minimises the "blast radius" of changes; if your production account is the only one that contains production resources, then a change to your dev account can't possibly affect your production service.
• Some AWS services, most notably IAM and Route53 are global in nature. For example, a change to an IAM role in an account shared between environments will affect all environments at the same time.

One option to avoid some of the IAM-related problems that can arise from having a shared account is to prefix or suffix the role name such that unique roles are created in each workspace. This is alluded to above, and more clearly shown in the next section when we create the roles necessary for OIDC authentication.

The code above creates 4 workspaces, each one representing a separate stage in the path to production.

Commit and push your changes to a branch, and raise a PR. The resulting Terraform Cloud plan should show 35 resources will be created. Go ahead and merge the PR, then apply the changes.

If you take a look at the new tfcloud-pipeline project in the Terraform Cloud UI, you'll see that it has the expected 4 workspaces configured and that each of them had a plan triggered by the initial commit to the new tfcloud-pipeline GitHub repo. Again, as expected at this stage, all of those runs failed due to a lack of Terraform code in the repository. The next section will bootstrap the OIDC authentication between Terraform Cloud and your AWS account so that Terraform Cloud can plan and apply changes successfully.

Configure OIDC Provider

In order to avoid having to create long-term credentials for Terraform Cloud to use so that it can make changes to your infrastructure we can configure Terraform Cloud to assume an IAM role instead.

Similarly to the previous chapter, we'll need to create some resources from our local machine until we've configured enough infrastructure to allow Terraform Cloud to take over management duties.

Clone the repository that was created in the last section:

git clone https://github.com/your-github-org/tfcloud-pipeline
cd tfcloud-pipeline

Firstly, we'll create a new module that will create the roles that Terraform Cloud will assume during its plan and apply runs.

Copy and paste the following into a new file, modules/tfcloud_role/variables.tf:

variable "apply_role_arns" {
  description = "List of role ARNs to the workspace's apply policy"
  type        = list(string)
}

variable "environment" {
  description = "Environment"
  type        = string
}

variable "oidc_provider_url" {
  description = "Terraform Cloud OIDC Provider URL"
  type        = string
  default     = "https://app.terraform.io"
}

variable "plan_role_arns" {
  description = "List of role ARNs to the workspace's plan policy"
  type        = list(string)
}

variable "tfcloud_org" {
  description = "Terraform Cloud Organization name"
  type        = string
}

variable "tfcloud_project" {
  description = "Terraform Cloud project name"
  type        = string
}

variable "workspace_name" {
  description = "Terraform Cloud workspace name"
  type        = string
}

Copy and paste the following into a new file, modules/tfcloud_role/main.tf

locals {
  oidc_provider_site_address = replace(data.aws_iam_openid_connect_provider.tfcloud.url, "https://", "")
}

data "aws_iam_openid_connect_provider" "tfcloud" {
  url = var.oidc_provider_url
}

data "aws_iam_policy_document" "tfcloud_pipeline_plan_assume_role" {
  statement {
    effect = "Allow"

    principals {
      type = "Federated"

      identifiers = [
        data.aws_iam_openid_connect_provider.tfcloud.arn,
      ]
    }

    actions = [
      "sts:AssumeRoleWithWebIdentity",
    ]

    condition {
      test     = "StringEquals"
      variable = "${local.oidc_provider_site_address}:aud"
      values   = data.aws_iam_openid_connect_provider.tfcloud.client_id_list
    }

    condition {
      test     = "StringLike"
      variable = "${local.oidc_provider_site_address}:sub"

      values = [
        "organization:${var.tfcloud_org}:project:${var.tfcloud_project}:workspace:${var.workspace_name}:run_phase:plan",
      ]
    }
  }
}

resource "aws_iam_role" "tfcloud_pipeline_plan" {
  name               = "${var.workspace_name}-plan"
  assume_role_policy = data.aws_iam_policy_document.tfcloud_pipeline_plan_assume_role.json
}

resource "aws_iam_role_policy_attachment" "tfcloud_pipeline_plan" {
  count      = length(var.plan_role_arns)
  role       = aws_iam_role.tfcloud_pipeline_plan.name
  policy_arn = var.plan_role_arns[count.index]
}

data "aws_iam_policy_document" "tfcloud_pipeline_apply_assume_role" {
  statement {
    effect = "Allow"

    principals {
      type = "Federated"

      identifiers = [
        data.aws_iam_openid_connect_provider.tfcloud.arn,
      ]
    }

    actions = [
      "sts:AssumeRoleWithWebIdentity",
    ]

    condition {
      test     = "StringEquals"
      variable = "${local.oidc_provider_site_address}:aud"
      values   = data.aws_iam_openid_connect_provider.tfcloud.client_id_list
    }

    condition {
      test     = "StringLike"
      variable = "${local.oidc_provider_site_address}:sub"

      values = [
        "organization:${var.tfcloud_org}:project:${var.tfcloud_project}:workspace:${var.workspace_name}:run_phase:apply",
      ]
    }
  }
}

resource "aws_iam_role" "tfcloud_pipeline_apply" {
  name               = "${var.workspace_name}-apply"
  assume_role_policy = data.aws_iam_policy_document.tfcloud_pipeline_apply_assume_role.json
}

resource "aws_iam_role_policy_attachment" "tfcloud_pipeline_apply" {
  count      = length(var.apply_role_arns)
  role       = aws_iam_role.tfcloud_pipeline_apply.name
  policy_arn = var.apply_role_arns[count.index]
}

With the module in place, we can set up the root module, which will create the OIDC provider and associated IAM roles.

Copy and paste the following into a new file, variables.tf:

variable "environment" {
  type        = string
  description = "Environment"
}

variable "oidc_provider" {
  description = "Terraform Cloud OIDC Provider details"
  type = object({
    url             = string
    site_address    = string
    client_id_list  = list(string)
    thumbprint_list = list(string)
  })
}

variable "region" {
  type        = string
  description = "AWS region"
}

variable "tfcloud_org" {
  type        = string
  description = "Terraform Cloud Organization name"
}

variable "tfcloud_project" {
  type        = string
  description = "Terraform Cloud Project names"
}

variable "workspace_name" {
  description = "Name of workspace"
  type        = string
}

Copy and paste the following into a new file, terraform.tf:

terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.20.0"
    }
  }

  required_version = "~> 1.6.0"
}

provider "aws" {
  region = var.region
}

Next, copy and paste the following into a new file, main.tf:

locals {
  configure_oidc = var.environment == "pre-dev" && var.region == "eu-west-2"
}

resource "aws_iam_openid_connect_provider" "terraform_cloud" {
  count           = local.configure_oidc ? 1 : 0
  url             = var.oidc_provider.url
  client_id_list  = var.oidc_provider.client_id_list
  thumbprint_list = var.oidc_provider.thumbprint_list
}

module "tfcloud_pipeline_roles" {
  source          = "./modules/tfcloud_role"
  environment     = var.environment
  tfcloud_project = var.tfcloud_project
  tfcloud_org     = var.tfcloud_org
  workspace_name  = var.workspace_name

  plan_role_arns = [
    "arn:aws:iam::aws:policy/IAMReadOnlyAccess",
    "arn:aws:iam::aws:policy/AmazonVPCReadOnlyAccess",
  ]

  apply_role_arns = [
    "arn:aws:iam::aws:policy/IAMFullAccess",
    "arn:aws:iam::aws:policy/AmazonVPCFullAccess",
  ]
}

Next, copy and paste the following into a new file, tfcloud_pipeline.auto.tfvars:

environment = "pre-dev"

oidc_provider = {
  client_id_list = [
    "aws.workload.identity",
  ]

  site_address = "app.terraform.io"

  thumbprint_list = [
    "9e99a48a9960b14926bb7f3b02e22da2b0ab7280",
  ]

  url = "https://app.terraform.io"
}

region          = "eu-west-2"
tfcloud_org     = "your-tfcloud-org"
tfcloud_project = "tfcloud-pipeline"

Run terraform init:

$ terraform init

Initializing the backend...

Initializing provider plugins...
- Finding hashicorp/aws versions matching "~> 5.20.0"...
- Installing hashicorp/aws v5.20.0...
- Installed hashicorp/aws v5.20.0 (signed by HashiCorp)

Terraform has created a lock file .terraform.lock.hcl to record the provider
selections it made above. Include this file in your version control repository
so that Terraform can guarantee to make the same selections by default when
you run "terraform init" in the future.

Terraform has been successfully initialized!

You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.

If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.

We'll need to create the roles for every workspace so that Terraform Cloud can assume them for their plan and apply runs. We'll start off in the pre-dev eu-west-2 workspace so that the OIDC provider is created:

$ terraform workspace new tfcloud-pipeline-pre-dev-eu-west-2
Created and switched to workspace "tfcloud-pipeline-pre-dev-eu-west-2"!

You're now on a new, empty workspace. Workspaces isolate their state,
so if you run "terraform plan" Terraform will not see any existing state
for this configuration.

Now we'll create the OIDC provider separately from the rest of the resources. This is required because there's an aws_openid_connect_provider data source in the tfcloud_role; running a terraform apply without a matching data source in AWS will error.

Run terraform apply -target aws_iam_openid_connect_provider.tfcloud to create the OIDC provider in your AWS account:

$ terraform apply -target aws_iam_openid_connect_provider.tfcloud

...
Terraform will perform the following actions:

  # aws_iam_openid_connect_provider.tfcloud[0] will be created
  + resource "aws_iam_openid_connect_provider" "tfcloud" {
      + arn             = (known after apply)
      + client_id_list  = [
          + "aws.workload.identity",
        ]
      + id              = (known after apply)
      + tags_all        = (known after apply)
      + thumbprint_list = [
          + "9e99a48a9960b14926bb7f3b02e22da2b0ab7280",
        ]
      + url             = "https://app.terraform.io"
    }

Plan: 1 to add, 0 to change, 0 to destroy.
╷
│ Warning: Resource targeting is in effect
│ 
│ You are creating a plan with the -target option, which means that the result of this plan may not represent all of the changes requested by
│ the current configuration.
│ 
│ The -target option is not for routine use, and is provided only for exceptional situations such as recovering from errors or mistakes, or when
│ Terraform specifically suggests to use it as part of an error message.
╵

Do you want to perform these actions in workspace "tfcloud-pipeline-pre-dev-eu-west-2"?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

aws_iam_openid_connect_provider.tfcloud[0]: Creating...
aws_iam_openid_connect_provider.tfcloud[0]: Creation complete after 1s [id=arn:aws:iam::012345678901:oidc-provider/app.terraform.io]
╷
│ Warning: Applied changes may be incomplete
│ 
│ The plan was created with the -target option in effect, so some changes requested in the configuration may have been ignored and the output
│ values may not be fully updated. Run the following command to verify that no other changes are pending:
│     terraform plan
│  
│ Note that the -target option is not suitable for routine use, and is provided only for exceptional situations such as recovering from errors
│ or mistakes, or when Terraform specifically suggests to use it as part of an error message.
╵

Apply complete! Resources: 1 added, 0 changed, 0 destroyed.

Planning failed. Terraform encountered an error while generating this plan.

╷
│ Error: No valid credential sources found
│ 
│   with provider["registry.terraform.io/hashicorp/aws"],
│   on terraform.tf line 10, in provider "aws":
│   10: provider "aws" {
│ 
│ Please see https://registry.terraform.io/providers/hashicorp/aws
│ for more information about providing credentials.
│ 
│ Error: failed to refresh cached credentials, no EC2 IMDS role found, operation error
│ ec2imds: GetMetadata, request canceled, context deadline exceeded

With the OIDC provider now in place, we can run a full terraform apply to create the IAM roles and attach the required policies to them:

$ terraform apply
module.tfcloud_pipeline_roles.data.aws_iam_openid_connect_provider.tfcloud: Reading...
aws_iam_openid_connect_provider.tfcloud[0]: Refreshing state... [id=arn:aws:iam::320797911953:oidc-provider/app.terraform.io]
module.tfcloud_pipeline_roles.data.aws_iam_openid_connect_provider.tfcloud: Read complete after 0s [id=arn:aws:iam::320797911953:oidc-provider/app.terraform.io]
module.tfcloud_pipeline_roles.data.aws_iam_policy_document.tfcloud_pipeline_plan_assume_role: Reading...
module.tfcloud_pipeline_roles.data.aws_iam_policy_document.tfcloud_pipeline_apply_assume_role: Reading...
module.tfcloud_pipeline_roles.data.aws_iam_policy_document.tfcloud_pipeline_apply_assume_role: Read complete after 0s [id=595414985]
module.tfcloud_pipeline_roles.data.aws_iam_policy_document.tfcloud_pipeline_plan_assume_role: Read complete after 0s [id=4136116602]

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:
...
module.tfcloud_pipeline_roles.aws_iam_role.tfcloud_pipeline_apply: Creating...
module.tfcloud_pipeline_roles.aws_iam_role.tfcloud_pipeline_plan: Creating...
module.tfcloud_pipeline_roles.aws_iam_role.tfcloud_pipeline_apply: Creation complete after 1s [id=tfcloud-pipeline-pre-dev-eu-west-2-apply]
module.tfcloud_pipeline_roles.aws_iam_role_policy_attachment.tfcloud_pipeline_apply[0]: Creating...
module.tfcloud_pipeline_roles.aws_iam_role_policy_attachment.tfcloud_pipeline_apply[1]: Creating...
module.tfcloud_pipeline_roles.aws_iam_role.tfcloud_pipeline_plan: Creation complete after 1s [id=tfcloud-pipeline-pre-dev-eu-west-2-plan]
module.tfcloud_pipeline_roles.aws_iam_role_policy_attachment.tfcloud_pipeline_plan[1]: Creating...
module.tfcloud_pipeline_roles.aws_iam_role_policy_attachment.tfcloud_pipeline_plan[0]: Creating...
module.tfcloud_pipeline_roles.aws_iam_role_policy_attachment.tfcloud_pipeline_apply[0]: Creation complete after 0s [id=tfcloud-pipeline-pre-dev-eu-west-2-apply-20231008194953744100000001]
module.tfcloud_pipeline_roles.aws_iam_role_policy_attachment.tfcloud_pipeline_apply[1]: Creation complete after 0s [id=tfcloud-pipeline-pre-dev-eu-west-2-apply-20231008194953786500000002]
module.tfcloud_pipeline_roles.aws_iam_role_policy_attachment.tfcloud_pipeline_plan[1]: Creation complete after 0s [id=tfcloud-pipeline-pre-dev-eu-west-2-plan-20231008194953905700000003]
module.tfcloud_pipeline_roles.aws_iam_role_policy_attachment.tfcloud_pipeline_plan[0]: Creation complete after 0s [id=tfcloud-pipeline-pre-dev-eu-west-2-plan-20231008194954014200000004]

Create Roles for all Workspaces

We'll now need to go around the remaining workspaces and create the roles necessary for Terraform Cloud to be able to run correctly.

$ terraform workspace new tfcloud-pipeline-dev-eu-west-2
$ sed -i 's/pre-dev/dev/' tfcloud_pipeline.auto.tfvars
$ terraform apply
...
Apply complete! Resources: 6 added, 0 changed, 0 destroyed.

$ terraform workspace new tfcloud-pipeline-test-eu-west-2
$ sed -i 's/dev/test/' tfcloud_pipeline.auto.tfvars
$ terraform apply
...
Apply complete! Resources: 6 added, 0 changed, 0 destroyed.

$ terraform workspace new tfcloud-pipeline-prod-eu-west-2
$ sed -i 's/test/prod/' tfcloud_pipeline.auto.tfvars
$ terraform apply
...
Apply complete! Resources: 6 added, 0 changed, 0 destroyed.

Migrate Local State To Terraform Cloud

Just as we had to in the last chapter, we'll need to let Terraform Cloud take ownership of the state information for the above resources. We can let Terraform migrate all of our local workspaces in one go by using tags instead of explicitly naming the target workspace.

Edit terraform.tf, adding the following inside the existing terraform {} block:

terraform {
  ...
  cloud {
    organization = "your_tfcloud_org_name"

    workspaces {
      tags = [
        "tfcloud-pipeline",
      ]
    }
  }
}

Initialize Terraform again, and confirm that you want the current (local) state to be migrated to the Terraform Cloud Workspace:

$ terraform init

Initializing Terraform Cloud...
Would you like to rename your workspaces?
  Unlike typical Terraform workspaces representing an environment associated with a particular
  configuration (e.g. production, staging, development), Terraform Cloud workspaces are named uniquely
  across all configurations used within an organization. A typical strategy to start with is
  <COMPONENT>-<ENVIRONMENT>-<REGION> (e.g. networking-prod-us-east, networking-staging-us-east).
  
  For more information on workspace naming, see https://www.terraform.io/docs/cloud/workspaces/naming.html
  
  When migrating existing workspaces from the backend "local" to Terraform Cloud, would you like to
  rename your workspaces? Enter 1 or 2.
  
  1. Yes, I'd like to rename all workspaces according to a pattern I will provide.
  2. No, I would not like to rename my workspaces. Migrate them as currently named.

  Enter a value: 2

Migration complete! Your workspaces are as follows:
  tfcloud-pipeline-dev-eu-west-2
  tfcloud-pipeline-pre-dev-eu-west-2
* tfcloud-pipeline-prod-eu-west-2
  tfcloud-pipeline-test-eu-west-2

Initializing modules...

Initializing provider plugins...
- Reusing previous version of hashicorp/aws from the dependency lock file
- Using previously-installed hashicorp/aws v5.20.0

Terraform Cloud has been successfully initialized!

You may now begin working with Terraform Cloud. Try running "terraform plan" to
see any changes that are required for your infrastructure.

If you ever set or change modules or Terraform Settings, run "terraform init"
again to reinitialize your working directory.

With the state migration complete, you can safely remove the cloud{} block that was temporarily added above; this is ignored by Terraform Cloud when the workspace is being managed by a VCS integration like GitHub.

Commit Code

As with the tfcloud-mgmt repository, the main branch in the tfcloud-pipeline repository is protected from being pushed to directly. Commit your code to a branch:

git checkout -b oidc-provider
git add .
git commit -m "Add OIDC provider and roles"
git push

This time, when you open a PR in GitHub, you should see 4 Terraform Cloud pipelines run, and all should report no changes being required. Go ahead and approve and merge the PR.