Chapter 2. Infrastructure as Code and Deployment Automation (Intermediate: Terraform & IaC)
Learning objectives
- Explain the operational limits of manual console configuration (ClickOps), and describe how Terraform's declarative model and state file (
terraform.tfstate) work together. - Configure an S3 remote backend with state locking (
use_lockfileand DynamoDB) to prevent concurrent runs from corrupting infrastructure state. - Design reusable Terraform modules, and compare directory-based environment separation against CLI workspaces to choose a safe production isolation layout.
- Distinguish in-place updates (
~) from destroy-and-recreate replacements (-/+) interraform planoutput, and refactor without data loss usinglifecycle { prevent_destroy = true }alongsidemovedandimportblocks. - Explain why
sensitive = trueor ephemeral values do not replace state file encryption, and design least-privilege access controls and secret injection flows for state storage. - Identify the root causes of configuration drift, and execute the proper workflow to either revert unauthorized manual changes or codify emergency hotfixes into HCL.
- Build a keyless Terraform deployment pipeline using GitHub Actions OIDC (
AssumeRoleWithWebIdentity) with strict claim conditions (subandaud) instead of long-lived IAM access keys.
Introduction
When teams first launch a cloud service, almost everyone signs in to the AWS, Cloudflare, or Supabase web console and clicks buttons with a mouse to provision resources. Opening a port in a security group, creating an S3 bucket, or adding a DNS record takes less than five minutes. This manual approach is widely known as ClickOps.
As soon as the service grows—when you need to spin up an identical staging environment, or when an undocumented firewall rule tweaked at 3 a.m. during an outage breaks another feature the next day—ClickOps turns into a nightmare. "Which checkbox did Alex toggle in the console last month?", "Why does this work in staging but fail with a permission error in production?", "Someone accidentally deleted a routing table and nobody remembers the original routes." Mouse clicks leave no Git commit history, no pull-request code review, and no automated rollback path.
Infrastructure as Code (IaC) and its industry-standard tool, Terraform, transform infrastructure from the fragile byproduct of mouse clicks into version-controlled, declarative code (.tf). Yet using Terraform without understanding its internal mechanics can trigger disasters even faster than ClickOps. A single careless terraform apply can destroy and recreate (-/+) a live production database, wiping out customer records; two engineers running commands at the same time can corrupt the state file; or committing terraform.tfstate to a Git repository can expose the database master password in plaintext. This chapter covers the ten essential concepts and CI/CD automation patterns needed to operate Terraform safely and predictably.
Key concepts
The declarative model and the core Terraform workflow
Approaches to automating infrastructure fall into two broad paradigms:
- Imperative (procedural) approach: Shell scripts or raw CLI commands specify how to achieve a goal step by step. For example: "Create a security group → open port 443 → launch 2 servers." The problem surfaces the second time you run the script: either it fails with an error because the security group already exists, or it launches 2 additional servers so you end up with 4 instead of 2.
- Declarative approach: Terraform (written in HCL, the HashiCorp Configuration Language) specifies only what the final target state should look like. If you declare "there must be 1 security group with port 443 open and 2 servers," Terraform creates 2 servers when 0 exist, does nothing when 2 already exist (idempotency), terminates 1 server if 3 exist, and computes the exact delta (diff) if you change port 443 to 8443.
Every Terraform change moves through a standard three-stage lifecycle:
terraform init: Initializes the working directory, downloads the required cloud provider plugins (AWS, Cloudflare, Supabase, and so on) and modules, and connects to the remote backend where the state file is stored.terraform plan: Without touching any live cloud resources yet, compares (a) your.tfcode (the desired state), (b) the state file (terraform.tfstate, the last recorded state), and (c) the actual live infrastructure queried via cloud APIs, producing an execution plan showing exactly what will be created (+), updated in place (~), destroyed (-), or replaced (-/+).terraform apply: Executes the reviewed execution plan against the cloud provider APIs and writes the resulting resource metadata back into the state file.
The role and structure of the state file (terraform.tfstate)
The secret behind Terraform's declarative engine is the state file (terraform.tfstate).
When your code contains a block reading resource "aws_instance" "web" { ... }, how does Terraform know which specific EC2 instance (i-0a1b2c3d4e5f) among hundreds of instances in your AWS account corresponds to that block? During the initial apply, Terraform records the real cloud resource ID and its current attributes inside a JSON ledger called terraform.tfstate.
The state file fulfills three indispensable roles.
| Role | Explanation | What happens if the state file is lost or corrupted |
|---|---|---|
| Mapping code to real-world resources | Binds the logical address in .tf (aws_db_instance.main) 1:1 to the physical cloud identifier (arn:aws:rds:...) |
Terraform forgets the production DB already exists and attempts to create a duplicate or fails with a naming collision |
| Preserving the dependency graph | Tracks which resources reference others so deletion and replacement order is computed accurately | When a block is removed from code, Terraform cannot determine the safe order in which dependent resources must be torn down |
| Baseline for performance and drift detection | Caches a snapshot of resource attributes last observed from cloud APIs | Every run must rescan all cloud resources from scratch, or loses the baseline needed to detect out-of-band drift |
Because the state file is the heart of your infrastructure, you must never open terraform.tfstate in a text editor to edit its JSON manually, and you must never store it on a local laptop disk or pass it around to teammates over chat messages.
Remote backends and state locking
Storing terraform.tfstate in a local folder works fine when you are experimenting alone in a sandbox, but the moment two or more engineers collaborate—or a CI/CD pipeline runs Terraform—a Remote Backend and State Locking become mandatory.
What happens if Engineer A and Engineer B run terraform apply from their laptops against the same infrastructure at the exact same moment? Both processes read the old state, make conflicting API calls, and overwrite the state file (a classic race condition)—leaving newly created resources orphaned outside the state file or corrupting the state JSON altogether.
To prevent concurrent corruption, you store the state file in a central remote storage backend with versioning and encryption enabled (such as AWS S3, Terraform Cloud, or Google Cloud Storage) and enforce State Locking so that whenever one plan or apply is holding the lock, any competing run is blocked until the first finishes.
terraform {
backend "s3" {
bucket = "company-prod-tfstate-apne2"
key = "network/prod/terraform.tfstate"
region = "ap-northeast-2"
encrypt = true
use_lockfile = true
}
}
When using the AWS S3 backend, you should know both state-locking mechanisms:
- Traditional DynamoDB locking (
dynamodb_table): Pairs the S3 bucket with a separate DynamoDB table keyed byLockID(a string partition key), using conditional writes in DynamoDB to coordinate locks. This pattern remains widespread in existing production codebases. - Modern S3 native locking (
use_lockfile = true): Because Amazon S3 now supports conditional object writes (If-None-Matchheaders for atomic object creation), modern Terraform can atomically create a.tflocklock file directly inside the S3 bucket simply by settinguse_lockfile = true, eliminating the need to provision and maintain a separate DynamoDB table.
In addition, your state S3 bucket must always have Bucket Versioning, server-side encryption (SSE-S3 or SSE-KMS), and Block Public Access enabled. With versioning turned on, even if a state file is accidentally overwritten or damaged, you can immediately roll back to the previous S3 object version.
Caution
When a CI pipeline crashes mid-run and leaves a lock behind, engineers are sometimes tempted to pass -lock=false or blindly run terraform force-unlock <LOCK_ID> without checking whether a teammate's deployment is still running. Passing -lock=false disables the safety lock entirely, and force-unlock must only be used as a last resort after you have verified with 100% certainty that no other Terraform process is actively running against that state.
Module design and version pinning
Once an infrastructure codebase grows to thousands of lines, the same architectural building blocks—such as a VPC with subnets and security groups, or a hardened S3 bucket with KMS encryption and access logging—repeat across services. A Terraform Module packages a group of related resources into a reusable abstraction that works like a function:
- Input variables (
variables.tf,variableblocks): Parameters injected from outside the module. Use explicittypeconstraints andvalidationblocks so invalid region codes or malformed CIDR blocks fail immediately duringterraform plan. - Local values (
localsblocks): Internal constants and computed expressions scoped strictly inside the module. - Output values (
outputs.tf,outputblocks): Return values—such as the created resource ID or endpoint hostname—exposed to the calling root module or downstream modules.
When consuming modules from the public Terraform Registry or an internal Git repository, the single most important reliability and supply-chain security rule is version pinning:
module "vpc" {
source = "terraform-aws-modules/vpc/aws"
version = "5.16.0"
name = "prod-vpc"
cidr = "10.20.0.0/16"
}
If you omit the version constraint or leave it wide open, running terraform init -upgrade for a minor tweak months later can pull in a major module upgrade that renames internal resources—triggering a full replacement (-/+) of live production infrastructure. Always pin exact module versions (or verified Git commit SHAs/tags) in production, and commit the dependency lock file (.terraform.lock.hcl) to Git so every developer laptop and CI runner executes the exact same cryptographically hashed provider binaries.
Environment isolation: Per-directory state separation vs. workspaces
When managing dev, staging, and prod environments with Terraform, how should you structure your repository and state files? There are two main approaches, and understanding their trade-offs is essential.
| Comparison dimension | Directory & backend separation (envs/dev, envs/prod) |
Single directory + CLI workspaces (terraform workspace) |
|---|---|---|
| State file isolation | Completely separate S3 buckets/keys with independent IAM permissions per environment | Stored in the same backend bucket under env:/dev/... and env:/prod/... prefixes |
| Supporting code/version skew | Can test a new module version in staging for several days before promoting it to prod |
Every workspace shares the exact same .tf code and module versions simultaneously |
| Operator human-error risk | Running commands inside envs/dev can never touch prod resources |
High risk of running destroy or apply while thinking you are in dev when the active workspace is actually prod |
| Recommended use case | Permanent isolation between production and non-production environments (best practice) | Spinning up short-lived ephemeral preview environments per pull request |
In short, your production (prod) environment must always be isolated in its own directory, backed by its own state file, and governed by dedicated cloud accounts/projects and dedicated IAM roles. Extract shared resource patterns into reusable modules (modules/app-stack), and let envs/stage/main.tf and envs/prod/main.tf invoke that module independently with their own backend configurations and pinned versions.
Reading terraform plan output and blocking destructive changes (prevent_destroy)
Roughly 90% of Infrastructure as Code outages happen when someone skims past the warnings in terraform plan and reflexively types yes—or worse, runs apply -auto-approve. You must memorize the four action symbols that appear next to resources in terraform plan output.
| Symbol | Meaning | What actually happens in the cloud | Risk level |
|---|---|---|---|
+ (Create) |
Create new resource | Provisions a newly declared resource in the cloud | Low (check cost and quota limits) |
- (Destroy) |
Permanent deletion | Permanently deletes a resource removed from code | Very high (permanent data loss) |
~ (Update in-place) |
In-place modification | Modifies settings via API (such as tags, security group rules, or storage expansion) without deleting the resource | Medium (verify whether brief restart occurs) |
-/+ or +/- (Replace) |
Destroy and recreate (Replace) | Completely deletes (-) the existing resource and creates (+) a brand-new one. Resource ID and all internal data are wiped! |
Catastrophic (wipes DBs, volumes, and static IPs) |
Why would changing a single line of .tf code produce -/+ (Replace) instead of ~ (Update in-place)? Because the underlying cloud provider API treats that attribute as immutable once created. For example, changing an AWS RDS database identifier or encryption flag, changing an EC2 instance's subnet_id or AMI, or changing an S3 bucket's bucket name cannot be done in place—so Terraform marks that attribute with # forces replacement and plans a -/+ action.
To prevent stateful crown-jewel resources (production databases, persistent storage buckets, KMS encryption keys, and core VPCs) from ever being accidentally deleted or replaced (-/+), you enforce a code-level safety rail using prevent_destroy = true inside the lifecycle block:
resource "aws_db_instance" "production" {
identifier = "aidan-prod-postgres"
engine = "postgres"
instance_class = "db.r6g.large"
lifecycle {
prevent_destroy = true
}
}
With prevent_destroy = true in place, if anyone modifies an immutable database attribute that forces a replacement (-/+) or deletes the resource block, Terraform halts immediately during plan with an error (Instance cannot be destroyed). In addition, for stateless resources like web servers or TLS certificates that require zero-downtime replacement, use create_before_destroy = true so the new resource is provisioned before the old one is removed (+/-), and use ignore_changes = [desired_capacity] for attributes managed dynamically outside Terraform by an autoscaler.
Zero-downtime refactoring and adopting existing resources: moved and import blocks
As an infrastructure codebase matures, you will inevitably need to rename a resource from aws_s3_bucket.old_name to aws_s3_bucket.assets, or move a resource from the root module into a child module (module.storage.aws_s3_bucket.assets).
If you simply edit the resource name in your .tf file and run terraform plan, the output is alarming: Terraform does not know you renamed the block—instead, it plans to destroy (-) the existing old_name resource and create (+) a brand-new assets resource! Running apply on that plan would delete your live production S3 bucket or database.
Historically, engineers had to run imperative terraform state mv CLI commands in a terminal to rewrite the state file manually, which bypassed pull-request code review and was error-prone in CI pipelines. Modern Terraform solves this declaratively inside your code using a moved block:
moved {
from = aws_s3_bucket.old_name
to = module.storage.aws_s3_bucket.assets
}
When you commit the moved block alongside the renamed resource, terraform plan recognizes the refactoring, updates the address mapping inside terraform.tfstate without touching the actual cloud resource, and completes with zero downtime.
Similarly, when you need to bring legacy cloud resources that were originally created by hand in the console under Terraform management, you use a declarative import block:
import {
to = aws_s3_bucket.legacy_uploads
id = "company-legacy-uploads-prod"
}
By adding the import block, generating or writing the matching resource block (for instance, with terraform plan -generate-config-out=generated.tf), adjusting the attributes until terraform plan reports zero unintended modifications (0 to change, 0 to destroy), and running apply, you adopt the live resource into your state file cleanly and safely.
Plaintext secret exposure in state files and managing sensitive variables
One of the most dangerous misconceptions in Terraform security is the belief that:
"Marking a variable or output with sensitive = true encrypts or hides that secret inside the terraform.tfstate file." That belief is completely false!
All sensitive = true does is redact the value as (sensitive value) in terminal console output when you run terraform plan and terraform apply. Because Terraform needs to compare the current value on the next plan run, database passwords, TLS private keys, and API tokens passed into regular resource attributes are written into the terraform.tfstate JSON file in literal plaintext.
Securing Terraform state therefore requires combining four defensive controls:
- Never commit
terraform.tfstate(or.terraform/) to Git under any circumstances (always include*.tfstate*and.terraform/in.gitignore). - Encrypt the remote backend storage itself (for example, S3 with KMS
encrypt = true), and lock down IAM read permissions on the state bucket strictly to the CI deployment role and a minimal set of administrators. Anyone who has permission to readterraform.tfstateeffectively has permission to read every secret managed by that state. - Avoid passing plaintext passwords through Terraform resource attributes in the first place: use cloud-managed master password rotation (such as
manage_master_user_password = trueon AWS RDS, which stores the secret directly in AWS Secrets Manager) or have your application fetch secrets dynamically at runtime from Secrets Manager, Parameter Store, or Vault. - Take advantage of ephemeral variables and resources (
ephemeral = true,ephemeralblocks) and write-only attributes (such aspassword_wo) supported in modern Terraform (1.10+), which pass secret values to the provider API during execution without ever persisting them into the state file or plan file.
Detecting and remediating configuration drift
Configuration drift occurs when the actual live infrastructure running in the cloud diverges from the configuration declared in your .tf code and recorded in terraform.tfstate.
How does drift happen? The most common cause is an on-call engineer logging directly into the AWS or Cloudflare console at 3 a.m. during an incident to open a firewall port, scale up an instance size, or tweak a WAF rule with the mouse—and forgetting to port that change back into the Terraform repository afterward.
If drift is left unresolved, the next time someone runs terraform apply weeks later to ship an unrelated feature, Terraform sees that the manual console hotfix does not match the .tf code and silently overwrites (reverts) the hotfix, causing the exact same outage to recur.
The disciplined engineering workflow for handling drift has two steps:
- Scheduled drift detection: Run a scheduled CI job (for example, every morning) executing
terraform plan -detailed-exitcode(which returns exit code0for no changes,1for errors, and2when drift or pending changes are detected) orterraform plan -refresh-only, firing an immediate alert to Slack or your issue tracker whenever out-of-band console edits are found. - Triage and two-way remediation:
- If the out-of-band change was unauthorized or a mistake: Run
terraform applyfrom the current.tfcode to revert the cloud resource back to the desired code state. - If the out-of-band change was a legitimate emergency hotfix that must be kept: Update the
.tfcode first (and addimportblocks for any newly created resources) so the code matches the live cloud setting, runterraform planto verify that it reportsNo changes. Your infrastructure matches the configuration., and merge the update into Git.
- If the out-of-band change was unauthorized or a mistake: Run
Keyless CI/CD pipelines with GitHub Actions OIDC
In older CI/CD setups, running Terraform from GitHub Actions against AWS required creating an IAM User, generating long-lived static credentials (AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY), and pasting them into GitHub Secrets. That pattern carries a severe security flaw: static keys do not expire automatically, so if a key leaks once through a misconfigured build log or a compromised third-party GitHub Action, an attacker can access your cloud account from anywhere at any time—and rotating static keys across dozens of repositories is painful.
The modern cloud security standard eliminates static long-lived keys altogether in favor of OpenID Connect (OIDC) short-lived token federation (AssumeRoleWithWebIdentity):
- Register GitHub's OIDC provider (
https://token.actions.githubusercontent.com) as a trusted Identity Provider in AWS IAM. - Grant
permissions: { id-token: write, contents: read }to your GitHub Actions job. When the job starts, GitHub mints a cryptographically signed, short-lived OIDC JWT asserting: "This workflow is executing inside repositoryorg/repoon branchmain." - The workflow presents this JWT to AWS Security Token Service (
sts:AssumeRoleWithWebIdentity). AWS verifies the signature against the IAM Role's Trust Policy conditions and returns temporary credentials valid for only one hour or less. Zero long-lived cloud keys are stored in GitHub!
Both on the exam and in production security reviews, the most critical detail is how tightly you scope the Condition block in the IAM Role's Trust Policy:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Federated": "arn:aws:iam::123456789012:oidc-provider/token.actions.githubusercontent.com"
},
"Action": "sts:AssumeRoleWithWebIdentity",
"Condition": {
"StringEquals": {
"token.actions.githubusercontent.com:aud": "sts.amazonaws.com",
"token.actions.githubusercontent.com:sub": "repo:acme-corp/infra-live:ref:refs/heads/main"
}
}
}
]
}
If you omit the sub (Subject) condition check, any GitHub repository belonging to any stranger in the world can assume your AWS IAM role! Even if you specify your repository name, using a loose wildcard such as "repo:acme-corp/infra-live:*" on a production apply role allows any developer who opens an unreviewed pull request from a feature branch to modify .github/workflows/deploy.yml and seize full production admin privileges. Therefore, you must enforce strict role separation between pull-request plan runs and post-merge main apply runs.
| Pipeline stage | Trigger event | IAM Role and sub claim condition |
Permission scope |
|---|---|---|---|
PR validation (plan) |
Pull request opened or updated | terraform-planner role (scoped to pull_request or a dedicated plan environment) |
Read-only permissions + state lock/read access; posts plan output as a PR comment |
Production deployment (apply) |
Merged into main after code review approval |
terraform-applier role (strictly locked to ref:refs/heads/main or a reviewer-protected environment:production) |
Write permissions to create, modify, and delete cloud resources and update the state file |
Going deeper
Attack surfaces in PR-driven infrastructure automation (the Atlantis pattern) and plan artifact integrity
Automating "run terraform plan automatically when a pull request opens, post the diff as a comment, and run apply upon approval" is enormously convenient, but from an attacker's perspective it introduces a subtle and dangerous attack surface.
terraform plan is not a passive syntax linter. During terraform plan, Terraform downloads and executes external modules and provider binaries, evaluates data "external" blocks and local expressions, and loads the entire remote state file into memory.
What happens if your CI pipeline automatically runs terraform plan with production state read credentials on an unreviewed pull request submitted from a fork or an untrusted feature branch? An attacker can add a few lines of HCL using an external data source or HTTP provider that reads a secret out of the loaded state and exfiltrates it to an external webhook—stealing every production secret during terraform plan before anyone ever approves or runs terraform apply!
A hardened PR automation pipeline therefore relies on three defensive layers:
- Never automatically trigger workflows carrying cloud credentials on pull requests originating from external repository forks.
- Restrict the PR
planrole to least-privilege read access without access to unrelated application secrets, and require explicit infrastructure maintainer approval (via GitHub CODEOWNERS and protected GitHub Environments) before running workflows on PRs that modify.github/workflows/*, provider plugins, or backend configurations. - To prevent Time-of-Check to Time-of-Use (TOCTOU) race conditions between the moment a
planis reviewed and the momentapplyexecutes, save the exact reviewed plan binary usingterraform plan -out=tfplan.binary, verify its hash, and runterraform apply tfplan.binary—while remembering thattfplan.binarycontains the same plaintext secrets asterraform.tfstateand must never be uploaded to publicly readable build artifact storage.
Reducing state file blast radius at scale
When a team first adopts Terraform, it is tempting to put the VPC network, RDS databases, compute clusters, and Cloudflare DNS/WAF rules inside a single directory backed by one monolithic terraform.tfstate file.
Once the resource count grows into the hundreds, however, a monolithic state file creates three painful bottlenecks:
- Slow
plantimes and API rate limiting: Everyterraform planrefreshes hundreds of cloud resources via provider APIs, stretching run times past 10 minutes and hitting cloud API throttling limits. - Lock contention: While a frontend engineer runs a
planto tweak a single CDN cache rule, the database team is blocked by the state lock from making urgent database parameter changes. - Massive blast radius: A mistake in an application-layer resource block risks destabilizing the core VPC or production database sitting in the same state file.
The architectural remedy is State Segmentation: splitting directories and state files along rate of change, lifecycle, and team ownership boundaries. For example, separate your infrastructure into (1) 01-network (VPCs, subnets, and ZTNA connectors that rarely change), (2) 02-data (RDS databases, S3 buckets, and KMS keys where data preservation is paramount), and (3) 03-app-edge (Cloudflare rules, application routing, and Workers that change multiple times a day). When a higher layer needs to reference the VPC ID or subnet IDs created by 01-network, pass them via read-only terraform_remote_state data sources, SSM Parameter Store entries, or tag-filtered data lookups—confining the blast radius of any single apply strictly to its own layer.
Workplace scenarios
Solbit Mobility: How renaming a resource nearly deleted the production DB, and how moved and prevent_destroy saved it
At Solbit Mobility, which operates an AI logistics dispatch platform, a DevOps engineer took on a cleanup task to refactor legacy Terraform code written hurriedly during the company's early days. The engineer opened a pull request that renamed aws_db_instance.db1 in the root module to the clearer name aws_db_instance.logistics_primary and increased its allocated storage from 100 GB to 200 GB.
At the bottom of the automated CI comment, the terraform plan summary read: Plan: 1 to add, 0 to change, 1 to destroy. Focusing only on the storage bump in the PR description, a reviewer clicked Approve without reading the full resource diff. Fortunately, right before triggering the production apply, a senior engineer scrolled through the plan log and froze: Terraform had interpreted the identifier rename as "permanently destroy (-) the existing production database (db1) and create (+) a brand-new empty database (logistics_primary)"! Had the apply proceeded, hundreds of thousands of live dispatch records would have been wiped out.
The team halted the deployment immediately and instituted three permanent safeguards:
- Zero-downtime refactoring with
movedblocks: They addedmoved { from = aws_db_instance.db1, to = aws_db_instance.logistics_primary }to the.tffile. Re-runningterraform planeliminated the1 to destroy, 1 to addactions, replacing them with a safe state-address move alongside a single in-place storage expansion (~,1 to change). - Mandatory
prevent_destroyon stateful resources: They addedlifecycle { prevent_destroy = true }to every production database, S3 bucket, and KMS key resource. Now, if any future PR accidentally renames a stateful resource or modifies an immutable attribute that forces replacement (-/+),terraform planfails immediately with a hard error. - Automated PR destruction warnings: They updated the CI bot so that any plan containing
to destroyormust be replaced(-/+) renders a high-visibility red warning banner and requires explicit sign-off from the infrastructure lead.
Daon Fintech: Eliminating departed employees' static AWS keys with GitHub OIDC and reconciling drift
During an external security audit, financial data analytics firm Daon Fintech was cited for two major findings. First, the AWS_SECRET_ACCESS_KEY stored in GitHub Actions Secrets belonged to an IAM user created two years earlier by an engineer who had since left the company, and the key had never been rotated. Second, during a DDoS incident the previous quarter, the security team had manually added 12 WAF and security group rules directly in the AWS console; because those rules were missing from the Terraform repository, the engineering team had been afraid to run terraform apply for two months lest they wipe out the security rules.
Over the course of one week, the infrastructure team resolved both findings in sequence.
| Initiative | Execution steps | Verified outcome |
|---|---|---|
| 1. Reconcile configuration drift | Ran terraform plan to inventory all 12 manual console changes → reviewed with security to codify 10 permanent rules into .tf (using import blocks for standalone resources) and remove 2 obsolete temporary rules |
terraform plan reached No changes; scheduled a daily 09:00 workflow running plan -detailed-exitcode to catch future drift immediately |
| 2. Harden S3 state locking | Migrated legacy state into a versioned, KMS-encrypted S3 backend with use_lockfile = true, restricting state read access to least-privilege roles |
Verified that concurrent runs immediately wait or abort via .tflock without corrupting state |
| 3. Migrate to keyless GitHub OIDC | Registered the GitHub OIDC provider in AWS IAM → created separate roles for PR read-only plan runs and ref:refs/heads/main production apply runs → deactivated and deleted the legacy static IAM user keys |
Zero long-lived cloud secret keys remain stored in GitHub Secrets; all deployments authenticate via 1-hour ephemeral OIDC tokens |
Common mistakes
- "Because I added
sensitive = trueto the database password variable, it is safe to uploadterraform.tfstateto a shared folder or Git repository." This is one of the most dangerous misconceptions in Infrastructure as Code. Settingsensitive = trueonly masks terminal stdout console output duringplanandapply; the literal secret value is still written in unencrypted plaintext insideterraform.tfstate. Never commit state files to Git, and always protect remote state backends with KMS encryption and strict least-privilege IAM access controls. - "The
-/+symbol interraform planjust means a minor configuration update, so it is safe to runapply." An in-place modification is represented by the~symbol. The-/+symbol means Replace (destroy the existing cloud resource completely and create a brand-new replacement from scratch), which causes catastrophic data loss if executed against a stateful production database or storage bucket. - "To rename a resource in Terraform, I just edit the resource name in the
.tffile and runapply." If you only change the resource identifier in your.tfcode, Terraform assumes you want to delete (-) the old resource and provision (+) a brand-new one. You must include a declarativemovedblock so Terraform updates the address mapping insideterraform.tfstatewithout touching the live cloud resource. - "Creating a CLI workspace (
terraform workspace new prod) completely isolates production from development." CLI workspaces share the exact same.tfsource files and backend storage configuration, making it dangerously easy for an operator to run a destructive command while sitting in the wrong active workspace. Production environments should be isolated using dedicated directories, separate backend state files, and separate cloud accounts/IAM roles. - "When configuring GitHub Actions OIDC with AWS IAM, checking
aud: sts.amazonaws.comin the trust policyConditionis sufficient." If you fail to restrict thesub(Subject) claim in the IAM trust policy to your exact GitHub organization, repository, and production branch or environment, workflows running in any other GitHub repository on earth can assume your AWS IAM role (AssumeRoleWithWebIdentity). - "If I make an urgent manual change in the cloud console during an incident, Terraform will automatically notice and preserve it on the next deploy." Terraform treats your
.tfcode as the single source of truth. Any manual console hotfix (configuration drift) that is not ported back into your.tfcodebase will be overwritten and reverted the next timeterraform applyruns.
Summary at a glance
| Core topic | Key takeaway for the exam and production |
|---|---|
| Declarative model & State | .tf code declares the desired end state, while terraform.tfstate maps logical code addresses 1:1 to physical cloud resource identifiers |
| Remote backend & locking | Store state in a versioned, KMS-encrypted remote backend (such as S3) and enable use_lockfile = true (or DynamoDB) to block concurrent runs |
Reading plan symbols |
+ create, - destroy, ~ update in-place, and -/+ destroy and recreate (Replace — causes catastrophic data loss on stateful resources) |
| Deletion protection | Add lifecycle { prevent_destroy = true } to production DBs, S3 buckets, and KMS keys so accidental deletions or replacements abort during plan |
| Zero-downtime refactoring | Use moved blocks when renaming resources or moving them into child modules, and use import blocks to bring manually created resources under code management |
| State file security | sensitive = true only redacts terminal output. Because state stores plaintext values, block Git commits, encrypt the backend with KMS, and use ephemeral/write-only attributes |
| Environment isolation | Isolate production using dedicated directories, separate state files, and separate cloud accounts/IAM roles rather than sharing CLI workspaces |
| Handling drift | Detect drift on a schedule with plan -detailed-exitcode; revert unauthorized edits via apply, or codify valid hotfixes into .tf until plan reports No changes |
| Keyless CI/CD with OIDC | Replace static access keys with AssumeRoleWithWebIdentity, verifying both aud and a tightly scoped sub (repo + main branch/environment) in the IAM trust policy |
Check the official docs
- HashiCorp Terraform — Purpose of Terraform State: Why state files are required and how resource metadata mapping works.
- HashiCorp Terraform — S3 Backend Configuration: Configuring S3 remote backend encryption and state locking with
use_lockfileor DynamoDB. - HashiCorp Terraform — The lifecycle Meta-Argument: How
prevent_destroy,create_before_destroy, andignore_changesbehave. - HashiCorp Terraform — Refactoring (
moved) &importBlocks: Moving resource addresses without downtime and declarative resource imports. - HashiCorp Terraform — Sensitive Data in State & Ephemeral Values: Why plaintext secrets appear in state files and how to protect or omit them.
- GitHub Docs — Configuring OpenID Connect in Amazon Web Services: Issuing OIDC tokens in GitHub Actions and scoping IAM trust policies (
subandaud).
Self-check
2-1. Explain what can go wrong when two teammates run terraform apply from their laptops at the same time against the same infrastructure, and state the two features (one for concurrency control and one for recovery) that must be enabled on an AWS S3 backend to prevent and recover from this problem.
2-2. Explain the difference between the ~ symbol and the -/+ symbol in terraform plan output, and write the HCL configuration block that should be added inside a production database resource to prevent -/+ or - from being executed accidentally.
2-3. Describe what happens if you rename a live resource in your .tf code from aws_instance.app to aws_instance.api_server and immediately run apply, and write the declarative HCL block required to rename the resource safely without service interruption.
2-4. Explain why committing terraform.tfstate to a public Git repository leaks secret values even when you have marked your input variables with sensitive = true, and list at least two security controls to prevent secret exposure through state files.
2-5. During a late-night outage, an on-call engineer manually added a port rule to an AWS security group in the web console (a hotfix), and the team decided to keep that rule permanently. Explain in order how you should update your Terraform code and what you must verify in terraform plan so the rule is not wiped out on the next deployment.
2-6. Design task: You want to run Terraform from GitHub Actions without storing long-lived AWS access keys. Explain why you must separate the IAM role used by (1) workflows triggered on pull requests from the IAM role used by (2) workflows triggered upon merging into main, and name the two OIDC claims that must be verified inside the Condition block of the production apply role's trust policy.
Hands-on tasks
Exercise
Audit terraform plan risks and write zero-downtime refactoring HCL (45 minutes)
- Assume your root module currently declares a live production database
resource "aws_db_instance" "legacy_db"and an S3 bucketresource "aws_s3_bucket" "legacy_bucket". - Write two
movedblocks that rename and relocate these resources toaws_db_instance.prod_primaryandmodule.storage.aws_s3_bucket.prod_assetswithout destroying or recreating (-/+) either resource. - Add the
lifecyclerule that blocks accidental deletion to both resource definitions, and write thebackend "s3"block enabling encryption (encrypt = true) and native S3 lockfiles (use_lockfile = true). - Write a three-item review checklist detailing what to inspect if a teammate's pull request outputs
aws_db_instance.prod_primary must be replaced (-/+).
Completion criteria: Both moved blocks contain exact from/to addresses, prevent_destroy = true and S3 locking are configured properly, and the checklist clearly identifies how to spot immutable attributes that force resource replacement (-/+).
Exercise
Design least-privilege GitHub Actions OIDC IAM Trust Policies (45 minutes)
- Write two separate AWS IAM Role Trust Policy JSON documents (
tf-pr-plan-roleandtf-main-apply-role) for an internal repository namedtrappist-org/cloud-infra. - Set
Actiontosts:AssumeRoleWithWebIdentityin both policies and enforce a condition verifying thataudequalssts.amazonaws.com. - Restrict the
subcondition ontf-main-apply-roleso it matches only runs on themainbranch (repo:trappist-org/cloud-infra:ref:refs/heads/main) or the protectedproductionenvironment (repo:trappist-org/cloud-infra:environment:production). - Write three sentences explaining the attack scenario that becomes possible if
tf-main-apply-roleuses a wildcard condition like"repo:trappist-org/cloud-infra:*".
Completion criteria: Both IAM trust policy JSON documents include exact aud and sub conditions, and the write-up clearly explains how an attacker or unreviewed feature-branch PR could hijack production credentials if wildcards are used.