Terraform Best Practices

Remote state outputs often begin as convenience and quietly become a platform API. When a network stack renames or reshapes an output, every consuming app stack can break in CI or attempt unexpected replacement. The damage is organizational: teams lose trust in infrastructure automation.

Local state files are the single biggest source of Terraform disasters in production environments. When state lives on a developer's laptop or in a CI runner's ephemeral filesystem, it cannot be shared across team members, has no locking mechanism to prevent concurrent modifications, and lacks versioning for rollback. A single terraform apply from a stale local state can destroy resources that another team member created hours ago because the local copy does not reflect the current infrastructure reality. In regulated industries, auditors specifically look for centralized state management as evidence of change control. Organizations that keep local state inevitably experience the nightmare scenario: an engineer runs apply from an outdated state file and Terraform attempts to reconcile by destroying production resources it does not know about. The S3 backend with DynamoDB locking provides atomic state operations that prevent the race conditions that corrupt state files during concurrent CI pipeline runs.

Terraform providers are released independently from Terraform core and can introduce breaking changes in any minor or major version. The AWS provider alone has had 47 breaking changes between versions 4.x and 5.x, including renamed attributes, changed default values for security groups, and altered lifecycle behaviors for RDS instances. Without pinned versions, running terraform init on a Monday might pull provider version 5.30.0 while the same command on Friday pulls 5.32.0 with a completely different security group default behavior. This creates non-deterministic infrastructure where the same configuration produces different results depending on when you run it. In production environments, this unpredictability is catastrophic: a routine terraform apply that was supposed to add a tag can silently modify security group rules because the provider changed its default handling. Version pinning ensures that every team member and every CI pipeline uses the exact same provider behavior, making infrastructure changes auditable, reproducible, and predictable across all environments.

Without input validation, Terraform will happily accept any string for a CIDR block, any number for an instance count, or any value for an environment name — only to fail 8 minutes into a 15-minute apply when AWS rejects the malformed input. By that point, half your resources may already be provisioned in an inconsistent state that requires manual cleanup. Variable validation blocks shift error detection to the earliest possible moment: before Terraform even generates a plan. This is especially critical in self-service platforms where application teams provide variable values through Terraform Cloud workspaces or CI pipeline parameters without deep infrastructure knowledge. A developer might accidentally set instance_count to 500 instead of 5, or paste a subnet CIDR that overlaps with an existing VPC. Catching these errors before plan prevents costly mistakes, reduces CI pipeline time wasted on doomed applies, and provides clear human-readable error messages that tell the operator exactly what went wrong and how to fix it rather than cryptic AWS API errors buried in 200 lines of apply output.

A monolithic state file that contains your VPC, EKS cluster, RDS databases, application deployments, and DNS records creates a catastrophic blast radius where a single misconfigured variable or accidental terraform destroy can wipe out your entire infrastructure in one operation. With 800+ resources in a single state, terraform plan takes 12-15 minutes as it refreshes every resource, making iterative development painfully slow and tying up DynamoDB locks for extended periods that block other teams. When state is split by blast radius, a bad apply to the application layer cannot possibly affect the VPC or database layer because those resources simply do not exist in the same state file. This isolation also enables independent team ownership: the network team manages VPC state, the platform team owns EKS state, and application teams manage their own service deployments without needing broad infrastructure permissions. State splitting reduces lock contention in CI pipelines by 80% because teams are no longer queuing behind each other for the same monolithic state lock.

Renaming a Terraform resource address — whether refactoring a module structure, changing a resource name for clarity, or migrating from count to for_each — traditionally causes Terraform to plan a destroy-then-create operation. For stateless resources like security group rules this is merely disruptive, but for stateful resources like RDS instances, EBS volumes, or S3 buckets with data, a destroy-and-recreate means data loss and downtime. Before Terraform 1.1 introduced moved blocks, teams had to use terraform state mv commands as manual operational steps that were error-prone, could not be code-reviewed, and left no audit trail in the configuration itself. The moved block solves this by declaratively recording the resource address change in your configuration, allowing Terraform to plan the rename as a no-op state update rather than a destructive operation. This makes refactoring safe, reviewable, and automated — the moved block appears in the pull request diff so reviewers can verify the rename is intentional. After one successful apply the moved block can be removed, but many teams keep them as documentation of the resource's history.

Automated terraform apply without human review is the infrastructure equivalent of deploying code directly to production without code review. Even experienced engineers produce plans that accidentally replace load balancers, modify security groups, or trigger RDS instance replacements due to parameter changes they did not realize were destructive. A terraform plan output is the last line of defense before irreversible infrastructure changes — but it is only useful if a human actually reads it and confirms the changes match intent. In a 2025 survey of infrastructure incidents, 34% of production Terraform outages were caused by engineers approving plans they did not fully read because the workflow made it too easy to auto-approve. The plan-then-approve pattern creates a mandatory pause that forces engineers to verify: Are any resources being destroyed? Are replacements expected? Does the number of changes match what I intended? This gate is especially critical for production where a single unexpected replacement of an ALB or NAT gateway can cause customer-facing downtime measured in minutes to hours.

Hardcoded ARNs, account IDs, VPC IDs, and subnet IDs scattered throughout Terraform configurations create brittle, environment-specific code that breaks the moment you deploy to a new region, a new AWS account, or when AWS regenerates resource IDs during maintenance operations. A hardcoded string like subnet-0a1b2c3d4e is meaningless to reviewers — they cannot verify if it is the correct subnet without logging into the console and checking. Hardcoded values also prevent code reuse across environments because every deployment requires finding and replacing dozens of IDs. Data sources solve this by querying AWS at plan time to resolve the current, correct values based on meaningful filters like tags or names. This makes configurations self-documenting (the filter criteria explain what you are looking for), environment-portable (the same code works in dev and prod because it queries dynamically), and drift-resistant (if a resource is recreated with a new ID, the data source picks up the new value automatically). In multi-team organizations, data sources create clean contracts between teams: the network team tags their subnets, and application teams reference them by tag rather than by opaque ID.

Without consistent tagging, your AWS bill is a black box where $47,000 per month disappears into an undifferentiated mass of EC2 instances, NAT gateway charges, and data transfer fees that no one can attribute to specific teams, services, or environments. When the CFO asks why infrastructure costs increased 40% this quarter, the answer should take minutes to find through Cost Explorer tag-based filtering — not weeks of forensic investigation. Tags are also critical for security (identifying untagged rogue resources), compliance (proving which resources belong to regulated workloads), and operations (knowing which team to page when a resource is unhealthy). AWS Cost Allocation Tags only work if applied consistently to every resource at creation time — retroactively tagging hundreds of resources after a cost audit is painful and error-prone. Terraform's default_tags feature at the provider level ensures that every resource automatically receives baseline tags without engineers remembering to add them manually to each resource block, eliminating the human error that creates cost attribution gaps.