Terraform Production Issues

All Terraform operations across every pipeline return 'Error acquiring the state lock' and no deployments can proceed. The lock has been held for over 4 hours with no active Terraform process running.

Production RDS database, ElastiCache cluster, and 14 EC2 instances were destroyed during what was supposed to be a staging environment teardown. Customer-facing services went down immediately and data loss was confirmed.

Terraform state shows duplicate resource entries, missing attributes, and lineage mismatch errors. Plans show resources that already exist as needing creation, and existing resources are not recognized.

After upgrading the AWS provider from v4.67.0 to v5.31.0, terraform plan shows 200+ resources marked for destruction and recreation. Critical production resources including RDS instances, EKS clusters, and ALBs are all flagged for replacement.

Terraform operations intermittently fail with lock acquisition timeouts during peak CI hours (9 AM - 11 AM UTC). Some pipelines succeed after retry but most fail, creating a backlog of 30+ queued deployments.

After upgrading the AWS provider from v5.30.0 to v5.31.0, terraform plan shows 53 unexpected in-place updates across resources that have not been modified. All changes are cosmetic attribute reorderings and default value additions, but they block the CI pipeline which enforces zero-diff plans.

Terraform apply failed midway through creating a new ECS service deployment. Some resources were created in AWS but not recorded in state. Subsequent plans attempt to create duplicate resources, and manual AWS console inspection shows orphaned load balancer target groups and ECS task definitions.

Production Terraform state file was written to an S3 bucket in the development AWS account (account 987654321098) instead of the production account (account 123456789012). Production infrastructure is unmanaged and development team discovers unknown state files in their bucket.

A production plan proposed replacing a shared security group immediately after an import cleanup.

Terraform plan shows phantom resources that do not exist in AWS, existing resources marked for recreation, and serial mismatch errors. Two deployments triggered within 8 seconds of each other for the payments-vpc workspace both reported success, but subsequent plans are inconsistent and show 23 resources needing recreation.

After a routine CI pipeline run on Monday morning, all new ECS tasks in the payments-api cluster failed to start with connection timeouts to the RDS database. Existing tasks continued working. No Terraform code had changed — the only difference was a provider version bump merged the previous Friday.

A production EC2 instance running the order-processing service was terminated and replaced during a terraform apply. The replacement instance came up with a fresh root volume, losing 6 hours of local SQS dead-letter queue processing state, active network connections, and the instance's IAM instance profile association took 4 minutes to propagate, causing a processing gap.

A terraform apply deleted 3 custom security group rules, removed a manually-added NAT gateway route, and reverted an RDS parameter group change that had been keeping the payments database stable for 4 months. The payments API immediately began experiencing connection pooling errors and 2x latency increase.

Every Terraform operation across all CI pipelines for the prod-eks workspace returns 'Error acquiring the state lock' and no deployments can proceed. The lock has been held for 7 hours with no active Terraform process. On-call pages are firing because a critical EKS node group scaling change cannot be deployed during a traffic spike.

Production PostgreSQL RDS instance (db.r6g.4xlarge, 2.3TB data) was destroyed during what was intended as a staging environment teardown. All customer-facing services returned HTTP 500 errors. Data loss confirmed for the 22 minutes between the last automated backup and the destroy event.

Running terraform plan against the platform stack takes over 30 minutes and never completes, consuming 100% CPU on the CI runner. The process must be manually killed. When run with TF_LOG=DEBUG, the log shows repeated evaluation cycles between the networking and compute modules, each iteration generating identical intermediate values without converging.

A security scan flagged the production RDS master password appearing in plaintext in 47 GitHub Actions workflow logs spanning the past 3 months. The password was visible in terraform plan and terraform output command outputs that were streamed to CI logs without redaction.