Production Issues

Security patch window stalled at 38% complete. Cluster autoscaler reported nodes could not be drained because multiple pods violated disruption budgets.

PagerDuty alert: 'payments-api pod restarts > 10 in 5m' at 02:14 UTC. Grafana showed memory usage hitting 512Mi limit, OOMKill events every 90 seconds, and HPA scaling from 4 to maxReplicas 20 within 3 minutes. Node CPU utilization spiked to 94% across all three nodes as 20 pods competed for CPU during startup. p99 latency jumped from 180ms to 12,400ms. Customer-facing error rate hit 38%.

Deployment rollout stuck at 50%. kubectl rollout status shows 'Waiting for deployment user-auth-service rollout to finish: 2 of 4 updated replicas are available.' PagerDuty alert: 'user-auth-service CrashLoopBackOff count > 5 in namespace identity.' New pods cycling through Running for 30s then Terminated with exit code 143 (SIGTERM from kubelet).

Multiple teams reporting intermittent 5xx errors across unrelated services starting at 14:22 UTC. Datadog alert: 'DNS resolution failure rate > 5% cluster-wide.' Services returning errors like 'connection refused' to downstream dependencies. Not all requests fail, creating a confusing pattern where 30% of requests succeed and 70% timeout or return 503. CoreDNS pods showing 2-4 restarts in the last hour.

All kubectl apply and helm upgrade commands hang for 30 seconds then fail. No new pods are being created across any namespace. CI/CD pipelines timing out on deployment steps. Existing running pods are unaffected. PagerDuty alert: 'Deployment rollout failure rate 100% across all namespaces for 12 minutes.'

Prometheus alert: 'KubeNodeDiskPressure firing on node ip-10-42-8-22 for 8 minutes.' Pods being evicted from the node with status 'Evicted' and reason 'The node was low on resource: ephemeral-storage.' Payments-api pods evicted first despite being tier-1, causing transaction failures. kubectl describe node shows 'DiskPressure: True'.

Grafana dashboard showing inventory-service replica count oscillating between 3 and 12 every 4-5 minutes in a sawtooth pattern. HPA events alternating between 'New size: 12; reason: cpu resource utilization above target' and 'New size: 3; reason: All metrics below target' in rapid succession. Customer complaints about intermittent slowness coinciding with scale-down phases.

During a Kubernetes version upgrade from 1.29 to 1.30, all kubectl commands started returning 'Error from server: etcdserver: request timed out' or 'Error from server: etcdserver: leader changed'. Helm deployments across all namespaces failing. ArgoCD showing 'ComparisonError' on all applications. Prometheus alert: 'etcd_server_leader_changes_seen_total increased by 47 in 5 minutes.' API server request latency p99 exceeding 30 seconds.

After migrating notification-service from the shared namespace to a dedicated notifications namespace, the service returns 504 Gateway Timeout on all requests. Health checks pass (pod is Running and Ready) but no traffic reaches the application. Other services in the old namespace report connection timeout errors when calling notification-service. PagerDuty alert: 'notification-service error rate 100% for 8 minutes.'