Best Practices

A blanket `minAvailable: 100%` feels safe until maintenance or autoscaler consolidation needs to evict a pod. At that point the cluster cannot move workloads, node repairs stall, and operators are tempted to delete safety controls during an incident. The failure gets worse at scale because many teams copy the same template into services with different replica counts and traffic tolerance.

When requests and limits diverge, Kubernetes assigns the pod a Burstable QoS class. Under node CPU pressure, the kubelet throttles Burstable pods first using CFS bandwidth quotas, which introduces unpredictable latency spikes. In a payments-api deployment processing card authorizations, a 200ms throttle window translates directly into declined transactions and customer-facing errors. The team at a fintech company lost $47K in failed authorizations during a Black Friday spike because their checkout-worker pods were Burstable and got throttled when a noisy neighbor batch job consumed spare CPU. Guaranteed QoS pods are the last to be evicted during node pressure and receive stable scheduling priority. Without matching requests and limits, you are silently accepting that your most critical workloads will degrade first during resource contention events that happen exactly when traffic matters most.

A single privileged container is a complete node compromise waiting to happen. When a container runs with privileged: true or hostPID: true, it can read all secrets from the kubelet, escape to the host filesystem, and pivot laterally to every pod on that node. In 2025, a supply-chain attack on a logging sidecar image gained code execution inside a container that happened to have hostNetwork: true. The attacker used the host network namespace to intercept traffic from 23 other pods, including one that held database credentials in environment variables. Pod Security Standards (PSS) at the restricted level prevent this entire class of attack by blocking privileged mode, host namespaces, hostPath mounts, and dangerous capabilities. Without PSS enforcement, you rely entirely on developers remembering not to add dangerous security contexts, which fails the moment someone copies a StackOverflow snippet that includes privileged: true to fix a permission error.

Without NetworkPolicy, every pod can talk to every other pod in the cluster on every port. This means a compromised frontend pod can directly query the database, a hijacked logging agent can reach the secrets manager, and lateral movement after any breach is trivial. In a real incident at a SaaS company, an SSRF vulnerability in the user-profile-service allowed an attacker to scan the internal network and discover an unpatched Redis instance in the cache namespace that had no authentication because it was supposed to be internal-only. The attacker extracted session tokens for 140K users. A default-deny policy would have blocked the user-profile-service from reaching the cache namespace entirely, containing the blast radius to a single service. Kubernetes does not apply any NetworkPolicy by default, so the moment you create a cluster, all pod-to-pod traffic is permitted unless you explicitly restrict it.

Without a PodDisruptionBudget, Kubernetes treats voluntary disruptions like node drains, cluster upgrades, and autoscaler scale-downs as unrestricted operations that can terminate all replicas simultaneously. During a routine EKS upgrade, a platform team discovered that their order-processing-service lost all three replicas when the autoscaler drained a node for consolidation. The PDB was missing, so the eviction API returned success immediately for all three pods. Customers saw a 4-minute outage during which 2,300 orders failed with 503 errors. The team had assumed that rolling update strategy alone would protect them, but rolling updates only govern Deployment rollouts, not voluntary evictions triggered by external controllers. PDBs are the only mechanism that tells the eviction API to respect application availability during disruptions that originate outside the Deployment controller. Every production Deployment, StatefulSet, and ReplicaSet serving traffic needs one.

Using only a liveness probe or using identical settings for all probes creates a dangerous failure mode where Kubernetes kills pods that are still initializing or restarts pods that are temporarily slow but recoverable. A Java Spring Boot application at a logistics company took 45 seconds to initialize its connection pool and warm its JIT compiler. The liveness probe had initialDelaySeconds: 30, which was not enough. During every rollout, pods were killed at second 31, restarted, killed again, creating a CrashLoopBackOff that cascaded across the rolling update. The team added more replicas thinking it was a capacity problem. The real issue was conflating startup health with runtime health. Startup probes exist specifically to handle slow-starting containers without compromising the fast failure detection that liveness probes provide during steady state. Without a startup probe, you must set a large initialDelaySeconds on liveness, which means genuinely crashed pods take that long to be detected.

Without consistent labels, you cannot correlate metrics, logs, and traces to the owning team, cannot filter Prometheus queries by environment, cannot build Grafana dashboards that aggregate by service tier, and cannot enforce NetworkPolicies or PDBs that use label selectors. A platform team at an e-commerce company discovered during a P1 incident that 40% of their pods had no team label. When the payments service degraded, they could not quickly identify which team owned the upstream dependency causing the issue because kubectl get pods showed 200 pods with inconsistent naming. The mean time to resolution doubled from 12 minutes to 28 minutes because three teams had to be paged to determine ownership. Labels are the foundation of every Kubernetes selector mechanism: Services route traffic by label, HPA targets by label, PDBs protect by label, and monitoring tools aggregate by label. Inconsistent labeling makes every operational tool partially blind.

Every package in a container image is attack surface. A standard ubuntu:22.04 base image contains 300+ packages including curl, wget, bash, apt, and network utilities that attackers use for reconnaissance and data exfiltration after gaining code execution. In 2025, a vulnerability in a libxml2 version shipped in a debian:bullseye base image gave attackers remote code execution in a notification-service that never used XML parsing directly. The library existed in the image purely because the base image included it. With a distroless image, that library would not have been present and the CVE would have been non-exploitable. Distroless images contain only your application binary and its runtime dependencies, reducing the CVE surface by 70-90% compared to full OS images. They also have no shell, so even if an attacker gains code execution, they cannot spawn an interactive session, install tools, or easily pivot. Additionally, smaller images mean faster pulls during scale-out events, reducing HPA response time from 45 seconds to 12 seconds in cold-start scenarios.

In a multi-tenant cluster where multiple teams deploy to separate namespaces, a single team can accidentally or intentionally consume all available cluster resources if no quotas exist. A data engineering team deployed a Spark job that requested 64 CPU cores and 256Gi of memory across 40 pods in their analytics namespace. Within minutes, the cluster autoscaler had provisioned 12 new nodes. Meanwhile, the payments team's HPA triggered a scale-out that could not schedule new pods because the scheduler saw no available capacity on existing nodes and the autoscaler hit its maxNodes limit. The payments service degraded for 18 minutes during peak checkout traffic. ResourceQuota caps total resource consumption per namespace, ensuring no team can starve others. LimitRange sets defaults and maximums per container, catching the developer who forgets to set resource requests entirely and deploys a pod that either gets BestEffort QoS or requests absurd amounts. Together they implement resource isolation without requiring separate clusters per team, saving significant infrastructure cost.