Jenkins

A production deployment completed successfully but shipped the previous commit. The build log showed tests ran on the new SHA while deploy copied an older artifact.

The Jenkins controller becomes unresponsive during peak hours (9-11 AM) when multiple teams trigger builds simultaneously. The Jenkins UI returns HTTP 503 errors, in-progress builds hang indefinitely, and the Jenkins process consumes 100% of available heap. The JVM eventually crashes with an OutOfMemoryError, requiring a manual restart that takes 15-20 minutes including job queue recovery.

Jenkins pipelines begin failing intermittently with 'Too many open files' errors after the controller has been running for 5-7 days without restart. The errors appear in random pipelines and affect different stages — sometimes during Git checkout, sometimes during artifact archival, sometimes during test report parsing. Restarting the Jenkins controller resolves the issue for another 5-7 days.

A security scan of Jenkins build logs revealed that production database connection strings, API keys, and cloud provider credentials were visible in plaintext in the console output of multiple pipelines. The credentials appeared in curl command outputs, docker build logs, environment variable dumps from debugging steps, and error messages from failed deployment scripts.

After a routine Jenkins plugin update performed on a Friday afternoon, all 200+ pipeline builds began failing on Monday morning. Every pipeline failed at the Git checkout stage with ClassNotFoundException and NoSuchMethodError exceptions. The Jenkins UI was functional but no builds could complete. Rollback attempts were complicated because the plugin manager had updated 35 plugins simultaneously as a batch.

The QA team reported that a bug fix merged to main was not actually fixed — the test that verified the fix passed in the Jenkins pipeline but the bug persisted in the deployed application. Investigation revealed that the test was passing because it was running against compiled artifacts from a previous build of a different branch, not the current commit.

The Jenkins controller server experienced a hardware failure (failed SSD) at 2 PM on a Wednesday, taking down the entire CI/CD platform. No builds could run, no deployments could proceed, and the deployment pipeline for a critical production hotfix was blocked. The team spent 4 hours provisioning a new server and restoring Jenkins from the most recent backup, which was 18 hours old, resulting in lost build history and configuration changes made during that window.

Pipelines become risky when stages assume a shared mutable workspace. Parallel stages, retries, and different agents can leave stale or missing files. The deployment may succeed operationally while releasing the wrong build output. Without following this practice, teams typically discover the problem during a production incident. The cost of fixing issues reactively is 10-100x higher than preventing them proactively. This becomes especially dangerous at scale when multiple teams depend on the same infrastructure, because one team's shortcut becomes another team's outage. Organizations that skip this practice often find themselves in a cycle of firefighting instead of building. The pattern is predictable: it works fine in development, survives staging, and fails spectacularly in production under real traffic and real failure conditions.