Apache Airflow

Symptom

Scheduler and worker processes across multiple, independent Airflow installations start failing at the same time; tasks stop being picked up, scheduler logs fill with permission-denied errors, and the UI shows tasks stuck in "scheduled"/"queued" indefinitely.

Error Message

PermissionError: [Errno 13] Permission denied: '/opt/airflow/logs/scheduler/latest'

Root Cause

The logs/ directory ownership was changed to root:root instead of the expected airflow:root (for example by a base-image update or a security-hardening automation run applied broadly), so the airflow user's scheduler and worker processes can no longer write task logs or the scheduler's own log files and begin crash-looping. A real incident matching this exact pattern affected 5 Airflow installations across 4 AWS accounts on the same day, reported in apache/airflow discussion #56417 and issue #56142 ("Changed ownership of the logs/ folder causing outage").

Diagnosis Steps

1. 1.Check scheduler and worker logs (via kubectl logs / docker logs, since the app-level logs directory itself may be unwritable) for PermissionError entries
2. 2.ls -la on the logs directory to compare current ownership against the expected airflow:root
3. 3.Review what changed recently: base image bump, config management run, security hardening script, host-level chown
4. 4.Compare timing across environments — near-simultaneous failures across independent installs points to a shared automation run, not app-level config drift

Solution

Restore correct ownership on the logs directory (chown -R airflow:root, matching the image's expected UID/GID) and restart the scheduler and worker processes. Confirm task logs are writable again before declaring the incident resolved.

Commands

ls -la /opt/airflow/logs

chown -R airflow:root /opt/airflow/logs

kubectl logs deploy/airflow-scheduler --tail=200

Symptom

As the number and complexity of DAGs grows, tasks intermittently fail immediately without running any task logic, and the failure rate correlates with which DAG files are large or slow to parse rather than with the task code itself.

Error Message

Timeout: DAG file processing exceeded configured timeout of 30 seconds for file '/opt/airflow/dags/reporting_pipeline.py'

Root Cause

Airflow enforces a hard timeout on parsing each DAG file (core.dagbag_import_timeout / dag_processor.dag_file_processor_timeout). DAGs that do expensive work at parse time — top-level API calls, database queries, or large loops generating tasks dynamically — exceed that timeout as the codebase grows, and the scheduler treats a DAG that failed to parse in time as an error rather than something to silently retry.

Diagnosis Steps

1. 1.Check scheduler/dag-processor logs for 'Process timed out' or import-timeout errors tied to a specific DAG file
2. 2.Time DAG parsing locally (e.g. `time python <dag_file>.py`) to reproduce outside the scheduler
3. 3.Identify what the slow DAG does at module/top level: API calls, DB queries, large dynamic task generation
4. 4.Check the DAG processor stats for per-file parse duration trend over time to see whether this is new or gradual

Solution

Increase dagbag_import_timeout and dag_file_processor_timeout to comfortably cover the slowest legitimate DAG as an immediate mitigation, then fix the offending DAG(s) to stop doing expensive work at import time — move API calls and dynamic task generation behind caching, lazy evaluation, or out of top-level module scope.

Commands

airflow dags list-import-errors

airflow dags report

time python /opt/airflow/dags/<slow_dag>.py

Symptom

A subset of long-running tasks (multi-hour ETL/ML jobs) occasionally run twice concurrently, or get marked failed and retried while the original execution is still healthy and running.

Error Message

Detected zombie job: <TaskInstance: etl.transform manual__2026-06-28T02:00:00+00:00 [running]> (See https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/tasks.html#zombie-undead-tasks)

Root Cause

Tasks send periodic heartbeats to the scheduler. If a heartbeat is missed for longer than the configured threshold — tied to the Celery broker's visibility_timeout when using the Celery executor — the scheduler considers the task a "zombie" and reschedules it, even though the original worker is still alive and working. This is a well-documented Airflow footgun for long-running tasks whose duration exceeds the broker's visibility_timeout.

Diagnosis Steps

1. 1.Search scheduler logs for 'zombie' detection events and correlate the task's actual duration against the configured visibility_timeout
2. 2.Check celery_broker_transport_options.visibility_timeout and scheduler.task_queued_timeout in airflow.cfg
3. 3.Confirm via task instance history whether the "duplicate" run is a true second execution or a retried zombie
4. 4.Audit whether the affected tasks are idempotent (safe to double-run) or require manual reconciliation

Solution

Increase celery_broker_transport_options.visibility_timeout to comfortably exceed the longest expected task duration, and align scheduler.task_queued_timeout / zombie-detection thresholds with that value. For tasks already duplicated, rely on task idempotency (or manual reconciliation) to resolve any double-written output.

Commands

grep -i zombie $AIRFLOW_HOME/logs/scheduler/latest/*.log

airflow config get-value celery_broker_transport_options visibility_timeout

airflow tasks states-for-dag-run <dag_id> <run_id>