Debugging Broken DAGs

airflow apache-airflow
arbazkhan002 picture arbazkhan002 · May 12, 2017 · Viewed 16.6k times · Source

When the airflow webserver shows up errors like Broken DAG: [<path/to/dag>] <error>, how and where can we find the full stacktrace for these exceptions?

I tried these locations:

/var/log/airflow/webserver -- had no logs in the timeframe of execution, other logs were in binary and decoding with strings gave no useful information.

/var/log/airflow/scheduler -- had some logs but were in binary form, tried to read them and looked to be mostly sqlalchemy logs probably for airflow's database.

/var/log/airflow/worker -- shows up the logs for running DAGs, (same as the ones you see on the airflow page)

and then also under /var/log/airflow/rotated -- couldn't find the stacktrace I was looking for.

I am using airflow v1.7.1.3

Answer

Babcool picture Babcool · Oct 7, 2017

Usually I used the command airflow list_dags which print the full stacktrace for python error found in dags.

That will work with almost any airflow command as airflow parse dags folder each time you use a airflow CLI command.

Related questions

How to stop/kill Airflow tasks from the UI

How can I stop/kill a running task on Airflow UI? I am using LocalExecutor. Even if I use CeleryExecutor, how do can I kill/stop the running task?

Read More
Airflow tasks get stuck at "queued" status and never gets running

I'm using Airflow v1.8.1 and run all components (worker, web, flower, scheduler) on kubernetes & Docker. I use Celery Executor with Redis and my tasks are looks like: (start) -> (do_work_for_product1) ├ -> (do_work_for_…

Read More
Apache Airflow DAG cannot import local module

I do not seem to understand how to import modules into an apache airflow DAG definition file. I would want to do this to be able to create a library which makes declaring tasks with similar settings less verbose, for …

Read More