Airflow not loading dags in /usr/local/airflow/dags

airflow
Jeremy Lewi picture Jeremy Lewi · Aug 6, 2017 · Viewed 22.1k times · Source

Airflow seems to be skipping the dags I added to /usr/local/airflow/dags.

When I run

airflow list_dags

The output shows

[2017-08-06 17:03:47,220] {models.py:168} INFO - Filling up the DagBag from /usr/local/airflow/dags


-------------------------------------------------------------------
DAGS
-------------------------------------------------------------------
example_bash_operator
example_branch_dop_operator_v3
example_branch_operator
example_http_operator
example_passing_params_via_test_command
example_python_operator
example_short_circuit_operator
example_skip_dag
example_subdag_operator
example_subdag_operator.section-1
example_subdag_operator.section-2
example_trigger_controller_dag
example_trigger_target_dag
example_xcom
latest_only
latest_only_with_trigger
test_utils
tutorial

But this doesn't include the dags in /usr/local/airflow/dags

ls -la /usr/local/airflow/dags/
total 20
drwxr-xr-x 3 airflow airflow 4096 Aug  6 17:08 .
drwxr-xr-x 4 airflow airflow 4096 Aug  6 16:57 ..
-rw-r--r-- 1 airflow airflow 1645 Aug  6 17:03 custom_example_bash_operator.py
drwxr-xr-x 2 airflow airflow 4096 Aug  6 17:08 __pycache__

Is there some other condition that neededs to be satisfied for airflow to identify a DAG and load it?

Answer

Jeremy Lewi picture Jeremy Lewi · Aug 6, 2017

My dag is being loaded but I had the name of the DAG wrong. I was expecting the dag to be named by the file but the name is determined by the first argument to the DAG constructor

dag = DAG(
    'tutorial', default_args=default_args, schedule_interval=timedelta(1))

Related questions

How do I restart airflow webserver?

I am using airflow for my data pipeline project. I have configured my project in airflow and start the airflow server as a backend process using following command airflow webserver -p 8080 -D True Server running successfully in backend. Now I …

Read More
execution_date in airflow: need to access as a variable

I am really a newbie in this forum. But I have been playing with airflow, for sometime, for our company. Sorry if this question sounds really dumb. I am writing a pipeline using bunch of BashOperators. Basically, for each Task, …

Read More
Airflow - How to pass xcom variable into Python function

I need to reference a variable that's returned by a BashOperator. In my task_archive_s3_file, I need to get the filename from get_s3_file. The task simply prints {{ ti.xcom_pull(task_ids=submit_file_to_spark) }} …

Read More