How to add new DAGs to Airflow?

python airflow apache-airflow
Aleksey Bilogur picture Aleksey Bilogur · Jul 1, 2017 · Viewed 18.3k times · Source

I have defined a DAG in a file called tutorial_2.py (actually a copy of the tutorial.py provided in the airflow tutorial, except with the dag_id changed to tutorial_2).

When I look inside my default, unmodified airflow.cfg (located in ~/airflow), I see that dags_folder is set to /home/alex/airflow/dags.

I do cd /home/alex/airflow; mkdir dags; cd dags; cp [...]/tutorial_2.py tutorial_2.py. Now I have a dags folder matching the path set in airflow.cfg , containing the tutorial_2.py file I created earlier.

However, when I run airflow list_dags, I only get the names corresponding with the default, tutorial DAGs.

I would like to have tutorial_2 show up in my DAG list, so that I can begin interacting with. Neither python tutorial_2.py nor airflow resetdb have caused it to appear in the list.

How do I remedy this?

Answer

Ace Haidrey picture Ace Haidrey · Jul 1, 2017

I think the reason for this is because you haven't exported AIRFLOW_HOME. Try doing: AIRFLOW_HOME="/home/alex/airflow/dags" airflow list_dags. If that's not working than do two steps export AIRFLOW_HOME="/home/alex/airflow/dags" airflow list_dags I believe this should work. Give it a go?

Related questions

How to stop/kill Airflow tasks from the UI

How can I stop/kill a running task on Airflow UI? I am using LocalExecutor. Even if I use CeleryExecutor, how do can I kill/stop the running task?

Read More
Apache Airflow DAG cannot import local module

I do not seem to understand how to import modules into an apache airflow DAG definition file. I would want to do this to be able to create a library which makes declaring tasks with similar settings less verbose, for …

Read More
Airflow default on_failure_callback

In my DAG file, I have define a on_failure_callback() function to post a Slack in case of failure. It works well if I specify for each operator in my DAG : on_failure_callback=on_failure_callback() Is there …

Read More