How to set dependencies between DAGs in Airflow?

python etl airflow
Conor picture Conor · Jun 24, 2016 · Viewed 32.2k times · Source

I am using Airflow to schedule batch jobs. I have one DAG (A) that runs every night and another DAG (B) that runs once per month. B depends on A having completed successfully. However B takes a long time to run and so I would like to keep it in a separate DAG to allow better SLA reporting.

How can I make running DAG B dependent on a successful run of DAG A on the same day?

Answer

p.magalhaes picture p.magalhaes · Jun 25, 2016

You can achieve this behavior using an operator called ExternalTaskSensor. Your task (B1) in DAG(B) will be scheduled and wait for a success on task (A2) in DAG(A)

External Task Sensor documentation

Related questions

What does 'killed' mean when a processing of a huge CSV with Python, which suddenly stops?

I have a Python script that imports a large CSV file and then counts the number of occurrences of each word in the file, then exports the counts to another CSV file. But what is happening is that once that …

Read More
Open Source ETL framework

I was asked to prototype two ETL frameworks. The requirements are as follows: Open Source Available to Linux Maintained Logs can be viewed on web browser (nice to have) Written in Perl, Python, Ruby or Java The raw file can …

Read More
What's the most efficient way to convert a MySQL result set to a NumPy array?

I'm using MySQLdb and Python. I have some basic queries such as this: c=db.cursor() c.execute("SELECT id, rating from video") results = c.fetchall() I need "results" to be a NumPy array, and I'm looking to be economical …

Read More