Schedule a DAG in airflow to run every 5 minutes

airflow
bsd picture bsd · Aug 15, 2017 · Viewed 31.4k times · Source

I have a DAG in airflow and for now it is running each hour (@hourly). Is it possible to have it running each 5 minutes ?

Answer

Mike picture Mike · Aug 15, 2017

Yes, here's an example of a DAG that I have running every 5 min:

dag = DAG(dag_id='eth_rates',
          default_args=args,
          schedule_interval='*/5 * * * *',
          dagrun_timeout=timedelta(seconds=5))

schedule_interval accepts a CRON expression: https://en.wikipedia.org/wiki/Cron#CRON_expression

Related questions

How do I restart airflow webserver?

I am using airflow for my data pipeline project. I have configured my project in airflow and start the airflow server as a backend process using following command airflow webserver -p 8080 -D True Server running successfully in backend. Now I …

Read More
execution_date in airflow: need to access as a variable

I am really a newbie in this forum. But I have been playing with airflow, for sometime, for our company. Sorry if this question sounds really dumb. I am writing a pipeline using bunch of BashOperators. Basically, for each Task, …

Read More
Airflow - How to pass xcom variable into Python function

I need to reference a variable that's returned by a BashOperator. In my task_archive_s3_file, I need to get the filename from get_s3_file. The task simply prints {{ ti.xcom_pull(task_ids=submit_file_to_spark) }} …

Read More