how do I use the --conf option in airflow

airflow apache-airflow airflow-scheduler
Praveen Singh picture Praveen Singh · Aug 29, 2017 · Viewed 14.8k times · Source

I am trying to run a airflow DAG and need to pass some parameters for the tasks.

How do I read the JSON string passed as the --conf parameter in the command line trigger_dag command, in the python DAG file.

ex: airflow trigger_dag 'dag_name' -r 'run_id' --conf '{"key":"value"}'

Answer

Daniel Huang picture Daniel Huang · Aug 31, 2017

Two ways. From inside a template field or file:

{{ dag_run.conf['key'] }}

Or when context is available, e.g. within a python callable of the PythonOperator:

context['dag_run'].conf['key']

Related questions

Airflow tasks get stuck at "queued" status and never gets running

I'm using Airflow v1.8.1 and run all components (worker, web, flower, scheduler) on kubernetes & Docker. I use Celery Executor with Redis and my tasks are looks like: (start) -> (do_work_for_product1) ├ -> (do_work_for_…

Read More
How to stop/kill airflow scheduler started in daemon mode

I and new to airflow and accidentally started airflow scheduler in daemon mode. Now, I want to kill the scheduler and possibly restart it. I tried doing sudo kill -9 <list of pids> pkill <name> nothing …

Read More
How to set a SLA in Airflow?

I would like to set a SLA in a Sensor Operator. The documentation is not too clear about the use of it. So I did a test using the S3KeySensor operator which is looking for a file that does …

Read More