Top "Airflow" questions

Apache Airflow is a workflow management platform to programmatically author, schedule, and monitor workflows as directed acyclic graphs (DAGs) of tasks.

How to create a conditional task in Airflow

I would like to create a conditional task in Airflow as described in the schema below. The expected scenario is …

python conditional-statements airflow
Proper way to create dynamic workflows in Airflow

Problem Is there any way in Airflow to create a workflow such that the number of tasks B.* is unknown …

python workflow airflow
Airflow pass parameters to dependent task

What is the way to pass parameter into dependent tasks in Airflow? I have a lot of bashes files, and …

bash airflow
How to prevent airflow from backfilling dag runs?

Say you have an airflow DAG that doesn't make sense to backfill, meaning that, after it's run once, running it …

python scheduled-tasks airflow
Python script scheduling in airflow

Hi everyone, I need to schedule my python files(which contains data extraction from sql and some joins) using airflow. …

python apache-spark scheduling reload airflow
How to set up Airflow Send Email?

I followed online tutorial to set up Email SMTP server in airflow.cfg as below: [email] email_backend = airflow.utils.…

python smtp airflow
Airflow s3 connection using UI

I've been trying to use Airflow to schedule a DAG. One of the DAG includes a task which loads data …

python-3.x airflow
How to set dependencies between DAGs in Airflow?

I am using Airflow to schedule batch jobs. I have one DAG (A) that runs every night and another DAG (…

python etl airflow
Airflow signals SIGTERM to subprocesses unexpectedly

I am using the PythonOperator to call a function that parallelizes data engineering process as an Airflow task. This is …

python airflow
Schedule a DAG in airflow to run every 5 minutes

I have a DAG in airflow and for now it is running each hour (@hourly). Is it possible to have …

airflow