Top "Airflow" questions

Apache Airflow is a workflow management platform to programmatically author, schedule, and monitor workflows as directed acyclic graphs (DAGs) of tasks.

Airflow: Creating a DAG in airflow via UI

Airflow veterans please help, I was looking for a cron replacement and came across apache airflow. We have a setup …

airflow airflow-scheduler
Airflow scheduler is slow to schedule subsequent tasks

When I try to run a DAG in Airflow 1.8.0 I find that it takes a lot of time between the …

airflow airflow-scheduler
Which one to choose Apache Oozie or Apache Airflow? Need a comparison

I am new to job schedulers and was looking out for one to run jobs on big data cluster. I …

jobs oozie airflow airflow-scheduler
How to obtain and process mysql records using Airflow?

I need to 1. run a select query on MYSQL DB and fetch the records. 2. Records are processed by python script. …

mysql python-3.x airflow apache-airflow
How to control the parallelism or concurrency of an Airflow installation?

In some of my Apache Airflow installations, DAGs or tasks that are scheduled to run do not run even when …

python google-cloud-platform airflow google-cloud-composer
Importing local module (python script) in Airflow DAG

I'm trying to import a local module (a python script) to my DAG. Directory structure: airflow/ ├── dag │   ├── __init__.py │   └── my_…

python airflow
Airflow depends_on_past for whole DAG

Is there a way in airflow of using the depends_on_past for an entire DagRun, not just applied to …

airflow apache-airflow airflow-scheduler
Airflow - run task regardless of upstream success/fail

I have a DAG which fans out to multiple independent units in parallel. This runs in AWS, so we have …

python airflow
Airflow + celery or dask. For what, when?

I read in the official Airflow documentation the following: What does this mean exactly? What do the authors mean by …

celery dask airflow
How to restart DAG in Airflow?

One of my DAGs with many steps stopped part way through because of a DB shutdown. I'd like to get …

airflow airflow-scheduler