Top "Airflow" questions

Apache Airflow is a workflow management platform to programmatically author, schedule, and monitor workflows as directed acyclic graphs (DAGs) of tasks.

Airflow - Broken DAG - Timeout

I have a DAG that executes a function that connects to a Postgres DB, deletes the contents in the table …

airflow airflow-scheduler
Use case of dummy operator

I was learning apache airflow and found that there is an operator called DummyOperator. I googled about its use case, …

airflow airflow-operator
What are the differences between airflow and Kubeflow pipeline?

Machine learning platform is one of the buzzwords in business, in order to boost develop ML or Deep learning. There …

machine-learning workflow airflow kubeflow-pipelines
Unable to start Airflow worker/flower and need clarification on Airflow architecture to confirm that the installation is correct

Running a worker on a different machine results in errors specified below. I have followed the configuration instructions and have …

python rabbitmq celery airflow
Remove Airflow Scheduler logs

I am using Docker Apache airflow VERSION 1.9.0-2 (https://github.com/puckel/docker-airflow). The scheduler produces a significant amount of …

docker docker-compose airflow docker-volume airflow-scheduler
Apache Airflow : airflow initdb throws ModuleNotFoundError: No module named 'werkzeug.wrappers.json'; 'werkzeug.wrappers' is not a package error

On Ubuntu 18.04 with python 3.6.8, trying to install airflow. When I ran airflow initdb command, below error is thrown Traceback (most …

airflow
Airflow structure/organization of Dags and tasks

My questions : What is a good directory structure in order to organize your dags and tasks? (the dags examples show …

airflow apache-airflow
Is there a way to create/modify connections through Airflow API

Going through Admin -> Connections, we have the ability to create/modify a connection's params, but I'm wondering if …

python airflow
Airflow: New DAG is not found by webserver

In Airflow, how should I handle the error "This DAG isn't available in the webserver DagBag object. It shows up …

airflow airflow-scheduler
Airflow: pass {{ ds }} as param to PostgresOperator

i would like to use execution date as parameter to my sql file: i tried dt = '{{ ds }}' s3_…

python scheduler airflow apache-airflow