How to restart DAG in Airflow?

airflow airflow-scheduler
Paymahn Moghadasian picture Paymahn Moghadasian · Feb 14, 2018 · Viewed 12k times · Source

One of my DAGs with many steps stopped part way through because of a DB shutdown. I'd like to get the DAG to start where it left off but all I can do is start the individual tasks of the DAG one by one. Is there a way to tell Airflow to start the DAG where it left off based on the success of the tasks already completed?

Here's a sample where one of the first tasks finished and the rest are queued or have no status: enter image description here

When I click on an individual task (runme_1) I can restart it: enter image description here

However, if I try to start a task which hasn't had it dependencies met (such as run_after_loop), I see the following error message: enter image description here

I'd like to be able to click on any task, ask it to be queued and have all dependencies run in the correct order. Is that possible?

I'm running Airflow 1.9.0.

Answer

jhnclvr picture jhnclvr · Feb 14, 2018

It sounds like you need to start a dag from the task where it failed.

Instead of going into the DAG and clicking on a task and clicking run, go to the Dag run with the failed task, click the failed task and click clear.

Clearing the task will re-run the task, and any other tasks after it will run.

Related questions

Airflow 1.9.0 is queuing but not launching tasks

Airflow is randomly not running queued tasks some tasks dont even get queued status. I keep seeing below in the scheduler logs [2018-02-28 02:24:58,780] {jobs.py:1077} INFO - No tasks to consider for execution. I do see tasks in database …

Read More
Airflow tasks get stuck at "queued" status and never gets running

I'm using Airflow v1.8.1 and run all components (worker, web, flower, scheduler) on kubernetes & Docker. I use Celery Executor with Redis and my tasks are looks like: (start) -> (do_work_for_product1) ├ -> (do_work_for_…

Read More
Why is a task stuck and not executed in airflow?

I'm trying out airflow with the BigQueryOperator. I thought I would use google composer later on, but I want it running locally first. I have airflow up and running an BashOperator works fine, I can also run airflow test <…

Read More