Airflow scheduler does not appear to be running after excute a task

airflow
DennisLi picture DennisLi · Aug 27, 2019 · Viewed 22.1k times · Source

When there is a task running, Airflow will pop a notice saying the scheduler does not appear to be running and it kept showing until the task finished:

The scheduler does not appear to be running. Last heartbeat was received 5 minutes ago.

The DAGs list may not update, and new tasks will not be scheduled.

Actually, the scheduler process is running, as I have checked the process. After the task finished, the notice will disappear and everything back to normal.

My task is kind of heavy, may running for couple hours. I appretiate any help. Thanks in advance.

Answer

Jarek Potiuk picture Jarek Potiuk · Mar 17, 2020

I think it is expected for Sequential Executor. Sequential Executor runs one thing at a time so it cannot run heartbeat and task at the same time.

Why do you need to use Sequential Executor / Sqlite? The advice to switch to other DB/Executor make perfect sense.

Related questions

How do I restart airflow webserver?

I am using airflow for my data pipeline project. I have configured my project in airflow and start the airflow server as a backend process using following command airflow webserver -p 8080 -D True Server running successfully in backend. Now I …

Read More
execution_date in airflow: need to access as a variable

I am really a newbie in this forum. But I have been playing with airflow, for sometime, for our company. Sorry if this question sounds really dumb. I am writing a pipeline using bunch of BashOperators. Basically, for each Task, …

Read More
Airflow - How to pass xcom variable into Python function

I need to reference a variable that's returned by a BashOperator. In my task_archive_s3_file, I need to get the filename from get_s3_file. The task simply prints {{ ti.xcom_pull(task_ids=submit_file_to_spark) }} …

Read More