Minimum hardware requirements for Apache Airflow cluster

airflow airflow-scheduler
Duleendra picture Duleendra · Nov 14, 2017 · Viewed 7.8k times · Source

What are the minimum hardware requirements for setting up an Apache Airflow cluster.

Eg. RAM, CPU, Disk etc for different types of nodes in the cluster.

Answer

jastang picture jastang · Jan 23, 2018

I have had no issues using very small instances in pseudo-distributed mode (32 parallel workers; Postgres backend):

  • RAM 4096 MB
  • CPU 1000 MHz
  • VCPUs 2 VCPUs
  • Disk 40 GB

If you want distributed mode, you should be more than fine with that if you keep it homogenous. Airflow shouldn't really do heavy lifting anyways; push the workload out to other things (Spark, EMR, BigQuery, etc).

You will also have to run some kind of messaging queue, like RabbitMQ. I think they take Redis too. However, this doesn't really dramatically impact how you size.

Related questions

Airflow 1.9.0 is queuing but not launching tasks

Airflow is randomly not running queued tasks some tasks dont even get queued status. I keep seeing below in the scheduler logs [2018-02-28 02:24:58,780] {jobs.py:1077} INFO - No tasks to consider for execution. I do see tasks in database …

Read More
Airflow tasks get stuck at "queued" status and never gets running

I'm using Airflow v1.8.1 and run all components (worker, web, flower, scheduler) on kubernetes & Docker. I use Celery Executor with Redis and my tasks are looks like: (start) -> (do_work_for_product1) ├ -> (do_work_for_…

Read More
Why is a task stuck and not executed in airflow?

I'm trying out airflow with the BigQueryOperator. I thought I would use google composer later on, but I want it running locally first. I have airflow up and running an BashOperator works fine, I can also run airflow test <…

Read More