How to set a SLA in Airflow?

airflow apache-airflow airflow-scheduler
ebertbm picture ebertbm · May 19, 2017 · Viewed 14.7k times · Source

I would like to set a SLA in a Sensor Operator. The documentation is not too clear about the use of it. So I did a test using the S3KeySensor operator which is looking for a file that does not exist. I set the sla to 30 seconds, I was hoping to see the record after 30 seconds in the UI - in SLA misses - but it did not happen. What am I doing wrong? 

inputsensor = S3KeySensor(
    task_id='check_for_files_in_s3',
    bucket_key='adp/backload/20136585/',
    wildcard_match=True,
    bucket_name='weblogs-raw',
    s3_conn_id='AWS_S3_CENTRAL',
    timeout=120,
    poke_interval=10,
    sla=timedelta(seconds=30),
    dag=dag)

inputsensor.set_downstream(next_step)

Answer

Manish Ranjan picture Manish Ranjan · Jun 17, 2017 
'sla': timedelta(hours=2),

An example from Airflow Git repository: airflow/example_dags/tutorial.py#L54.

Related questions

Airflow tasks get stuck at "queued" status and never gets running

I'm using Airflow v1.8.1 and run all components (worker, web, flower, scheduler) on kubernetes & Docker. I use Celery Executor with Redis and my tasks are looks like: (start) -> (do_work_for_product1) ├ -> (do_work_for_…

Read More
How to stop/kill airflow scheduler started in daemon mode

I and new to airflow and accidentally started airflow scheduler in daemon mode. Now, I want to kill the scheduler and possibly restart it. I tried doing sudo kill -9 <list of pids> pkill <name> nothing …

Read More
how do I use the --conf option in airflow

I am trying to run a airflow DAG and need to pass some parameters for the tasks. How do I read the JSON string passed as the --conf parameter in the command line trigger_dag command, in the python DAG …

Read More