How do I force a task on airflow to fail?

python airflow
Mask picture Mask · Mar 30, 2017 · Viewed 20k times · Source

I have a python callable process_csv_entries that processes csv file entries. I want my task to complete successfully only if all entries were processed successfully. Task should fail otherwise

def process_csv_entries(csv_file):
    # Boolean 
    file_completely_parsed = <call_to_module_to_parse_csv>
    return not file_completely_parsed

CSV_FILE=<Sets path to csv file>
t1 = PythonOperator(dag=dag,
                      task_id='parse_csv_completely',
                      python_operator=process_csv_entries,
                      op_args=[CSV_FILE])

t1 seems to complete successfully irrespective of returned value. How do I force PythonOperator task to fail?

Answer

Priyank Mehta picture Priyank Mehta · Mar 30, 2017

raise exception when you meet the error condition ( in your case: when file is not sucesfully parsed)

raise ValueError('File not parsed completely/correctly')

raise relevant error type with suitable message

Related questions

How to run Airflow on Windows

The usual instructions for running Airflow do not apply on a Windows environment: # airflow needs a home, ~/airflow is the default, # but you can lay foundation somewhere else if you prefer # (optional) export AIRFLOW_HOME=~/airflow # install from pypi using …

Read More
How to stop/kill Airflow tasks from the UI

How can I stop/kill a running task on Airflow UI? I am using LocalExecutor. Even if I use CeleryExecutor, how do can I kill/stop the running task?

Read More
How to Run a Simple Airflow DAG

I am totally new to Airflow. I would like to run a simple DAG at a specified date. I'm struggling to make difference between the start date, the execution date, and backfilling. And what is the command to run the …

Read More