Adding logs to Airflow Logs

user3542930 picture user3542930 · Oct 19, 2016 · Viewed 14.5k times · Source

How can I add my own logs onto the Apache Airflow logs that are automatically generated? any print statements wont get logged in there, so I was wondering how I can add my logs so that it shows up on the UI as well?

Answer

Ivan Gozali picture Ivan Gozali · Jan 25, 2017

I think you can work around this by using the logging module and trusting the configuration to Airflow.

Something like:

import ...

dag = ...

def print_params_fn(**kwargs):
    import logging
    logging.info(kwargs)
    return None

print_params = PythonOperator(task_id="print_params",
                              python_callable=print_params_fn,
                              provide_context=True,
                              dag=dag)