Errno 13 Permission denied when Airflow tries to write to logs

Question

We're running into a permission error when using Airflow, receiving the following error:

PermissionError: [Errno 13] Permission denied: '/usr/local/airflow/logs/scheduler/2019-12-18/../../../../home

We've tried using chmod 777 -R on the /usr/local/airflow/logs/schedule directory within the container but this doesn't seem to have done the trick.

We have this piece in our entrypoint.sh script:

export AIRFLOW__CORE__BASE_LOGS_FOLDER="/usr/local/airflow/logs

Has anyone else run into this airflow log permission issue? Can't seem to find much about this one in particular online.

Answer 1

I also have the same problem using Apache Airflow 1.10.7.

Traceback (most recent call last):
  File "/usr/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap
    self.run()
  File "/usr/lib/python3.7/multiprocessing/process.py", line 99, in run
    self._target(*self._args, **self._kwargs)
  File "/home/radifar/.virtualenvs/airflow/lib/python3.7/site-packages/airflow/jobs/scheduler_job.py", line 135, in _run_file_processor
    set_context(log, file_path)
  File "/home/radifar/.virtualenvs/airflow/lib/python3.7/site-packages/airflow/utils/log/logging_mixin.py", line 198, in set_context
    handler.set_context(value)
  File "/home/radifar/.virtualenvs/airflow/lib/python3.7/site-packages/airflow/utils/log/file_processor_handler.py", line 65, in set_context
    local_loc = self._init_file(filename)
  File "/home/radifar/.virtualenvs/airflow/lib/python3.7/site-packages/airflow/utils/log/file_processor_handler.py", line 148, in _init_file
    os.makedirs(directory)
  File "/home/radifar/.virtualenvs/airflow/lib/python3.7/os.py", line 211, in makedirs
    makedirs(head, exist_ok=exist_ok)
  File "/home/radifar/.virtualenvs/airflow/lib/python3.7/os.py", line 211, in makedirs
    makedirs(head, exist_ok=exist_ok)
  File "/home/radifar/.virtualenvs/airflow/lib/python3.7/os.py", line 211, in makedirs
    makedirs(head, exist_ok=exist_ok)
  [Previous line repeated 5 more times]
  File "/home/radifar/.virtualenvs/airflow/lib/python3.7/os.py", line 221, in makedirs
    mkdir(name, mode)
PermissionError: [Errno 13] Permission denied: '/media/radifar/radifar-dsl/Workflow/Airflow/airflow-home/logs/scheduler/2020-01-04/../../../../../../../home'

After checking how file_processor_handler.py works I find that the error was caused by the different directory location of example dag and our dag folder settings. In my case 7 folder above the folder 2020-01-04 is /media/radifar. In your case 4 folder above the folder 2019-12-18 is /usr/local. That's why the PermissionError was raised.

I was able to solve this problem by cleaning the AIRFLOW_HOME folder then run airflow version, set the load_example to False in airflow.cfg. Then run airflow initdb. After that I can use airflow without error.