How to use apache airflow in a virtual environment?

Question

I am quite new to using apache airflow. I use pycharm as my IDE. I create a project (anaconda environment), create a python script that includes DAG definitions and Bash operators. When I open my airflow webserver, my DAGS are not shown. Only the default example DAGs are shown. My AIRFLOW_HOME variable contains ~/airflow. So i stored my python script there and now it shows.

How do I use this in a project environment?

Do I change the environment variable at the start of every project?

Is there a way to add specific airflow home directories for each project?

I dont wanna be storing my DAGs in the default airflow directory since I would wanna add it to my git repository. Kindly help me out.

Answer 1

You can set/override airflow options specified in ${AIRFLOW_HOME}/airflow.cfg with environment variables by using this format: $AIRFLOW__{SECTION}__{KEY} (note the double underscores). Here is a link to airflow docs. So you can simply do

export AIRFLOW__CORE__DAGS_FOLDER=/path/to/dags/folder

However, it is tedious and error prone to do this for different projects. As alternative, you can consider using pipenv for managing virtual environments instead of Anaconda. Here is a nice guide about pipenv and problems it solves. One of the default features of pipenv is that it automatically loads variables defined in .env file when you spawn a shell with the virtualenv activated. So here is what your workflow with pipenv could look like:

cd /path/to/my_project

# Creates venv with python 3.7 
pipenv install --python=3.7 Flask==1.0.3 apache-airflow==1.10.3

# Set home for airflow in a root of your project (specified in .env file)
echo "AIRFLOW_HOME=${PWD}/airflow" >> .env

# Enters created venv and loads content of .env file 
pipenv shell

# Initialize airflow
airflow initdb
mkdir -p ${AIRFLOW_HOME}/dags/

Note: usage of Flask==1.03 I will explain at the end, but this is because pipenv checks whether sub-dependencies are compatible in order to ensure reproducibility.

So after these steps you would get the following project structure

my_project
├── airflow
│   ├── airflow.cfg
│   ├── airflow.db
│   ├── dags
│   ├── logs
│   │   └── scheduler
│   │       ├── 2019-07-07
│   │       └── latest -> /path/to/my_project/airflow/logs/scheduler/2019-07-07
│   └── unittests.cfg
├── .env
├── Pipfile
└── Pipfile.lock

Now when you initialize airflow for the first time it will create ${AIRFLOW_HOME}/airflow.cfg file and will use/expand ${AIRFLOW_HOME}/dags as value for dags_folder. In case you still need a different location for dags_folder, you can use .env file again

echo "AIRFLOW__CORE__DAGS_FOLDER=/different/path/to/dags/folder" >> .env

Thus, you .env file will look like:

AIRFLOW_HOME=/path/to/my_project/airflow
AIRFLOW__CORE__DAGS_FOLDER=/different/path/to/dags/folder

What have we accomplished and why this would work just fine

1. Since you installed airflow in virtual environment, you would need to activate it in order to use airflow
2. Since you did it with pipenv, you would need to use pipenv shell in order to activate venv
3. Since you use pipenv shell, you would always get variables defined in .env exported into your venv. On top of that pipenv will still be a subshell, therefore, when you exit it, all additional environmental variables would be cleared as well.
4. Different projects that use airflow would have different locations for their log files etc.

Additional notes on pipenv

1. In order to use venv created with pipenv as your IDE's project interpreter, use path provided by pipenv --py.
2. By default, pipenv creates all venvs in the same global location like conda does, but you can change that behavior to creating .venv in a project's root by adding export PIPENV_VENV_IN_PROJECT=1 into your .bashrc (or other rc). Then PyCharm would be able to automatically pick it up when you go into settings of project interpreter.

Note on usage of Flask==1.0.3

Airflow 1.10.3 from PyPi depends on flask>=1.0, <2.0 and on jinja2>=2.7.3, <=2.10.0. Today, when I tested code snippets the latest available flask was 1.1.0 which depends on jinja2>=2.10.1. This means that although pipenv can install all required software, but it fails to lock dependencies. So for clean use of my code samples, I had to specify version of flask that requires version of jinja2 compatible with airflow requirements. But there is nothing to worry about. The latest version of airflow on GitHub is already fixed that.