Why do I get no such table error when installing Apache Airflow on Mac?
It was so hard to put that right title. Ok, here it goes. I was following this tutorial to install Apache Airflow on my Mac (Mojave version) -
https://towardsdatascience.com/getting-started-with-apache-airflow-df1aa77d7b1b
Right at the first step after performing the pip install airflow task, when I run the airflow version command I am getting the following error and then the airflow version appears -
ERROR - Failed on pre-execution callback using Traceback (most recent call last): File "/Users/karthikv/anaconda3/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1244, in _execute_context cursor, statement, parameters, context File "/Users/karthikv/anaconda3/lib/python3.7/site-packages/sqlalchemy/engine/default.py", line 552, in do_execute cursor.execute(statement, parameters) sqlite3.OperationalError: no such table: log
The above exception was the direct cause of the following exception:
Traceback (most recent call last): File "/Users/karthikv/anaconda3/lib/python3.7/site-packages/airflow/utils/cli_action_loggers.py", line 68, in on_pre_execution cb(**kwargs) File "/Users/karthikv/anaconda3/lib/python3.7/site-packages/airflow/utils/cli_action_loggers.py", line 99, in default_action_log session.add(log) File "/Users/karthikv/anaconda3/lib/python3.7/contextlib.py", line 119, in exit next(self.gen) File "/Users/karthikv/anaconda3/lib/python3.7/site-packages/airflow/utils/db.py", line 45, in create_session session.commit() File "/Users/karthikv/anaconda3/lib/python3.7/site-packages/sqlalchemy/orm/session.py", line 1026, in commit self.transaction.commit() File "/Users/karthikv/anaconda3/lib/python3.7/site-packages/sqlalchemy/orm/session.py", line 493, in commit self._prepare_impl() File "/Users/karthikv/anaconda3/lib/python3.7/site-packages/sqlalchemy/orm/session.py", line 472, in _prepare_impl self.session.flush() File "/Users/karthikv/anaconda3/lib/python3.7/site-packages/sqlalchemy/orm/session.py", line 2451, in flush self._flush(objects) File "/Users/karthikv/anaconda3/lib/python3.7/site-packages/sqlalchemy/orm/session.py", line 2589, in _flush transaction.rollback(_capture_exception=True) File "/Users/karthikv/anaconda3/lib/python3.7/site-packages/sqlalchemy/util/langhelpers.py", line 68, in exit compat.reraise(exc_type, exc_value, exc_tb) File "/Users/karthikv/anaconda3/lib/python3.7/site-packages/sqlalchemy/util/compat.py", line 129, in reraise raise value File "/Users/karthikv/anaconda3/lib/python3.7/site-packages/sqlalchemy/orm/session.py", line 2549, in _flush flush_context.execute() File "/Users/karthikv/anaconda3/lib/python3.7/site-packages/sqlalchemy/orm/unitofwork.py", line 422, in execute rec.execute(self) File "/Users/karthikv/anaconda3/lib/python3.7/site-packages/sqlalchemy/orm/unitofwork.py", line 589, in execute uow, File "/Users/karthikv/anaconda3/lib/python3.7/site-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj insert, File "/Users/karthikv/anaconda3/lib/python3.7/site-packages/sqlalchemy/orm/persistence.py", line 1120, in _emit_insert_statements statement, params File "/Users/karthikv/anaconda3/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 988, in execute return meth(self, multiparams, params) File "/Users/karthikv/anaconda3/lib/python3.7/site-packages/sqlalchemy/sql/elements.py", line 287, in _execute_on_connection return connection._execute_clauseelement(self, multiparams, params) File "/Users/karthikv/anaconda3/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1107, in _execute_clauseelement distilled_params, File "/Users/karthikv/anaconda3/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1248, in _execute_context e, statement, parameters, cursor, context File "/Users/karthikv/anaconda3/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1466, in _handle_dbapi_exception util.raise_from_cause(sqlalchemy_exception, exc_info) File "/Users/karthikv/anaconda3/lib/python3.7/site-packages/sqlalchemy/util/compat.py", line 383, in raise_from_cause reraise(type(exception), exception, tb=exc_tb, cause=cause) File "/Users/karthikv/anaconda3/lib/python3.7/site-packages/sqlalchemy/util/compat.py", line 128, in reraise raise value.with_traceback(tb) File "/Users/karthikv/anaconda3/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1244, in _execute_context cursor, statement, parameters, context File "/Users/karthikv/anaconda3/lib/python3.7/site-packages/sqlalchemy/engine/default.py", line 552, in do_execute cursor.execute(statement, parameters) sqlalchemy.exc.OperationalError: (sqlite3.OperationalError) no such table: log [SQL: INSERT INTO log (dttm, dag_id, task_id, event, execution_date, owner, extra) VALUES (?, ?, ?, ?, ?, ?, ?)] [parameters: ('2019-08-12 20:50:24.960006', None, None, 'cli_version', None, 'karthikv', '{"host_name": "192-168-1-6.tpgi.com.au", "full_command": "[\'/Users/karthikv/anaconda3/bin/airflow\', \'version\']"}')]
(Background on this error at: http://sqlalche.me/e/e3q8)
Can someone help me what this error means and how to solve it? I understand from the instructions that by default SQLLite db gets installed and a single DAG restrictions would be in place before we get into setting up backend database say PostgreSQL.
I tried to uninstall using pip uninstall airflow to perform clean installation again. I get the following error -
WARNING: Skipping airflow as it is not installed.
Kindly help me in solving the issue (or) pointing me to resources where I can do further reading.
Answer
You need to perform initialization after installation:
$ export AIRFLOW_HOME=some/dir
$ airflow db init # or `airflow initdb` for the legacy 1.X
If AIRFLOW_HOME is unset, ~/airflow/ will be created and used. This is where the config and logs will be stored; if you want to reset the configuration, remove the dir stored in AIRFLOW_HOME and rerun airflow db init.
Now other commands should work, e.g.
$ airflow db check
[2021-02-06 23:35:21,577] {db.py:756} INFO - Connection successful.
Source: Initializing Airflow Database section from airflow docs. Kudos to Anas Tiour who notified about the command line interface change in 2.0.0 in this comment!