Where is Python's sys.path initialized from?

Question

Where is Python's sys.path initialized from?

UPD: Python is adding some paths before refering to PYTHONPATH:

    >>> import sys
    >>> from pprint import pprint as p
    >>> p(sys.path)
    ['',
     'C:\\Python25\\lib\\site-packages\\setuptools-0.6c9-py2.5.egg',
     'C:\\Python25\\lib\\site-packages\\orbited-0.7.8-py2.5.egg',
     'C:\\Python25\\lib\\site-packages\\morbid-0.8.6.1-py2.5.egg',
     'C:\\Python25\\lib\\site-packages\\demjson-1.4-py2.5.egg',
     'C:\\Python25\\lib\\site-packages\\stomper-0.2.2-py2.5.egg',
     'C:\\Python25\\lib\\site-packages\\uuid-1.30-py2.5.egg',
     'C:\\Python25\\lib\\site-packages\\stompservice-0.1.0-py2.5.egg',
     'C:\\Python25\\lib\\site-packages\\cherrypy-3.0.1-py2.5.egg',
     'C:\\Python25\\lib\\site-packages\\pyorbited-0.2.2-py2.5.egg',
     'C:\\Python25\\lib\\site-packages\\flup-1.0.1-py2.5.egg',
     'C:\\Python25\\lib\\site-packages\\wsgilog-0.1-py2.5.egg',
     'c:\\testdir',
     'C:\\Windows\\system32\\python25.zip',
     'C:\\Python25\\DLLs',
     'C:\\Python25\\lib',
     'C:\\Python25\\lib\\plat-win',
     'C:\\Python25\\lib\\lib-tk',
     'C:\\Python25',
     'C:\\Python25\\lib\\site-packages',
     'C:\\Python25\\lib\\site-packages\\PIL',
     'C:\\Python25\\lib\\site-packages\\win32',
     'C:\\Python25\\lib\\site-packages\\win32\\lib',
     'C:\\Python25\\lib\\site-packages\\Pythonwin']

My PYTHONPATH is:

    PYTHONPATH=c:\testdir

I wonder where those paths before PYTHONPATH's ones come from?

Answer 1

Python really tries hard to intelligently set sys.path. How it is set can get really complicated. The following guide is a watered-down, somewhat-incomplete, somewhat-wrong, but hopefully-useful guide for the rank-and-file python programmer of what happens when python figures out what to use as the initial values of sys.path, sys.executable, sys.exec_prefix, and sys.prefix on a normal python installation.

First, python does its level best to figure out its actual physical location on the filesystem based on what the operating system tells it. If the OS just says "python" is running, it finds itself in $PATH. It resolves any symbolic links. Once it has done this, the path of the executable that it finds is used as the value for sys.executable, no ifs, ands, or buts.

Next, it determines the initial values for sys.exec_prefix and sys.prefix.

If there is a file called pyvenv.cfg in the same directory as sys.executable or one directory up, python looks at it. Different OSes do different things with this file.

One of the values in this config file that python looks for is the configuration option home = <DIRECTORY>. Python will use this directory instead of the directory containing sys.executable when it dynamically sets the initial value of sys.prefix later. If the applocal = true setting appears in the pyvenv.cfg file on Windows, but not the home = <DIRECTORY> setting, then sys.prefix will be set to the directory containing sys.executable.

Next, the PYTHONHOME environment variable is examined. On Linux and Mac, sys.prefix and sys.exec_prefix are set to the PYTHONHOME environment variable, if it exists, superseding any home = <DIRECTORY> setting in pyvenv.cfg. On Windows, sys.prefix and sys.exec_prefix is set to the PYTHONHOME environment variable, if it exists, unless a home = <DIRECTORY> setting is present in pyvenv.cfg, which is used instead.

Otherwise, these sys.prefix and sys.exec_prefix are found by walking backwards from the location of sys.executable, or the home directory given by pyvenv.cfg if any.

If the file lib/python<version>/dyn-load is found in that directory or any of its parent directories, that directory is set to be to be sys.exec_prefix on Linux or Mac. If the file lib/python<version>/os.py is is found in the directory or any of its subdirectories, that directory is set to be sys.prefix on Linux, Mac, and Windows, with sys.exec_prefix set to the same value as sys.prefix on Windows. This entire step is skipped on Windows if applocal = true is set. Either the directory of sys.executable is used or, if home is set in pyvenv.cfg, that is used instead for the initial value of sys.prefix.

If it can't find these "landmark" files or sys.prefix hasn't been found yet, then python sets sys.prefix to a "fallback" value. Linux and Mac, for example, use pre-compiled defaults as the values of sys.prefix and sys.exec_prefix. Windows waits until sys.path is fully figured out to set a fallback value for sys.prefix.

Then, (what you've all been waiting for,) python determines the initial values that are to be contained in sys.path.

1. The directory of the script which python is executing is added to sys.path. On Windows, this is always the empty string, which tells python to use the full path where the script is located instead.
2. The contents of PYTHONPATH environment variable, if set, is added to sys.path, unless you're on Windows and applocal is set to true in pyvenv.cfg.
3. The zip file path, which is <prefix>/lib/python35.zip on Linux/Mac and os.path.join(os.dirname(sys.executable), "python.zip") on Windows, is added to sys.path.
4. If on Windows and no applocal = true was set in pyvenv.cfg, then the contents of the subkeys of the registry key HK_CURRENT_USER\Software\Python\PythonCore\<DLLVersion>\PythonPath\ are added, if any.
5. If on Windows and no applocal = true was set in pyvenv.cfg, and sys.prefix could not be found, then the core contents of the of the registry key HK_CURRENT_USER\Software\Python\PythonCore\<DLLVersion>\PythonPath\ is added, if it exists;
6. If on Windows and no applocal = true was set in pyvenv.cfg, then the contents of the subkeys of the registry key HK_LOCAL_MACHINE\Software\Python\PythonCore\<DLLVersion>\PythonPath\ are added, if any.
7. If on Windows and no applocal = true was set in pyvenv.cfg, and sys.prefix could not be found, then the core contents of the of the registry key HK_CURRENT_USER\Software\Python\PythonCore\<DLLVersion>\PythonPath\ is added, if it exists;
8. If on Windows, and PYTHONPATH was not set, the prefix was not found, and no registry keys were present, then the relative compile-time value of PYTHONPATH is added; otherwise, this step is ignored.
9. Paths in the compile-time macro PYTHONPATH are added relative to the dynamically-found sys.prefix.
10. On Mac and Linux, the value of sys.exec_prefix is added. On Windows, the directory which was used (or would have been used) to search dynamically for sys.prefix is added.

At this stage on Windows, if no prefix was found, then python will try to determine it by searching all the directories in sys.path for the landmark files, as it tried to do with the directory of sys.executable previously, until it finds something. If it doesn't, sys.prefix is left blank.

Finally, after all this, Python loads the site module, which adds stuff yet further to sys.path:

It starts by constructing up to four directories from a head and a tail part. For the head part, it uses sys.prefix and sys.exec_prefix; empty heads are skipped. For the tail part, it uses the empty string and then lib/site-packages (on Windows) or lib/pythonX.Y/site-packages and then lib/site-python (on Unix and Macintosh). For each of the distinct head-tail combinations, it sees if it refers to an existing directory, and if so, adds it to sys.path and also inspects the newly added path for configuration files.