Parse config files, environment, and command-line arguments, to get a single collection of options

Question

Python's standard library has modules for configuration file parsing (configparser), environment variable reading (os.environ), and command-line argument parsing (argparse). I want to write a program that does all those, and also:

• Has a cascade of option values:

  • default option values, overridden by
  • config file options, overridden by
  • environment variables, overridden by
  • command-line options.

• Allows one or more configuration file locations specified on the command line with e.g. --config-file foo.conf, and reads that (either instead of, or additional to, the usual configuration file). This must still obey the above cascade.

• Allows option definitions in a single place to determine the parsing behaviour for configuration files and the command line.

• Unifies the parsed options into a single collection of option values for the rest of the program to access without caring where they came from.

Everything I need is apparently in the Python standard library, but they don't work together smoothly.

How can I achieve this with minimum deviation from the Python standard library?

Answer 1

The argparse module makes this not nuts, as long as you're happy with a config file that looks like command line. (I think this is an advantage, because users will only have to learn one syntax.) Setting fromfile_prefix_chars to, for example, @, makes it so that,

my_prog --foo=bar

is equivalent to

my_prog @baz.conf

if @baz.conf is,

--foo
bar

You can even have your code look for foo.conf automatically by modifying argv

if os.path.exists('foo.conf'):
    argv = ['@foo.conf'] + argv
args = argparser.parse_args(argv)

The format of these configuration files is modifiable by making a subclass of ArgumentParser and adding a convert_arg_line_to_args method.