How to add package data recursively in Python setup.py?

Question

I have a new library that has to include a lot of subfolders of small datafiles, and I'm trying to add them as package data. Imagine I have my library as so:

 library
    - foo.py
    - bar.py
 data
   subfolderA
      subfolderA1
      subfolderA2
   subfolderB
      subfolderB1 
      ...

I want to add all of the data in all of the subfolders through setup.py, but it seems like I manually have to go into every single subfolder (there are 100 or so) and add an init.py file. Furthermore, will setup.py find these files recursively, or do I need to manually add all of these in setup.py like:

package_data={
  'mypackage.data.folderA': ['*'],
  'mypackage.data.folderA.subfolderA1': ['*'],
  'mypackage.data.folderA.subfolderA2': ['*']
   },

I can do this with a script, but seems like a super pain. How can I achieve this in setup.py?

PS, the hierarchy of these folders is important because this is a database of material files and we want the file tree to be preserved when we present them in a GUI to the user, so it would be to our advantage to keep this file structure intact.

Answer 1

The problem with the glob answer is that it only does so much. I.e. it's not fully recursive. The problem with the copy_tree answer is that the files that are copied will be left behind on an uninstall.

The proper solution is a recursive one which will let you set the package_data parameter in the setup call.

I've written this small method to do this:

import os

def package_files(directory):
    paths = []
    for (path, directories, filenames) in os.walk(directory):
        for filename in filenames:
            paths.append(os.path.join('..', path, filename))
    return paths

extra_files = package_files('path_to/extra_files_dir')

setup(
    ...
    packages = ['package_name'],
    package_data={'': extra_files},
    ....
)

You'll notice that when you do a pip uninstall package_name, that you'll see your additional files being listed (as tracked with the package).