Package only binary compiled .so files of a python library compiled with Cython

Question

I have a package named mypack which inside has a module mymod.py, and the __init__.py. For some reason that is not in debate, I need to package this module compiled (nor .py or .pyc files are allowed). That is, the __init__.py is the only source file allowed in the distributed compressed file.

The folder structure is:

. 
│  
├── mypack
│   ├── __init__.py
│   └── mymod.py
├── setup.py

I find that Cython is able to do this, by converting each .py file in a .so library that can be directly imported with python.

The question is: how the setup.py file must be in order to allow an easy packaging and installation?

The target system has a virtualenv where the package must be installed with whatever method that allows easy install and uninstall (easy_install, pip, etc are all welcome).

I tried all that was at my reach. I read setuptools and distutils documentation, all stackoverflow related questions, and tried with all kind of commands (sdist, bdist, bdist_egg, etc), with lots of combinations of setup.cfg and MANIFEST.in file entries.

The closest I got was with the below setup file, that would subclass the bdist_egg command in order to remove also .pyc files, but that is breaking the installation.

A solution that installs "manually" the files in the venv is also good, provided that all ancillary files that are included in a proper installation are covered (I need to run pip freeze in the venv and see mymod==0.0.1).

Run it with:

python setup.py bdist_egg --exclude-source-files

and (try to) install it with

easy_install mymod-0.0.1-py2.7-linux-x86_64.egg

As you may notice, the target is linux 64 bits with python 2.7.

from Cython.Distutils import build_ext
from setuptools import setup, find_packages
from setuptools.extension import Extension
from setuptools.command import bdist_egg
from setuptools.command.bdist_egg import  walk_egg, log 
import os

class my_bdist_egg(bdist_egg.bdist_egg):

    def zap_pyfiles(self):
        log.info("Removing .py files from temporary directory")
        for base, dirs, files in walk_egg(self.bdist_dir):
            for name in files:
                if not name.endswith('__init__.py'):
                    if name.endswith('.py') or name.endswith('.pyc'):
                        # original 'if' only has name.endswith('.py')
                        path = os.path.join(base, name)
                        log.info("Deleting %s",path)
                        os.unlink(path)

ext_modules=[
    Extension("mypack.mymod", ["mypack/mymod.py"]),
]

setup(
  name = 'mypack',
  cmdclass = {'build_ext': build_ext, 
              'bdist_egg': my_bdist_egg },
  ext_modules = ext_modules,
  version='0.0.1',
  description='This is mypack compiled lib',
  author='Myself',
  packages=['mypack'],
)

UPDATE. Following @Teyras answer, it was possible to build a wheel as requested in the answer. The setup.py file contents are:

import os
import shutil
from setuptools.extension import Extension
from setuptools import setup
from Cython.Build import cythonize
from Cython.Distutils import build_ext

class MyBuildExt(build_ext):
    def run(self):
        build_ext.run(self)
        build_dir = os.path.realpath(self.build_lib)
        root_dir = os.path.dirname(os.path.realpath(__file__))
        target_dir = build_dir if not self.inplace else root_dir
        self.copy_file('mypack/__init__.py', root_dir, target_dir)

    def copy_file(self, path, source_dir, destination_dir):
        if os.path.exists(os.path.join(source_dir, path)):
            shutil.copyfile(os.path.join(source_dir, path), 
                            os.path.join(destination_dir, path))


setup(
  name = 'mypack',
  cmdclass = {'build_ext': MyBuildExt},
  ext_modules = cythonize([Extension("mypack.*", ["mypack/*.py"])]),
  version='0.0.1',
  description='This is mypack compiled lib',
  author='Myself',
  packages=[],
  include_package_data=True )

The key point was to set packages=[],. The overwriting of the build_ext class run method was needed to get the __init__.py file inside the wheel.

Answer 1

While packaging as a wheel is definitely what you want, the original question was about excluding .py source files from the package. This is addressed in Using Cython to protect a Python codebase by @Teyras, but his solution uses a hack: it removes the packages argument from the call to setup(). This prevents the build_py step from running which does, indeed, exclude the .py files but it also excludes any data files you want included in the package. (For example my package has a data file called VERSION which contains the package version number.) A better solution would be replacing the build_py setup command with a custom command which only copies the data files.

You also need the __init__.py file as described above. So the custom build_py command should create the __init_.py file. I found that the compiled __init__.so runs when the package is imported so all that is needed is an empty __init__.py file to tell Python that the directory is a module which is ok to import.

Your custom build_py class would look like:

import os
from setuptools.command.build_py import build_py

class CustomBuildPyCommand(build_py):
    def run(self):
        # package data files but not .py files
        build_py.build_package_data(self)
        # create empty __init__.py in target dirs
        for pdir in self.packages:
            open(os.path.join(self.build_lib, pdir, '__init__.py'), 'a').close()

And configure setup to override the original build_py command:

setup(
   ...
   cmdclass={'build_py': CustomBuildPyCommand},
)