coverage.py does not cover script if py.test executes it from another directory

Question

I got a python script which takes command line arguments, working with some files. I'm writing succeeding tests with py.test putting this script through its paces, executing it with subprocess.call.

Now I want to analyze code coverage with coverage.py. Coverage, when used via the pytest-cov plugin (which has subprocess-handling built-in), does not see/cover my script when it is called from a temporary testing directory created with py.test's tmpdir fixture. Coverage does see my script when it's called in the directory it resides in (and the filename argument points to a remote path).

In both situations, my tests pass! Coverage 3.6, pytest-2.3.5, pytest-cov 1.6, all from PyPi.

Question: How can I get coverage to recognize my script even if it's executed in another directory? Is this a bug in coverage, or something which is just not possible to do? Would be surprised if the latter, after all, tmpdir is a stock mechanism of py.test...

Minimal example:

I got a script my_script.py which just echoes the contents of a file arg_file.txt supplied via command-line argument. In two different tests, this is once called in a tmpdir, and once in the script's location. Both tests pass, but the in the tmpdir test, I get no coverage information!

Test run:

~/pytest_experiment$ py.test -s
=================================== test session starts ====================================
platform linux2 -- Python 2.7.4 -- pytest-2.3.5
plugins: cov
collected 2 items 

tests/test_in_scriptdir.py 
set_up: In directory /tmp/pytest-52/test_10
Running in directory /home/cbuchner/pytest_experiment
Command: ./my_script.py /tmp/pytest-52/test_10/arg_file.txt
--Contents of arg_file.txt--

.
tests/test_in_tmpdir.py 
set_up: In directory /tmp/pytest-52/test_11
Running in directory /tmp/pytest-52/test_11
Command: /home/cbuchner/pytest_experiment/my_script.py arg_file.txt
--Contents of arg_file.txt--

.

================================= 2 passed in 0.06 seconds =================================

Coverage:

~/pytest_experiment$ py.test --cov=my_script.py tests/test_in_scriptdir.py=================================== test session starts ====================================
platform linux2 -- Python 2.7.4 -- pytest-2.3.5
plugins: cov
collected 1 items 

tests/test_in_scriptdir.py .
--------------------- coverage: platform linux2, python 2.7.4-final-0 ----------------------
Name        Stmts   Miss  Cover
-------------------------------
my_script       3      0   100%

================================= 1 passed in 0.09 seconds =================================
~/pytest_experiment$ py.test --cov=my_script.py tests/test_in_tmpdir.py=================================== test session starts ====================================
platform linux2 -- Python 2.7.4 -- pytest-2.3.5
plugins: cov
collected 1 items 

tests/test_in_tmpdir.py .Coverage.py warning: No data was collected.

--------------------- coverage: platform linux2, python 2.7.4-final-0 ----------------------
Name    Stmts   Miss  Cover
---------------------------

================================= 1 passed in 0.09 seconds =================================

The files are here: https://gist.github.com/bilderbuchi/6412754

Edit: Interstingly, when running the coverage tests with -s, too, there's more curious output - coverage warns that No data was collected, when obviously it was collected, and in the tmpdir test warns that Module my_script.py was never imported.??

~/pytest_experiment$ py.test -s --cov=my_script.py tests/test_in_scriptdir.py
=================================== test session starts ====================================
platform linux2 -- Python 2.7.4 -- pytest-2.3.5
plugins: cov
collected 1 items 

tests/test_in_scriptdir.py 
set_up: In directory /tmp/pytest-63/test_10
Running in directory /home/cbuchner/pytest_experiment
Command: ./my_script.py /tmp/pytest-63/test_10/arg_file.txt
--Contents of arg_file.txt--

Coverage.py warning: No data was collected.
.
--------------------- coverage: platform linux2, python 2.7.4-final-0 ----------------------
Name        Stmts   Miss  Cover
-------------------------------
my_script       3      0   100%

================================= 1 passed in 0.09 seconds =================================
~/pytest_experiment$ py.test -s --cov=my_script.py tests/test_in_tmpdir.py=================================== test session starts ====================================
platform linux2 -- Python 2.7.4 -- pytest-2.3.5
plugins: cov
collected 1 items 

tests/test_in_tmpdir.py 
set_up: In directory /tmp/pytest-64/test_10
Running in directory /tmp/pytest-64/test_10
Command: /home/cbuchner/pytest_experiment/my_script.py arg_file.txt
--Contents of arg_file.txt--

Coverage.py warning: Module my_script.py was never imported.
Coverage.py warning: No data was collected.
Coverage.py warning: Module my_script.py was never imported.
Coverage.py warning: No data was collected.
.Coverage.py warning: No data was collected.

--------------------- coverage: platform linux2, python 2.7.4-final-0 ----------------------
Name    Stmts   Miss  Cover
---------------------------

================================= 1 passed in 0.09 seconds =================================

Answer 1

I encountered the same issue when calling "py.test --cov ..." from tox. I found a hint on this page: http://blog.ionelmc.ro/2014/05/25/python-packaging/ even though it does not mention this explicitly. Using "--develop" for tox will make sure that coverage data gathering is called from the same directory as coverage analysis. This section in tox.ini made it work for me to have a test environment for coverage:

[tox]
envlist = ...,py34,cov

[testenv:cov]
# necessary to make cov find the .coverage file
# see http://blog.ionelmc.ro/2014/05/25/python-packaging/
usedevelop = true
commands = py.test --cov=<MODULE_NAME>
deps = pytest pytest-cov