pytest - rerun failed test after all other tests

Question

In my scenario, I have one test that writes a file, and one (but potentially many more) tests that would want to read that file. I cannot simply extract writing that file to a function/fixture, because it involves some other fixtures that inside are starting some other binary and it that binary that writes that file. So I have a fixture, that checks if the file is already there.

What I tried so far:

• flaky and pytest-rerunfailures plugins - not suitable, as they both rerun the test instantly on failure (when the file still isn't there), and I want to append it at end of test queue.
• manually modifying the test queue, like this:

...

request.session.items.append(request.node)
pytest.xfail("file not present yet")

this kind of works, but only when I run on single runner (without xdist, or turning it on, by passing -n0 cli arg, in my test report I see something like this:

test_load_file_before_save xfail
test_save_file PASSED        
test_load_file PASSED        
test_load_file_before_save PASSED    

when run with xdist, the xfailed test does not get repeated. Anyone know how to proceed? Some way to force xdist refresh the test list?

Answer 1

You can use pytest.cache to get the test run status and append that test to the queue in case of failure.

if request.config.cache.get(request.node):
    request.session.items.append(request.node)
    pytest.xfail("file not present yet")

You can also set custom values in pytest cache to be used across different runs with request.config.cache.set(data,val).

If you are writing a file which is in the test directory, you can use --looponfail switch of pytest-xdist. It watches the directory and reruns the test until the tests pass. From documentation: distributed and subprocess testing: -f, --looponfail run tests in subprocess, wait for modified files and re-run failing test set until all pass.

Links that could be helpful: Pytest-cache

As a friendly suggestion, I would recommend you to make your tests independent of each other if you plan it to be run in parallel threads.