How do I handle multiple asserts within a single Python unittest?

Question

This is a problem that came up when performing a single test that had multiple independent failure modes, due to having multiple output streams. I also wanted to show the results of asserting the data on all those modes, regardless of which failed first. Python's unittest has no such feature outside of using a Suite to represent the single test, which was unacceptable since my single test always needed to be run as a single unit; it just doesn't capture the nature of the thing.

A practical example is testing an object that also generates a log. You want to assert the output of it's methods, but you also want to assert the log output. The two outputs require different tests, which can be neatly expressed as two of the stock asserts expressions, but you also don't want the failure of one to hide the possible failure of the other within the test. So you really need to test both at the same time.

I cobbled together this useful little widget to solve my problem.

def logFailures(fnList):
    failurelog = []
    for fn in fnList:
        try:
            fn()
        except AssertionError as e:
            failurelog.append("\nFailure %d: %s" % (len(failurelog)+1,str(e)))

    if len(failurelog) != 0:
        raise AssertionError(
            "%d failures within test.\n %s" % (len(failurelog),"\n".join(failurelog))
        )

Which is used like so:

def test__myTest():
    # do some work here
    logFailures([
        lambda: assert_(False,"This test failed."),
        lambda: assert_(False,"This test also failed."),
    ])

The result is that logFailures() will raise an exception that contains a log of all the assertions that were raised in methods within the list.

The question: While this does the job, I'm left wondering if there's a better way to handle this, other than having to go to the length of creating nested suites of tests and so forth?

Answer 1

With using a subtest, execution would not stop after the first failure https://docs.python.org/3/library/unittest.html#subtests

Here is example with two fail asserts:

class TestMultipleAsserts(unittest.TestCase):

    def test_multipleasserts(self):
        with self.subTest():
            self.assertEqual(1, 0)
        with self.subTest():
            self.assertEqual(2, 0)

Output will be:

======================================================================
FAIL: test_multipleasserts (__main__.TestMultipleAsserts) (<subtest>)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "./test.py", line 9, in test_multipleasserts
    self.assertEqual(1, 0)
AssertionError: 1 != 0

======================================================================
FAIL: test_multipleasserts (__main__.TestMultipleAsserts) (<subtest>)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "./test.py", line 11, in test_multipleasserts
    self.assertEqual(2, 0)
AssertionError: 2 != 0

----------------------------------------------------------------------
Ran 1 test in 0.000s

FAILED (failures=2)

You can easy wrap subtest as following

class MyTestCase(unittest.TestCase):
    def expectEqual(self, first, second, msg=None):
        with self.subTest():
            self.assertEqual(first, second, msg)

class TestMA(MyTestCase):
    def test_ma(self):
        self.expectEqual(3, 0)
        self.expectEqual(4, 0)