Discussion about how subtests failures should be displayed

[nicoddemus]

One more test case that is interesting.
This file:

import unittest

class T(unittest.TestCase):
    def test_fail(self):
        with self.subTest():
           self.assertEqual(1, 2)

No passing subtests, just one failure. Still shows PASSED.
I would expect FAILED and the bottom line to show 1 failed
Maybe pytest itself is calling pytest_runtest_logreport() at the end and causing an extra test to be counted.

pytest -v test_sub.py
=============================== test session starts ===============================
platform win32 -- Python 3.7.1, pytest-4.4.0, ...
plugins: subtests-0.2.0
collected 1 item

test_sub.py::T::test_fail FAILED                                         [100%]
test_sub.py::T::test_fail PASSED                                             [100%]

==================================== FAILURES =====================================
_____________________________ T.test_fail (<subtest>) _____________________________

self = <test_sub.T testMethod=test_fail>

    def test_fail(self):
        with self.subTest():
>          self.assertEqual(1, 2)
E          AssertionError: 1 != 2

test_sub.py:6: AssertionError
======================= 1 failed, 1 passed in 0.05 seconds ========================

Originally posted by @okken in #7 (comment)

cc @jurisbu @bskinn

[nicoddemus]

(Continuing the discussion here because the verbosity problem that started the discussion in #7 has been fixed).

Maybe pytest itself is calling pytest_runtest_logreport() at the end and causing an extra test to be counted.

Behind the scenes, what happens is:

1. Each exception inside the with self.subTest(): block is captured and reported via pytest_runtest_logreport(), with context information about the subtest (msg and kwargs parameters passed to self.subTest()).

2. When the test reaches the end, it triggers the normal pytest_runtest_logreport() hook. It will show up as "passed" because the exceptions raised in the with block by the subtests have been suppressed and reported at this point.

This is the same behavior we see with unittest. If we change your example slightly:

import unittest

class T(unittest.TestCase):
    def test_fail(self):
        with self.subTest():
           self.assertEqual(1, 2)

        self.assertEqual(1, 2)

if __name__ == '__main__':
    unittest.main()

We get:

λ python  .tmp\test_ut2.py
F
======================================================================
FAIL: test_fail (__main__.T) (<subtest>)
----------------------------------------------------------------------
Traceback (most recent call last):
  File ".tmp\test_ut2.py", line 6, in test_fail
    self.assertEqual(1, 2)
AssertionError: 1 != 2

======================================================================
FAIL: test_fail (__main__.T)
----------------------------------------------------------------------
Traceback (most recent call last):
  File ".tmp\test_ut2.py", line 8, in test_fail
    self.assertEqual(1, 2)
AssertionError: 1 != 2

----------------------------------------------------------------------
Ran 1 test in 0.001s

FAILED (failures=2)

So we have: 1 sub-test failure, and 1 normal failure. This is better seen here because we clearly see the failure context for the subtest ((<subtest>)).

We don't see 2 passes because unittest does not show "passed" tests when there are no failures:

.
----------------------------------------------------------------------
Ran 1 test in 0.000s

OK

So "passes" don't get special reporting, but behind the scenes the main test still passed, even if the subtest failed (they are two separate reports, just as with the subtests fixture).

As we have discussed in #7 (comment), perhaps after we have improved the output to get this:

.tmp/test_ut.py::test [custom message] (i=0) PASSED                                   
.tmp/test_ut.py::test [custom message] (i=1) FAILED                                   
.tmp/test_ut.py::test [custom message] (i=2) PASSED                                   
.tmp/test_ut.py::test [custom message] (i=3) FAILED                                   
.tmp/test_ut.py::test [custom message] (i=4) PASSED                                   
.tmp/test_ut.py::test PASSED                                   [100%]

Things won't be so confusing anymore?

To achieve this I will need further changes in pytest though (pytest-dev/pytest#5047).

[okken]

Well, that's definitely more clear and makes subtests more usable and less surprising.
There still is a bit of surprise, though, especially with the number of tests increasing, but the above option would be definitely better.

[bskinn]

Unfortunately, I think there's a lot here that needs untangling, some of which derives from (IMO undesirable) idiosyncrasy of unittest. Things get even hairier when you add xfails to the mix.

For these, MAIN BODY PASS means that there either were no asserts in the main function body, or all asserts there passed; MAIN BODY FAIL means a failing assert was present in the main body.

• Non-xfail unittest-style tests (without @unittest.expectedFailure)

  • unittest
    • SUBTEST FAIL: Report, as separate FAIL(s)
    • SUBTEST PASS: NO report
    • MAIN BODY PASS: NO report
    • MAIN BODY FAIL: Report, as separate fail
  • pytest
    • SUBTEST FAIL: Report as separate FAIL(s)
    • SUBTEST PASS: NO report
    • MAIN BODY PASS: Always report, as separate PASS
    • MAIN BODY FAIL: Always report, as separate FAIL

• Non-xfail pytest-style tests (subtests fixture, no @pytest.mark.xfail)

  • pytest (subtests fixture)
    • SUBTEST FAIL: Report, as separate FAIL(s)
    • SUBTEST PASS: Report, as separate PASS(es)
    • MAIN BODY PASS: Always report, as separate PASS
    • MAIN BODY FAIL: Always report, as separate FAIL

• unittest-style tests, WITH XFAIL (@unittest.expectedFailure)

  • unittest
    • SUBTEST FAIL: Report ENTIRE TEST, once, as XFAIL
    • SUBTEST PASS: Report ENTIRE TEST, once, as XPASS
  • pytest
    • SUBTEST FAIL: Report ENTIRE TEST, once, as XFAIL
    • SUBTEST PASS: Report ENTIRE TEST, once, as FAIL (not XPASS!)

• pytest-style tests (subtests fixture), WITH XFAIL (@pytest.mark.xfail)

  • pytest (subtests fixture)
    • SUBTESTS: Report each subtest outcome, separately, PASS or FAIL
    • ENTIRE TEST: Entire test reports XFAIL or XPASS based just on any asserts in the main body

---

Stuff used to figure out the above

pytest config:

[pytest]
markers =
 ...

addopts = --strict --doctest-glob="README.rst" -rsxX -p no:warnings

norecursedirs = .* env* src *.egg dist build

xfail_strict = False

Contents of tests\unittest_test.py:

import unittest

import pytest


class UT(unittest.TestCase):
    def test_fail_subtest_fail_test(self):
        with self.subTest("Fail"):
            self.assertFalse("SUBTEST")

        self.assertFalse("MAIN TEST")

    def test_fail_subtest_ok_test(self):
        with self.subTest("Fail"):
            self.assertFalse("SUBTEST")

        self.assertTrue("MAIN TEST")

    def test_fail_subtests_1x(self):
        with self.subTest("Pass"):
            self.assertTrue(True)

        with self.subTest("Fail"):
            self.assertFalse("SUBTEST 1")

    def test_fail_subtests_2x(self):
        with self.subTest("Pass"):
            self.assertTrue("PASS SUBTEST")

        with self.subTest("Fail 1"):
            self.assertFalse("SUBTEST 1")

        with self.subTest("Fail 2"):
            self.assertFalse("SUBTEST 2")

    @unittest.expectedFailure
    def test_xfail_subtests(self):
        with self.subTest(i=1):
            self.assertFalse("SUBTEST 1")

        with self.subTest(i=2):
            self.assertFalse("SUBTEST 2")

    @unittest.expectedFailure
    def test_xpass_subtests(self):
        with self.subTest(i=1):
            self.assertTrue("SUBTEST")


def test_pt_fail_subtest_fail_test(subtests):
    with subtests.test(msg="Fail"):
        assert not "SUBTEST"

    assert not "MAIN TEST"


def test_pt_fail_subtest_ok_test(subtests):
    with subtests.test(msg="Fail"):
        assert not "SUBTEST"

    assert "MAIN TEST"


def test_pt_fail_subtests_1x(subtests):
    with subtests.test(msg="Pass"):
        assert "PASS SUBTEST"

    with subtests.test(msg="Fail"):
        assert not "FAIL SUBTEST"


def test_pt_fail_subtests_2x(subtests):
    with subtests.test(msg="Pass"):
        assert "PASS SUBTEST"

    with subtests.test(msg="Fail 1"):
        assert not "FAIL SUBTEST 1"

    with subtests.test(msg="Fail 2"):
        assert not "FAIL SUBTEST 2"


@pytest.mark.xfail()
def test_pt_xfail_mixed_subtests_fail_main(subtests):
    with subtests.test(msg="Pass 1"):
        assert "PASS SUBTEST 1"

    with subtests.test(msg="Fail 2"):
        assert not "FAIL SUBTEST 2"

    assert not "MAIN TEST"


@pytest.mark.xfail()
def test_pt_xfail_mixed_subtests(subtests):
    with subtests.test(msg="Pass 1"):
        assert "PASS SUBTEST 1"

    with subtests.test(msg="Fail 2"):
        assert not "FAIL SUBTEST 2"


@pytest.mark.xfail()
def test_pt_xfail_subtests(subtests):
    with subtests.test(msg="Fail 1"):
        assert not "FAIL SUBTEST 1"

    with subtests.test(msg="Fail 2"):
        assert not "FAIL SUBTEST 2"


@pytest.mark.xfail
def test_pt_xpass_subtests(subtests):
    with subtests.test(msg="Fail"):
        assert "XPASS SUBTEST"


if __name__ == "__main__":
    unittest.main()

unittest execution:

>python tests\unittest_tests.py -v
test_fail_subtest_fail_test (__main__.UT) ... FAIL
test_fail_subtest_ok_test (__main__.UT) ... test_fail_subtests_1x (__main__.UT) ... test_fail_subtests_2x (__main__.UT) ... test_xfail_subtests (__main__.UT) ... expected failure
test_xpass_subtests (__main__.UT) ... unexpected success

======================================================================
FAIL: test_fail_subtest_fail_test (__main__.UT) [Fail]
----------------------------------------------------------------------
Traceback (most recent call last):
  File "tests\unittest_tests.py", line 9, in test_fail_subtest_fail_test
    self.assertFalse("SUBTEST")
AssertionError: 'SUBTEST' is not false

======================================================================
FAIL: test_fail_subtest_fail_test (__main__.UT)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "tests\unittest_tests.py", line 11, in test_fail_subtest_fail_test
    self.assertFalse("MAIN TEST")
AssertionError: 'MAIN TEST' is not false

======================================================================
FAIL: test_fail_subtest_ok_test (__main__.UT) [Fail]
----------------------------------------------------------------------
Traceback (most recent call last):
  File "tests\unittest_tests.py", line 15, in test_fail_subtest_ok_test
    self.assertFalse("SUBTEST")
AssertionError: 'SUBTEST' is not false

======================================================================
FAIL: test_fail_subtests_1x (__main__.UT) [Fail]
----------------------------------------------------------------------
Traceback (most recent call last):
  File "tests\unittest_tests.py", line 24, in test_fail_subtests_1x
    self.assertFalse("SUBTEST 1")
AssertionError: 'SUBTEST 1' is not false

======================================================================
FAIL: test_fail_subtests_2x (__main__.UT) [Fail 1]
----------------------------------------------------------------------
Traceback (most recent call last):
  File "tests\unittest_tests.py", line 31, in test_fail_subtests_2x
    self.assertFalse("SUBTEST 1")
AssertionError: 'SUBTEST 1' is not false

======================================================================
FAIL: test_fail_subtests_2x (__main__.UT) [Fail 2]
----------------------------------------------------------------------
Traceback (most recent call last):
  File "tests\unittest_tests.py", line 34, in test_fail_subtests_2x
    self.assertFalse("SUBTEST 2")
AssertionError: 'SUBTEST 2' is not false

----------------------------------------------------------------------
Ran 6 tests in 0.000s

FAILED (failures=6, expected failures=1, unexpected successes=1)

pytest execution of just the unittest-style tests:

>pytest -v --tb=line tests\unittest_tests.py -k UT
======================================= test session starts ========================================
platform win32 -- Python 3.6.3, pytest-4.4.0, py-1.8.0, pluggy-0.9.0 -- c:\temp\git\sphobjinv\env\scripts\python.exe
cachedir: .pytest_cache
rootdir: C:\Temp\git\sphobjinv, inifile: tox.ini
plugins: timeout-1.3.3, subtests-0.2.1, ordering-0.6, cov-2.6.1
collected 10 items / 4 deselected / 6 selected

tests/unittest_tests.py::UT::test_fail_subtest_fail_test FAILED          [ 16%]
tests/unittest_tests.py::UT::test_fail_subtest_fail_test FAILED                               [ 16%]
tests/unittest_tests.py::UT::test_fail_subtest_ok_test FAILED            [ 33%]
tests/unittest_tests.py::UT::test_fail_subtest_ok_test PASSED                                 [ 33%]
tests/unittest_tests.py::UT::test_fail_subtests_1x FAILED                [ 50%]
tests/unittest_tests.py::UT::test_fail_subtests_1x PASSED                                     [ 50%]
tests/unittest_tests.py::UT::test_fail_subtests_2x FAILED                [ 66%]
tests/unittest_tests.py::UT::test_fail_subtests_2x FAILED                [ 66%]
tests/unittest_tests.py::UT::test_fail_subtests_2x PASSED                                     [ 66%]
tests/unittest_tests.py::UT::test_xfail_subtests XFAIL                                        [ 83%]
tests/unittest_tests.py::UT::test_xpass_subtests FAILED                                       [100%]

============================================= FAILURES =============================================
C:\Temp\git\sphobjinv\tests\unittest_tests.py:9: AssertionError: 'SUBTEST' is not false
C:\Temp\git\sphobjinv\tests\unittest_tests.py:11: AssertionError: 'MAIN TEST' is not false
C:\Temp\git\sphobjinv\tests\unittest_tests.py:15: AssertionError: 'SUBTEST' is not false
C:\Temp\git\sphobjinv\tests\unittest_tests.py:24: AssertionError: 'SUBTEST 1' is not false
C:\Temp\git\sphobjinv\tests\unittest_tests.py:31: AssertionError: 'SUBTEST 1' is not false
C:\Temp\git\sphobjinv\tests\unittest_tests.py:34: AssertionError: 'SUBTEST 2' is not false
Unexpected success
===================================== short test summary info ======================================
XFAIL tests/unittest_tests.py::UT::test_xfail_subtests
  reason:
=================== 7 failed, 3 passed, 4 deselected, 1 xfailed in 0.21 seconds ====================

pytest execution of just the pytest-style tests:

>pytest -v --tb=line tests\unittest_tests.py -k _pt_
======================================= test session starts ========================================
platform win32 -- Python 3.6.3, pytest-4.4.0, py-1.8.0, pluggy-0.9.0 -- c:\temp\git\sphobjinv\env\scripts\python.exe
cachedir: .pytest_cache
rootdir: C:\Temp\git\sphobjinv, inifile: tox.ini
plugins: timeout-1.3.3, subtests-0.2.1, ordering-0.6, cov-2.6.1
collected 14 items / 6 deselected / 8 selected

tests/unittest_tests.py::test_pt_fail_subtest_fail_test FAILED                                [ 12%]
tests/unittest_tests.py::test_pt_fail_subtest_fail_test FAILED                                [ 12%]
tests/unittest_tests.py::test_pt_fail_subtest_ok_test FAILED                                  [ 25%]
tests/unittest_tests.py::test_pt_fail_subtest_ok_test PASSED                                  [ 25%]
tests/unittest_tests.py::test_pt_fail_subtests_1x PASSED                                      [ 37%]
tests/unittest_tests.py::test_pt_fail_subtests_1x FAILED                                      [ 37%]
tests/unittest_tests.py::test_pt_fail_subtests_1x PASSED                                      [ 37%]
tests/unittest_tests.py::test_pt_fail_subtests_2x PASSED                                      [ 50%]
tests/unittest_tests.py::test_pt_fail_subtests_2x FAILED                                      [ 50%]
tests/unittest_tests.py::test_pt_fail_subtests_2x FAILED                                      [ 50%]
tests/unittest_tests.py::test_pt_fail_subtests_2x PASSED                                      [ 50%]
tests/unittest_tests.py::test_pt_xfail_mixed_subtests_fail_main PASSED                        [ 62%]
tests/unittest_tests.py::test_pt_xfail_mixed_subtests_fail_main FAILED                        [ 62%]
tests/unittest_tests.py::test_pt_xfail_mixed_subtests_fail_main XFAIL                         [ 62%]
tests/unittest_tests.py::test_pt_xfail_mixed_subtests PASSED                                  [ 75%]
tests/unittest_tests.py::test_pt_xfail_mixed_subtests FAILED                                  [ 75%]
tests/unittest_tests.py::test_pt_xfail_mixed_subtests XPASS                                   [ 75%]
tests/unittest_tests.py::test_pt_xfail_subtests FAILED                                        [ 87%]
tests/unittest_tests.py::test_pt_xfail_subtests FAILED                                        [ 87%]
tests/unittest_tests.py::test_pt_xfail_subtests XPASS                                         [ 87%]
tests/unittest_tests.py::test_pt_xpass_subtests PASSED                                        [100%]
tests/unittest_tests.py::test_pt_xpass_subtests XPASS                                         [100%]

============================================= FAILURES =============================================
C:\Temp\git\sphobjinv\tests\unittest_tests.py:52: AssertionError: assert not 'SUBTEST'
C:\Temp\git\sphobjinv\tests\unittest_tests.py:54: AssertionError: assert not 'MAIN TEST'
C:\Temp\git\sphobjinv\tests\unittest_tests.py:59: AssertionError: assert not 'SUBTEST'
C:\Temp\git\sphobjinv\tests\unittest_tests.py:69: AssertionError: assert not 'FAIL SUBTEST'
C:\Temp\git\sphobjinv\tests\unittest_tests.py:77: AssertionError: assert not 'FAIL SUBTEST 1'
C:\Temp\git\sphobjinv\tests\unittest_tests.py:80: AssertionError: assert not 'FAIL SUBTEST 2'
C:\Temp\git\sphobjinv\tests\unittest_tests.py:89: AssertionError: assert not 'FAIL SUBTEST 2'
C:\Temp\git\sphobjinv\tests\unittest_tests.py:100: AssertionError: assert not 'FAIL SUBTEST 2'
C:\Temp\git\sphobjinv\tests\unittest_tests.py:106: AssertionError: assert not 'FAIL SUBTEST 1'
C:\Temp\git\sphobjinv\tests\unittest_tests.py:109: AssertionError: assert not 'FAIL SUBTEST 2'
===================================== short test summary info ======================================
XFAIL tests/unittest_tests.py::test_pt_xfail_mixed_subtests_fail_main
XPASS tests/unittest_tests.py::test_pt_xfail_mixed_subtests
XPASS tests/unittest_tests.py::test_pt_xfail_subtests
XPASS tests/unittest_tests.py::test_pt_xpass_subtests
============= 10 failed, 3 passed, 6 deselected, 1 xfailed, 3 xpassed in 0.19 seconds ==============

[bskinn]

Separately from the above reporting and unittest-compat logistics, I would like having the ability to switch on a mode where any failing subtest causes the encapsulating test also to fail. This IMO would greatly improve using xfails with subtests, because then I wouldn't have to care which subtest failed to trigger the xfail.

Alternatively (and/or additionally), it would be nice to have an xfail argument to subtests.test, allowing selective xfail-ing of subtests. This relates to a broader pain I've been feeling, where AFAIK pytest has no mechanism for selectively marking test functions as xfail ("@pytest.mark.xfail_if(...)", so to speak).

[nicoddemus]

(I'm short on time so I will read your post more carefully later @bskinn, thanks!)

This relates to a broader pain I've been feeling, where AFAIK pytest has no mechanism for selectively marking test functions as xfail ("@pytest.mark.xfail_if(...)", so to speak).

Just wanted to mention that xfail supports a condition argument for exactly that purpose. 😉

[bskinn]

🤦‍♂️ Thank you. 😃

[sterliakov]

Sorry for bumping an old issue, but I believe that there is one more factor that was not considered: currently subtests do not fail the whole test, breaking pytest --lf: you can't ask to "retry what failed last time", because all failed subtests will be ignored, and this is annoying.