Leaderboard  /  pytest-10081

pytest-dev__pytest-10081

pytest-dev/pytest · v7.2  ·  15 steps · 150,095 tokens · 198s
resolved
1.00
⬤  Audit flag — broken-tests

3 of 64 test ids are corrupt - SWE-bench's log parser truncated them at a space, e.g. 'testing/test_unittest.py::test_setup_inheritance_skipping[test_setup_skip.py-1'. pytest aborts the whole run on an unknown id, so the task would falsely score 0; the harness quarantines them before scoring.

what does this mean? →

The task

The bug report the agent was given — plus the gold fix and hidden tests, shown here for reference. The agent saw only the problem statement.

Problem statementgithub issue
unittest.TestCase.tearDown executed for classes marked with `unittest.skip` when running --pdb
<!--
Thanks for submitting an issue!

Quick check-list while reporting bugs:
-->

- [x] a detailed description of the bug or problem you are having
- [x] output of `pip list` from the virtual environment you are using
- [x] pytest and operating system versions
- [x] minimal example if possible

Running `pytest --pdb` will run the `tearDown()` of `unittest.TestCase` classes that are decorated with `unittest.skip` on the class level.

Identical to #7215 , but with the `skip()` on the class level rather than on the function level.

Minimal test (adapted from #7215), `test_repro_skip_cla …
Gold patchheld back · the reference fix
diff --git a/src/_pytest/unittest.py b/src/_pytest/unittest.py--- a/src/_pytest/unittest.py+++ b/src/_pytest/unittest.py@@ -316,7 +316,10 @@ def runtest(self) -> None: # Arguably we could always postpone tearDown(), but this changes the moment where the # TestCase instance interacts with the results object, so better to only do it # when absolutely needed.- if self.config.getoption("usepdb") and not _is_skipped(self.obj):+ # We need to consider if the test itself is skipped, or the whole class.+ assert isinstance(self.parent, UnitTestCase)+ skipped = _is_skipped(self.obj) or _is_skipped(self.parent.obj)+ if self.config.getoption("usepdb") and not skipped: self._explicit_tearDown = self._testcase.tearDown setattr(self._testcase, "tearDown", lambda *args: None)
Hidden testtest_patch · applied only to grade
diff --git a/testing/test_unittest.py b/testing/test_unittest.py--- a/testing/test_unittest.py+++ b/testing/test_unittest.py@@ -1241,12 +1241,15 @@ def test_2(self): @pytest.mark.parametrize("mark", ["@unittest.skip", "@pytest.mark.skip"])-def test_pdb_teardown_skipped(+def test_pdb_teardown_skipped_for_functions( pytester: Pytester, monkeypatch: MonkeyPatch, mark: str ) -> None:- """With --pdb, setUp and tearDown should not be called for skipped tests."""+ """+ With --pdb, setUp and tearDown should not be called for tests skipped+ via a decorator (#7215).+ """ tracked: List[str] = []- monkeypatch.setattr(pytest, "test_pdb_teardown_skipped", tracked, raising=False)+ monkeypatch.setattr(pytest, "track_pdb_teardown_skipped", tracked, raising=False) pytester.makepyfile( """@@ -1256,10 +1259,10 @@ def test_pdb_teardown_skipped( class MyTestCase(unittest.TestCase): def setUp(self):- pytest.test_pdb_teardown_skipped.append("setUp:" + self.id())+ pytest.track_pdb_teardown_skipped.append("setUp:" + self.id()) def tearDown(self):- pytest.test_pdb_teardown_skipped.append("tearDown:" + self.id())+ pytest.track_pdb_teardown_skipped.append("tearDown:" + self.id()) {mark}("skipped for reasons") def test_1(self):@@ -1274,6 +1277,43 @@ def test_1(self): assert tracked == [] +@pytest.mark.parametrize("mark", ["@unittest.skip", "@pytest.mark.skip"])+def test_pdb_teardown_skipped_for_classes(+ pytester: Pytester, monkeypatch: MonkeyPatch, mark: str+) -> None:+ """+ With --pdb, setUp and tearDown should not be called for tests skipped+ via a decorator on the class (#10060).+ """+ tracked: List[str] = []+ monkeypatch.setattr(pytest, "track_pdb_teardown_skipped", tracked, raising=False)++ pytester.makepyfile(+ """+ import unittest+ import pytest++ {mark}("skipped for reasons")+ class MyTestCase(unittest.TestCase):++ def setUp(self):+ pytest.track_pdb_teardown_skipped.append("setUp:" + self.id())++ def tearDown(self):+ pytest.track_pdb_teardown_skipped.append("tearDown:" + self.id())++ def test_1(self):+ pass++ """.format(+ mark=mark+ )+ )+ result = pytester.runpytest_inprocess("--pdb")+ result.stdout.fnmatch_lines("* 1 skipped in *")+ assert tracked == []++ def test_async_support(pytester: Pytester) -> None: pytest.importorskip("unittest.async_case")
1FAIL → PASS
60PASS → PASS
61 / 61green

The agent run

The agent's step-by-step attempt — its reasoning, the tools it called, and the patch it produced.

System promptharness.agent.build_system_prompt
You are an AI coding assistant working inside a code repository that contains a bug. Investigate the code, find the bug, and fix it.

Your working directory is: /testbed
All four tools (read_file, write_file, edit_file, run_bash) already run from that directory. Use plain relative paths; do not prefix shell commands with cd. Fix the bug by editing source files only; do not create or edit test files.
Step 01 tool use
Let me understand the bug: When using `pytest --pdb` with `unittest.TestCase` classes that are decorated with `unittest.skip` at the class level, the `tearDown()` method is being executed even though the class is skipped. This shouldn't happen - if a test is skipped, neither `setUp` nor `tearDown` should be called. Let me investigate the pytest codebase to understand how this works. The issue is similar to #7215 but with class-level skip instead of function-level skip. Let me start by exploring the structure of the repository.
run_bash{ command: "ls /testbed" }
tool result · run_bash
exitcode: 0 stdout: AUTHORS CHANGELOG.rst CITATION CODE_OF_CONDUCT.md CONTRIBUTING.rst LICENSE OPENCOLLECTIVE.rst README.rst RELEASING.rst TIDELIFT.rst bench changelog codecov.yml doc extra pyproject.toml scripts setup.cfg setup.py src testing tox.ini stderr:
Step 02 tool use
Let's look at the pytest source code, particularly the unittest integration.
run_bash{ command: "ls /testbed/src/pytest/" }
tool result · run_bash
exitcode: 0 stdout: __init__.py __main__.py py.typed stderr:
Step 03 tool use
It seems the source structure is different. Let me check.
run_bash{ command: "find /testbed/src -name \"*.py\" | head -30" }
tool result · run_bash
exitcode: 0 stdout: /testbed/src/_pytest/timing.py /testbed/src/_pytest/fixtures.py /testbed/src/_pytest/pytester.py /testbed/src/_pytest/capture.py /testbed/src/_pytest/unraisableexception.py /testbed/src/_pytest/nodes.py /testbed/src/_pytest/outcomes.py /testbed/src/_pytest/python_path.py /testbed/src/_pytest/debugging.py /testbed/src/_pytest/hookspec.py /testbed/src/_pytest/_argcomplete.py /testbed/src/_pytest/tmpdir.py /testbed/src/_pytest/_code/source.py /testbed/src/_py …
Step 04 tool use
The source is in `/testbed/src/_pytest/`. Let me look at the unittest integration file and the debugging/pdb related code.
run_bash{ command: "find /testbed/src -name \"*.py\" | xargs grep -l \"unittest\\|TestCase\\|tearDown\\|setUp\" 2>/dev/null" }
tool result · run_bash
exitcode: 0 stdout: /testbed/src/_pytest/fixtures.py /testbed/src/_pytest/capture.py /testbed/src/_pytest/python.py /testbed/src/_pytest/nose.py /testbed/src/_pytest/config/__init__.py /testbed/src/_pytest/runner.py /testbed/src/_pytest/compat.py /testbed/src/_pytest/unittest.py /testbed/src/_pytest/python_api.py stderr:
Step 05 tool use
The key file is `/testbed/src/_pytest/unittest.py`. Let me read it.
read_file{ path: "/testbed/src/_pytest/unittest.py" }
tool result · read_file
"""Discover and run std-library "unittest" style tests.""" import sys import traceback import types from typing import Any from typing import Callable from typing import Generator from typing import Iterable from typing import List from typing import Optional from typing import Tuple from typing import Type from typing import TYPE_CHECKING from typing import Union import _pytest._code import pytest from _pytest.compat import getimfunc from _pytest.compat import is_async_func …
Step 06 tool use
Now I can see the issue. Let me look at the `runtest` method in `TestCaseFunction`: ```python def runtest(self) -> None: ... if self.config.getoption("usepdb") and not _is_skipped(self.obj): self._explicit_tearDown = self._testcase.tearDown setattr(self._testcase, "tearDown", lambda *args: None) ... ``` The check `not _is_skipped(self.obj)` is checking `self.obj`, which is the test method (e.g., `test_one`). When the skip is on the class level, `self.obj` (the method) doesn't have `__unittest_skip__`, only the class does. So `_is_skipped(self.obj)` returns `False`, and the code proceeds to replace `tearDown` with a no-op lambda. Wait, actually let me re-read the logic. The issue is: 1. When `--pdb` is given, the code postpones tearDown by storing the original tearDown in `self._explicit_tearDown` and replacing `self._testcase.tearDown` with a no-op lambda. 2. This is only done when `not _is_skipped(self.obj)` - i.e., when the test method itself is not skipped. 3. When the class is skipped, `self.obj` (the test method) is NOT marked as skipped, so the condition passes, and tearDown is replaced with a no-op. 4. But then, since the class is skipped, the test is skipped anyway, and... Wait, actually, the issue is the opposite. Let me re-read. The problem is: `tearDown` is being executed when it shouldn't be. Let me trace through the code: 1. The class `MyTestCase` is decorated with `@unittest.skip("hello")`. 2. In `UnitTestCase.collect()`, `_is_skipped(cls)` returns True (the class has `__unittest_skip__`). 3. Because it's skipped, `_inject_setup_teardown_fixtures` and `_inject_setup_class_fixture` are NOT called. 4. The test methods are still yielded. Now, in `TestCaseFunction.setup()`: - `self._testcase = self.parent.obj(self.name)` - this creates an instance of the test case class. In `TestCaseFunction.runtest()`: - `if self.config.getoption("usepdb") and not _is_skipped(self.obj):` - `self.obj` is the test method (e.g., `test_one`), which is NOT skipped. - So `self._explicit_tearDown = self._testcase.tearDown` and tearDown is replaced with a lambda. But wait, the skip is on the class. How does the skip actually get triggered? Looking at the fixture that's injected in `_make_xunit_fixture`: ```python def fixture(self, request: FixtureRequest) -> Generator[None, None, None]: if _is_skipped(self): reason = self.__unittest_skip_why__ raise pytest.skip.Exception(reason, _use_item_location=True) ... ``` But this fixture is only injected when the class is NOT skipped. Since the class IS skipped, the fixture is never injected. So where does the skip happen? Let me check the `TestCaseFunction` more carefully. Actually, looking at `pytest_runtest_makereport`, it handles `unittest.SkipTest`. But the skip is on the class, so when a test method is called on an instance of a skipped class... Wait, when `@unittest.skip` is applied to a class, what happens? The class gets `__unittest_skip__ = True` and `__unittest_skip_why__ = "hello"`. When you try to instantiate the class and call a test method, the standard unittest runner would check for this. But pytest is calling `self._testcase(result=self)` which calls `unittest.TestCase.__call__` which calls `self.run(result)`, and `TestCase.run` checks `__unittest_skip__` on the instance. Actually, let me check: when you create an instance of a class decorated with `@unittest.skip`, does the instance also have `__unittest_skip__`? Yes, because it's a class attribute. So `self._testcase` (the instance) has `__unittest_skip__ = True`. Now in `runtest()`: ```python if self.config.getoption("usepdb") and not _is_skipped(self.obj): self._explicit_tearDown = self._testcase.tearDown setattr(self._testcase, "tearDown", lambda *args: None) ``` `self.obj` is the test method (unbound), not the instance. The test method doesn't have `__unittest_skip__` because the skip is on the class. So the condition passes, and `self._testcase.t ... [truncated 7,450 characters]
Now I understand the issue. Let me trace through the code to confirm the bug.
read_file{ path: "/testbed/src/_pytest/unittest.py", offset: "225", limit: "35" }
tool result · read_file
ERROR: bad arguments for 'read_file': Toolbox.read_file() got an unexpected keyword argument 'offset'
Step 07 tool use
Let me read the file correctly.
read_file{ path: "/testbed/src/_pytest/unittest.py" }
tool result · read_file
"""Discover and run std-library "unittest" style tests.""" import sys import traceback import types from typing import Any from typing import Callable from typing import Generator from typing import Iterable from typing import List from typing import Optional from typing import Tuple from typing import Type from typing import TYPE_CHECKING from typing import Union import _pytest._code import pytest from _pytest.compat import getimfunc from _pytest.compat import is_async_func …
Step 08 tool use
Now I understand the bug clearly. Here's the flow: 1. A test class is decorated with `@unittest.skip("hello")`. This sets `__unittest_skip__` and `__unittest_skip_why__` on the class. 2. In `UnitTestCase.collect()`, the class is recognized as skipped, so fixtures are NOT injected. 3. `TestCaseFunction` items are still yielded for each test method. 4. In `TestCaseFunction.setup()`, an instance of the test class is created (`self._testcase = self.parent.obj(self.name)`). 5. In `TestCaseFunction.runtest()`: - The check `if self.config.getoption("usepdb") and not _is_skipped(self.obj)` is performed. - `self.obj` is the test *method* (e.g., `test_one`), which does NOT have `__unittest_skip__` because the skip is on the class, not on the method. - So the condition passes, and `self._explicit_tearDown` is set to the original `tearDown`, and `tearDown` on the instance is replaced with a no-op lambda. 6. Then `self._testcase(result=self)` is called, which eventually calls `unittest.TestCase.run()`. - `run()` checks `self.__class__.__unittest_skip__` → True. - It calls `result.addSkip(self, reason)` which raises `pytest.skip.Exception`. - The test is skipped, and the no-op `tearDown` is... well, it's not called by `run()` since it was skipped. 7. Then `TestCaseFunction.teardown()` is called: - `self._explicit_tearDown` is not None (it was set in step 5). - So it calls `self._explicit_tearDown()` which is the original `tearDown` method! - This executes `tearDown` even though the test was skipped. The fix: In `runtest()`, the condition should also check whether the test case class (or instance) is skipped. The check `not _is_skipped(self.obj)` only checks the test method. We need to also check the class. I can fix this by also checking `self._testcase` (the instance) or `self.parent.obj` (the class). Let me use `self._testcase` since it's already available at that point (asserted just above). Actually, looking more carefully at the code, the condition should be: "only postpone tearDown if the test is actually going to run." The test won't run if either: - The method is skipped (`_is_skipped(self.obj)`) - The class is skipped (`_is_skipped(self._testcase)` or `_is_skipped(self.parent.obj)`) Let me make the fix:
Now I can see the bug clearly. Let me trace the flow: 1. When a class is decorated with `@unittest.skip`, the `__unittest_skip__` attribute is set on the **class**, not on its methods. 2. In `TestCaseFunction.runtest()` (line ~253), the check `not _is_skipped(self.obj)` only checks the test **method** for `__unittest_skip__`. Since the skip is on the class, this returns `False` (meaning "not skipped"), and the code proceeds to postpone `tearDown` by storing it in `self._explicit_tearDown`. 3. When `self._testcase(result=self)` is called, `unittest.TestCase.run()` detects the class-level skip and raises `pytest.skip.Exception` without calling `tearDown`. 4. But then in `TestCaseFunction.teardown()`, `self._explicit_tearDown` is not `None`, so the original `tearDown` is executed — even though the test was skipped. The fix is to also check the test case instance (or its class) for the skip marker:
edit_file{ path: "/testbed/src/_pytest/unittest.py", old_string: " if self.config.getoption(\"usepdb\") and not _is_skipped(self.obj):\n self._explicit_tearDown = self._testcase.tearDown\n …", new_string: " if (\n self.config.getoption(\"usepdb\")\n and not _is_skipped(self.obj)\n and not _is_skipped(self._testcas…" }
tool result · edit_file
File /testbed/src/_pytest/unittest.py edited successfully.
Step 09 tool use
Now let me verify the fix by creating a test file and running it to confirm the bug is fixed. Let me also check if there are similar issues in the `_make_xunit_fixture` function. Looking at it: ```python def fixture(self, request: FixtureRequest) -> Generator[None, None, None]: if _is_skipped(self): reason = self.__unittest_skip_why__ raise pytest.skip.Exception(reason, _use_item_location=True) ... ``` This fixture is only injected when the class is NOT skipped (see `UnitTestCase.collect()`). So that path is fine. Now let me run the reproduction test.
Now let me verify the fix by creating and running the reproduction test:
run_bash{ command: "cd /testbed && cat > /tmp/test_repro_skip_class.py << 'EOF'\nimport unittest\n\n@unittest.skip(\"hello\")\nclass MyTestCase(unittest.TestCase):\n def setUp(self):\n …" }
tool result · run_bash
exitcode: 0 stdout: ============================= test session starts ============================== platform linux -- Python 3.9.20, pytest-7.2.0.dev173+gda9a2b584, pluggy-0.13.1 rootdir: /tmp collected 1 item ../tmp/test_repro_skip_class.py s [100%] ============================== 1 skipped in 0.01s ============================== stderr:
Step 10 tool use
The test is properly skipped without entering pdb or calling tearDown. The fix works. Let me also run the existing test suite to make sure nothing is broken.
The test is now properly skipped without entering PDB or executing `tearDown`. Let me also run the existing unittest-related tests to make sure we haven't broken anything:
run_bash{ command: "cd /testbed && python -m pytest testing/test_unittest.py -x -q 2>&1 | tail -20" }
tool result · run_bash
exitcode: 0 stdout: ......................sssssss...............................s........s. [100%] 62 passed, 9 skipped in 1.82s stderr:
Step 11 tool use
All 62 tests pass (9 skipped). Let me also check if there are any tests specifically related to this issue — maybe tests for the --pdb interaction with unittest skips.
All 62 tests pass. Let me also verify that the original issue #7215 scenario (method-level skip with `--pdb`) still works correctly:
run_bash{ command: "cd /testbed && cat > /tmp/test_repro_skip_method.py << 'EOF'\nimport unittest\n\nclass MyTestCase(unittest.TestCase):\n @unittest.skip(\"hello\")\n def test_one(…" }
tool result · run_bash
exitcode: 0 stdout: ============================= test session starts ============================== platform linux -- Python 3.9.20, pytest-7.2.0.dev173+gda9a2b584, pluggy-0.13.1 rootdir: /tmp collected 1 item ../tmp/test_repro_skip_method.py s [100%] ============================== 1 skipped in 0.01s ============================== stderr:
Step 12 tool use
Both method-level and class-level skips work correctly now. Let me also check if there are any tests in the test suite that specifically test the `--pdb` + unittest skip interaction:
Both class-level and method-level skips with `--pdb` now work correctly. Let me also check if there are any existing tests covering the `--pdb` + skip interaction to make sure they still pass:
run_bash{ command: "cd /testbed && grep -n \"pdb\\|tearDown\\|_explicit_tearDown\\|_is_skipped\" testing/test_unittest.py | head -20" }
tool result · run_bash
exitcode: 0 stdout: 96: def tearDownModule(): 118: def tearDownModule(): 177: def tearDown(self): 229: def tearDownClass(self): 242: def tearDown(self): 252: "*tearDown*", 295: def tearDownClass(cls): 316: def tearDownClass(cls): 599: def test_trial_pdb(self, pytester: Pytester) -> None: 606: assert 0, "hellopdb" 610: child.expect("hellopdb") 683: def tearDo …
Step 13 tool use
Let me check the test `test_pdb_teardown_called` at line 1208:
read_file{ path: "/testbed/testing/test_unittest.py", offset: "1208", limit: "60" }
tool result · read_file
ERROR: bad arguments for 'read_file': Toolbox.read_file() got an unexpected keyword argument 'offset'
Step 14 tool use
Hmm, the read_file doesn't support offset. Let me just use run_bash to extract lines.
run_bash{ command: "cd /testbed && sed -n '1208,1270p' testing/test_unittest.py" }
tool result · run_bash
exitcode: 0 stdout: def test_pdb_teardown_called(pytester: Pytester, monkeypatch: MonkeyPatch) -> None: """Ensure tearDown() is always called when --pdb is given in the command-line. We delay the normal tearDown() calls when --pdb is given, so this ensures we are calling tearDown() eventually to avoid memory leaks when using --pdb. """ teardowns: List[str] = [] monkeypatch.setattr( pytest, "test_pdb_teardown_called_teardowns", teardowns, raisi …
Step 15 tool use
Let me see the rest of that test:
run_bash{ command: "cd /testbed && sed -n '1270,1295p' testing/test_unittest.py" }
tool result · run_bash
exitcode: 0 stdout: ) ) result = pytester.runpytest_inprocess("--pdb") result.stdout.fnmatch_lines("* 1 skipped in *") assert tracked == [] def test_async_support(pytester: Pytester) -> None: pytest.importorskip("unittest.async_case") pytester.copy_example("unittest/test_unittest_asyncio.py") reprec = pytester.inline_run() reprec.assertoutcome(failed=1, passed=2) def test_asynctest_support(pytester: Pytester) -> None: """Check …
Agent diffcaptured before grading
diff --git a/src/_pytest/unittest.py b/src/_pytest/unittest.pyindex 851e4943b..586b2fc78 100644--- a/src/_pytest/unittest.py+++ b/src/_pytest/unittest.py@@ -316,7 +316,11 @@ class TestCaseFunction(Function): # Arguably we could always postpone tearDown(), but this changes the moment where the # TestCase instance interacts with the results object, so better to only do it # when absolutely needed.- if self.config.getoption("usepdb") and not _is_skipped(self.obj):+ if (+ self.config.getoption("usepdb")+ and not _is_skipped(self.obj)+ and not _is_skipped(self._testcase)+ ): self._explicit_tearDown = self._testcase.tearDown setattr(self._testcase, "tearDown", lambda *args: None)