Repository navigation
Fix DumpError when a report value is a str subclass - #1389
DawnofGenX wants to merge 3 commits into
Conversation
execnet serializes by exact type, so a str subclass -- notably a StrEnum member, which *is* a str -- raised ``DumpError: can't serialize <enum '...' >`` when it crossed the worker->controller channel. Report payloads can carry such values; a pytest-subtests message is one real way in. Coerce str subclasses to plain str in ``WorkerInteractor.sendevent``, the single choke point for worker->controller sends. Containers are walked so nested values are converted too. Tuple subclasses are rebuilt field-by-field (``type(value)(converted)`` would raise TypeError on a namedtuple, and report ``location`` is one), and anything execnet genuinely cannot handle -- a non-str Enum, a set -- is still passed through so its own error surfaces rather than being masked. Fixes pytest-dev#1161.
Two problems with the previous commit: - ``enum.StrEnum`` is 3.11+, and xdist supports 3.9, so the tests failed at import on the whole 3.9/3.10 CI matrix. The condition under test is "a str subclass", so the fixtures use ``class X(str, enum.Enum)`` instead, which exists on every supported version. - ``str()`` is the wrong conversion for such a class: mixing ``str`` with ``enum.Enum`` gives ``Enum.__str__``, so ``str(X.RED)`` is ``"X.RED"`` -- the value would have been replaced by its repr. Use ``str.__str__``, which always yields the underlying string data. (The f-string form differs between 3.11 and 3.12, so it cannot be used here either.) Covered by a test asserting ``str(_Colour.RED) != "red"``. Verified on CPython 3.9, 3.10 and 3.12.
pre-commit's mypy (strict, warn_unreachable) rejected three things: - a generator passed where a sequence was required, for the splatted namedtuple reconstruction; - ``type(value)(...)`` on a ``type[tuple]``, which mypy types as taking a single iterable. A namedtuple's __new__ takes positional fields, so a cast to Any is the accurate description of the call; - now-unnecessary ``type: ignore`` comments in the tests. The list and tuple branches are also split, which removes the shared list/tuple special case entirely.
|
The two failing jobs ( What fails These live at lines 1449-1600. This PR only appends to Reproduction Cloned this branch and ran
Master fails identically, so the branch is not the trigger. The only job that fails is the one pinned to pytest built from main; every released-pin job is green. Cause pytest turned @property
def nodeid(self) -> str:
return str(self._id)There is no setter. item._nodeid = f"{item.nodeid}@{'_'.join(sorted(gnames))}"That write now lands on an attribute nothing reads, so the Asking before touching anything The one-line fix is How would you like to handle it? Options as I see them:
For reference, the 22 passing checks here do include the two new tests from this PR, so the |
|
Closing due to inactivity — 6d idle, blocked upstream. Feel free to reopen if still wanted. |
|
🤖 Written by Claude Opus 5.5 via Claude Code for the pytest-xdist maintainers; I prompted it, it did the work, I read it. Thanks for the investigation. As I understand it, the right place for this fix is pytest, not xdist. pytest requires reports to be serializable (see the discussion in #1273), and execnet's serializer is intentionally limited to exact builtin types, so coercing every payload in pytest already handles the related cases:
The remaining gap is the native I started a branch for it in pytest: pytest-dev/pytest@main...claude/project-thread-pe7mtw. It converts Generated by Claude Code |
|
Acknowledged, and thanks for the detailed redirect — agreed the fix belongs in pytest, not xdist. I see you've opened pytest-dev/pytest#15134 for the |
2 participants
Fix
DumpErrorwhen a report value is astrsubclassCloses #1161.
execnetserializes by exact type, so astrsubclass — notably aStrEnummember, which is a
str— raisedwhen it crossed the worker→controller channel. The report payload can carry
such values; a
pytest-subtestsmessage is one real way in, and the issue'sreproducer (a
StrEnumused as a subtest message) still fails on currentmaster:The same test without
-npasses.The fix
WorkerInteractor.sendeventis the single choke point for worker→controllersends, so
strsubclasses are coerced to plainstrthere. Three details thatare easy to get wrong, and that the tests below pin:
report._subtest.context['msg']).type(value)(converted)raises
TypeError: NT.__new__() missing 1 required positional argumenton anamedtuple, and report
locationis one — my first version had exactly thatbug and the unit test caught it.
str.__str__(value), notstr(value). A class mixingstrwithenum.EnuminheritsEnum.__str__, sostr(X.RED)is"X.RED"— the valuewould have been replaced by its repr. The f-string form is not usable either,
since it differs between 3.11 and 3.12.
Values execnet genuinely cannot serialize — a non-
strEnum, aset— arepassed through untouched, so their own error still surfaces instead of being
masked by a lossy conversion.
Tests
testing/test_remote.py— unit tests for the conversion:strsubclass,nested dict/list/tuple, the
str()-vs-str.__str__trap, namedtuple round-trip,and the pass-through cases.
testing/acceptance_test.py— two end-to-end tests under-n 2: a reportcarrying a
str-subclass value now passes, and a report carrying a trulyunserializable value still fails the run, so the coercion cannot mask real bugs.
Both acceptance tests fail on
masterand pass here.Verified on CPython 3.9, 3.10 and 3.12: 11 targeted tests pass, and the full
suite shows the same 17 pre-existing failures as
master(TestLocking,test_looponfail,test_remote.TestWorkerInteractor::test_basic_collect_and_runtests),with 7 new tests added and no new failures.
The
py311-pytestmainjob also fails, but identically onmasterand on thisbranch (
TestGroupScope, 5 tests, caused by pytestmain) — not from thischange.