diff --git a/Doc/c-api/gcsupport.rst b/Doc/c-api/gcsupport.rst index fed795b1e8c963..f6ce2c52b2e9b3 100644 --- a/Doc/c-api/gcsupport.rst +++ b/Doc/c-api/gcsupport.rst @@ -220,54 +220,223 @@ The :c:member:`~PyTypeObject.tp_traverse` handler accepts a function parameter o detection; it's not expected that users will need to write their own visitor functions. -The :c:member:`~PyTypeObject.tp_traverse` handler must have the following type: +The :c:member:`~PyTypeObject.tp_clear` handler must be of the :c:type:`inquiry` type, or ``NULL`` +if the object is immutable. +.. c:type:: int (*inquiry)(PyObject *self) + + Drop references that may have created reference cycles. Immutable objects + do not have to define this method since they can never directly create + reference cycles. Note that the object must still be valid after calling + this method (don't just call :c:func:`Py_DECREF` on a reference). The + collector will call this method if it detects that this object is involved + in a reference cycle. + + +.. _gc-traversal: + +Traversal +--------- + +The :c:member:`~PyTypeObject.tp_traverse` handler must have the following type: + .. c:type:: int (*traverseproc)(PyObject *self, visitproc visit, void *arg) - Traversal function for a container object. Implementations must call the + Traversal function for a garbage-collected object, used by the garbage + collector to detect reference cycles. + Implementations must call the *visit* function for each object directly contained by *self*, with the parameters to *visit* being the contained object and the *arg* value passed to the handler. The *visit* function must not be called with a ``NULL`` - object argument. If *visit* returns a non-zero value that value should be + object argument. If *visit* returns a non-zero value, that value should be returned immediately. - The traversal function must not have any side effects. Implementations - may not modify the reference counts of any Python objects nor create or - destroy any Python objects. + A typical :c:member:`!tp_traverse` function calls the :c:func:`Py_VISIT` + convenience macro on each of the instance's members that are Python + objects that the instance owns. + For example, this is a (slightly outdated) traversal function for + the :py:class:`threading.local` class:: + + static int + local_traverse(PyObject *op, visitproc visit, void *arg) + { + localobject *self = (localobject *) op; + Py_VISIT(Py_TYPE(self)); + Py_VISIT(self->args); + Py_VISIT(self->kw); + Py_VISIT(self->dict); + return 0; + } + + .. note:: + :c:func:`Py_VISIT` requires the *visit* and *arg* parameters to + :c:func:`!local_traverse` to have these specific names; don't name them just + anything. + + Instances of :ref:`heap-allocated types ` hold a reference to + their type. Their traversal function must therefore visit the type:: + + Py_VISIT(Py_TYPE(self)); + + Alternately, the type may delegate this responsibility by + calling ``tp_traverse`` of a heap-allocated superclass (or another + heap-allocated type, if applicable). + If they do not, the type object may not be garbage-collected. + + If the :c:macro:`Py_TPFLAGS_MANAGED_DICT` bit is set in the + :c:member:`~PyTypeObject.tp_flags` field, the traverse function must call + :c:func:`PyObject_VisitManagedDict` like this:: + + PyObject_VisitManagedDict((PyObject*)self, visit, arg); + + Only the members that the instance *owns* (by having + :term:`strong references ` to them) must be + visited. For instance, if an object supports weak references via the + :c:member:`~PyTypeObject.tp_weaklist` slot, the pointer supporting + the linked list (what *tp_weaklist* points to) must **not** be + visited as the instance does not directly own the weak references to itself. + + The traversal function has a limitation: + + .. warning:: + + The traversal function must not have any side effects. Implementations + may not modify the reference counts of any Python objects nor create or + destroy any Python objects, directly or indirectly. + + This means that *most* Python C API functions may not be used, since + they can raise a new exception, return a new reference to a result object, + have internal logic that uses side effects. + Also, unless documented otherwise, functions that happen to not have side + effects may start having them in future versions, without warning. + + For a list of safe functions, see a + :ref:`separate section ` below. + + .. note:: + + The :c:func:`Py_VISIT` call may be skipped for those members that provably + cannot participate in reference cycles. + In the ``local_traverse`` example above, there is also a ``self->key`` + member, but it can only be ``NULL`` or a Python string and therefore + cannot be part of a reference cycle. -To simplify writing :c:member:`~PyTypeObject.tp_traverse` handlers, a :c:func:`Py_VISIT` macro is -provided. In order to use this macro, the :c:member:`~PyTypeObject.tp_traverse` implementation -must name its arguments exactly *visit* and *arg*: + On the other hand, even if you know a member can never be part of a cycle, + as a debugging aid you may want to visit it anyway just so the :mod:`gc` + module's :func:`~gc.get_referents` function will include it. + .. note:: + + The :c:member:`~PyTypeObject.tp_traverse` function can be called from any + thread. + + .. versionchanged:: 3.9 + + Heap-allocated types are expected to visit ``Py_TYPE(self)`` in + ``tp_traverse``. In earlier versions of Python, due to + `bug 40217 `_, doing this + may lead to crashes in subclasses. + +To simplify writing :c:member:`~PyTypeObject.tp_traverse` handlers, +a :c:func:`Py_VISIT` macro is provided. +In order to use this macro, the :c:member:`~PyTypeObject.tp_traverse` +implementation must name its arguments exactly *visit* and *arg*: .. c:macro:: Py_VISIT(o) - If the :c:expr:`PyObject *` *o* is not ``NULL``, call the *visit* callback, with arguments *o* - and *arg*. If *visit* returns a non-zero value, then return it. - Using this macro, :c:member:`~PyTypeObject.tp_traverse` handlers - look like:: + If the :c:expr:`PyObject *` *o* is not ``NULL``, call the *visit* + callback, with arguments *o* and *arg*. + If *visit* returns a non-zero value, then return it. - static int - my_traverse(Noddy *self, visitproc visit, void *arg) - { - Py_VISIT(self->foo); - Py_VISIT(self->bar); - return 0; - } + This corresponds roughly to:: -The :c:member:`~PyTypeObject.tp_clear` handler must be of the :c:type:`inquiry` type, or ``NULL`` -if the object is immutable. + #define Py_VISIT(o) \ + if (op) { \ + int visit_result = visit(o, arg); \ + if (visit_result != 0) { \ + return visit_result; \ + } \ + } +.. _durniggc-functions: -.. c:type:: int (*inquiry)(PyObject *self) +Traversal-safe functions +^^^^^^^^^^^^^^^^^^^^^^^^ - Drop references that may have created reference cycles. Immutable objects - do not have to define this method since they can never directly create - reference cycles. Note that the object must still be valid after calling - this method (don't just call :c:func:`Py_DECREF` on a reference). The - collector will call this method if it detects that this object is involved - in a reference cycle. +The following functions and macros are safe to use in a +:c:member:`~PyTypeObject.tp_traverse` handler: + +* the *visit* function passed to ``tp_traverse`` +* :c:func:`Py_VISIT` +* :c:func:`Py_SIZE` +* :c:func:`Py_TYPE` +* :c:func:`PyObject_VisitManagedDict` +* :c:func:`PyObject_TypeCheck`, :c:func:`PyType_IsSubtype`, + :c:func:`PyType_HasFeature` +* :samp:`Py{}_Check` and :samp:`Py{}_CheckExact` -- for example, + :c:func:`PyTuple_Check` + +The following functions should *only* used in a +:c:member:`~PyTypeObject.tp_traverse` handler; calling them in other +contexts may have unintended consequences: + +.. c:function:: void *PyObject_GetTypeData_DuringGC(PyObject *o, PyTypeObject *cls) + void *PyObject_GetItemData_DuringGC(PyObject *o) + void *PyType_GetModuleState_DuringGC(PyTypeObject *type) + void *PyModule_GetState_DuringGC(PyObject *module) + int PyModule_GetToken_DuringGC(PyObject *module, void** result) + + These functions act like their counterparts without the ``_DuringGC`` suffix, + but they are guaranteed to not have side effects, and they do not + set an exception on failure. + + Note that these functions may fail (return ``NULL`` or -1). + Only creating and setting the exception is suppressed. + + .. versionadded:: next + + .. seealso:: + + :c:func:`PyObject_GetTypeData`, + :c:func:`PyObject_GetItemData`, + :c:func:`PyType_GetModuleState`, + :c:func:`PyModule_GetState`, + :c:func:`PyModule_GetToken`, + :c:func:`PyType_GetBaseByToken` + +.. c:function:: int PyType_GetBaseByToken_DuringGC(PyTypeObject *type, void *tp_token, PyTypeObject **result) + + Acts like :c:func:`PyType_GetBaseByToken`, + but is guaranteed to not have side effects, does not + set an exception on failure, and sets *\*result* to + a :term:`borrowed reference` rather than a strong one. + The reference is valid for the duration + of the :c:member:`!tp_traverse` handler call. + + Note that this function may fail (return -1). + Only creating and setting the exception is suppressed. + + .. versionadded:: next + +.. c:function:: PyObject* PyType_GetModule_DuringGC(PyTypeObject *type) + PyObject* PyType_GetModuleByToken_DuringGC(PyTypeObject *type, const void *mod_token) + + These functions act like their counterparts without the ``_DuringGC`` suffix, + but they are guaranteed to not have side effects, they never set an + exception on failure, and they return a :term:`borrowed reference`. + The returned reference is valid for the duration + of the :c:member:`!tp_traverse` handler call. + + Note that these functions may fail (return ``NULL``). + Only creating and setting the exception is suppressed. + + .. versionadded:: next + + .. seealso:: + + :c:func:`PyType_GetModule`, + :c:func:`PyType_GetModuleByToken` Controlling the Garbage Collector State diff --git a/Doc/c-api/typeobj.rst b/Doc/c-api/typeobj.rst index 87b488912653b9..ac4e892d390ebf 100644 --- a/Doc/c-api/typeobj.rst +++ b/Doc/c-api/typeobj.rst @@ -1563,93 +1563,9 @@ and :c:data:`PyType_Type` effectively act as defaults.) .. corresponding-type-slot:: Py_tp_traverse An optional pointer to a traversal function for the garbage collector. This is - only used if the :c:macro:`Py_TPFLAGS_HAVE_GC` flag bit is set. The signature is:: + only used if the :c:macro:`Py_TPFLAGS_HAVE_GC` flag bit is set. - int tp_traverse(PyObject *self, visitproc visit, void *arg); - - More information about Python's garbage collection scheme can be found - in section :ref:`supporting-cycle-detection`. - - The :c:member:`~PyTypeObject.tp_traverse` pointer is used by the garbage collector to detect - reference cycles. A typical implementation of a :c:member:`~PyTypeObject.tp_traverse` function - simply calls :c:func:`Py_VISIT` on each of the instance's members that are Python - objects that the instance owns. For example, this is function :c:func:`!local_traverse` from the - :mod:`!_thread` extension module:: - - static int - local_traverse(PyObject *op, visitproc visit, void *arg) - { - localobject *self = (localobject *) op; - Py_VISIT(self->args); - Py_VISIT(self->kw); - Py_VISIT(self->dict); - return 0; - } - - Note that :c:func:`Py_VISIT` is called only on those members that can participate - in reference cycles. Although there is also a ``self->key`` member, it can only - be ``NULL`` or a Python string and therefore cannot be part of a reference cycle. - - On the other hand, even if you know a member can never be part of a cycle, as a - debugging aid you may want to visit it anyway just so the :mod:`gc` module's - :func:`~gc.get_referents` function will include it. - - Heap types (:c:macro:`Py_TPFLAGS_HEAPTYPE`) must visit their type with:: - - Py_VISIT(Py_TYPE(self)); - - It is only needed since Python 3.9. To support Python 3.8 and older, this - line must be conditional:: - - #if PY_VERSION_HEX >= 0x03090000 - Py_VISIT(Py_TYPE(self)); - #endif - - If the :c:macro:`Py_TPFLAGS_MANAGED_DICT` bit is set in the - :c:member:`~PyTypeObject.tp_flags` field, the traverse function must call - :c:func:`PyObject_VisitManagedDict` like this:: - - PyObject_VisitManagedDict((PyObject*)self, visit, arg); - - .. warning:: - When implementing :c:member:`~PyTypeObject.tp_traverse`, only the - members that the instance *owns* (by having :term:`strong references - ` to them) must be - visited. For instance, if an object supports weak references via the - :c:member:`~PyTypeObject.tp_weaklist` slot, the pointer supporting - the linked list (what *tp_weaklist* points to) must **not** be - visited as the instance does not directly own the weak references to itself - (the weakreference list is there to support the weak reference machinery, - but the instance has no strong reference to the elements inside it, as they - are allowed to be removed even if the instance is still alive). - - .. warning:: - The traversal function must not have any side effects. It must not - modify the reference counts of any Python objects nor create or destroy - any Python objects. - - Note that :c:func:`Py_VISIT` requires the *visit* and *arg* parameters to - :c:func:`!local_traverse` to have these specific names; don't name them just - anything. - - Instances of :ref:`heap-allocated types ` hold a reference to - their type. Their traversal function must therefore either visit - :c:func:`Py_TYPE(self) `, or delegate this responsibility by - calling ``tp_traverse`` of another heap-allocated type (such as a - heap-allocated superclass). - If they do not, the type object may not be garbage-collected. - - .. note:: - - The :c:member:`~PyTypeObject.tp_traverse` function can be called from any - thread. - - .. versionchanged:: 3.9 - - Heap-allocated types are expected to visit ``Py_TYPE(self)`` in - ``tp_traverse``. In earlier versions of Python, due to - `bug 40217 `_, doing this - may lead to crashes in subclasses. + See :ref:`gc-traversal` for documentation. **Inheritance:** diff --git a/Doc/data/refcounts.dat b/Doc/data/refcounts.dat index 01b064f3e617ff..2a6e6b963134bb 100644 --- a/Doc/data/refcounts.dat +++ b/Doc/data/refcounts.dat @@ -2430,10 +2430,17 @@ PyType_GetName:PyTypeObject*:type:0: PyType_GetModule:PyObject*::0: PyType_GetModule:PyTypeObject*:type:0: +PyType_GetModule_DuringGC:PyObject*::0: +PyType_GetModule_DuringGC:PyTypeObject*:type:0: + PyType_GetModuleByToken:PyObject*::+1: PyType_GetModuleByToken:PyTypeObject*:type:0: PyType_GetModuleByToken:PyModuleDef*:def:: +PyType_GetModuleByToken_DuringGC:PyObject*::0: +PyType_GetModuleByToken_DuringGC:PyTypeObject*:type:0: +PyType_GetModuleByToken_DuringGC:PyModuleDef*:mod_token:: + PyType_GetModuleByDef:PyObject*::0: PyType_GetModuleByDef:PyTypeObject*:type:0: PyType_GetModuleByDef:PyModuleDef*:def:: diff --git a/Doc/data/stable_abi.dat b/Doc/data/stable_abi.dat index 510e683c87e8b9..4a7fbdf60bfb43 100644 --- a/Doc/data/stable_abi.dat +++ b/Doc/data/stable_abi.dat @@ -495,7 +495,9 @@ func,PyModule_GetName,3.2,, func,PyModule_GetNameObject,3.7,, func,PyModule_GetState,3.2,, func,PyModule_GetStateSize,3.15,, +func,PyModule_GetState_DuringGC,3.15,, func,PyModule_GetToken,3.15,, +func,PyModule_GetToken_DuringGC,3.15,, func,PyModule_New,3.2,, func,PyModule_NewObject,3.7,, func,PyModule_SetDocString,3.7,, @@ -598,6 +600,7 @@ func,PyObject_GetIter,3.2,, func,PyObject_GetOptionalAttr,3.13,, func,PyObject_GetOptionalAttrString,3.13,, func,PyObject_GetTypeData,3.12,, +func,PyObject_GetTypeData_DuringGC,3.15,, func,PyObject_HasAttr,3.2,, func,PyObject_HasAttrString,3.2,, func,PyObject_HasAttrStringWithError,3.13,, @@ -750,13 +753,17 @@ func,PyType_FromSpecWithBases,3.3,, func,PyType_GenericAlloc,3.2,, func,PyType_GenericNew,3.2,, func,PyType_GetBaseByToken,3.14,, +func,PyType_GetBaseByToken_DuringGC,3.15,, func,PyType_GetFlags,3.2,, func,PyType_GetFullyQualifiedName,3.13,, func,PyType_GetModule,3.10,, func,PyType_GetModuleByDef,3.13,, func,PyType_GetModuleByToken,3.15,, +func,PyType_GetModuleByToken_DuringGC,3.15,, func,PyType_GetModuleName,3.13,, func,PyType_GetModuleState,3.10,, +func,PyType_GetModuleState_DuringGC,3.15,, +func,PyType_GetModule_DuringGC,3.15,, func,PyType_GetName,3.11,, func,PyType_GetQualName,3.11,, func,PyType_GetSlot,3.4,, diff --git a/Include/cpython/object.h b/Include/cpython/object.h index 61cdb93d1d5354..7d98007f54045a 100644 --- a/Include/cpython/object.h +++ b/Include/cpython/object.h @@ -442,6 +442,7 @@ PyAPI_FUNC(void) _PyTrash_thread_destroy_chain(PyThreadState *tstate); PyAPI_FUNC(void *) PyObject_GetItemData(PyObject *obj); +PyAPI_FUNC(void *) PyObject_GetItemData_DuringGC(PyObject *obj); PyAPI_FUNC(int) PyObject_VisitManagedDict(PyObject *obj, visitproc visit, void *arg); PyAPI_FUNC(void) PyObject_ClearManagedDict(PyObject *obj); diff --git a/Include/internal/pycore_moduleobject.h b/Include/internal/pycore_moduleobject.h index 7882ce03323561..5bcfd17cec4627 100644 --- a/Include/internal/pycore_moduleobject.h +++ b/Include/internal/pycore_moduleobject.h @@ -53,11 +53,13 @@ static inline PyModuleDef *_PyModule_GetDefOrNull(PyObject *arg) { return NULL; } +// Get md_token. Used in _DuringGC functions; must have no side effects. static inline PyModuleDef *_PyModule_GetToken(PyObject *arg) { PyModuleObject *mod = _PyModule_CAST(arg); return (PyModuleDef *)mod->md_token; } +// Get md_state. Used in _DuringGC functions; must have no side effects. static inline void* _PyModule_GetState(PyObject* mod) { return _PyModule_CAST(mod)->md_state; } diff --git a/Include/moduleobject.h b/Include/moduleobject.h index d1dde7982ad50d..8758aa3f71713f 100644 --- a/Include/moduleobject.h +++ b/Include/moduleobject.h @@ -123,6 +123,8 @@ PyAPI_FUNC(PyObject *) PyModule_FromSlotsAndSpec(const PyModuleDef_Slot *slots, PyAPI_FUNC(int) PyModule_Exec(PyObject *module); PyAPI_FUNC(int) PyModule_GetStateSize(PyObject *module, Py_ssize_t *result); PyAPI_FUNC(int) PyModule_GetToken(PyObject *module, void **result); +PyAPI_FUNC(void*) PyModule_GetState_DuringGC(PyObject*); +PyAPI_FUNC(int) PyModule_GetToken_DuringGC(PyObject *module, void **result); #endif #ifndef _Py_OPAQUE_PYOBJECT diff --git a/Include/object.h b/Include/object.h index 3fb28035a50547..c09d7ae8150f71 100644 --- a/Include/object.h +++ b/Include/object.h @@ -856,6 +856,14 @@ PyAPI_FUNC(int) PyType_Freeze(PyTypeObject *type); #if !defined(Py_LIMITED_API) || Py_LIMITED_API+0 >= _Py_PACK_VERSION(3, 15) PyAPI_FUNC(PyObject *) PyType_GetModuleByToken(PyTypeObject *type, const void *token); +PyAPI_FUNC(void *) PyObject_GetTypeData_DuringGC(PyObject *obj, + PyTypeObject *cls); +PyAPI_FUNC(void *) PyType_GetModuleState_DuringGC(PyTypeObject *); +PyAPI_FUNC(int) PyType_GetBaseByToken_DuringGC(PyTypeObject *, + void *, PyTypeObject **); +PyAPI_FUNC(PyObject *) PyType_GetModule_DuringGC(PyTypeObject *); +PyAPI_FUNC(PyObject *) PyType_GetModuleByToken_DuringGC(PyTypeObject *type, + const void *token); #endif #ifdef __cplusplus diff --git a/Lib/test/test_stable_abi_ctypes.py b/Lib/test/test_stable_abi_ctypes.py index 28f5dd11130c70..ed0868e0017fce 100644 --- a/Lib/test/test_stable_abi_ctypes.py +++ b/Lib/test/test_stable_abi_ctypes.py @@ -488,7 +488,9 @@ def test_windows_feature_macros(self): "PyModule_GetNameObject", "PyModule_GetState", "PyModule_GetStateSize", + "PyModule_GetState_DuringGC", "PyModule_GetToken", + "PyModule_GetToken_DuringGC", "PyModule_New", "PyModule_NewObject", "PyModule_SetDocString", @@ -586,6 +588,7 @@ def test_windows_feature_macros(self): "PyObject_GetOptionalAttr", "PyObject_GetOptionalAttrString", "PyObject_GetTypeData", + "PyObject_GetTypeData_DuringGC", "PyObject_HasAttr", "PyObject_HasAttrString", "PyObject_HasAttrStringWithError", @@ -740,13 +743,17 @@ def test_windows_feature_macros(self): "PyType_GenericAlloc", "PyType_GenericNew", "PyType_GetBaseByToken", + "PyType_GetBaseByToken_DuringGC", "PyType_GetFlags", "PyType_GetFullyQualifiedName", "PyType_GetModule", "PyType_GetModuleByDef", "PyType_GetModuleByToken", + "PyType_GetModuleByToken_DuringGC", "PyType_GetModuleName", "PyType_GetModuleState", + "PyType_GetModuleState_DuringGC", + "PyType_GetModule_DuringGC", "PyType_GetName", "PyType_GetQualName", "PyType_GetSlot", diff --git a/Misc/NEWS.d/next/C_API/2026-03-13-16-37-54.gh-issue-145921.ssA7HZ.rst b/Misc/NEWS.d/next/C_API/2026-03-13-16-37-54.gh-issue-145921.ssA7HZ.rst new file mode 100644 index 00000000000000..4a512832027703 --- /dev/null +++ b/Misc/NEWS.d/next/C_API/2026-03-13-16-37-54.gh-issue-145921.ssA7HZ.rst @@ -0,0 +1,9 @@ +Add functions that are guaranteed to be safe for use in +:c:member:`~PyTypeObject.tp_traverse` handlers: +:c:func:`PyObject_GetTypeData_DuringGC`, +:c:func:`PyObject_GetItemData_DuringGC`, +:c:func:`PyType_GetModuleState_DuringGC`, +:c:func:`PyModule_GetState_DuringGC`, :c:func:`PyModule_GetToken_DuringGC`, +:c:func:`PyType_GetBaseByToken_DuringGC`, +:c:func:`PyType_GetModule_DuringGC`, +:c:func:`PyType_GetModuleByToken_DuringGC`. diff --git a/Misc/stable_abi.toml b/Misc/stable_abi.toml index 63fd83868b644f..606a0a88d26cf2 100644 --- a/Misc/stable_abi.toml +++ b/Misc/stable_abi.toml @@ -2666,6 +2666,20 @@ [function.Py_SET_SIZE] # Before 3.15, this was a macro that accessed the PyObject member added = '3.15' +[function.PyObject_GetTypeData_DuringGC] + added = '3.15' +[function.PyType_GetModuleState_DuringGC] + added = '3.15' +[function.PyModule_GetState_DuringGC] + added = '3.15' +[function.PyModule_GetToken_DuringGC] + added = '3.15' +[function.PyType_GetBaseByToken_DuringGC] + added = '3.15' +[function.PyType_GetModule_DuringGC] + added = '3.15' +[function.PyType_GetModuleByToken_DuringGC] + added = '3.15' # PEP 757 import/export API. diff --git a/Modules/_ctypes/_ctypes.c b/Modules/_ctypes/_ctypes.c index 2c691c3766fc4d..02d12be28b146d 100644 --- a/Modules/_ctypes/_ctypes.c +++ b/Modules/_ctypes/_ctypes.c @@ -467,11 +467,7 @@ class _ctypes.CType_Type "PyObject *" "clinic_state()->CType_Type" static int CType_Type_traverse(PyObject *self, visitproc visit, void *arg) { - StgInfo *info = _PyStgInfo_FromType_NoState(self); - if (!info) { - PyErr_FormatUnraisable("Exception ignored while " - "calling ctypes traverse function %R", self); - } + StgInfo *info = _PyStgInfo_FromType_DuringGC(self); if (info) { Py_VISIT(info->proto); Py_VISIT(info->argtypes); @@ -515,11 +511,7 @@ ctype_free_stginfo_members(StgInfo *info) static int CType_Type_clear(PyObject *self) { - StgInfo *info = _PyStgInfo_FromType_NoState(self); - if (!info) { - PyErr_FormatUnraisable("Exception ignored while " - "clearing ctypes %R", self); - } + StgInfo *info = _PyStgInfo_FromType_DuringGC(self); if (info) { ctype_clear_stginfo(info); } @@ -529,11 +521,7 @@ CType_Type_clear(PyObject *self) static void CType_Type_dealloc(PyObject *self) { - StgInfo *info = _PyStgInfo_FromType_NoState(self); - if (!info) { - PyErr_FormatUnraisable("Exception ignored while " - "deallocating ctypes %R", self); - } + StgInfo *info = _PyStgInfo_FromType_DuringGC(self); if (info) { ctype_free_stginfo_members(info); } diff --git a/Modules/_ctypes/ctypes.h b/Modules/_ctypes/ctypes.h index 478daecad55b94..21743eb80ee976 100644 --- a/Modules/_ctypes/ctypes.h +++ b/Modules/_ctypes/ctypes.h @@ -614,15 +614,14 @@ PyStgInfo_FromAny(ctypes_state *state, PyObject *obj, StgInfo **result) * state is torn down. */ static inline StgInfo * -_PyStgInfo_FromType_NoState(PyObject *type) +_PyStgInfo_FromType_DuringGC(PyObject *type) { PyTypeObject *PyCType_Type; - if (_PyType_GetBaseByToken_Borrow(Py_TYPE(type), &pyctype_type_spec, &PyCType_Type) < 0 || - PyCType_Type == NULL) { + PyType_GetBaseByToken_DuringGC(Py_TYPE(type), &pyctype_type_spec, &PyCType_Type); + if (PyCType_Type == NULL) { return NULL; } - - return PyObject_GetTypeData(type, PyCType_Type); + return PyObject_GetTypeData_DuringGC(type, PyCType_Type); } // Initialize StgInfo on a newly created type diff --git a/Modules/_testcapi/heaptype.c b/Modules/_testcapi/heaptype.c index 4fdcc850a339b4..bb52f0a947070f 100644 --- a/Modules/_testcapi/heaptype.c +++ b/Modules/_testcapi/heaptype.c @@ -403,6 +403,7 @@ static PyObject * pyobject_getitemdata(PyObject *self, PyObject *o) { void *pointer = PyObject_GetItemData(o); + assert(pointer == PyObject_GetItemData_DuringGC(o)); if (pointer == NULL) { return NULL; } @@ -485,17 +486,27 @@ pytype_getbasebytoken(PyObject *self, PyObject *args) mro_save = type->tp_mro; type->tp_mro = NULL; } - void *token = PyLong_AsVoidPtr(py_token); + if (PyErr_Occurred()) { + return NULL; + } + + void *result_duringgc; + int ret_duringgc = PyType_GetBaseByToken_DuringGC( + type, token, (PyTypeObject **)&result_duringgc); + assert(!PyErr_Occurred()); + PyObject *result; int ret; if (need_result == Py_True) { ret = PyType_GetBaseByToken(type, token, (PyTypeObject **)&result); + assert(result == result_duringgc); } else { result = NULL; ret = PyType_GetBaseByToken(type, token, NULL); } + assert(ret == ret_duringgc); if (use_mro != Py_True) { type->tp_mro = mro_save; @@ -518,6 +529,7 @@ pytype_getbasebytoken(PyObject *self, PyObject *args) error: Py_XDECREF(py_ret); Py_XDECREF(result); + assert(PyErr_Occurred()); return NULL; } @@ -525,6 +537,7 @@ static PyObject * pytype_getmodulebydef(PyObject *self, PyObject *type) { PyObject *mod = PyType_GetModuleByDef((PyTypeObject *)type, _testcapimodule); + assert(mod == PyType_GetModuleByDef((PyTypeObject *)type, _testcapimodule)); return Py_XNewRef(mod); } @@ -540,7 +553,9 @@ pytype_getmodulebytoken(PyObject *self, PyObject *args) if ((!token) && PyErr_Occurred()) { return NULL; } - return PyType_GetModuleByToken((PyTypeObject *)type, token); + PyObject *result = PyType_GetModuleByToken((PyTypeObject *)type, token); + assert(result == PyType_GetModuleByToken_DuringGC((PyTypeObject *)type, token)); + return result; } @@ -799,6 +814,7 @@ heapctypesubclasswithfinalizer_finalize(PyObject *self) PyObject *exc = PyErr_GetRaisedException(); PyObject *m = PyType_GetModule(Py_TYPE(self)); + assert(m == PyType_GetModule_DuringGC(Py_TYPE(self))); if (m == NULL) { goto cleanup_finalize; } @@ -1250,6 +1266,7 @@ HeapCCollection_new(PyTypeObject *subtype, PyObject *args, PyObject *kwds) goto finally; } PyObject **data = PyObject_GetItemData(self); + assert(data == PyObject_GetItemData_DuringGC(self)); if (!data) { goto finally; } @@ -1279,6 +1296,7 @@ HeapCCollection_item(PyObject *self, Py_ssize_t i) return PyErr_Format(PyExc_IndexError, "index %zd out of range", i); } PyObject **data = PyObject_GetItemData(self); + assert(data == PyObject_GetItemData_DuringGC(self)); if (!data) { return NULL; } @@ -1289,6 +1307,7 @@ static int HeapCCollection_traverse(PyObject *self, visitproc visit, void *arg) { PyObject **data = PyObject_GetItemData(self); + assert(data == PyObject_GetItemData_DuringGC(self)); if (!data) { return -1; } @@ -1302,6 +1321,7 @@ static int HeapCCollection_clear(PyObject *self) { PyObject **data = PyObject_GetItemData(self); + assert(data == PyObject_GetItemData_DuringGC(self)); if (!data) { return -1; } diff --git a/Modules/_testcapi/module.c b/Modules/_testcapi/module.c index 3411b21e942a19..82278c5edc8239 100644 --- a/Modules/_testcapi/module.c +++ b/Modules/_testcapi/module.c @@ -143,6 +143,8 @@ module_from_slots_token(PyObject *self, PyObject *spec) return NULL; } assert(got_token == &test_token); + assert(PyModule_GetToken_DuringGC(mod, &got_token) >= 0); + assert(got_token == &test_token); return mod; } @@ -371,7 +373,12 @@ static PyObject * pymodule_get_token(PyObject *self, PyObject *module) { void *token; - if (PyModule_GetToken(module, &token) < 0) { + int res = PyModule_GetToken(module, &token); + void *token_duringgc; + int res_duringgc = PyModule_GetToken_DuringGC(module, &token_duringgc); + assert(res == res_duringgc); + assert(token == token_duringgc); + if (res < 0) { return NULL; } return PyLong_FromVoidPtr(token); diff --git a/Modules/_testlimitedcapi/heaptype_relative.c b/Modules/_testlimitedcapi/heaptype_relative.c index fc278a70b77d31..c02a52368b5324 100644 --- a/Modules/_testlimitedcapi/heaptype_relative.c +++ b/Modules/_testlimitedcapi/heaptype_relative.c @@ -1,7 +1,7 @@ -// Need limited C API version 3.12 for PyType_FromMetaclass() +// Need limited C API version 3.15 for _DuringGC functions #include "pyconfig.h" // Py_GIL_DISABLED #if !defined(Py_GIL_DISABLED) && !defined(Py_LIMITED_API) -# define Py_LIMITED_API 0x030c0000 +# define Py_LIMITED_API 0x030f0000 #endif #include "parts.h" @@ -55,6 +55,8 @@ make_sized_heaptypes(PyObject *module, PyObject *args) goto finally; } char *data_ptr = PyObject_GetTypeData(instance, (PyTypeObject *)sub); + assert(data_ptr == PyObject_GetTypeData_DuringGC(instance, + (PyTypeObject *)sub)); if (!data_ptr) { goto finally; } @@ -80,6 +82,7 @@ var_heaptype_set_data_to_3s( PyObject *const *args, Py_ssize_t nargs, PyObject *kwnames) { void *data_ptr = PyObject_GetTypeData(self, defining_class); + assert(data_ptr == PyObject_GetTypeData_DuringGC(self, defining_class)); if (!data_ptr) { return NULL; } @@ -96,6 +99,7 @@ var_heaptype_get_data(PyObject *self, PyTypeObject *defining_class, PyObject *const *args, Py_ssize_t nargs, PyObject *kwnames) { void *data_ptr = PyObject_GetTypeData(self, defining_class); + assert(data_ptr == PyObject_GetTypeData_DuringGC(self, defining_class)); if (!data_ptr) { return NULL; } @@ -259,6 +263,7 @@ heapctypewithrelativedict_dealloc(PyObject* self) { PyTypeObject *tp = Py_TYPE(self); HeapCTypeWithDictStruct *data = PyObject_GetTypeData(self, tp); + assert(data == PyObject_GetTypeData_DuringGC(self, tp)); Py_XDECREF(data->dict); PyObject_Free(self); Py_DECREF(tp); @@ -297,6 +302,7 @@ heapctypewithrelativeweakref_dealloc(PyObject* self) { PyTypeObject *tp = Py_TYPE(self); HeapCTypeWithWeakrefStruct *data = PyObject_GetTypeData(self, tp); + assert(data == PyObject_GetTypeData_DuringGC(self, tp)); if (data->weakreflist != NULL) { PyObject_ClearWeakRefs(self); } diff --git a/Modules/_testmultiphase.c b/Modules/_testmultiphase.c index e286eaae820b2b..1647b674804ad7 100644 --- a/Modules/_testmultiphase.c +++ b/Modules/_testmultiphase.c @@ -152,10 +152,13 @@ _testmultiphase_StateAccessType_get_defining_module_impl(StateAccessTypeObject * { PyObject *retval; retval = PyType_GetModule(cls); + assert(retval == PyType_GetModule_DuringGC(cls)); if (retval == NULL) { return NULL; } assert(PyType_GetModuleByDef(Py_TYPE(self), &def_meth_state_access) == retval); + assert(PyType_GetModuleByToken_DuringGC(Py_TYPE(self), &def_meth_state_access) + == retval); return Py_NewRef(retval); } @@ -172,8 +175,13 @@ _testmultiphase_StateAccessType_getmodulebydef_bad_def_impl(StateAccessTypeObjec PyTypeObject *cls) /*[clinic end generated code: output=64509074dfcdbd31 input=edaff09aa4788204]*/ { - PyType_GetModuleByDef(Py_TYPE(self), &def_nonmodule); // should raise + // DuringGC: does not raise + assert(PyType_GetModuleByToken_DuringGC(Py_TYPE(self), &def_nonmodule) == NULL); + assert(!PyErr_Occurred()); + // should raise: + PyObject *m = PyType_GetModuleByDef(Py_TYPE(self), &def_nonmodule); assert(PyErr_Occurred()); + assert(m == NULL); return NULL; } @@ -200,6 +208,7 @@ _testmultiphase_StateAccessType_increment_count_clinic_impl(StateAccessTypeObjec /*[clinic end generated code: output=3b34f86bc5473204 input=551d482e1fe0b8f5]*/ { meth_state *m_state = PyType_GetModuleState(cls); + assert(m_state == PyType_GetModuleState_DuringGC(cls)); if (twice) { n *= 2; } @@ -249,6 +258,7 @@ _StateAccessType_increment_count_noclinic(PyObject *self, n *= 2; } meth_state *m_state = PyType_GetModuleState(defining_class); + assert(m_state == PyType_GetModuleState_DuringGC(defining_class)); m_state->counter += n; Py_RETURN_NONE; @@ -268,6 +278,7 @@ _testmultiphase_StateAccessType_get_count_impl(StateAccessTypeObject *self, /*[clinic end generated code: output=64600f95b499a319 input=d5d181f12384849f]*/ { meth_state *m_state = PyType_GetModuleState(cls); + assert(m_state == PyType_GetModuleState_DuringGC(cls)); return PyLong_FromLong(m_state->counter); } @@ -879,6 +890,7 @@ meth_state_access_exec(PyObject *m) meth_state *m_state; m_state = PyModule_GetState(m); + assert(m_state == PyModule_GetState_DuringGC(m)); if (m_state == NULL) { return -1; } @@ -1125,6 +1137,7 @@ modexport_smoke_exec(PyObject *mod) return 0; } int *state = PyModule_GetState(mod); + assert(state == PyModule_GetState_DuringGC(mod)); if (!state) { return -1; } @@ -1142,6 +1155,7 @@ static PyObject * modexport_smoke_get_state_int(PyObject *mod, PyObject *arg) { int *state = PyModule_GetState(mod); + assert(state == PyModule_GetState_DuringGC(mod)); if (!state) { return NULL; } @@ -1171,6 +1185,7 @@ modexport_smoke_free(void *op) { PyObject *mod = (PyObject *)op; int *state = PyModule_GetState(mod); + assert(state == PyModule_GetState_DuringGC(mod)); if (!state) { PyErr_FormatUnraisable("Exception ignored in module %R free", mod); } diff --git a/Modules/_testsinglephase.c b/Modules/_testsinglephase.c index 7ea77c6312c59e..49ab2ad55398f0 100644 --- a/Modules/_testsinglephase.c +++ b/Modules/_testsinglephase.c @@ -254,6 +254,7 @@ get_module_state(PyObject *module) } else { module_state *state = (module_state*)PyModule_GetState(module); + assert(state == PyModule_GetState_DuringGC(module)); assert(state != NULL); return state; } diff --git a/Objects/moduleobject.c b/Objects/moduleobject.c index e3868097c0ba9f..86e6724e3cb312 100644 --- a/Objects/moduleobject.c +++ b/Objects/moduleobject.c @@ -900,6 +900,17 @@ PyModule_GetStateSize(PyObject *m, Py_ssize_t *size_p) return 0; } +int +PyModule_GetToken_DuringGC(PyObject *m, void **token_p) +{ + *token_p = NULL; + if (!PyModule_Check(m)) { + return -1; + } + *token_p = _PyModule_GetToken(m); + return 0; +} + int PyModule_GetToken(PyObject *m, void **token_p) { @@ -1055,6 +1066,15 @@ PyModule_GetDef(PyObject* m) return _PyModule_GetDefOrNull(m); } +void* +PyModule_GetState_DuringGC(PyObject* m) +{ + if (!PyModule_Check(m)) { + return NULL; + } + return _PyModule_GetState(m); +} + void* PyModule_GetState(PyObject* m) { diff --git a/Objects/typeobject.c b/Objects/typeobject.c index 7b4318e79fb2be..817db638f40c5e 100644 --- a/Objects/typeobject.c +++ b/Objects/typeobject.c @@ -5737,6 +5737,20 @@ PyType_GetSlot(PyTypeObject *type, int slot) return *(void**)((char*)parent_slot + pyslot_offsets[slot].subslot_offset); } +PyObject * +PyType_GetModule_DuringGC(PyTypeObject *type) +{ + assert(PyType_Check(type)); + if (!_PyType_HasFeature(type, Py_TPFLAGS_HEAPTYPE)) { + return NULL; + } + PyHeapTypeObject* et = (PyHeapTypeObject*)type; + if (!et->ht_module) { + return NULL; + } + return et->ht_module; +} + PyObject * PyType_GetModule(PyTypeObject *type) { @@ -5758,7 +5772,16 @@ PyType_GetModule(PyTypeObject *type) return NULL; } return et->ht_module; +} +void * +PyType_GetModuleState_DuringGC(PyTypeObject *type) +{ + PyObject *m = PyType_GetModule_DuringGC(type); + if (m == NULL) { + return NULL; + } + return _PyModule_GetState(m); } void * @@ -5771,19 +5794,18 @@ PyType_GetModuleState(PyTypeObject *type) return _PyModule_GetState(m); } - /* Return borrowed ref to the module of the first superclass where the module * has the given token. */ -static PyObject * -borrow_module_by_token(PyTypeObject *type, const void *token) +PyObject * +PyType_GetModuleByToken_DuringGC(PyTypeObject *type, const void *token) { assert(PyType_Check(type)); if (!_PyType_HasFeature(type, Py_TPFLAGS_HEAPTYPE)) { // type_ready_mro() ensures that no heap type is // contained in a static type MRO. - goto error; + return NULL; } else { PyHeapTypeObject *ht = (PyHeapTypeObject*)type; @@ -5823,27 +5845,29 @@ borrow_module_by_token(PyTypeObject *type, const void *token) } END_TYPE_LOCK(); - if (res != NULL) { - return res; - } -error: - PyErr_Format( - PyExc_TypeError, - "PyType_GetModuleByDef: No superclass of '%s' has the given module", - type->tp_name); - return NULL; + return res; } PyObject * -PyType_GetModuleByDef(PyTypeObject *type, PyModuleDef *def) +PyType_GetModuleByToken(PyTypeObject *type, const void *token) { - return borrow_module_by_token(type, def); + PyObject *mod = PyType_GetModuleByToken_DuringGC(type, token); + if (!mod) { + PyErr_Format( + PyExc_TypeError, + "PyType_GetModuleByDef: No superclass of '%s' has the given module", + type->tp_name); + return NULL; + } + return Py_NewRef(mod); } PyObject * -PyType_GetModuleByToken(PyTypeObject *type, const void *token) +PyType_GetModuleByDef(PyTypeObject *type, PyModuleDef *def) { - return Py_XNewRef(borrow_module_by_token(type, token)); + PyObject *mod = PyType_GetModuleByToken(type, def); + Py_XDECREF(mod); // return borrowed ref + return mod; } @@ -5872,14 +5896,17 @@ get_base_by_token_recursive(PyObject *bases, void *token) } int -_PyType_GetBaseByToken_Borrow(PyTypeObject *type, void *token, PyTypeObject **result) +PyType_GetBaseByToken_DuringGC(PyTypeObject *type, void *token, PyTypeObject **result) { - assert(token != NULL); - assert(PyType_Check(type)); - if (result != NULL) { *result = NULL; } + if (token == NULL) { + return -1; + } + if (!PyType_Check(type)) { + return -1; + } if (!_PyType_HasFeature(type, Py_TPFLAGS_HEAPTYPE)) { // No static type has a heaptype superclass, @@ -5940,7 +5967,7 @@ PyType_GetBaseByToken(PyTypeObject *type, void *token, PyTypeObject **result) return -1; } - int res = _PyType_GetBaseByToken_Borrow(type, token, result); + int res = PyType_GetBaseByToken_DuringGC(type, token, result); if (res > 0 && result) { Py_INCREF(*result); } @@ -5949,12 +5976,18 @@ PyType_GetBaseByToken(PyTypeObject *type, void *token, PyTypeObject **result) void * -PyObject_GetTypeData(PyObject *obj, PyTypeObject *cls) +PyObject_GetTypeData_DuringGC(PyObject *obj, PyTypeObject *cls) { assert(PyObject_TypeCheck(obj, cls)); return (char *)obj + _align_up(cls->tp_base->tp_basicsize); } +void * +PyObject_GetTypeData(PyObject *obj, PyTypeObject *cls) +{ + return PyObject_GetTypeData_DuringGC(obj, cls); +} + Py_ssize_t PyType_GetTypeDataSize(PyTypeObject *cls) { @@ -5965,6 +5998,15 @@ PyType_GetTypeDataSize(PyTypeObject *cls) return result; } +void * +PyObject_GetItemData_DuringGC(PyObject *obj) +{ + if (!PyType_HasFeature(Py_TYPE(obj), Py_TPFLAGS_ITEMS_AT_END)) { + return NULL; + } + return (char *)obj + Py_TYPE(obj)->tp_basicsize; +} + void * PyObject_GetItemData(PyObject *obj) { @@ -5974,7 +6016,7 @@ PyObject_GetItemData(PyObject *obj) Py_TYPE(obj)->tp_name); return NULL; } - return (char *)obj + Py_TYPE(obj)->tp_basicsize; + return PyObject_GetItemData_DuringGC(obj); } /* Internal API to look for a name through the MRO, bypassing the method cache. diff --git a/PC/python3dll.c b/PC/python3dll.c index b23bc2b8f4382f..abbe35c342c13e 100755 --- a/PC/python3dll.c +++ b/PC/python3dll.c @@ -437,8 +437,10 @@ EXPORT_FUNC(PyModule_GetFilenameObject) EXPORT_FUNC(PyModule_GetName) EXPORT_FUNC(PyModule_GetNameObject) EXPORT_FUNC(PyModule_GetState) +EXPORT_FUNC(PyModule_GetState_DuringGC) EXPORT_FUNC(PyModule_GetStateSize) EXPORT_FUNC(PyModule_GetToken) +EXPORT_FUNC(PyModule_GetToken_DuringGC) EXPORT_FUNC(PyModule_New) EXPORT_FUNC(PyModule_NewObject) EXPORT_FUNC(PyModule_SetDocString) @@ -523,6 +525,7 @@ EXPORT_FUNC(PyObject_GetIter) EXPORT_FUNC(PyObject_GetOptionalAttr) EXPORT_FUNC(PyObject_GetOptionalAttrString) EXPORT_FUNC(PyObject_GetTypeData) +EXPORT_FUNC(PyObject_GetTypeData_DuringGC) EXPORT_FUNC(PyObject_HasAttr) EXPORT_FUNC(PyObject_HasAttrString) EXPORT_FUNC(PyObject_HasAttrStringWithError) @@ -678,13 +681,17 @@ EXPORT_FUNC(PyType_FromSpecWithBases) EXPORT_FUNC(PyType_GenericAlloc) EXPORT_FUNC(PyType_GenericNew) EXPORT_FUNC(PyType_GetBaseByToken) +EXPORT_FUNC(PyType_GetBaseByToken_DuringGC) EXPORT_FUNC(PyType_GetFlags) EXPORT_FUNC(PyType_GetFullyQualifiedName) EXPORT_FUNC(PyType_GetModule) +EXPORT_FUNC(PyType_GetModule_DuringGC) EXPORT_FUNC(PyType_GetModuleByDef) EXPORT_FUNC(PyType_GetModuleByToken) +EXPORT_FUNC(PyType_GetModuleByToken_DuringGC) EXPORT_FUNC(PyType_GetModuleName) EXPORT_FUNC(PyType_GetModuleState) +EXPORT_FUNC(PyType_GetModuleState_DuringGC) EXPORT_FUNC(PyType_GetName) EXPORT_FUNC(PyType_GetQualName) EXPORT_FUNC(PyType_GetSlot)