62. Reference Ownership Rules

Reference ownership is the core memory discipline of the Python C API. Every PyObject * points to a Python object whose lifetime is controlled by a reference count. Native extension code must follow ownership rules exactly. A single missing Py_DECREF can leak memory. A single extra Py_DECREF can free a live object and corrupt the interpreter.

62.1 Why Ownership Rules Exist

CPython stores a reference count in every normal Python object header.

typedef struct _object {
    Py_ssize_t ob_refcnt;
    PyTypeObject *ob_type;
} PyObject;

The count records how many active owners hold the object alive.

When native code receives a PyObject *, the pointer alone does not say whether the caller owns a reference. The API contract of the function determines ownership.

There are three main cases:

62.2 New References

A new reference means the caller owns a strong reference to the object.

Example:

PyObject *x = PyLong_FromLong(42);

PyLong_FromLong returns a new reference. The caller must eventually release it:

Py_DECREF(x);

Common new-reference APIs:

Pattern:

PyObject *obj = PyUnicode_FromString("hello");
if (obj == NULL) {
    return NULL;
}

/* use obj */

Py_DECREF(obj);
Py_RETURN_NONE;

62.3 Borrowed References

A borrowed reference means the caller can use the object temporarily, but does not own it.

Example:

PyObject *item = PyList_GetItem(list, 0);

PyList_GetItem returns a borrowed reference. The list still owns the reference. The caller must not call Py_DECREF(item) unless it first calls Py_INCREF(item).

Correct:

PyObject *item = PyList_GetItem(list, 0);
if (item == NULL) {
    return NULL;
}

/* use item, no DECREF */

Incorrect:

PyObject *item = PyList_GetItem(list, 0);
Py_DECREF(item);  /* wrong */

Borrowed references are safe only while the owner remains alive and unchanged. If the container can mutate or release the object, convert the borrowed reference into an owned reference:

PyObject *item = PyList_GetItem(list, 0);
if (item == NULL) {
    return NULL;
}

Py_INCREF(item);

/* item is now owned by this code */

Py_DECREF(item);

62.4 Stolen References

A stolen reference means the callee takes ownership of a reference that the caller already owns.

Example:

PyObject *x = PyLong_FromLong(42);
if (x == NULL) {
    return NULL;
}

if (PyList_SetItem(list, 0, x) < 0) {
    Py_DECREF(x);
    return NULL;
}

/* do not DECREF x here */

PyList_SetItem steals a reference to x on success. The list now owns it.

Common stealing APIs:

Stealing APIs are a frequent source of bugs because they do not all behave the same way.

62.5 Py_INCREF

Py_INCREF(obj) increments an object’s reference count.

Use it when you need to keep an object alive beyond the lifetime of a borrowed reference.

PyObject *item = PyList_GetItem(list, 0);
if (item == NULL) {
    return NULL;
}

Py_INCREF(item);
return item;

This function returns a new reference to Python code because the extension increments the borrowed reference before returning it.

62.6 Py_DECREF

Py_DECREF(obj) releases one owned reference.

If the reference count reaches zero, CPython deallocates the object immediately.

PyObject *s = PyUnicode_FromString("data");
if (s == NULL) {
    return NULL;
}

/* use s */

Py_DECREF(s);

After Py_DECREF(s), the pointer s must be treated as invalid unless another owned reference is known to exist.

62.7 Py_XDECREF

Py_XDECREF(obj) is like Py_DECREF, but accepts NULL.

PyObject *obj = maybe_create_object();
Py_XDECREF(obj);

This is useful in cleanup paths where some variables may not have been initialized.

Example:

PyObject *a = NULL;
PyObject *b = NULL;

a = PyLong_FromLong(1);
if (a == NULL) {
    goto error;
}

b = PyLong_FromLong(2);
if (b == NULL) {
    goto error;
}

/* success */

Py_DECREF(a);
Py_DECREF(b);
Py_RETURN_NONE;

error:
Py_XDECREF(a);
Py_XDECREF(b);
return NULL;

62.8 Returning References

A C function exposed to Python must return a new reference or NULL.

Correct:

static PyObject *
make_answer(PyObject *self, PyObject *args)
{
    return PyLong_FromLong(42);
}

Also correct:

static PyObject *
first_item(PyObject *self, PyObject *args)
{
    PyObject *list;

    if (!PyArg_ParseTuple(args, "O", &list)) {
        return NULL;
    }

    PyObject *item = PyList_GetItem(list, 0);
    if (item == NULL) {
        return NULL;
    }

    Py_INCREF(item);
    return item;
}

The function returns a borrowed list item only after converting it into a new reference.

Incorrect:

return PyList_GetItem(list, 0);  /* returns borrowed reference */

This may appear to work, then fail later under different object lifetime conditions.

62.9 Error Paths

Ownership bugs often appear on error paths.

Example:

PyObject *a = PyLong_FromLong(1);
PyObject *b = PyLong_FromLong(2);
PyObject *sum = PyNumber_Add(a, b);

Every allocation can fail. Correct code must clean up partially created objects.

PyObject *a = NULL;
PyObject *b = NULL;
PyObject *sum = NULL;

a = PyLong_FromLong(1);
if (a == NULL) {
    goto error;
}

b = PyLong_FromLong(2);
if (b == NULL) {
    goto error;
}

sum = PyNumber_Add(a, b);
if (sum == NULL) {
    goto error;
}

Py_DECREF(a);
Py_DECREF(b);
return sum;

error:
Py_XDECREF(a);
Py_XDECREF(b);
Py_XDECREF(sum);
return NULL;

The cleanup block releases only owned references.

62.10 Borrowed Reference Lifetime Hazards

Borrowed references are fragile when code can execute arbitrary Python.

This is dangerous:

PyObject *item = PyList_GetItem(list, 0);

/* This call may run Python code */
PyObject_CallObject(callback, NULL);

/* item may no longer be valid */

The callback may mutate the list, delete the item, or trigger garbage collection.

Safer:

PyObject *item = PyList_GetItem(list, 0);
if (item == NULL) {
    return NULL;
}

Py_INCREF(item);

PyObject *res = PyObject_CallObject(callback, NULL);
if (res == NULL) {
    Py_DECREF(item);
    return NULL;
}

Py_DECREF(res);

/* item is still alive */

Py_DECREF(item);
Py_RETURN_NONE;

Rule: before calling arbitrary Python code, own the references you still need afterward.

62.11 Reference Ownership and Containers

Container APIs differ.

PyList_Append

Does not steal.

PyObject *x = PyLong_FromLong(1);
if (x == NULL) {
    return NULL;
}

if (PyList_Append(list, x) < 0) {
    Py_DECREF(x);
    return NULL;
}

Py_DECREF(x);

The list receives its own reference. The caller still owns x.

PyList_SetItem

Steals on success.

PyObject *x = PyLong_FromLong(1);
if (x == NULL) {
    return NULL;
}

if (PyList_SetItem(list, 0, x) < 0) {
    Py_DECREF(x);
    return NULL;
}

/* x stolen */

These two APIs look similar but have different ownership contracts.

62.12 Reference Ownership and Tuples

Tuple construction often uses stealing APIs for speed.

PyObject *t = PyTuple_New(2);
if (t == NULL) {
    return NULL;
}

PyObject *a = PyLong_FromLong(1);
if (a == NULL) {
    Py_DECREF(t);
    return NULL;
}

PyObject *b = PyLong_FromLong(2);
if (b == NULL) {
    Py_DECREF(a);
    Py_DECREF(t);
    return NULL;
}

PyTuple_SET_ITEM(t, 0, a);
PyTuple_SET_ITEM(t, 1, b);

return t;

PyTuple_SET_ITEM steals references and performs fewer checks than PyTuple_SetItem. It should be used only when constructing a fresh tuple and when indexes are known valid.

62.13 Reference Ownership and Dictionaries

Dictionary setters do not steal references.

PyObject *d = PyDict_New();
PyObject *k = PyUnicode_FromString("answer");
PyObject *v = PyLong_FromLong(42);

if (d == NULL || k == NULL || v == NULL) {
    Py_XDECREF(d);
    Py_XDECREF(k);
    Py_XDECREF(v);
    return NULL;
}

if (PyDict_SetItem(d, k, v) < 0) {
    Py_DECREF(d);
    Py_DECREF(k);
    Py_DECREF(v);
    return NULL;
}

Py_DECREF(k);
Py_DECREF(v);

return d;

The dict increments references internally. The caller releases its own references afterward.

62.14 Py_RETURN_NONE and Singleton Returns

None, True, and False are singleton objects.

Returning None must still return a new reference. The macro handles this:

Py_RETURN_NONE;

Equivalent conceptually to:

Py_INCREF(Py_None);
return Py_None;

Boolean macros:

Py_RETURN_TRUE;
Py_RETURN_FALSE;

These prevent common ownership mistakes.

62.15 Reference Leaks

A reference leak happens when code owns a reference but never releases it.

Example:

PyObject *x = PyLong_FromLong(42);
Py_RETURN_NONE;  /* x leaked */

Correct:

PyObject *x = PyLong_FromLong(42);
if (x == NULL) {
    return NULL;
}

Py_DECREF(x);
Py_RETURN_NONE;

Leaks are especially costly in long-running processes:

web servers
workers
notebooks
databases
embedded runtimes
daemon processes

62.16 Use-After-Free

Use-after-free happens when code uses an object after its last reference has been released.

PyObject *x = PyLong_FromLong(42);
Py_DECREF(x);

PyObject_Repr(x);  /* invalid */

This is undefined behavior. It may crash, corrupt memory, or appear to work during testing.

62.17 Double DECREF

A double DECREF releases ownership twice.

PyObject *x = PyLong_FromLong(42);

Py_DECREF(x);
Py_DECREF(x);  /* wrong */

If the first Py_DECREF destroys the object, the second touches freed memory.

Double decrefs are often caused by unclear ownership after stealing APIs.

62.18 Debugging Reference Bugs

Debug builds of CPython help find reference leaks and memory errors.

Useful techniques:

build CPython in debug mode
run targeted tests repeatedly
use refleak test mode
use sanitizers
use gdb or lldb
check failure paths
audit borrowed references around callbacks

Extension authors should also test under allocation failure when possible, because many ownership bugs occur only after partial initialization.

62.19 Ownership Rules as Local Invariants

Good extension code keeps ownership visible.

Prefer this style:

PyObject *name = NULL;
PyObject *value = NULL;
PyObject *result = NULL;

Then maintain a local invariant:

Every non-NULL variable either owns exactly one reference or owns none by documented convention.

When ownership transfers, make the code obvious:

if (PyList_SetItem(list, i, value) < 0) {
    Py_DECREF(value);
    return NULL;
}

value = NULL;  /* ownership transferred */

Setting the variable to NULL after transfer prevents accidental cleanup later.

62.20 Chapter Summary

Reference ownership rules define how native code keeps Python objects alive. The pointer type PyObject * is not enough. The API contract tells you whether a reference is new, borrowed, or stolen.

The safe discipline is simple:

Most C API crashes come from violating these rules. Mastering them is required before writing extension modules, implementing new types, or reading CPython object code.