# 62. Reference Ownership Rules

# 62. Reference Ownership Rules

Reference ownership is the core memory discipline of the Python C API. Every `PyObject *` points to a Python object whose lifetime is controlled by a reference count. Native extension code must follow ownership rules exactly. A single missing `Py_DECREF` can leak memory. A single extra `Py_DECREF` can free a live object and corrupt the interpreter.

## 62.1 Why Ownership Rules Exist

CPython stores a reference count in every normal Python object header.

```c
typedef struct _object {
    Py_ssize_t ob_refcnt;
    PyTypeObject *ob_type;
} PyObject;
```

The count records how many active owners hold the object alive.

When native code receives a `PyObject *`, the pointer alone does not say whether the caller owns a reference. The API contract of the function determines ownership.

There are three main cases:

| Reference kind | Meaning | Caller must decref? |
|---|---|---:|
| New reference | Caller owns one reference | Yes |
| Borrowed reference | Caller observes someone else’s reference | No |
| Stolen reference | Callee takes ownership from caller | Usually no, after transfer |

## 62.2 New References

A new reference means the caller owns a strong reference to the object.

Example:

```c
PyObject *x = PyLong_FromLong(42);
```

`PyLong_FromLong` returns a new reference. The caller must eventually release it:

```c
Py_DECREF(x);
```

Common new-reference APIs:

| API | Result |
|---|---|
| `PyLong_FromLong` | New integer object |
| `PyUnicode_FromString` | New Unicode object |
| `PyList_New` | New list object |
| `PyDict_New` | New dict object |
| `PyObject_GetAttrString` | New reference to attribute |
| `PyImport_ImportModule` | New reference to module |
| `PyObject_CallObject` | New reference to call result |

Pattern:

```c
PyObject *obj = PyUnicode_FromString("hello");
if (obj == NULL) {
    return NULL;
}

/* use obj */

Py_DECREF(obj);
Py_RETURN_NONE;
```

## 62.3 Borrowed References

A borrowed reference means the caller can use the object temporarily, but does not own it.

Example:

```c
PyObject *item = PyList_GetItem(list, 0);
```

`PyList_GetItem` returns a borrowed reference. The list still owns the reference. The caller must not call `Py_DECREF(item)` unless it first calls `Py_INCREF(item)`.

Correct:

```c
PyObject *item = PyList_GetItem(list, 0);
if (item == NULL) {
    return NULL;
}

/* use item, no DECREF */
```

Incorrect:

```c
PyObject *item = PyList_GetItem(list, 0);
Py_DECREF(item);  /* wrong */
```

Borrowed references are safe only while the owner remains alive and unchanged. If the container can mutate or release the object, convert the borrowed reference into an owned reference:

```c
PyObject *item = PyList_GetItem(list, 0);
if (item == NULL) {
    return NULL;
}

Py_INCREF(item);

/* item is now owned by this code */

Py_DECREF(item);
```

## 62.4 Stolen References

A stolen reference means the callee takes ownership of a reference that the caller already owns.

Example:

```c
PyObject *x = PyLong_FromLong(42);
if (x == NULL) {
    return NULL;
}

if (PyList_SetItem(list, 0, x) < 0) {
    Py_DECREF(x);
    return NULL;
}

/* do not DECREF x here */
```

`PyList_SetItem` steals a reference to `x` on success. The list now owns it.

Common stealing APIs:

| API | Behavior |
|---|---|
| `PyList_SetItem` | Steals item reference on success |
| `PyTuple_SetItem` | Steals item reference on success |
| `PyModule_AddObject` | Steals object reference on success in common usage |
| `PyCell_Set` | Does not steal |
| `PyDict_SetItem` | Does not steal |

Stealing APIs are a frequent source of bugs because they do not all behave the same way.

## 62.5 `Py_INCREF`

`Py_INCREF(obj)` increments an object’s reference count.

Use it when you need to keep an object alive beyond the lifetime of a borrowed reference.

```c
PyObject *item = PyList_GetItem(list, 0);
if (item == NULL) {
    return NULL;
}

Py_INCREF(item);
return item;
```

This function returns a new reference to Python code because the extension increments the borrowed reference before returning it.

## 62.6 `Py_DECREF`

`Py_DECREF(obj)` releases one owned reference.

If the reference count reaches zero, CPython deallocates the object immediately.

```c
PyObject *s = PyUnicode_FromString("data");
if (s == NULL) {
    return NULL;
}

/* use s */

Py_DECREF(s);
```

After `Py_DECREF(s)`, the pointer `s` must be treated as invalid unless another owned reference is known to exist.

## 62.7 `Py_XDECREF`

`Py_XDECREF(obj)` is like `Py_DECREF`, but accepts `NULL`.

```c
PyObject *obj = maybe_create_object();
Py_XDECREF(obj);
```

This is useful in cleanup paths where some variables may not have been initialized.

Example:

```c
PyObject *a = NULL;
PyObject *b = NULL;

a = PyLong_FromLong(1);
if (a == NULL) {
    goto error;
}

b = PyLong_FromLong(2);
if (b == NULL) {
    goto error;
}

/* success */

Py_DECREF(a);
Py_DECREF(b);
Py_RETURN_NONE;

error:
Py_XDECREF(a);
Py_XDECREF(b);
return NULL;
```

## 62.8 Returning References

A C function exposed to Python must return a new reference or `NULL`.

Correct:

```c
static PyObject *
make_answer(PyObject *self, PyObject *args)
{
    return PyLong_FromLong(42);
}
```

Also correct:

```c
static PyObject *
first_item(PyObject *self, PyObject *args)
{
    PyObject *list;

    if (!PyArg_ParseTuple(args, "O", &list)) {
        return NULL;
    }

    PyObject *item = PyList_GetItem(list, 0);
    if (item == NULL) {
        return NULL;
    }

    Py_INCREF(item);
    return item;
}
```

The function returns a borrowed list item only after converting it into a new reference.

Incorrect:

```c
return PyList_GetItem(list, 0);  /* returns borrowed reference */
```

This may appear to work, then fail later under different object lifetime conditions.

## 62.9 Error Paths

Ownership bugs often appear on error paths.

Example:

```c
PyObject *a = PyLong_FromLong(1);
PyObject *b = PyLong_FromLong(2);
PyObject *sum = PyNumber_Add(a, b);
```

Every allocation can fail. Correct code must clean up partially created objects.

```c
PyObject *a = NULL;
PyObject *b = NULL;
PyObject *sum = NULL;

a = PyLong_FromLong(1);
if (a == NULL) {
    goto error;
}

b = PyLong_FromLong(2);
if (b == NULL) {
    goto error;
}

sum = PyNumber_Add(a, b);
if (sum == NULL) {
    goto error;
}

Py_DECREF(a);
Py_DECREF(b);
return sum;

error:
Py_XDECREF(a);
Py_XDECREF(b);
Py_XDECREF(sum);
return NULL;
```

The cleanup block releases only owned references.

## 62.10 Borrowed Reference Lifetime Hazards

Borrowed references are fragile when code can execute arbitrary Python.

This is dangerous:

```c
PyObject *item = PyList_GetItem(list, 0);

/* This call may run Python code */
PyObject_CallObject(callback, NULL);

/* item may no longer be valid */
```

The callback may mutate the list, delete the item, or trigger garbage collection.

Safer:

```c
PyObject *item = PyList_GetItem(list, 0);
if (item == NULL) {
    return NULL;
}

Py_INCREF(item);

PyObject *res = PyObject_CallObject(callback, NULL);
if (res == NULL) {
    Py_DECREF(item);
    return NULL;
}

Py_DECREF(res);

/* item is still alive */

Py_DECREF(item);
Py_RETURN_NONE;
```

Rule: before calling arbitrary Python code, own the references you still need afterward.

## 62.11 Reference Ownership and Containers

Container APIs differ.

### `PyList_Append`

Does not steal.

```c
PyObject *x = PyLong_FromLong(1);
if (x == NULL) {
    return NULL;
}

if (PyList_Append(list, x) < 0) {
    Py_DECREF(x);
    return NULL;
}

Py_DECREF(x);
```

The list receives its own reference. The caller still owns `x`.

### `PyList_SetItem`

Steals on success.

```c
PyObject *x = PyLong_FromLong(1);
if (x == NULL) {
    return NULL;
}

if (PyList_SetItem(list, 0, x) < 0) {
    Py_DECREF(x);
    return NULL;
}

/* x stolen */
```

These two APIs look similar but have different ownership contracts.

## 62.12 Reference Ownership and Tuples

Tuple construction often uses stealing APIs for speed.

```c
PyObject *t = PyTuple_New(2);
if (t == NULL) {
    return NULL;
}

PyObject *a = PyLong_FromLong(1);
if (a == NULL) {
    Py_DECREF(t);
    return NULL;
}

PyObject *b = PyLong_FromLong(2);
if (b == NULL) {
    Py_DECREF(a);
    Py_DECREF(t);
    return NULL;
}

PyTuple_SET_ITEM(t, 0, a);
PyTuple_SET_ITEM(t, 1, b);

return t;
```

`PyTuple_SET_ITEM` steals references and performs fewer checks than `PyTuple_SetItem`. It should be used only when constructing a fresh tuple and when indexes are known valid.

## 62.13 Reference Ownership and Dictionaries

Dictionary setters do not steal references.

```c
PyObject *d = PyDict_New();
PyObject *k = PyUnicode_FromString("answer");
PyObject *v = PyLong_FromLong(42);

if (d == NULL || k == NULL || v == NULL) {
    Py_XDECREF(d);
    Py_XDECREF(k);
    Py_XDECREF(v);
    return NULL;
}

if (PyDict_SetItem(d, k, v) < 0) {
    Py_DECREF(d);
    Py_DECREF(k);
    Py_DECREF(v);
    return NULL;
}

Py_DECREF(k);
Py_DECREF(v);

return d;
```

The dict increments references internally. The caller releases its own references afterward.

## 62.14 `Py_RETURN_NONE` and Singleton Returns

`None`, `True`, and `False` are singleton objects.

Returning `None` must still return a new reference. The macro handles this:

```c
Py_RETURN_NONE;
```

Equivalent conceptually to:

```c
Py_INCREF(Py_None);
return Py_None;
```

Boolean macros:

```c
Py_RETURN_TRUE;
Py_RETURN_FALSE;
```

These prevent common ownership mistakes.

## 62.15 Reference Leaks

A reference leak happens when code owns a reference but never releases it.

Example:

```c
PyObject *x = PyLong_FromLong(42);
Py_RETURN_NONE;  /* x leaked */
```

Correct:

```c
PyObject *x = PyLong_FromLong(42);
if (x == NULL) {
    return NULL;
}

Py_DECREF(x);
Py_RETURN_NONE;
```

Leaks are especially costly in long-running processes:

```text
web servers
workers
notebooks
databases
embedded runtimes
daemon processes
```

## 62.16 Use-After-Free

Use-after-free happens when code uses an object after its last reference has been released.

```c
PyObject *x = PyLong_FromLong(42);
Py_DECREF(x);

PyObject_Repr(x);  /* invalid */
```

This is undefined behavior. It may crash, corrupt memory, or appear to work during testing.

## 62.17 Double `DECREF`

A double `DECREF` releases ownership twice.

```c
PyObject *x = PyLong_FromLong(42);

Py_DECREF(x);
Py_DECREF(x);  /* wrong */
```

If the first `Py_DECREF` destroys the object, the second touches freed memory.

Double decrefs are often caused by unclear ownership after stealing APIs.

## 62.18 Debugging Reference Bugs

Debug builds of CPython help find reference leaks and memory errors.

Useful techniques:

```text
build CPython in debug mode
run targeted tests repeatedly
use refleak test mode
use sanitizers
use gdb or lldb
check failure paths
audit borrowed references around callbacks
```

Extension authors should also test under allocation failure when possible, because many ownership bugs occur only after partial initialization.

## 62.19 Ownership Rules as Local Invariants

Good extension code keeps ownership visible.

Prefer this style:

```c
PyObject *name = NULL;
PyObject *value = NULL;
PyObject *result = NULL;
```

Then maintain a local invariant:

```text
Every non-NULL variable either owns exactly one reference or owns none by documented convention.
```

When ownership transfers, make the code obvious:

```c
if (PyList_SetItem(list, i, value) < 0) {
    Py_DECREF(value);
    return NULL;
}

value = NULL;  /* ownership transferred */
```

Setting the variable to `NULL` after transfer prevents accidental cleanup later.

## 62.20 Chapter Summary

Reference ownership rules define how native code keeps Python objects alive. The pointer type `PyObject *` is not enough. The API contract tells you whether a reference is new, borrowed, or stolen.

The safe discipline is simple:

| Situation | Rule |
|---|---|
| You receive a new reference | Release it with `Py_DECREF` |
| You receive a borrowed reference | Do not release it |
| You need to keep a borrowed reference | Call `Py_INCREF` |
| An API steals your reference | Do not release it after successful transfer |
| You return an object to Python | Return a new reference |
| You call arbitrary Python code | Own anything you need afterward |

Most C API crashes come from violating these rules. Mastering them is required before writing extension modules, implementing new types, or reading CPython object code.