66. Capsules

A capsule is a CPython object that stores an opaque C pointer. It allows extension modules to expose native pointers to other extension modules without making those pointers directly visible to normal Python code.

Capsules are used when C code needs to share a function table, context pointer, type registry, library handle, allocator, or native resource across module boundaries.

At Python level, a capsule usually appears as an object with no useful behavior:

<capsule object "module.name" at 0x...>

At C level, it carries a pointer:

PyCapsule object
    name string
    C pointer
    optional destructor
    optional context

66.1 Why Capsules Exist

Extension modules often need to cooperate.

Suppose one module owns a native library context:

database_core
    owns native DB handle

Another module wants to use that handle:

database_extra
    adds fast analytics functions

Passing the raw pointer through Python code would be unsafe and meaningless. Python cannot validate, dereference, or manage such a pointer directly.

A capsule gives CPython a controlled container for that pointer.

native pointer
    wrapped in PyCapsule
        stored in module attribute
            imported by another C extension

This gives native code an agreed lookup mechanism while preserving Python-level opacity.

66.2 What a Capsule Stores

A capsule can store:

The name is important. It prevents unrelated code from accidentally retrieving a pointer of the wrong kind.

A common name format is:

"module.attribute"

Example:

"fastdb._C_API"

66.3 Creating a Capsule

Use PyCapsule_New.

PyObject *
PyCapsule_New(void *pointer,
              const char *name,
              PyCapsule_Destructor destructor);

Example:

static void *
api_table = NULL;

static PyObject *
make_capsule(void)
{
    return PyCapsule_New(
        api_table,
        "fastdb._C_API",
        NULL
    );
}

If creation succeeds, the result is a new reference. The caller owns it.

If creation fails, it returns NULL and sets an exception.

66.4 Adding a Capsule to a Module

A common pattern is to create the capsule during module initialization and expose it as a module attribute.

static int
export_api(PyObject *m)
{
    PyObject *capsule;

    capsule = PyCapsule_New(
        &FastDB_API,
        "fastdb._C_API",
        NULL
    );

    if (capsule == NULL) {
        return -1;
    }

    if (PyModule_AddObject(m, "_C_API", capsule) < 0) {
        Py_DECREF(capsule);
        return -1;
    }

    return 0;
}

PyModule_AddObject steals the reference on success. On failure, the caller must release it.

Python can see the attribute:

import fastdb
print(fastdb._C_API)

but cannot use the pointer directly.

66.5 Retrieving a Pointer

A consuming extension imports the module, gets the capsule attribute, then extracts the pointer.

PyObject *mod;
PyObject *capsule;
void *ptr;

mod = PyImport_ImportModule("fastdb");
if (mod == NULL) {
    return -1;
}

capsule = PyObject_GetAttrString(mod, "_C_API");
Py_DECREF(mod);

if (capsule == NULL) {
    return -1;
}

ptr = PyCapsule_GetPointer(capsule, "fastdb._C_API");
Py_DECREF(capsule);

if (ptr == NULL) {
    return -1;
}

The name passed to PyCapsule_GetPointer must match the capsule name.

This check is a core safety mechanism.

66.6 Name Validation

Capsule name validation prevents accidental pointer misuse.

Producer:

PyCapsule_New(ptr, "fastdb._C_API", NULL);

Consumer:

PyCapsule_GetPointer(capsule, "fastdb._C_API");

Correct name:

fastdb._C_API

Wrong name:

otherdb._C_API

If the names do not match, CPython raises an exception and returns NULL.

This does not prove the pointer is safe in a cryptographic sense. It prevents ordinary extension bugs and accidental ABI mismatches.

66.7 Exporting Function Tables

A capsule often stores a pointer to a C API table.

Example:

typedef struct {
    int api_version;
    PyObject *(*connect)(const char *path);
    PyObject *(*execute)(PyObject *conn, const char *sql);
    void (*close)(PyObject *conn);
} FastDB_APIObject;

static FastDB_APIObject FastDB_API = {
    1,
    fastdb_connect,
    fastdb_execute,
    fastdb_close
};

Export:

PyObject *capsule =
    PyCapsule_New(&FastDB_API,
                  "fastdb._C_API",
                  NULL);

Consumer:

FastDB_APIObject *api;

api = PyCapsule_GetPointer(capsule,
                           "fastdb._C_API");
if (api == NULL) {
    return -1;
}

if (api->api_version != 1) {
    PyErr_SetString(PyExc_RuntimeError,
                    "unsupported fastdb C API version");
    return -1;
}

The function table acts like a small C-level interface.

66.8 API Versioning

Capsules do not automatically solve ABI compatibility.

If the exported pointer refers to a struct, both producer and consumer must agree on its layout.

Bad design:

typedef struct {
    void (*a)(void);
    void (*b)(void);
} API;

Later changing it to:

typedef struct {
    void (*a)(void);
    int flags;
    void (*b)(void);
} API;

can break consumers compiled against the older layout.

Better design:

typedef struct {
    int version;
    size_t size;
    void (*a)(void);
    void (*b)(void);
} API;

Then consumers can check:

if (api->version != EXPECTED_VERSION ||
    api->size < sizeof(ExpectedPrefix)) {
    PyErr_SetString(PyExc_RuntimeError,
                    "incompatible C API");
    return -1;
}

A capsule stores a pointer. It does not describe or validate the memory layout behind that pointer.

66.9 Capsule Destructors

A capsule can have a destructor.

static void
capsule_destructor(PyObject *capsule)
{
    void *ptr;

    ptr = PyCapsule_GetPointer(capsule,
                               "fastdb.Context");
    if (ptr == NULL) {
        PyErr_Clear();
        return;
    }

    fastdb_context_free(ptr);
}

Create:

PyObject *capsule =
    PyCapsule_New(ctx,
                  "fastdb.Context",
                  capsule_destructor);

The destructor runs when the capsule is destroyed.

Use destructors for capsule-owned resources. Do not use them for pointers whose lifetime is managed elsewhere.

66.10 Destructor Error Handling

Capsule destructors cannot return errors.

If a destructor calls PyCapsule_GetPointer and the name check fails, it sets an exception. Since destructors cannot propagate that exception normally, they should clear it if they intentionally ignore failure.

ptr = PyCapsule_GetPointer(capsule, "fastdb.Context");
if (ptr == NULL) {
    PyErr_Clear();
    return;
}

Destructor code should be small, deterministic, and safe during interpreter shutdown.

66.11 Capsule Context

A capsule can store an additional context pointer.

Set:

PyCapsule_SetContext(capsule, context);

Get:

void *context = PyCapsule_GetContext(capsule);

The context can hold metadata, configuration, ABI information, or an owning module pointer.

Example:

typedef struct {
    int version;
    const char *build_id;
} APIContext;

The main capsule pointer might be the function table, while the context stores metadata.

66.12 Mutating a Capsule

Capsules can be modified:

PyCapsule_SetPointer(capsule, ptr);
PyCapsule_SetName(capsule, name);
PyCapsule_SetDestructor(capsule, destructor);
PyCapsule_SetContext(capsule, context);

In most API-export cases, mutation should be avoided after publication.

A published capsule should behave like an immutable capability:

create once
publish as module attribute
consume by exact name
do not mutate

Mutation makes consumers harder to reason about.

66.13 Capsules as Private C APIs

A common pattern is:

module exposes Python API
module also exposes _C_API capsule
other extension modules import _C_API
normal Python users ignore it

Example layout:

fastdb/
    __init__.py
    _core.so       exposes fastdb._C_API
    _analytics.so  imports fastdb._C_API

This allows multiple extension modules to share native internals without exposing unsupported Python APIs.

66.14 Capsules vs Python Objects

A capsule is useful when the consumer is C code.

It is usually poor Python API design to require normal Python users to handle capsules directly.

Better Python-level API:

conn = fastdb.connect("data.db")
fastdb_analytics.scan(conn)

Internal C-level implementation:

conn object contains native pointer
analytics extension imports fastdb C API capsule
analytics reads native pointer through supported API

Capsules should normally stay at module boundaries, not application boundaries.

66.15 Capsules vs ctypes

ctypes can pass raw pointers around at Python level.

Capsules serve a different purpose.

Capsules are best for trusted extension modules compiled against a shared contract.

They are not a general-purpose safe pointer mechanism for untrusted Python code.

66.16 Capsules and Lifetime

Pointer lifetime must be designed explicitly.

Common patterns:

A capsule does not automatically keep arbitrary native state alive unless the destructor and ownership model are correct.

For shared API tables, the pointer usually lives for the process lifetime.

For resource handles, the capsule may own the resource and clean it up.

66.17 Capsules and Module State

Modern extensions often use per-module state rather than process-wide globals. Capsules can expose API tables whose functions retrieve module state from objects or modules rather than relying on global variables.

Design:

capsule exports function table
function table functions accept PyObject * module or PyObject * instance
state is read from module/instance
no global mutable runtime state

This works better with subinterpreters and module reinitialization.

Avoid storing interpreter-specific mutable state in a static capsule pointer unless the extension is explicitly single-interpreter only.

66.18 Capsules and Subinterpreters

Subinterpreters complicate capsule design.

A process may contain multiple interpreters. A static C API table can be shared safely if it contains only function pointers and immutable metadata.

Unsafe capsule contents include:

borrowed Python object pointers
interpreter-specific exception objects
module state pointers from one interpreter
thread state pointers
mutable global runtime state

Safer capsule contents:

function pointers
ABI version numbers
constant metadata
pure native stateless helpers

If a capsule must expose interpreter-specific state, the consumer must retrieve it from the correct module instance in the current interpreter.

66.19 Capsules and Error Reporting

A C API exported through a capsule should follow Python C API error conventions.

For functions returning PyObject *:

non-NULL = success
NULL = exception set

For integer status:

0 = success
-1 = exception set

Example:

typedef struct {
    int version;
    int (*parse)(PyObject *obj, Result *out);
} API;

If parse fails, it should set a Python exception and return -1.

This keeps consuming extensions compatible with CPython’s normal error propagation model.

66.20 Practical Minimal Producer

Producer module:

#define PY_SSIZE_T_CLEAN
#include <Python.h>

typedef struct {
    int version;
    PyObject *(*add_one)(PyObject *);
} DemoAPI;

static PyObject *
demo_add_one(PyObject *obj)
{
    long v = PyLong_AsLong(obj);
    if (v == -1 && PyErr_Occurred()) {
        return NULL;
    }

    return PyLong_FromLong(v + 1);
}

static DemoAPI demo_api = {
    1,
    demo_add_one
};

static struct PyModuleDef module = {
    PyModuleDef_HEAD_INIT,
    "demo",
    "Demo module exporting a C API",
    -1,
    NULL
};

PyMODINIT_FUNC
PyInit_demo(void)
{
    PyObject *m;
    PyObject *capsule;

    m = PyModule_Create(&module);
    if (m == NULL) {
        return NULL;
    }

    capsule = PyCapsule_New(&demo_api,
                            "demo._C_API",
                            NULL);
    if (capsule == NULL) {
        Py_DECREF(m);
        return NULL;
    }

    if (PyModule_AddObject(m, "_C_API", capsule) < 0) {
        Py_DECREF(capsule);
        Py_DECREF(m);
        return NULL;
    }

    return m;
}

The module publishes _C_API as an opaque capsule.

66.21 Practical Minimal Consumer

Consumer extension:

#define PY_SSIZE_T_CLEAN
#include <Python.h>

typedef struct {
    int version;
    PyObject *(*add_one)(PyObject *);
} DemoAPI;

static DemoAPI *demo_api = NULL;

static int
import_demo_api(void)
{
    PyObject *mod;
    PyObject *capsule;

    mod = PyImport_ImportModule("demo");
    if (mod == NULL) {
        return -1;
    }

    capsule = PyObject_GetAttrString(mod, "_C_API");
    Py_DECREF(mod);

    if (capsule == NULL) {
        return -1;
    }

    demo_api = PyCapsule_GetPointer(capsule,
                                    "demo._C_API");
    Py_DECREF(capsule);

    if (demo_api == NULL) {
        return -1;
    }

    if (demo_api->version != 1) {
        PyErr_SetString(PyExc_RuntimeError,
                        "unsupported demo C API version");
        demo_api = NULL;
        return -1;
    }

    return 0;
}

static PyObject *
call_add_one(PyObject *self, PyObject *args)
{
    PyObject *obj;

    if (!PyArg_ParseTuple(args, "O", &obj)) {
        return NULL;
    }

    if (demo_api == NULL) {
        if (import_demo_api() < 0) {
            return NULL;
        }
    }

    return demo_api->add_one(obj);
}

The consumer never dereferences an unvalidated pointer.

66.22 Common Capsule Bugs

The most serious bugs come from treating a capsule as if it gives lifetime and type safety automatically. It gives neither. It gives a structured pointer transport mechanism.

66.23 Design Guidelines

For API capsules:

A good capsule API is boring: fixed name, fixed versioning rules, clear ownership, small surface area.

66.24 Chapter Summary

Capsules are CPython objects that wrap opaque C pointers. They are used to share native APIs and resources between extension modules while keeping the pointer opaque at Python level.

A capsule has a pointer, name, optional destructor, and optional context. The name validates that producer and consumer agree on the pointer type. The destructor can release owned native resources. The context can carry metadata.

Capsules are powerful but intentionally low-level. They do not provide automatic ABI safety, lifetime safety, or thread safety. Those properties must be designed into the C API that the capsule exposes.