71. Calling C From Python

Calling C from Python means exposing native functions, objects, or libraries so Python code can invoke them as if they were ordinary Python APIs.

This is the most common direction for native integration:

Python code
    calls wrapper
        wrapper calls C function
            C returns native result
        wrapper converts result to Python object

CPython supports several ways to do this:

Mechanism	Main use
C extension module	Fast, direct, compiled CPython integration
`ctypes`	Dynamic foreign function calls from Python
`cffi`	C interface with cleaner declaration model
Cython	Python-like syntax compiled to C extension code
Generated bindings	Large native library wrappers
Buffer protocol	Pass raw memory to native code
Capsules	Share C APIs between extension modules

This chapter focuses on the runtime idea: Python invokes a native function through a boundary layer that converts objects, manages errors, and controls ownership.

71.1 The Basic Shape

A Python call:

result = native.add(2, 3)

may reach a C function like:

static PyObject *
native_add(PyObject *self, PyObject *args)
{
    long a;
    long b;

    if (!PyArg_ParseTuple(args, "ll", &a, &b)) {
        return NULL;
    }

    return PyLong_FromLong(a + b);
}

The C function is wrapped in a method table:

static PyMethodDef methods[] = {
    {"add", native_add, METH_VARARGS, "Add two integers"},
    {NULL, NULL, 0, NULL}
};

From Python’s perspective, native.add is just callable.

From CPython’s perspective, it is a PyCFunctionObject pointing to native code.

71.2 What Happens During the Call

The call path looks like this:

Python bytecode
    LOAD_ATTR native.add
    CALL
        CPython recognizes C function object
        enters C function pointer
            parses Python arguments
            calls native logic
            converts native result
        returns PyObject *

The boundary does four jobs:

validate arguments
convert Python objects to C values
call native code
convert C result back to Python

Errors follow CPython’s normal convention:

success:
    return new PyObject *

failure:
    set Python exception
    return NULL

71.3 C Extension Modules

The most direct method is a C extension module.

Example module:

#define PY_SSIZE_T_CLEAN
#include <Python.h>

static PyObject *
add(PyObject *self, PyObject *args)
{
    long a;
    long b;

    if (!PyArg_ParseTuple(args, "ll", &a, &b)) {
        return NULL;
    }

    return PyLong_FromLong(a + b);
}

static PyMethodDef methods[] = {
    {"add", add, METH_VARARGS, "Add two integers"},
    {NULL, NULL, 0, NULL}
};

static struct PyModuleDef module = {
    PyModuleDef_HEAD_INIT,
    "native",
    "Native example module",
    -1,
    methods
};

PyMODINIT_FUNC
PyInit_native(void)
{
    return PyModule_Create(&module);
}

Python usage:

import native

print(native.add(2, 3))

This is the lowest-overhead and most CPython-specific route.

71.4 Argument Conversion

Python values are objects. C functions usually want primitive values, pointers, structs, or buffers.

Argument parsing converts between the two.

long value;

if (!PyArg_ParseTuple(args, "l", &value)) {
    return NULL;
}

Common conversions:

Format	Python input	C output
`i`	`int`	`int`
`l`	`int`	`long`
`d`	`float`	`double`
`s`	`str`	`const char *`
`y#`	bytes-like	`const char *`, length
`O`	any object	`PyObject *`

For binary data, prefer buffer-aware parsing or PyObject_GetBuffer when the function should accept bytes, bytearray, memoryview, arrays, or NumPy arrays.

71.5 Returning Values

Native results must be converted back to Python objects.

return PyLong_FromLong(value);

Examples:

C result	Python result API
`long`	`PyLong_FromLong`
`double`	`PyFloat_FromDouble`
`char *`	`PyUnicode_FromString` or `PyBytes_FromString`
buffer	`PyBytes_FromStringAndSize` or custom object
native handle	extension type or capsule
no result	`Py_RETURN_NONE`

Returning raw C values directly is invalid.

Incorrect:

return 42;

Correct:

return PyLong_FromLong(42);

71.6 Error Translation

C libraries often report errors through return codes, errno, status structs, or null pointers. The wrapper must translate those errors into Python exceptions.

Example with return code:

int rc = native_open(path);
if (rc < 0) {
    PyErr_SetString(PyExc_RuntimeError, "native_open failed");
    return NULL;
}

Example with errno:

FILE *fp = fopen(path, "rb");
if (fp == NULL) {
    return PyErr_SetFromErrnoWithFilename(
        PyExc_OSError,
        path
    );
}

Good wrappers preserve useful error information:

operation
native error code
filename or resource name
system errno
library error message

71.7 Ownership at the Boundary

Calling C from Python creates two ownership systems at once:

Python ownership:
    PyObject reference counts

Native ownership:
    malloc/free, handles, library-specific lifetimes

A wrapper must define where ownership moves.

Example native handle:

typedef struct {
    PyObject_HEAD
    NativeHandle *handle;
} ConnectionObject;

Deallocation:

static void
Connection_dealloc(ConnectionObject *self)
{
    if (self->handle != NULL) {
        native_close(self->handle);
        self->handle = NULL;
    }

    Py_TYPE(self)->tp_free((PyObject *)self);
}

Here Python owns the native handle through the wrapper object.

When the Python object dies, the native handle is closed.

71.8 Wrapping Native Handles

A C library might return:

NativeHandle *native_connect(const char *path);

The Python wrapper should not expose the pointer directly. It should wrap it in a Python object.

static PyObject *
connect(PyObject *self, PyObject *args)
{
    const char *path;
    NativeHandle *h;
    ConnectionObject *obj;

    if (!PyArg_ParseTuple(args, "s", &path)) {
        return NULL;
    }

    h = native_connect(path);
    if (h == NULL) {
        PyErr_SetString(PyExc_RuntimeError, "connect failed");
        return NULL;
    }

    obj = PyObject_New(ConnectionObject, &ConnectionType);
    if (obj == NULL) {
        native_close(h);
        return NULL;
    }

    obj->handle = h;
    return (PyObject *)obj;
}

Python usage:

conn = native.connect("data.db")

The pointer remains hidden inside the extension type.

71.9 `ctypes`

ctypes allows Python code to load shared libraries and call C functions dynamically.

Example:

import ctypes

libc = ctypes.CDLL(None)
libc.puts.argtypes = [ctypes.c_char_p]
libc.puts.restype = ctypes.c_int

libc.puts(b"hello from C")

ctypes is useful for:

small integrations
experiments
system library calls
simple C APIs
tools and scripts

It is weaker for large, complex, performance-sensitive bindings.

Problems include:

manual signature declarations
easy crashes from wrong types
limited C macro support
hard callback lifetime rules
less natural Python object integration

ctypes runs inside the Python process. A wrong pointer or wrong signature can crash the interpreter.

71.10 `cffi`

cffi provides a higher-level foreign function interface.

Example shape:

from cffi import FFI

ffi = FFI()
ffi.cdef("""
    int add(int a, int b);
""")

lib = ffi.dlopen("./libdemo.so")
print(lib.add(2, 3))

Compared with ctypes, cffi often gives a cleaner C declaration model.

It works well for:

wrapping existing C libraries
keeping declarations close to C headers
avoiding hand-written CPython C API code
supporting PyPy more naturally

The tradeoff is another dependency and a different build/runtime model.

71.11 Cython

Cython compiles Python-like code into C extension modules.

Example:

def add(long a, long b):
    return a + b

Cython generates the CPython C API wrapper code.

It is commonly used when:

Python code needs C-level speed
typed loops matter
native libraries need wrappers
manual C API code would be verbose

Cython still produces CPython extension modules in common usage, so reference counting, GIL rules, and ABI compatibility remain relevant underneath.

71.12 Passing Buffers to C

For data-heavy code, avoid converting every element into a Python object.

Use the buffer protocol.

Python:

data = bytearray(b"abcdef")
native.process(data)

static PyObject *
process(PyObject *self, PyObject *args)
{
    PyObject *obj;
    Py_buffer view;

    if (!PyArg_ParseTuple(args, "O", &obj)) {
        return NULL;
    }

    if (PyObject_GetBuffer(obj, &view, PyBUF_SIMPLE) < 0) {
        return NULL;
    }

    native_process(view.buf, view.len);

    PyBuffer_Release(&view);
    Py_RETURN_NONE;
}

This accepts many objects without copying:

bytes
bytearray
memoryview
array.array
mmap
NumPy arrays with compatible layout
custom buffer exporters

71.13 Releasing the GIL Around Native Work

If native code runs for a long time and does not touch Python objects, release the GIL.

Py_BEGIN_ALLOW_THREADS

native_process(view.buf, view.len);

Py_END_ALLOW_THREADS

Full pattern:

if (PyObject_GetBuffer(obj, &view, PyBUF_SIMPLE) < 0) {
    return NULL;
}

Py_BEGIN_ALLOW_THREADS

native_process(view.buf, view.len);

Py_END_ALLOW_THREADS

PyBuffer_Release(&view);
Py_RETURN_NONE;

While the GIL is released, native code must not call Python APIs, mutate Python objects, or touch reference counts.

71.14 Callbacks From C Into Python

Some C libraries accept callbacks.

Native library concept:

typedef void (*event_callback)(int code, void *user_data);

void native_set_callback(event_callback cb, void *user_data);

A Python wrapper may store a Python callable and pass a C trampoline.

static void
callback_trampoline(int code, void *user_data)
{
    PyObject *callback = (PyObject *)user_data;

    PyGILState_STATE state = PyGILState_Ensure();

    PyObject *arg = PyLong_FromLong(code);
    if (arg != NULL) {
        PyObject *res = PyObject_CallOneArg(callback, arg);
        Py_DECREF(arg);

        if (res == NULL) {
            PyErr_Print();
        } else {
            Py_DECREF(res);
        }
    } else {
        PyErr_Print();
    }

    PyGILState_Release(state);
}

The wrapper must hold an owned reference to the callback for as long as the C library may call it.

71.15 Callback Lifetime

Callback lifetime bugs are common.

Unsafe:

Python callback object is garbage-collected
C library still stores function pointer and user_data
native event fires
trampoline uses dangling PyObject *
crash

Safe wrapper design:

store callback as owned PyObject *
register trampoline with native library
unregister callback before wrapper deallocation
decref callback after unregistering

Deallocation must consider whether the native library may still call back from another thread.

71.16 Native Threads Calling Python

If a C library calls a callback from a native thread, the trampoline must acquire the GIL.

PyGILState_STATE state = PyGILState_Ensure();

/* Python API calls */

PyGILState_Release(state);

Without this, the callback may corrupt interpreter state.

This applies even if the original Python call registered the callback while holding the GIL. The later native callback may happen on a different thread.

71.17 Blocking C Calls

A C library call may block:

network read
database query
compression
GPU synchronization
file operation
lock wait

If the call does not use Python APIs, release the GIL around it.

Py_BEGIN_ALLOW_THREADS

rc = native_blocking_call(handle);

Py_END_ALLOW_THREADS

This allows other Python threads to continue.

But be careful with object lifetimes. Convert Python objects to stable native values before releasing the GIL.

71.18 Borrowed Data and GIL Release

This is unsafe:

const char *path = PyUnicode_AsUTF8(py_path);

Py_BEGIN_ALLOW_THREADS
native_open(path);
Py_END_ALLOW_THREADS

The UTF-8 pointer is tied to the Python object. If no owned reference keeps py_path alive, or if assumptions change, this can become fragile.

Safer pattern:

PyObject *bytes = PyUnicode_AsUTF8String(py_path);
if (bytes == NULL) {
    return NULL;
}

const char *path = PyBytes_AsString(bytes);
if (path == NULL) {
    Py_DECREF(bytes);
    return NULL;
}

Py_BEGIN_ALLOW_THREADS
rc = native_open(path);
Py_END_ALLOW_THREADS

Py_DECREF(bytes);

The bytes object owns the memory during the native call.

71.19 Mapping C Structs to Python

There are several ways to expose C structs.

Approach	Python view
Copy fields into dict	Simple but loses identity
Expose as tuple	Compact but less readable
Expose as dataclass-like object	Pythonic but needs wrapper
Extension type	Best for identity and methods
Buffer protocol	Best for raw arrays
Capsule	Best for opaque pointer sharing

For stable user-facing APIs, prefer extension types or ordinary Python objects over raw capsules.

71.20 Mapping Python Objects to C Structs

A wrapper may parse Python objects into native structs.

Example:

native.draw_rect({"x": 10, "y": 20, "w": 100, "h": 50})

C conversion:

typedef struct {
    int x;
    int y;
    int w;
    int h;
} Rect;

The wrapper should validate all fields before calling native code.

check object type
read attributes or mapping keys
convert each value
validate ranges
only then call native function

Do not let partially converted invalid data reach native code.

71.21 Memory Allocation Across Boundaries

Avoid freeing memory with a different allocator than the one that allocated it.

Bad pattern:

C library allocates with native_alloc
wrapper frees with PyMem_Free

Correct pattern:

C library allocates with native_alloc
wrapper frees with native_free

If exposing memory to Python, decide ownership clearly:

Memory owner	Cleanup site
Python object	`tp_dealloc`
Native library	native release function
Caller-owned buffer	caller lifetime
Shared buffer	explicit reference model

Allocator mismatches cause heap corruption.

71.22 Translating Native Error Codes

A native library may return domain-specific errors.

Example:

int rc = db_query(handle, sql);

if (rc != DB_OK) {
    PyErr_SetString(DbError, db_error_message(handle));
    return NULL;
}

Better wrappers define exception classes:

native.Error
native.ConnectionError
native.TimeoutError
native.ProtocolError
native.InvalidStateError

Python users should not need to inspect raw C error codes for normal control flow.

71.23 Signals and Interrupts

Long native calls can delay Python signal handling, including KeyboardInterrupt.

If a C loop runs while holding the GIL, periodically check signals:

if (PyErr_CheckSignals() < 0) {
    return NULL;
}

If the loop releases the GIL for a long blocking call, signal behavior depends on the native operation and platform. A robust wrapper may need cancellation support at the native library level.

71.24 Python API Shape

A good Python wrapper should not expose the C library too literally.

C APIs often use:

out parameters
integer status codes
manual init/free
raw pointers
global error state
flags bitmasks

Python APIs should prefer:

return values
exceptions
context managers
objects with methods
keyword arguments
enums
bytes-like inputs
iterators where natural

Example C style:

db_handle *h;
db_open(path, &h);
db_exec(h, sql);
db_close(h);

Python style:

with native.connect(path) as conn:
    conn.execute(sql)

The wrapper should translate not only types, but also idioms.

71.25 Context Managers for Native Resources

Native resources should often support with.

with native.open_resource(path) as r:
    r.process()

C type methods:

__enter__ returns self
__exit__ closes resource
close method is idempotent
dealloc closes if still open

This gives deterministic cleanup without relying only on object finalization.

71.26 Common Bugs

Bug	Cause
Interpreter crash	Wrong pointer, wrong signature, or ABI mismatch
Memory leak	Native handle never freed
Double free	Python and native code both own resource
Use-after-free	Python wrapper outlives native object
Deadlock	GIL held during blocking native call
Race	Native thread calls Python without GIL
Data corruption	Buffer assumed contiguous or writable incorrectly
Wrong results	C integer overflow or unchecked conversion
Poor Python API	C error codes exposed directly
Shutdown crash	Native callbacks fire after interpreter finalization

71.27 Practical Design Checklist

Before exposing a C function to Python:

Question	Required decision
Who owns each native pointer?	Python, native library, or shared
How are errors translated?	Exception classes and messages
Can the call block?	Release GIL if safe
Can native code call back?	Store callback and acquire GIL
Does it need raw memory?	Use buffer protocol
Can memory resize during use?	Pin or copy
Which allocator frees memory?	Match allocator family
What is the Pythonic API shape?	Objects, context managers, exceptions
What happens during shutdown?	Unregister callbacks and close handles

71.28 Chapter Summary

Calling C from Python is the foundation of CPython’s native extension ecosystem. Python code invokes a wrapper, the wrapper converts Python objects into native values, calls C code, converts the result back to Python, and translates native errors into Python exceptions.

The direct route is a C extension module, but Python also supports ctypes, cffi, Cython, generated bindings, capsules, and buffer-based interfaces.

Correct wrappers need more than a working function call. They need clear ownership, exact type conversion, disciplined error handling, GIL management, callback lifetime control, allocator consistency, and a Python API that hides unsafe C details behind normal Python semantics.

71. Calling C From Python

71.1 The Basic Shape

71.2 What Happens During the Call

71.3 C Extension Modules

71.4 Argument Conversion

71.5 Returning Values

71.6 Error Translation

71.7 Ownership at the Boundary

71.8 Wrapping Native Handles

71.9 ctypes

71.10 cffi

71.11 Cython

71.12 Passing Buffers to C

71.13 Releasing the GIL Around Native Work

71.14 Callbacks From C Into Python

71.15 Callback Lifetime

71.16 Native Threads Calling Python

71.17 Blocking C Calls

71.18 Borrowed Data and GIL Release

71.19 Mapping C Structs to Python

71.20 Mapping Python Objects to C Structs

71.21 Memory Allocation Across Boundaries

71.22 Translating Native Error Codes

71.23 Signals and Interrupts

71.24 Python API Shape

71.25 Context Managers for Native Resources

71.26 Common Bugs

71.27 Practical Design Checklist

71.28 Chapter Summary

71.9 `ctypes`

71.10 `cffi`