# 71. Calling C From Python

# 71. Calling C From Python

Calling C from Python means exposing native functions, objects, or libraries so Python code can invoke them as if they were ordinary Python APIs.

This is the most common direction for native integration:

```text
Python code
    calls wrapper
        wrapper calls C function
            C returns native result
        wrapper converts result to Python object
```

CPython supports several ways to do this:

| Mechanism | Main use |
|---|---|
| C extension module | Fast, direct, compiled CPython integration |
| `ctypes` | Dynamic foreign function calls from Python |
| `cffi` | C interface with cleaner declaration model |
| Cython | Python-like syntax compiled to C extension code |
| Generated bindings | Large native library wrappers |
| Buffer protocol | Pass raw memory to native code |
| Capsules | Share C APIs between extension modules |

This chapter focuses on the runtime idea: Python invokes a native function through a boundary layer that converts objects, manages errors, and controls ownership.

## 71.1 The Basic Shape

A Python call:

```python
result = native.add(2, 3)
```

may reach a C function like:

```c
static PyObject *
native_add(PyObject *self, PyObject *args)
{
    long a;
    long b;

    if (!PyArg_ParseTuple(args, "ll", &a, &b)) {
        return NULL;
    }

    return PyLong_FromLong(a + b);
}
```

The C function is wrapped in a method table:

```c
static PyMethodDef methods[] = {
    {"add", native_add, METH_VARARGS, "Add two integers"},
    {NULL, NULL, 0, NULL}
};
```

From Python’s perspective, `native.add` is just callable.

From CPython’s perspective, it is a `PyCFunctionObject` pointing to native code.

## 71.2 What Happens During the Call

The call path looks like this:

```text
Python bytecode
    LOAD_ATTR native.add
    CALL
        CPython recognizes C function object
        enters C function pointer
            parses Python arguments
            calls native logic
            converts native result
        returns PyObject *
```

The boundary does four jobs:

```text
validate arguments
convert Python objects to C values
call native code
convert C result back to Python
```

Errors follow CPython’s normal convention:

```text
success:
    return new PyObject *

failure:
    set Python exception
    return NULL
```

## 71.3 C Extension Modules

The most direct method is a C extension module.

Example module:

```c
#define PY_SSIZE_T_CLEAN
#include <Python.h>

static PyObject *
add(PyObject *self, PyObject *args)
{
    long a;
    long b;

    if (!PyArg_ParseTuple(args, "ll", &a, &b)) {
        return NULL;
    }

    return PyLong_FromLong(a + b);
}

static PyMethodDef methods[] = {
    {"add", add, METH_VARARGS, "Add two integers"},
    {NULL, NULL, 0, NULL}
};

static struct PyModuleDef module = {
    PyModuleDef_HEAD_INIT,
    "native",
    "Native example module",
    -1,
    methods
};

PyMODINIT_FUNC
PyInit_native(void)
{
    return PyModule_Create(&module);
}
```

Python usage:

```python
import native

print(native.add(2, 3))
```

This is the lowest-overhead and most CPython-specific route.

## 71.4 Argument Conversion

Python values are objects. C functions usually want primitive values, pointers, structs, or buffers.

Argument parsing converts between the two.

```c
long value;

if (!PyArg_ParseTuple(args, "l", &value)) {
    return NULL;
}
```

Common conversions:

| Format | Python input | C output |
|---|---|---|
| `i` | `int` | `int` |
| `l` | `int` | `long` |
| `d` | `float` | `double` |
| `s` | `str` | `const char *` |
| `y#` | bytes-like | `const char *`, length |
| `O` | any object | `PyObject *` |

For binary data, prefer buffer-aware parsing or `PyObject_GetBuffer` when the function should accept `bytes`, `bytearray`, `memoryview`, arrays, or NumPy arrays.

## 71.5 Returning Values

Native results must be converted back to Python objects.

```c
return PyLong_FromLong(value);
```

Examples:

| C result | Python result API |
|---|---|
| `long` | `PyLong_FromLong` |
| `double` | `PyFloat_FromDouble` |
| `char *` | `PyUnicode_FromString` or `PyBytes_FromString` |
| buffer | `PyBytes_FromStringAndSize` or custom object |
| native handle | extension type or capsule |
| no result | `Py_RETURN_NONE` |

Returning raw C values directly is invalid.

Incorrect:

```c
return 42;
```

Correct:

```c
return PyLong_FromLong(42);
```

## 71.6 Error Translation

C libraries often report errors through return codes, `errno`, status structs, or null pointers. The wrapper must translate those errors into Python exceptions.

Example with return code:

```c
int rc = native_open(path);
if (rc < 0) {
    PyErr_SetString(PyExc_RuntimeError, "native_open failed");
    return NULL;
}
```

Example with `errno`:

```c
FILE *fp = fopen(path, "rb");
if (fp == NULL) {
    return PyErr_SetFromErrnoWithFilename(
        PyExc_OSError,
        path
    );
}
```

Good wrappers preserve useful error information:

```text
operation
native error code
filename or resource name
system errno
library error message
```

## 71.7 Ownership at the Boundary

Calling C from Python creates two ownership systems at once:

```text
Python ownership:
    PyObject reference counts

Native ownership:
    malloc/free, handles, library-specific lifetimes
```

A wrapper must define where ownership moves.

Example native handle:

```c
typedef struct {
    PyObject_HEAD
    NativeHandle *handle;
} ConnectionObject;
```

Deallocation:

```c
static void
Connection_dealloc(ConnectionObject *self)
{
    if (self->handle != NULL) {
        native_close(self->handle);
        self->handle = NULL;
    }

    Py_TYPE(self)->tp_free((PyObject *)self);
}
```

Here Python owns the native handle through the wrapper object.

When the Python object dies, the native handle is closed.

## 71.8 Wrapping Native Handles

A C library might return:

```c
NativeHandle *native_connect(const char *path);
```

The Python wrapper should not expose the pointer directly. It should wrap it in a Python object.

```c
static PyObject *
connect(PyObject *self, PyObject *args)
{
    const char *path;
    NativeHandle *h;
    ConnectionObject *obj;

    if (!PyArg_ParseTuple(args, "s", &path)) {
        return NULL;
    }

    h = native_connect(path);
    if (h == NULL) {
        PyErr_SetString(PyExc_RuntimeError, "connect failed");
        return NULL;
    }

    obj = PyObject_New(ConnectionObject, &ConnectionType);
    if (obj == NULL) {
        native_close(h);
        return NULL;
    }

    obj->handle = h;
    return (PyObject *)obj;
}
```

Python usage:

```python
conn = native.connect("data.db")
```

The pointer remains hidden inside the extension type.

## 71.9 `ctypes`

`ctypes` allows Python code to load shared libraries and call C functions dynamically.

Example:

```python
import ctypes

libc = ctypes.CDLL(None)
libc.puts.argtypes = [ctypes.c_char_p]
libc.puts.restype = ctypes.c_int

libc.puts(b"hello from C")
```

`ctypes` is useful for:

```text
small integrations
experiments
system library calls
simple C APIs
tools and scripts
```

It is weaker for large, complex, performance-sensitive bindings.

Problems include:

```text
manual signature declarations
easy crashes from wrong types
limited C macro support
hard callback lifetime rules
less natural Python object integration
```

`ctypes` runs inside the Python process. A wrong pointer or wrong signature can crash the interpreter.

## 71.10 `cffi`

`cffi` provides a higher-level foreign function interface.

Example shape:

```python
from cffi import FFI

ffi = FFI()
ffi.cdef("""
    int add(int a, int b);
""")

lib = ffi.dlopen("./libdemo.so")
print(lib.add(2, 3))
```

Compared with `ctypes`, `cffi` often gives a cleaner C declaration model.

It works well for:

```text
wrapping existing C libraries
keeping declarations close to C headers
avoiding hand-written CPython C API code
supporting PyPy more naturally
```

The tradeoff is another dependency and a different build/runtime model.

## 71.11 Cython

Cython compiles Python-like code into C extension modules.

Example:

```cython
def add(long a, long b):
    return a + b
```

Cython generates the CPython C API wrapper code.

It is commonly used when:

```text
Python code needs C-level speed
typed loops matter
native libraries need wrappers
manual C API code would be verbose
```

Cython still produces CPython extension modules in common usage, so reference counting, GIL rules, and ABI compatibility remain relevant underneath.

## 71.12 Passing Buffers to C

For data-heavy code, avoid converting every element into a Python object.

Use the buffer protocol.

Python:

```python
data = bytearray(b"abcdef")
native.process(data)
```

C:

```c
static PyObject *
process(PyObject *self, PyObject *args)
{
    PyObject *obj;
    Py_buffer view;

    if (!PyArg_ParseTuple(args, "O", &obj)) {
        return NULL;
    }

    if (PyObject_GetBuffer(obj, &view, PyBUF_SIMPLE) < 0) {
        return NULL;
    }

    native_process(view.buf, view.len);

    PyBuffer_Release(&view);
    Py_RETURN_NONE;
}
```

This accepts many objects without copying:

```text
bytes
bytearray
memoryview
array.array
mmap
NumPy arrays with compatible layout
custom buffer exporters
```

## 71.13 Releasing the GIL Around Native Work

If native code runs for a long time and does not touch Python objects, release the GIL.

```c
Py_BEGIN_ALLOW_THREADS

native_process(view.buf, view.len);

Py_END_ALLOW_THREADS
```

Full pattern:

```c
if (PyObject_GetBuffer(obj, &view, PyBUF_SIMPLE) < 0) {
    return NULL;
}

Py_BEGIN_ALLOW_THREADS

native_process(view.buf, view.len);

Py_END_ALLOW_THREADS

PyBuffer_Release(&view);
Py_RETURN_NONE;
```

While the GIL is released, native code must not call Python APIs, mutate Python objects, or touch reference counts.

## 71.14 Callbacks From C Into Python

Some C libraries accept callbacks.

Native library concept:

```c
typedef void (*event_callback)(int code, void *user_data);

void native_set_callback(event_callback cb, void *user_data);
```

A Python wrapper may store a Python callable and pass a C trampoline.

```c
static void
callback_trampoline(int code, void *user_data)
{
    PyObject *callback = (PyObject *)user_data;

    PyGILState_STATE state = PyGILState_Ensure();

    PyObject *arg = PyLong_FromLong(code);
    if (arg != NULL) {
        PyObject *res = PyObject_CallOneArg(callback, arg);
        Py_DECREF(arg);

        if (res == NULL) {
            PyErr_Print();
        } else {
            Py_DECREF(res);
        }
    } else {
        PyErr_Print();
    }

    PyGILState_Release(state);
}
```

The wrapper must hold an owned reference to the callback for as long as the C library may call it.

## 71.15 Callback Lifetime

Callback lifetime bugs are common.

Unsafe:

```text
Python callback object is garbage-collected
C library still stores function pointer and user_data
native event fires
trampoline uses dangling PyObject *
crash
```

Safe wrapper design:

```text
store callback as owned PyObject *
register trampoline with native library
unregister callback before wrapper deallocation
decref callback after unregistering
```

Deallocation must consider whether the native library may still call back from another thread.

## 71.16 Native Threads Calling Python

If a C library calls a callback from a native thread, the trampoline must acquire the GIL.

```c
PyGILState_STATE state = PyGILState_Ensure();

/* Python API calls */

PyGILState_Release(state);
```

Without this, the callback may corrupt interpreter state.

This applies even if the original Python call registered the callback while holding the GIL. The later native callback may happen on a different thread.

## 71.17 Blocking C Calls

A C library call may block:

```text
network read
database query
compression
GPU synchronization
file operation
lock wait
```

If the call does not use Python APIs, release the GIL around it.

```c
Py_BEGIN_ALLOW_THREADS

rc = native_blocking_call(handle);

Py_END_ALLOW_THREADS
```

This allows other Python threads to continue.

But be careful with object lifetimes. Convert Python objects to stable native values before releasing the GIL.

## 71.18 Borrowed Data and GIL Release

This is unsafe:

```c
const char *path = PyUnicode_AsUTF8(py_path);

Py_BEGIN_ALLOW_THREADS
native_open(path);
Py_END_ALLOW_THREADS
```

The UTF-8 pointer is tied to the Python object. If no owned reference keeps `py_path` alive, or if assumptions change, this can become fragile.

Safer pattern:

```c
PyObject *bytes = PyUnicode_AsUTF8String(py_path);
if (bytes == NULL) {
    return NULL;
}

const char *path = PyBytes_AsString(bytes);
if (path == NULL) {
    Py_DECREF(bytes);
    return NULL;
}

Py_BEGIN_ALLOW_THREADS
rc = native_open(path);
Py_END_ALLOW_THREADS

Py_DECREF(bytes);
```

The bytes object owns the memory during the native call.

## 71.19 Mapping C Structs to Python

There are several ways to expose C structs.

| Approach | Python view |
|---|---|
| Copy fields into dict | Simple but loses identity |
| Expose as tuple | Compact but less readable |
| Expose as dataclass-like object | Pythonic but needs wrapper |
| Extension type | Best for identity and methods |
| Buffer protocol | Best for raw arrays |
| Capsule | Best for opaque pointer sharing |

For stable user-facing APIs, prefer extension types or ordinary Python objects over raw capsules.

## 71.20 Mapping Python Objects to C Structs

A wrapper may parse Python objects into native structs.

Example:

```python
native.draw_rect({"x": 10, "y": 20, "w": 100, "h": 50})
```

C conversion:

```c
typedef struct {
    int x;
    int y;
    int w;
    int h;
} Rect;
```

The wrapper should validate all fields before calling native code.

```text
check object type
read attributes or mapping keys
convert each value
validate ranges
only then call native function
```

Do not let partially converted invalid data reach native code.

## 71.21 Memory Allocation Across Boundaries

Avoid freeing memory with a different allocator than the one that allocated it.

Bad pattern:

```text
C library allocates with native_alloc
wrapper frees with PyMem_Free
```

Correct pattern:

```text
C library allocates with native_alloc
wrapper frees with native_free
```

If exposing memory to Python, decide ownership clearly:

| Memory owner | Cleanup site |
|---|---|
| Python object | `tp_dealloc` |
| Native library | native release function |
| Caller-owned buffer | caller lifetime |
| Shared buffer | explicit reference model |

Allocator mismatches cause heap corruption.

## 71.22 Translating Native Error Codes

A native library may return domain-specific errors.

Example:

```c
int rc = db_query(handle, sql);

if (rc != DB_OK) {
    PyErr_SetString(DbError, db_error_message(handle));
    return NULL;
}
```

Better wrappers define exception classes:

```text
native.Error
native.ConnectionError
native.TimeoutError
native.ProtocolError
native.InvalidStateError
```

Python users should not need to inspect raw C error codes for normal control flow.

## 71.23 Signals and Interrupts

Long native calls can delay Python signal handling, including `KeyboardInterrupt`.

If a C loop runs while holding the GIL, periodically check signals:

```c
if (PyErr_CheckSignals() < 0) {
    return NULL;
}
```

If the loop releases the GIL for a long blocking call, signal behavior depends on the native operation and platform. A robust wrapper may need cancellation support at the native library level.

## 71.24 Python API Shape

A good Python wrapper should not expose the C library too literally.

C APIs often use:

```text
out parameters
integer status codes
manual init/free
raw pointers
global error state
flags bitmasks
```

Python APIs should prefer:

```text
return values
exceptions
context managers
objects with methods
keyword arguments
enums
bytes-like inputs
iterators where natural
```

Example C style:

```c
db_handle *h;
db_open(path, &h);
db_exec(h, sql);
db_close(h);
```

Python style:

```python
with native.connect(path) as conn:
    conn.execute(sql)
```

The wrapper should translate not only types, but also idioms.

## 71.25 Context Managers for Native Resources

Native resources should often support `with`.

```python
with native.open_resource(path) as r:
    r.process()
```

C type methods:

```text
__enter__ returns self
__exit__ closes resource
close method is idempotent
dealloc closes if still open
```

This gives deterministic cleanup without relying only on object finalization.

## 71.26 Common Bugs

| Bug | Cause |
|---|---|
| Interpreter crash | Wrong pointer, wrong signature, or ABI mismatch |
| Memory leak | Native handle never freed |
| Double free | Python and native code both own resource |
| Use-after-free | Python wrapper outlives native object |
| Deadlock | GIL held during blocking native call |
| Race | Native thread calls Python without GIL |
| Data corruption | Buffer assumed contiguous or writable incorrectly |
| Wrong results | C integer overflow or unchecked conversion |
| Poor Python API | C error codes exposed directly |
| Shutdown crash | Native callbacks fire after interpreter finalization |

## 71.27 Practical Design Checklist

Before exposing a C function to Python:

| Question | Required decision |
|---|---|
| Who owns each native pointer? | Python, native library, or shared |
| How are errors translated? | Exception classes and messages |
| Can the call block? | Release GIL if safe |
| Can native code call back? | Store callback and acquire GIL |
| Does it need raw memory? | Use buffer protocol |
| Can memory resize during use? | Pin or copy |
| Which allocator frees memory? | Match allocator family |
| What is the Pythonic API shape? | Objects, context managers, exceptions |
| What happens during shutdown? | Unregister callbacks and close handles |

## 71.28 Chapter Summary

Calling C from Python is the foundation of CPython’s native extension ecosystem. Python code invokes a wrapper, the wrapper converts Python objects into native values, calls C code, converts the result back to Python, and translates native errors into Python exceptions.

The direct route is a C extension module, but Python also supports `ctypes`, `cffi`, Cython, generated bindings, capsules, and buffer-based interfaces.

Correct wrappers need more than a working function call. They need clear ownership, exact type conversion, disciplined error handling, GIL management, callback lifetime control, allocator consistency, and a Python API that hides unsafe C details behind normal Python semantics.