59. `multiprocessing`

The multiprocessing module provides process-based concurrency. It lets Python programs create child processes, communicate between them, share limited state, coordinate execution, and distribute work across CPU cores.

For CPython internals, multiprocessing matters because it is the standard library answer to a central runtime constraint: ordinary Python threads in one traditional CPython interpreter are limited by the Global Interpreter Lock during bytecode execution. Separate processes have separate interpreters, separate heaps, separate GILs, and separate address spaces.

59.1 The Role of multiprocessing

multiprocessing gives Python a high-level API for OS processes.

Example:

from multiprocessing import Process

def worker():
    print("child process")

p = Process(target=worker)
p.start()
p.join()

This creates a child process, runs worker() inside it, and waits for it to finish.

Common uses include:

CPU-bound parallel work
isolation between tasks
fault containment
parallel data processing
background workers
producer-consumer pipelines
process pools

The core model is:

parent process
    ↓ starts
child process
    ↓ runs Python code in separate interpreter
parent waits or communicates

Each process has its own Python runtime state.

59.2 Process Isolation

A process is an operating system execution unit with its own virtual address space.

That means this code does not share ordinary Python objects:

from multiprocessing import Process

x = []

def worker():
    x.append(1)
    print("child:", x)

p = Process(target=worker)
p.start()
p.join()

print("parent:", x)

Typical output:

child: [1]
parent: []

The child modifies its own copy or independently created version of x. The parent’s list remains unchanged.

This is the most important distinction between threads and processes.

59.3 Relationship to the GIL

Traditional CPython uses a Global Interpreter Lock per interpreter. In one interpreter, only one thread runs Python bytecode at a time.

multiprocessing avoids that limitation by using multiple OS processes.

process A
    CPython interpreter
    GIL A

process B
    CPython interpreter
    GIL B

process C
    CPython interpreter
    GIL C

Since each process has its own interpreter and GIL, CPU-bound Python code can run in parallel on multiple cores.

Example:

from multiprocessing import Pool

def square(x):
    return x * x

with Pool() as pool:
    print(pool.map(square, range(10)))

The work is distributed across worker processes.

59.4 Process Creation Methods

CPython supports several start methods.

Check available methods:

import multiprocessing as mp

print(mp.get_all_start_methods())
print(mp.get_start_method())

Set a method:

import multiprocessing as mp

if __name__ == "__main__":
    mp.set_start_method("spawn")

The start method strongly affects semantics, performance, and safety.

59.5 fork

With fork, the child process starts as a copy of the parent process.

Conceptually:

parent process memory
    ↓ fork
child process sees copied memory

Modern operating systems usually implement this with copy-on-write pages. Physical memory is not copied immediately. Pages are copied only when modified.

Advantages:

fast startup
inherits loaded modules
inherits initialized data
low initial memory cost with copy-on-write

Risks:

unsafe after threads have started
inherits locks in unknown states
inherits open file descriptors
inherits partially initialized runtime state
can interact badly with native libraries

Forking a multi-threaded process is especially delicate. Only the thread that calls fork() survives in the child, but locks held by other threads may remain locked.

59.6 spawn

With spawn, the child starts a fresh Python interpreter.

Conceptually:

parent process
    ↓ create new process
fresh Python interpreter
    ↓ import main module
    ↓ unpickle target and arguments

Advantages:

clean interpreter state
safer with threads
portable to Windows
avoids inherited lock problems

Costs:

slower startup
requires picklable targets
imports main module again
does not inherit live Python objects directly

With spawn, this guard is essential:

if __name__ == "__main__":
    ...

Without it, importing the main module in the child may recursively create new child processes.

59.7 forkserver

With forkserver, a dedicated server process is started. Later children are forked from that server.

Conceptually:

main process
    ↓ starts fork server
fork server
    ↓ forks clean workers on request
worker processes

This combines some benefits of fork and spawn.

It avoids forking from a complex multi-threaded main process while still allowing relatively efficient child creation.

It is mainly available on Unix-like platforms.

59.8 The Main Module Rule

When using spawn, child processes import the main module.

Therefore, process creation must be protected:

from multiprocessing import Process

def worker():
    print("work")

if __name__ == "__main__":
    p = Process(target=worker)
    p.start()
    p.join()

Top-level process creation is unsafe:

# Bad with spawn
p = Process(target=worker)
p.start()

Why:

parent imports main module
    ↓ creates process
child imports main module
    ↓ creates process again
recursive process creation

The guard makes top-level code import-safe.

59.9 Pickling and Process Boundaries

Processes do not share normal Python objects. Arguments and results usually cross process boundaries through serialization.

multiprocessing mostly uses pickle.

Example:

from multiprocessing import Process

def worker(data):
    print(data)

if __name__ == "__main__":
    p = Process(target=worker, args=({"x": 1},))
    p.start()
    p.join()

The dictionary is serialized in the parent and reconstructed in the child.

This implies:

target function must be importable
arguments must be picklable
return values through pools must be picklable
closures and lambdas often fail with spawn
large objects have serialization cost

Good target:

def worker(x):
    return x * x

Problematic target:

worker = lambda x: x * x

Top-level functions are easier to pickle than local functions or lambdas.

59.10 Process

Process is the basic unit.

from multiprocessing import Process

def run(name):
    print("hello", name)

if __name__ == "__main__":
    p = Process(target=run, args=("worker-1",))
    p.start()
    p.join()

    print(p.exitcode)

Important methods and attributes:

A process object in the parent is a controller for the child. It is not the child’s memory.

59.11 Exit Codes

A child process has an exit code.

from multiprocessing import Process
import sys

def worker():
    sys.exit(3)

if __name__ == "__main__":
    p = Process(target=worker)
    p.start()
    p.join()

    print(p.exitcode)

Typical output:

3

Conventions:

If a child raises an unhandled exception, it exits with a nonzero code and prints a traceback to its stderr.

59.12 Queues

multiprocessing.Queue provides process-safe message passing.

from multiprocessing import Process, Queue

def worker(q):
    q.put("done")

if __name__ == "__main__":
    q = Queue()

    p = Process(target=worker, args=(q,))
    p.start()

    print(q.get())

    p.join()

A queue serializes objects with pickle and sends them through an inter-process communication channel.

Conceptually:

producer process
    pickle object
    send bytes through pipe
consumer process
    receive bytes
    unpickle object

Queues are good for task pipelines and result collection.

59.13 Pipes

multiprocessing.Pipe creates connected endpoints.

from multiprocessing import Process, Pipe

def worker(conn):
    conn.send("hello")
    conn.close()

if __name__ == "__main__":
    parent_conn, child_conn = Pipe()

    p = Process(target=worker, args=(child_conn,))
    p.start()

    print(parent_conn.recv())

    p.join()

A pipe is lower-level than a queue.

Use pipes for direct two-party communication. Use queues for many producers or consumers.

59.14 Pools

Pool manages a group of worker processes.

from multiprocessing import Pool

def square(x):
    return x * x

if __name__ == "__main__":
    with Pool(processes=4) as pool:
        results = pool.map(square, range(10))

    print(results)

Common pool methods:

Pool model:

parent
    ↓ submit tasks
worker processes
    ↓ execute tasks
parent
    ↓ collect results

Pools are convenient, but serialization and scheduling overhead matter.

59.15 Chunking

For Pool.map(), work is sent in chunks.

Too-small chunks cause overhead. Too-large chunks hurt load balancing.

Example:

with Pool(4) as pool:
    results = pool.map(square, range(1000), chunksize=50)

Chunking tradeoff:

For short tasks, chunking can dominate performance.

59.16 Shared Values and Arrays

multiprocessing.Value and Array allocate shared memory wrappers for simple C-style data.

from multiprocessing import Process, Value

def worker(counter):
    with counter.get_lock():
        counter.value += 1

if __name__ == "__main__":
    counter = Value("i", 0)

    processes = [Process(target=worker, args=(counter,)) for _ in range(4)]

    for p in processes:
        p.start()

    for p in processes:
        p.join()

    print(counter.value)

The type code "i" means C int.

Shared objects need synchronization when mutated from multiple processes.

59.17 Shared Memory

multiprocessing.shared_memory provides named shared memory blocks.

from multiprocessing import shared_memory

shm = shared_memory.SharedMemory(create=True, size=10)

try:
    shm.buf[:5] = b"hello"
    print(bytes(shm.buf[:5]))
finally:
    shm.close()
    shm.unlink()

Shared memory is useful for large binary data because it avoids pickling and copying.

Conceptually:

process A
    maps shared memory block

process B
    maps same shared memory block

Only bytes are shared. You must define the data layout and synchronization.

59.18 Managers

A manager process hosts Python objects and exposes proxies to other processes.

from multiprocessing import Manager, Process

def worker(shared_list):
    shared_list.append("x")

if __name__ == "__main__":
    with Manager() as manager:
        xs = manager.list()

        p = Process(target=worker, args=(xs,))
        p.start()
        p.join()

        print(list(xs))

Managers are flexible but slower than queues or shared memory.

They work by proxy calls:

worker process
    ↓ proxy method call
manager process
    ↓ mutates real object
worker process
    ↓ receives result

Use managers for coordination, not high-throughput data paths.

59.19 Locks

multiprocessing provides synchronization primitives.

from multiprocessing import Process, Lock

def worker(lock):
    with lock:
        print("critical section")

if __name__ == "__main__":
    lock = Lock()

    processes = [Process(target=worker, args=(lock,)) for _ in range(4)]

    for p in processes:
        p.start()

    for p in processes:
        p.join()

Common primitives:

These map to OS-level or multiprocessing-managed synchronization mechanisms.

59.20 Daemon Processes

A process can be marked daemon.

p = Process(target=worker)
p.daemon = True

Daemon child processes are terminated when the parent process exits.

They are not allowed to create child processes themselves.

Daemon processes are useful for auxiliary background work, but they are poor for work requiring reliable cleanup.

59.21 Termination

terminate() stops a process abruptly.

p.terminate()
p.join()

This does not run normal Python cleanup reliably in the child.

Consequences may include:

finally blocks skipped
locks left acquired
queues corrupted
temporary files not cleaned
shared resources leaked

Prefer cooperative shutdown:

from multiprocessing import Event

stop = Event()

def worker(stop):
    while not stop.is_set():
        do_work()

Then signal:

stop.set()

and join.

59.22 Process Pools and Shutdown

A pool should be closed or used as a context manager.

Good:

with Pool() as pool:
    results = pool.map(square, range(10))

Manual form:

pool = Pool()
try:
    results = pool.map(square, range(10))
finally:
    pool.close()
    pool.join()

Important methods:

Using the context manager handles shutdown more safely.

59.23 Exceptions in Pools

Exceptions raised in workers are sent back to the parent.

from multiprocessing import Pool

def fail(x):
    raise ValueError(x)

if __name__ == "__main__":
    with Pool(2) as pool:
        try:
            pool.map(fail, [1, 2, 3])
        except ValueError as exc:
            print("caught:", exc)

The exception is reconstructed in the parent process.

Tracebacks may be less direct than ordinary in-process exceptions because execution happened elsewhere.

For async calls:

result = pool.apply_async(fail, (1,))
result.get()

get() re-raises the worker exception.

59.24 Initializers

Pools can run initializer functions in each worker.

from multiprocessing import Pool

_state = None

def init_worker(value):
    global _state
    _state = value

def work(x):
    return _state + x

if __name__ == "__main__":
    with Pool(initializer=init_worker, initargs=(10,)) as pool:
        print(pool.map(work, [1, 2, 3]))

Initializers are useful for per-process setup:

open database connections
load models
initialize caches
set process-global config
ignore signals
configure logging

Remember each worker has separate state.

59.25 Logging

Multiprocessing complicates logging because several processes may write concurrently.

Naive logging:

print("message")

can interleave output.

A robust pattern is to send log records through a queue to a single logging process or listener thread.

Conceptually:

worker process
    ↓ log record queue
logging listener
    ↓ writes files or stdout

This avoids corrupted or interleaved output.

59.26 Signals

Signals interact with multiprocessing at the OS level.

On Unix-like systems, signals are delivered to processes, not Python tasks. A parent may receive SIGINT, while children may also receive it depending on process groups and terminal state.

Robust process systems usually handle:

SIGINT
SIGTERM
child cleanup
queue draining
pool termination
graceful shutdown deadlines

Signal behavior differs across platforms, especially Windows.

59.27 File Descriptors and Handles

Child processes may inherit file descriptors or handles, depending on platform and start method.

With fork, the child inherits many open resources.

With spawn, inheritance is more controlled.

Inherited resources can include:

files
sockets
pipes
locks
database connections
random generator state
logging handlers

Some inherited resources are safe. Others must be reopened in the child.

This is one reason pool initializers are useful.

59.28 Randomness

With fork, pseudo-random generator state may be copied into children.

If multiple children inherit the same random state, they may produce identical sequences unless reseeded.

Example mitigation:

import os
import random

def init_worker():
    random.seed(os.getpid())

For cryptographic randomness, use APIs backed by OS entropy sources.

59.29 Memory Cost

Processes are heavier than threads.

Costs include:

separate interpreter state
separate heaps
separate module imports
serialization overhead
IPC buffers
OS process scheduling

With fork, copy-on-write reduces initial memory cost, but writing to inherited pages creates real copies.

With spawn, each process imports modules independently, often using more memory.

For large read-only data, strategies include:

load before fork to exploit copy-on-write
use shared memory
memory-map files
send small indexes instead of large objects
use worker initializers

59.30 Performance Model

multiprocessing improves performance when parallel work is large enough to amortize overhead.

Costs:

process startup
pickling arguments
IPC transfer
unpickling arguments
scheduling work
pickling results
unpickling results

Good workload:

large CPU-bound tasks
limited communication
independent inputs
small result objects
long-lived worker pool

Poor workload:

tiny functions
large objects copied repeatedly
shared mutable state
high-frequency synchronization
tasks requiring many round trips

Use measurement before assuming multiprocessing helps.

59.31 Relationship to subprocess

multiprocessing runs Python functions in child Python processes.

subprocess runs external programs.

Example contrast:

# multiprocessing
Process(target=worker).start()

# subprocess
subprocess.run(["python", "script.py"])

multiprocessing provides Python object communication. subprocess provides process execution and byte streams.

59.32 Relationship to concurrent.futures

concurrent.futures.ProcessPoolExecutor provides a simpler pool API.

from concurrent.futures import ProcessPoolExecutor

def square(x):
    return x * x

if __name__ == "__main__":
    with ProcessPoolExecutor() as ex:
        print(list(ex.map(square, range(10))))

It is built on process-based execution and shares many constraints:

picklable functions
picklable arguments
process startup costs
separate memory
main module guard

Use multiprocessing when you need lower-level process control. Use ProcessPoolExecutor when a future-based API is enough.

59.33 Relationship to CPython Internals

multiprocessing touches several CPython internals:

This module is a runtime boundary layer between Python and the operating system process model.

59.34 Common Mistakes

Common errors include:

Most multiprocessing bugs come from forgetting the process boundary.

59.35 Practical Design Rules

Use these rules for robust multiprocessing code:

put process creation behind the main guard
prefer top-level functions as targets
send small immutable messages
use queues for communication
use shared memory for large numeric or binary data
initialize per-process resources in workers
close and join pools
prefer cooperative shutdown
avoid global mutable state
measure serialization cost

A clean process program looks like message passing, not shared-object programming.

59.36 Chapter Summary

The multiprocessing module provides process-based concurrency for CPython. It creates child interpreters, communicates through queues, pipes, shared memory, managers, and serialized messages, and provides process pools for parallel work.

For CPython internals, multiprocessing is important because it bypasses one-interpreter GIL limits by using separate processes. It also exposes the runtime consequences of process isolation: pickling, interpreter startup, import behavior, memory separation, file descriptor inheritance, signal handling, and explicit communication.