Reduce before theorizing
The repository isolates the failure from the original application so each process transition can be observed.
ShippedConcurrency · 2025
Systems debugging
Celery Fork-Safety Investigation
A vector-store operation inside a Celery task appeared to hang for no reason. The real failure crossed Python, Rust, SQLite, Tokio, Celery's prefork pool, and two operating systems. I reduced it to a public reproduction and followed it down to the inherited synchronization state.
System map
Initialize Chroma
Celery forks
Locks are copied
Owner threads vanish
Hang or crash
Concurrency
- Linux
- Permanent futex wait
- macOS
- libdispatch SIGTRAP
- Root cause
- Forked synchronization state
01
The misleading symptom
The task stopped while adding documents to ChromaDB. Treating that as an application-level timeout or a database bug would have hidden the mechanism: Chroma's Rust and SQLite internals had already created threads and synchronization primitives before Celery called fork.
02
What fork preserves—and what it does not
The child receives a copy of the parent's address space, including mutex and semaphore state, but only the thread that called fork survives. A lock can therefore remain marked as owned by a thread that no longer exists in the child.
The first later operation touching that inherited state exposes the damage. The place where execution stops is not necessarily the place where the bug was introduced.
03
Same cause, platform-specific failure
Inside Linux containers, GDB traced the child to a futex wait that could never complete. On macOS, LLDB showed libdispatch detecting corrupted semaphore state and aborting with SIGTRAP.
The repository includes both paths because a single-platform reproduction makes this class of failure look library-specific. Seeing both makes the process-model bug much harder to misdiagnose.
04
The fixes follow from the mechanism
Initialize fork-unsafe resources inside the worker after the fork, or choose a pool that does not fork. Restarting, adding retries, or increasing timeouts cannot release a lock whose owning thread ceased to exist.
Engineering choices
The decisions that shaped it.
The implementation details matter because each one closes a specific failure mode or keeps an important boundary visible.
Debug both operating systems
GDB and LLDB evidence ties the different surface failures back to the same inherited-state mechanism.
Publish executable evidence
Docker and native commands let another engineer reproduce the deadlock and crash instead of trusting a postmortem.
Where it landed
A concrete explanation for a class of 'Celery hung' failures, plus reproductions and fixes that generalize to database clients, async runtimes, thread pools, and other stateful libraries initialized before fork.
Next case study
Ghostty × Yazi theme sync