Circuit breakers stop you from hammering a failing downstream service. Bulkheads stop a failing downstream service from consuming all the resources you have -- even if you're not hammering it.
The name comes from ship design. A ship's hull is divided into watertight compartments. One compartment floods, the others stay dry. The ship stays afloat. Without bulkheads, one breach sinks everything.
The Problem Without Bulkheads
Imagine a service that calls three downstream APIs -- payment, inventory, and recommendations. You have a thread pool with 100 threads handling all outbound requests.
The recommendations API starts responding slowly. Requests pile up waiting. Threads get tied up waiting for responses that take 10 seconds instead of 100ms. Within minutes all 100 threads are occupied waiting on recommendations. Now payment requests and inventory requests can't get a thread either. Your service is down for all three functions because one non-critical API got slow.
A circuit breaker would help here once the recommendations API is actually failing -- but slow isn't failing. The requests are completing, just slowly. The circuit never trips. The thread pool still drains.
The Fix: Separate Resource Pools
Partition your resources so that one downstream dependency can only consume its own allocation.
Instead of one pool of 100 threads, you have three pools:
- Payment: 40 threads
- Inventory: 40 threads
- Recommendations: 20 threads
The recommendations API goes slow. It consumes all 20 of its threads. Payment and inventory are unaffected -- they have their own pools. The non-critical dependency is isolated in its own compartment.
Two Implementation Styles
Thread pool isolation is what Hystrix made popular. Each downstream call goes through a dedicated thread pool. Callers block on their own pool, not a shared one. Overhead: context switches between the caller thread and the pool thread for every call.
Semaphore isolation is lighter. Instead of separate thread pools, you put a semaphore in front of each dependency with a max concurrent count. Callers execute on their own thread but must acquire the semaphore first. No thread switching overhead. The tradeoff: if the call actually blocks the thread (synchronous I/O), you're still tying up the caller's thread. Better for fast calls, worse for slow ones.
For services making lots of downstream calls with variable latency, thread pool isolation is generally safer. The overhead is real but the isolation is stronger.
Combining with Circuit Breakers
Bulkheads and circuit breakers are complementary, not alternatives. They protect against different failure modes:
- Bulkhead: limits resource consumption from a slow or partially degraded dependency
- Circuit breaker: stops calls entirely when a dependency crosses a failure threshold
The typical pattern is both together. The bulkhead caps how many threads/connections one dependency can consume. The circuit breaker trips when that dependency starts returning errors. Between the two, you handle the full range of failure behavior from "slightly slow" to "completely down."
Netflix Hystrix combined both in a single abstraction. Resilience4j does the same. In a service mesh like Istio, connection pool settings are the bulkhead and outlier detection is the circuit breaker -- same ideas, configured at the mesh level rather than the application level.
What to Size the Pools At
This is the hard part. Too small and you're rejecting valid traffic. Too large and the isolation breaks down.
A reasonable starting point: look at your normal concurrency for each dependency, add headroom for spikes, and set the max at roughly 2x your normal peak. Monitor pool utilization in steady state -- if the recommendations pool is regularly at 18/20, you either need to size up or your recommendations calls are too slow.
Bulkheads don't fix slow dependencies. They contain the blast radius while you do.