Circuit Breakers

State Machine Implementation

struct CircuitBreaker:
    state: "CLOSED" | "OPEN" | "HALF_OPEN"
    failure_count: int
    failure_threshold: int        // N failures → trip to OPEN
    success_count: int
    success_threshold: int        // N successes in HALF_OPEN → CLOSED
    open_at: timestamp            // when did circuit open?
    reset_timeout: duration       // how long before trying HALF_OPEN?

function call_through(cb, dependency_fn):
    if cb.state == "OPEN":
        // Check if it's time to try HALF_OPEN
        if current_time() - cb.open_at >= cb.reset_timeout:
            transition_to_half_open(cb)
        else:
            raise CircuitOpenError("dependency unavailable — fast-fail")

    if cb.state == "HALF_OPEN":
        // Only one probe request passes through
        try:
            result = dependency_fn()
            on_half_open_success(cb)
            return result
        except Exception as e:
            on_half_open_failure(cb)
            raise

    // State: CLOSED — pass through normally
    try:
        result = dependency_fn()
        on_success(cb)
        return result
    except Exception as e:
        on_failure(cb)
        raise

function on_failure(cb):
    cb.failure_count += 1
    if cb.failure_count >= cb.failure_threshold:
        transition_to_open(cb)

function on_success(cb):
    cb.failure_count = 0   // reset on success in CLOSED state

function transition_to_open(cb):
    cb.state = "OPEN"
    cb.open_at = current_time()
    log("Circuit breaker OPENED for dependency: " + cb.name)
    emit_metric("circuit_breaker.opened", cb.name)

function transition_to_half_open(cb):
    cb.state = "HALF_OPEN"
    cb.success_count = 0
    log("Circuit breaker transitioning to HALF_OPEN: " + cb.name)

function on_half_open_success(cb):
    cb.success_count += 1
    if cb.success_count >= cb.success_threshold:
        cb.state = "CLOSED"
        cb.failure_count = 0
        log("Circuit breaker CLOSED: dependency recovered")
        emit_metric("circuit_breaker.closed", cb.name)

function on_half_open_failure(cb):
    cb.state = "OPEN"
    cb.open_at = current_time()
    log("Circuit breaker returned to OPEN: probe failed")

Sliding Window Failure Rate

// Count-based threshold is naive: 10 failures in 10 requests = 100% failure rate
// But 10 failures in 10,000 requests = 0.1% failure rate
// A count threshold does not distinguish between these cases

// Better: use a sliding window to measure failure rate, not count

struct SlidingWindowCB:
    window: RingBuffer[bool]   // true = success, false = failure
    window_size: int           // e.g., 100 most recent calls
    failure_rate_threshold: float   // e.g., 0.5 = 50% failure rate
    minimum_calls: int         // don't trip on first 5 calls of a cold start

function measure_failure_rate(cb):
    if len(cb.window) < cb.minimum_calls:
        return 0.0    // insufficient data
    failures = count(cb.window, where=false)
    return failures / len(cb.window)

function on_call_result(cb, success: bool):
    cb.window.push(success)  // replaces oldest entry if full

    if measure_failure_rate(cb) >= cb.failure_rate_threshold:
        if cb.state == "CLOSED":
            transition_to_open(cb)

Circuit Breaker per Dependency

// One circuit breaker per dependency — not one global breaker
// A slow payments service should not affect the recommendations service

struct ServiceClient:
    circuit_breakers: HashMap[dependency_name -> CircuitBreaker]

function call_dependency(client, dependency_name, fn):
    cb = client.circuit_breakers.get_or_create(
        dependency_name,
        CircuitBreaker(failure_threshold=5, reset_timeout=30s)
    )
    return call_through(cb, fn)

// Example: payment service failing should not affect user profile reads
function checkout(user_id, cart):
    // These calls have independent circuit breakers
    try:
        user_profile = call_dependency("user_service", () -> user_service.get(user_id))
    except CircuitOpenError:
        raise CheckoutError("user service unavailable")

    try:
        payment = call_dependency("payment_service", () -> payment_service.charge(cart.total))
    except CircuitOpenError:
        raise CheckoutError("payment service unavailable")

    // Recommendations failure should not block checkout
    try:
        upsells = call_dependency("recommendation_service",
                                  () -> recommendation_service.get(user_id))
    except CircuitOpenError:
        upsells = []   // graceful degradation: skip upsells if circuit is open

Bulkhead Complement

// Circuit breaker: detects failure and stops sending requests (time-based isolation)
// Bulkhead: limits the resources any one dependency can consume (resource isolation)
// Together: comprehensive protection against dependency failures

// Circuit breaker alone: a slow dependency keeps threads waiting until circuit opens
// During the N failures needed to trip the breaker, threads are held
// Bulkhead limits how many threads are held simultaneously

struct BulkheadedCircuitBreaker:
    circuit_breaker: CircuitBreaker
    semaphore: Semaphore    // limits concurrent calls (e.g., max 10 concurrent)

function call_with_bulkhead_and_cb(bcb, fn):
    // First check circuit breaker (cheap — no thread hold)
    if bcb.circuit_breaker.state == "OPEN":
        raise CircuitOpenError()

    // Try to acquire a slot in the bulkhead
    if not bcb.semaphore.try_acquire(timeout=10ms):
        raise BulkheadFullError("too many concurrent requests")

    try:
        return bcb.circuit_breaker.call_through(fn)
    finally:
        bcb.semaphore.release()

Level 2 — Apply

A user-facing API service calls three downstream services: AuthService (authentication), ProductService (product data), and RecommendationService (personalised suggestions). All three calls are required on every page load. The circuit breaker configuration is: failure threshold=5, reset timeout=30 seconds.

1. AuthService begins returning errors for 30% of requests. After 5 consecutive failures, the circuit opens. Draw the state transition diagram and label the transitions with the triggering conditions. What does the API service return to users during the OPEN window?

2. With the circuit open, the HALF-OPEN probe fires after 30 seconds. The probe succeeds. The circuit closes. But AuthService resumes failing again 10 seconds later. How does the circuit breaker detect the second failure wave? What is the minimum time the circuit is open during the second failure wave?

3. The RecommendationService is optional — the page can load without personalised recommendations (just show defaults). Modify the checkout pseudocode example to implement graceful degradation when the RecommendationService circuit is open. What does the fallback return?

4. All three circuit breakers are OPEN simultaneously (a network partition affecting the data center). The API service returns errors for all requests. How does the health endpoint help operations engineers understand what is happening? What specific metrics and state information should the health endpoint expose?

Level 3 — Design

Design the circuit breaker configuration for a payment processing service that calls: (1) FraudDetectionService — P99 latency 200ms, expected failure rate < 0.1%; (2) PaymentGateway — P99 latency 500ms, expected failure rate < 0.5%; (3) NotificationService — P99 latency 100ms, expected failure rate < 1%; (4) AuditService — P99 latency 50ms, expected failure rate < 0.1%.

1. For each dependency, specify the circuit breaker configuration: failure threshold, failure rate threshold, slow-call threshold, reset timeout, and whether the call is required or optional. Justify each parameter choice based on the dependency’s characteristics.

2. The PaymentGateway begins returning HTTP 503 errors during a maintenance window. The circuit breaker trips open. During the open window, incoming payment requests must not be silently dropped — they must be queued for retry when the circuit closes. Design the queue-based fallback mechanism. Specify: queue storage, retry trigger (how does the circuit closing trigger queue processing?), idempotency handling, and maximum queue depth before overflow.

3. A “slow circuit breaker” variant opens when a dependency’s P99 latency exceeds a threshold rather than when errors exceed a threshold. The PaymentGateway is returning 200 OK but with 3-second latency (6× normal). Describe why a standard error-based circuit breaker would not trip in this scenario. Specify the slow-call configuration that would detect this situation and trip the breaker.

4. The AuditService is mandatory for regulatory compliance — every payment must generate an audit record. Unlike other dependencies, the AuditService circuit breaker cannot use graceful degradation. Design the failure behaviour when AuditService is unavailable. Should the circuit breaker still be used? If so, what is the fallback? If payments are blocked until AuditService recovers, how do you prevent the payment service from accumulating a backlog of waiting requests?

A complete answer will: (1) specify distinct circuit breaker configurations for each dependency with each parameter justified — FraudDetectionService and AuditService configured as required, NotificationService as optional with a graceful degradation fallback, (2) identify FM2 (cascading failure — PaymentGateway queue overflows and backs up into the payment service thread pool) as the failure mode the queue depth limit in part (b) must prevent, (3) address the AT3 tradeoff between the slow-call circuit breaker (catches latency degradation before it causes P99 SLA violations) and the standard error-based breaker (misses slow-but-successful responses), and (4) design the AuditService failure mode — block new payments and write to a local transactional outbox for guaranteed delivery when AuditService recovers — with a backlog prevention mechanism (reject incoming payments with HTTP 503 when the outbox queue reaches a limit).