An order is paid through a third-party gateway. One evening the gateway does not go down completely, it just gets slow: every call takes 30 seconds instead of 300 milliseconds. Tomcat threads are held waiting, and within minutes the whole website stops responding even though only one feature depends on that gateway. This is a cascading failure, and the Circuit Breaker exists to stop it: when a dependency keeps failing, we stop calling it for a while instead of queueing up to wait. This article uses Resilience4j 2.3.0 on Spring Boot 3.4.4 and JDK 21 to implement a Circuit Breaker, Retry and Bulkhead, run them for real, measure them, and point out three traps: composition order, which errors deserve to be counted, and how a Bulkhead interacts with a Circuit Breaker.

This article follows CQRS and Event Sourcing in Java: when they are worth it, where the write and read models are separate but still have to call each other over the network, and Upgrading to Spring Boot 3: Jakarta EE, Java 17 and observability, because we will read the breaker state through Actuator.

The problem: calling an external service with no limits

Martin Fowler explains in his article on the Circuit Breaker that remote calls can hang until a timeout expires, and when many such calls pile up at once, resources (threads, connections) run out and cause a chain of failures. Michael Nygard popularized this pattern in the book Release It!. The three Resilience4j mechanisms answer three different questions:

MechanismThe question it answersAction
RetryCould this failure be only transient?Call again a few times, with growing intervals
Circuit BreakerIs the other service really broken?Stop calling for a while and fail fast
BulkheadCan one slow feature swallow all the resources?Limit concurrent calls and reject the excess

The three complement each other: Retry without a breaker amplifies load on a service that is already broken, and a breaker without a bulkhead lets slow calls occupy every thread before it has time to open.

How a Circuit Breaker works: three states

According to the Resilience4j CircuitBreaker documentation, a Circuit Breaker is a finite state machine. In the CLOSED state, calls pass through normally and results are recorded in a sliding window; when the failure rate or the slow-call rate reaches a threshold, the breaker switches to OPEN and rejects every call with CallNotPermittedException. After waitDurationInOpenState, the Circuit Breaker moves to HALF_OPEN and lets a number of trial calls through; if the failure rate still exceeds the threshold it goes back to OPEN, otherwise to CLOSED.

The key parameters and their defaults according to the documentation: failureRateThreshold 50%, slidingWindowSize 100, minimumNumberOfCalls 100, waitDurationInOpenState 60 seconds, permittedNumberOfCallsInHalfOpenState 10, slowCallDurationThreshold 60 seconds. With the defaults, the breaker evaluates nothing until 100 calls have been made; and it only moves to HALF_OPEN when a call arrives, unless automaticTransitionFromOpenToHalfOpenEnabled is turned on.

Trying the Circuit Breaker with the core library

Before using Spring, let us look at each mechanism in a plain Java program (code/s6-27, class Plain). The fake payment gateway has three modes: ok, fail (throws a 503 error) and slow (300 ms delay). The Circuit Breaker configuration uses a small window so it is easy to observe:

        CircuitBreakerConfig cbCfg = CircuitBreakerConfig.custom()
                .slidingWindowType(CircuitBreakerConfig.SlidingWindowType.COUNT_BASED)
                .slidingWindowSize(10).minimumNumberOfCalls(5).failureRateThreshold(50)
                .waitDurationInOpenState(Duration.ofMillis(1000))
                .permittedNumberOfCallsInHalfOpenState(2).build();
        CircuitBreaker cb = CircuitBreaker.of("payment", cbCfg);
        cb.getEventPublisher().onStateTransition(e ->
                System.out.println("   [event] " + e.getStateTransition()));

We send eight calls while the gateway is broken, then let the gateway recover and wait for the OPEN period to end. Real output:

=== 1. Circuit Breaker: CLOSED -> OPEN -> HALF_OPEN -> CLOSED ===
call 1 -> error: gateway 503                       state=CLOSED    real gateway calls=1  took 1722 µs
call 2 -> error: gateway 503                       state=CLOSED    real gateway calls=2  took 35 µs
call 3 -> error: gateway 503                       state=CLOSED    real gateway calls=3  took 23 µs
call 4 -> error: gateway 503                       state=CLOSED    real gateway calls=4  took 20 µs
   [event] State transition from CLOSED to OPEN
call 5 -> error: gateway 503                       state=OPEN      real gateway calls=5  took 1991 µs
call 6 -> BLOCKED (CallNotPermittedException)      state=OPEN      real gateway calls=5  took 155 µs
call 7 -> BLOCKED (CallNotPermittedException)      state=OPEN      real gateway calls=5  took 21 µs
call 8 -> BLOCKED (CallNotPermittedException)      state=OPEN      real gateway calls=5  took 20 µs
failureRate=100%  failed=5  notPermitted=3
-- gateway recovers, wait 1.1 seconds --
   [event] State transition from OPEN to HALF_OPEN
probe 1 -> PAID:DH state=HALF_OPEN
   [event] State transition from HALF_OPEN to CLOSED
probe 2 -> PAID:DH state=CLOSED
probe 3 -> PAID:DH state=CLOSED

The first five calls all reached the gateway and all failed; after the fifth call (exactly minimumNumberOfCalls) the failure rate was 100%, so the breaker opened. The next three calls were blocked instantly with CallNotPermittedException, and the number of real calls reaching the gateway stayed at 5. That is the value of a Circuit Breaker: the broken gateway receives no more load, and your threads get an error immediately (the "took" column is measured with System.nanoTime: blocked calls took 20 to 155 microseconds in this run, and this number varies between runs) instead of waiting for a timeout. After waiting 1.1 seconds, the breaker moved to HALF_OPEN, the first probe succeeded, and the second probe closed the Circuit Breaker again.

Retry: controlled repeat calls

Retry handles transient errors such as a lost packet or a single 503 while the gateway restarts. According to the Retry documentation, the default maxAttempts is 3 (counting the first call) and waitDuration is 500 ms; you can use IntervalFunction.ofExponentialBackoff to make the wait grow, or ofRandomized to add jitter so many clients do not retry at the same moment. The retryExceptions and ignoreExceptions parameters decide which errors are worth retrying.

        RetryConfig rCfg = RetryConfig.custom().maxAttempts(3)
                .intervalFunction(IntervalFunction.ofExponentialBackoff(Duration.ofMillis(100), 2.0))
                .retryExceptions(IllegalStateException.class).build();
        Retry retry = Retry.of("payment", rCfg);

With a 100 ms backoff that doubles, the three real calls happen as follows (real measurements, the clock starts before the first call):

=== 2. Retry: 3 attempts, backoff 100ms x2, with the time of each attempt ===
   real call #1 at t=1 ms
   real call #2 at t=108 ms
   real call #3 at t=314 ms
Result: failed after 3 real calls: gateway 503

The second call comes about 100 ms after the first and the third about 200 ms after the second, exactly a geometric progression. The total is over 300 ms (314 ms in this run) for one failed payment, a figure that must fit inside the timeout budget of the original request. Retry assumes the operation is idempotent: calling a payment gateway again without an idempotency key can charge the customer twice if the first call actually succeeded but the response was lost.

Bulkhead: watertight compartments

The name comes from the partitions in a ship's hull: one flooded compartment does not sink the whole ship. Resilience4j has two kinds: SemaphoreBulkhead limits concurrent calls with a semaphore (default maxConcurrentCalls is 25, maxWaitDuration is 0), and ThreadPoolBulkhead uses its own thread pool with a bounded queue. The semaphore type is lighter and fits when you already manage the client's thread pool yourself.

Test with 10 threads calling a gateway that takes 300 ms, with the bulkhead allowing at most 3 concurrent calls and no waiting:

=== 3. Bulkhead: at most 3 concurrent calls, 10 threads at once, 300 ms per call ===
succeeded=3  rejected immediately (BulkheadFullException)=7  gateway actually received=3

Three calls got in, seven were rejected at once with BulkheadFullException, and the gateway really received only three. The slow gateway now occupies at most 3 threads, and the remaining threads serve other features.

Composition order: Retry outside or Circuit Breaker outside

When using Resilience4j through Spring Boot, the default order of the aspects according to the Spring Boot 3 documentation is Retry wrapping CircuitBreaker, wrapping RateLimiter, wrapping TimeLimiter, wrapping Bulkhead, with your function innermost. The question is how this order differs from the reverse. We ran six requests against a completely broken gateway, with at most 3 attempts per request and a breaker window of 6 calls:

            Supplier<String> chain = retryOutside
                    ? Retry.decorateSupplier(r, CircuitBreaker.decorateSupplier(c, call))   // Retry outside
                    : CircuitBreaker.decorateSupplier(c, Retry.decorateSupplier(r, call));  // CircuitBreaker outside
=== 4. Composition order, gateway completely broken, 6 user requests ===
Retry( CircuitBreaker )              gateway received  6 | breaker recorded 6 failures, blocked 12 | final: OPEN
CircuitBreaker( Retry )              gateway received 18 | breaker recorded 6 failures, blocked  0 | final: OPEN
Retry(CB), ignore CallNotPermitted   gateway received  6 | breaker recorded 6 failures, blocked  4 | final: OPEN

The two orders give very different results:

  • Retry( CircuitBreaker ): every retry goes through the Circuit Breaker and is counted by the breaker. The gateway receives only 6 calls, then the Circuit Breaker opens. But the 12 retries after that all hit the open breaker and are blocked; with a real backoff (hundreds of milliseconds), the user's request is slowed down for nothing.
  • CircuitBreaker( Retry ): the Circuit Breaker sees only the final result of the whole retry sequence, so each request counts as a single failure. The breaker needs six requests to open, and during that time the gateway takes 18 calls, three times as many. The breaker reacts slowly and the broken gateway gets more load.
  • Retry( CircuitBreaker ) plus ignoring CallNotPermittedException: the gateway receives 6 calls, the breaker opens at the right moment, and the remaining 4 requests are blocked immediately without retrying. This is the configuration we recommend and will use in Spring.

The rule that follows: keep Retry outside the Circuit Breaker (as in the default), but always declare CallNotPermittedException in Retry's ignoreExceptions, because retrying a call the breaker has rejected only wastes time.

Integrating with Spring Boot 3 using annotations

With Spring Boot 3, the dependency is io.github.resilience4j:resilience4j-spring-boot3, together with spring-boot-starter-aop and Actuator. The configuration goes in application.yml; this example names the instance payment for all three mechanisms:

resilience4j:
  circuitbreaker:
    instances:
      payment:
        registerHealthIndicator: true
        slidingWindowType: COUNT_BASED
        slidingWindowSize: 10
        minimumNumberOfCalls: 5
        failureRateThreshold: 50
        waitDurationInOpenState: 3s
        permittedNumberOfCallsInHalfOpenState: 2
        ignoreExceptions:
          - demo.r4j.InvalidOrderException
  retry:
    instances:
      payment:
        maxAttempts: 3
        waitDuration: 100ms
        enableExponentialBackoff: true
        exponentialBackoffMultiplier: 2
        retryExceptions:
          - demo.r4j.GatewayException
        ignoreExceptions:
          - io.github.resilience4j.circuitbreaker.CallNotPermittedException
          - demo.r4j.InvalidOrderException
  bulkhead:
    instances:
      payment:
        maxConcurrentCalls: 3
        maxWaitDuration: 0

This file contains three important decisions. First, InvalidOrderException (a 4xx error caused by customer data, such as an invalid order) is in ignoreExceptions for both the Circuit Breaker and Retry: a customer error says nothing about the health of the gateway, and retrying will not change the result. Fowler also notes that not every error should trip the breaker. Second, retryExceptions lists only GatewayException (a transient error), which means unknown errors are not retried by default. Third, Retry ignores CallNotPermittedException as analyzed above. The service uses three annotations:

@Service
public class PaymentService {
    private final FakeGateway gateway;
    PaymentService(FakeGateway gateway) { this.gateway = gateway; }

    // Resilience4j default order: Retry( CircuitBreaker( Bulkhead( function ) ) )
    // The fallback goes on the OUTERMOST layer (@Retry). Putting it on @CircuitBreaker swallows the exception and Retry never sees the error.
    @Retry(name = "payment", fallbackMethod = "pending")
    @CircuitBreaker(name = "payment")
    @Bulkhead(name = "payment")
    public String pay(String orderId) {
        return gateway.charge(orderId);
    }

    /** Fallback: same signature + one Throwable parameter. It does not say "paid", it says "pending". */
    String pending(String orderId, Throwable t) {
        if (t instanceof InvalidOrderException e) throw e;      // customer error: do not hide it behind a fallback
        return "PENDING:" + orderId + " (" + t.getClass().getSimpleName() + ")";
    }
}

The pending method is the fallback: same signature as the original method plus one Throwable parameter, exactly as the documentation requires. The fallback sits on the outermost annotation (@Retry); if it were on @CircuitBreaker, it would swallow the exception before Retry could see the error and Retry would never run. And the fallback must not "hide" every error: a customer error (InvalidOrderException) is rethrown, while an infrastructure error returns the PENDING status rather than PAID, because reporting "paid" when you are not sure is a business trap more serious than any technical failure.

Observing real behavior through HTTP and Actuator

To rerun it yourself: in code/s6-27 run mvn -q package, then bash drive.sh (JDK 21; edit JAVA_HOME in the script to match your machine). The drive.sh script starts the application and sends requests following a scenario. Real output (JDK 21, Apple M4, 16 GB):

--- A. healthy gateway
PAID:DH001
--- B. invalid order data (4xx): no retry, not counted by the breaker
Invalid order: BAD1 (HTTP 422)
   gateway received 1 call
--- C. broken gateway: 8 consecutive requests (each request makes at most 3 real calls thanks to Retry)
req 1 -> PENDING:DH001 (GatewayException) | gateway received 3 more calls
req 2 -> PENDING:DH002 (CallNotPermittedException) | gateway received 1 more call
req 3 -> PENDING:DH003 (CallNotPermittedException) | gateway received 0 more calls
req 4 -> PENDING:DH004 (CallNotPermittedException) | gateway received 0 more calls
req 5 -> PENDING:DH005 (CallNotPermittedException) | gateway received 0 more calls
req 6 -> PENDING:DH006 (CallNotPermittedException) | gateway received 0 more calls
req 7 -> PENDING:DH007 (CallNotPermittedException) | gateway received 0 more calls
req 8 -> PENDING:DH008 (CallNotPermittedException) | gateway received 0 more calls
   breaker: OPEN | failureRate 80.0% | failed 4 | notPermitted 7
--- D. wait 3.2 seconds, then the gateway recovers
probe 1 -> PAID:OK1
   breaker: HALF_OPEN | failureRate -1.0% | failed 0 | notPermitted 0
probe 2 -> PAID:OK2
   breaker: CLOSED | failureRate -1.0% | failed 0 | notPermitted 0
probe 3 -> PAID:OK3
   breaker: CLOSED | failureRate -1.0% | failed 0 | notPermitted 0

Four things deserve a careful read. One, order BAD1 returns HTTP 422 and the gateway receives it only once: no retry. Two, the first request while the gateway is broken makes the gateway receive 3 calls (exactly maxAttempts), and the second request touches the gateway only once before the Circuit Breaker opens: after that, the next 6 requests return PENDING immediately while the gateway receives 0 calls. Three, the failureRate 80.0% figure with 4 failures means the window holds 5 recorded calls: one success from step A and four failures (the BAD1 call is not counted because it is on the ignore list). Four, after 3.2 seconds the breaker moves to HALF_OPEN when the first probe arrives, then to CLOSED after the second; failureRate -1.0% is how Resilience4j says "not enough calls yet to compute a rate".

A breaker should be monitored. Fowler recommends alerting whenever a Circuit Breaker opens, logging every state transition, and letting operations intervene manually. Resilience4j exports metrics through Micrometer, so you can put the breaker state on Grafana next to your other metrics as in Kubernetes monitoring with kube-prometheus and Grafana, and tie it to health-check endpoints as in Health endpoint for Laravel on Kubernetes: what to check and which probe to use. Note: if an OPEN breaker makes /actuator/health return DOWN, the readiness probe may remove the pod from the Service and turn a problem in a secondary service into a problem for the whole application; only do this when that service is mandatory.

The trap: the Bulkhead pushes the Circuit Breaker open

Scenario E in drive.sh fires 10 concurrent requests at a gateway that takes 400 ms. The result: 3 succeed, the rest are rejected, and the breaker opens even though the gateway never failed. The cause: the bulkhead sits inside the Circuit Breaker, so BulkheadFullException runs through the breaker and is counted as a failure. The call was rejected because of a limit we set ourselves, not because the gateway is broken. Compare the two configurations on the same scenario (script drive-bulkhead.sh):

[default] 10 concurrent requests, gateway slow 400 ms:
   3 PAID:SLn
   7 PENDING:SLn (BulkheadFullException)
gateway actually received: 3 calls
breaker: OPEN | failed 7 | notPermitted 0

[ignore] 10 concurrent requests, gateway slow 400 ms:
   3 PAID:SLn
   7 PENDING:SLn (BulkheadFullException)
gateway actually received: 3 calls
breaker: CLOSED | failed 0 | notPermitted 0

The default configuration lets the Circuit Breaker count 7 failures and open, whereas adding BulkheadFullException to the breaker's ignoreExceptions keeps the state CLOSED and the bulkhead does only its own job. Different runs give slightly different splits between calls rejected by the bulkhead and by the breaker (we recorded two runs: 5 rejected by the bulkhead and 2 by the breaker in drive.sh, then 7 by the bulkhead and 0 by the breaker in the table above) because requests arrive asynchronously; test the invariant (the gateway receives no more than 3 calls) rather than exact numbers.

When not to use a Circuit Breaker, Retry and Bulkhead

  • Non-idempotent operations: do not enable Retry without an idempotency key or a deduplication mechanism.
  • Low-traffic services: with the default minimumNumberOfCalls of 100, the breaker almost never opens; lower the threshold.
  • An existing service mesh: Envoy or Istio already retries for you, and repeating it in the application multiplies the number of attempts.

Checklist for a Circuit Breaker, Retry and Bulkhead

  • The Circuit Breaker has a minimumNumberOfCalls, window and waitDurationInOpenState that fit real traffic.
  • 4xx errors and business errors are in ignoreExceptions.
  • Retry is used only for idempotent operations, with exponential backoff plus jitter, and ignores CallNotPermittedException.
  • The Bulkhead limits concurrency to match the dependency's capacity; consider ignoring BulkheadFullException in the Circuit Breaker.
  • The fallback returns an honest status (pending), and does not mask business errors.
  • There are metrics and an alert when the Circuit Breaker opens; decide explicitly whether the breaker should affect readiness.

In short, a Circuit Breaker protects you only when it is configured correctly: swapping the order or missing a single ignoreExceptions line is enough for the broken gateway to take three times the load, or for the Circuit Breaker to open because of errors we caused ourselves.

Next article

Resilience4j helps a single service protect itself, but when one business operation spans several services that all write data, a different approach is needed to keep things consistent. Continue with Saga pattern and Transactional Outbox for microservices.