Saga pattern and Transactional Outbox for microservices

When an order system is split into microservices, a request that sounds simple, like "place an order", suddenly touches three databases: inventory reservation, payment, and shipping. There is no single ACID transaction that spans all of them, and the question "what happens to the first two steps if the third one fails?" becomes a real design problem. The Saga pattern is the most common answer, and the Transactional Outbox is the missing piece that lets a saga send messages reliably. This article builds both with Java 21 and plain JDBC on in-memory H2, runs the failure scenarios for real, and pastes the exact output, so you can watch each mechanism work instead of just reading a diagram.

A note on scope: the two halves of the Saga pattern and the Outbox are demonstrated separately. The Outbox runs in Demo2Outbox, while the orchestrator in Demo3Saga calls the services with synchronous method calls so the focus stays on compensation and recovery. In a real system, the commands the orchestrator sends to services should go through the outbox.

This article follows Circuit Breaker, Retry and Bulkhead with Resilience4j: Resilience4j makes a single network call more fault tolerant, while this article keeps data consistent when several such calls must all succeed or all be undone. It is also a natural next step after CQRS and Event Sourcing in Java: when they are worth it, since both put events at the center.

Why there is no shared transaction anymore

Inside a single service with a single database, @Transactional gives you all four ACID properties. Microservices usually follow the "one database per service" rule, so a business operation that spans several services cannot use one local transaction. The two-phase commit (2PC) protocol solves the problem in theory, but in practice many brokers and NoSQL databases do not support it, it holds locks for a long time, and it turns the coordinator into a bottleneck. That is why Chris Richardson's microservices.io defines the Saga pattern as "a sequence of local transactions that is coordinated reliably", and the Azure Architecture Center documentation describes the same idea: each step commits in its own database, and if a step fails, a chain of compensating transactions undoes the effects of the steps that already finished.

There are two big differences from an ordinary rollback. First, the earlier steps have really committed, so "undoing" them is a new business operation (refund the money, return the items to stock) rather than erasing the trace. Second, the Saga pattern has no isolation (the "I" in ACID): while it runs, intermediate data is visible to other operations. We come back to both points in the trade-offs section.

Two ways to coordinate the Saga pattern: choreography and orchestration

The table above is based on the Azure documentation for the Saga pattern; the row "Where the saga state lives" is our own observation, not the documentation's. For an order flow with three steps and compensation branches, this article implements the Saga pattern as an orchestrator, because the state sits in one place, which suits the goal of explaining. The general structure is this: the orchestrator runs each step in order, writes a journal to a saga_log table, and when it hits a business error it runs the compensation of the completed steps in reverse order.

The problem before writing a saga: dual write

Whichever style of Saga pattern you use, every service has to do two things: update its own database and send a message to another service. Those two things belong to two different systems and cannot sit in the same transaction. This is the dual write bug. The Demo1DualWrite code tries both orders:

// Option 1: commit, then publish; the broker fails at exactly that moment
Db.exec(db, "insert into orders values ('DH001','PENDING')");
db.commit();
broker.failNextPublish = true;
broker.publish("order.events", Msg.of("OrderCreated", "DH001", ""));   // throws

// Option 2: publish, then commit; the transaction is rolled back after the message has gone out
broker.publish("order.events", Msg.of("OrderCreated", "DH002", ""));
Db.exec(db, "insert into orders values ('DH002','PENDING')");
db.rollback();

The actual output of the dual write run:

Option 1 (commit, then publish): publish failed -> broker not responding
  Orders in DB: 1, messages in broker: 0  => the order exists but nobody knows
Option 2 (publish, then commit): rollback after publish
  Order DH002 in DB: 0, messages in broker: 1  => a message about an order that does not exist

Option 1 leaves an "orphan" order that inventory and payment never hear about; option 2 makes the other services process an order that does not exist. Both break the Saga pattern at the very first step, so message delivery has to be made reliable first.

Transactional Outbox: write the message in the same transaction as the data

According to the microservices.io description, the sender stores each outgoing message in an outbox table within the same transaction that updates the business data; a separate process (the message relay) reads the outbox and pushes the messages to the broker. A message exists only if the transaction commits, and it is delivered in the order it was created. The outbox table used in this article:

create table outbox(
  seq bigint auto_increment primary key,
  topic varchar(40), msg_id varchar(40), type varchar(40),
  order_id varchar(40), data varchar(200),
  published boolean default false)

The write function and the relay function (a polling publisher) in Outbox.java:

public static void add(Connection c, String topic, Msg m) throws SQLException {
    Db.exec(c, "insert into outbox(topic,msg_id,type,order_id,data) values (?,?,?,?,?)",
            topic, m.id(), m.type(), m.orderId(), m.data());   // same connection, same transaction
}

// Relay: read by seq, publish, and only THEN mark as sent
for (Object[] r : rows) {
    broker.publish((String) r[1], (Msg) r[2]);
    Db.exec(c, "update outbox set published=true where seq=?", r[0]);
    c.commit();
}

The key to the Saga pattern and the outbox working together is the order inside the relay: publish first, mark second. If the relay dies between those two actions, the message is sent again on the next run. In other words, the Outbox trades "a message can be lost" for "a message can be duplicated" (at-least-once delivery). This is a deliberate trade-off, and Demo2Outbox proves it step by step:

--- 1. If the transaction rolls back, the outbox rolls back too ---
orders=0, outbox=0
--- 2. If it commits, both exist ---
orders=1, unsent outbox rows=1
--- 3. Broker failure: the message is not lost, the relay retries later ---
relay attempt 1 failed: broker not responding
unsent outbox rows=1, broker=0
relay attempt 2 delivered: 1 message
--- 4. Relay dies after publish, before marking: duplicate delivery ---
incident: relay died after publish, before marking
Restarting relay: sent 1 message (DH103 is delivered a second time)
messages in broker=3

Steps 1 and 2 solve both dual write cases: on rollback there is no message, on commit there is certainly a message. Step 3 shows that a broker failure does not lose the message. Step 4 is the price: DH103 is delivered twice.

Idempotent consumer: the other half of the promise

With the Saga pattern, a duplicate message is harmless only if the receiver handles it idempotently, meaning that processing it again does not change the result. The simplest approach is a processed_messages table whose primary key is the message id, written in the same transaction as the business effect:

public static boolean firstTime(Connection c, String msgId) throws SQLException {
    try {
        Db.exec(c, "insert into processed_messages(msg_id) values (?)", msgId);
        return true;
    } catch (SQLException e) {
        if (e.getErrorCode() == 23505) return false; // primary key violation
        throw e;
    }
}

--- 5. Idempotent consumer: each msg_id is processed only once ---
processing OrderCreated[cd30763b, order=DH102, amount=500000]
processing OrderCreated[589af2a4, order=DH103, amount=900000]
skipping duplicate OrderCreated[589af2a4, order=DH103, amount=900000]
invoices created=2 (order DH103 has only 1 invoice: 1)

Code 23505 is H2's unique constraint violation code (MySQL uses 1062), so adjust it for your real database. Marking a message as "processed" must happen in the same transaction as the side effect; otherwise you recreate the dual write problem on the receiving side.

Implementing the Saga pattern with an orchestrator

The three services in the Saga pattern example (Services.java) each use their own H2 database: Inventory reserves stock, Payment charges money (the gateway rejects anything over 20 million VND), and Shipping creates shipments (it rejects a few regions). Each service has a main action and, where needed, a compensating action. The step names are Vietnamese slugs: giu-hang means reserve stock, thanh-toan means payment, and van-chuyen means shipping. The orchestrator declares the Saga pattern as a list of steps:

List.of(
    new Step("giu-hang",   () -> inv.reserve(id, "AO-THUN", qty), () -> inv.release(id)),
    new Step("thanh-toan", () -> pay.charge(id, amount),        () -> pay.refund(id)),
    new Step("van-chuyen", () -> ship.create(id, city),         null));  // last step, no compensation needed

The main loop of the Saga pattern records the state before and after each step, and on a business error it compensates in reverse order (this excerpt omits the crashBefore parameter and the tracing from Orchestrator.java):

for (Step s : steps) {
    if ("DONE".equals(statusInLog(orderId, s.name()))) { done.add(s); continue; }  // on re-run, skip
    try {
        log(orderId, s.name(), "STARTED");
        s.action().run();
        log(orderId, s.name(), "DONE");
        done.add(s);
    } catch (Services.StepFailed e) {
        log(orderId, s.name(), "FAILED");
        Collections.reverse(done);
        for (Step d : done) {
            if (d.compensation() == null) continue;   // step without compensation (the last step)
            d.compensation().run(); log(orderId, d.name(), "COMPENSATED");
        }
        return "CANCELLED";
    }
}
return "COMPLETED";

The output of Demo3Saga for the first three Saga pattern scenarios (the warehouse starts with 10 T-shirts):

--- Scenario 1: success ---
  giu-hang OK
  thanh-toan OK
  van-chuyen OK
  => DH201: COMPLETED | stock AO-THUN=8, payment=CHARGED, shipments=1
--- Scenario 2: payment rejected (compensate the reserve step) ---
  giu-hang OK
  thanh-toan FAILED: payment gateway rejected 25000000 VND (over the limit)
  compensate giu-hang
  => DH202: CANCELLED | stock AO-THUN=8, payment=NONE, shipments=1
--- Scenario 3: shipping fails (refund, then return stock, in reverse order) ---
  giu-hang OK
  thanh-toan OK
  van-chuyen FAILED: the carrier does not deliver to Con Dao
  compensate thanh-toan
  compensate giu-hang
  => DH203: CANCELLED | stock AO-THUN=8, payment=REFUNDED, shipments=1

In scenario 2, stock goes back to 8 (2 were deducted for DH201 and the 3 reserved for DH202 were returned), and payment has no record. In scenario 3, the payment step really committed, so it has to be refunded, and the final state is REFUNDED rather than "nothing". That is exactly the difference between compensation and rollback: the trace of the transaction remains in the books, just as in real accounting.

Recovering the Saga pattern when the orchestrator dies midway

The orchestrator of a Saga pattern is a process that can be killed at any moment, so its state has to live in the database, not in memory. Scenario 4 simulates the process dying right before the shipping step, then restarting and calling run again with the same order id:

--- Scenario 4: orchestrator dies midway, restart resumes ---
  incident: process died before step van-chuyen
  saga_log: thanh-toan=DONE giu-hang=DONE
  giu-hang OK
  thanh-toan OK
  skip giu-hang (already DONE in saga_log)
  skip thanh-toan (already DONE in saga_log)
  van-chuyen OK
  => DH204: COMPLETED | stock AO-THUN=7, payment=CHARGED, shipments=2
  saga_log DH204 after the re-run: van-chuyen=DONE thanh-toan=DONE giu-hang=DONE
  saga_log DH203: van-chuyen=FAILED thanh-toan=COMPENSATED giu-hang=COMPENSATED

What matters is that stock drops only once (from 8 to 7) and there is only one payment for DH204, thanks to two layers of protection: the orchestrator skips steps that are already DONE, and each service also checks for duplicates by order id (reserve and charge return immediately if a record already exists). If the orchestrator dies after the service has committed but before it writes DONE, the step runs again and only idempotency in the service saves us. Scenario 5 verifies exactly this by calling reserve repeatedly with the same order id DH204 (already committed in scenario 4):

--- Scenario 5: step committed in the service but the orchestrator has not written DONE, so reserve is called again ---
  stock before=7, after calling reserve(DH204) again=7

Stock is still 7, so the repeated call in the Saga pattern does not deduct any more items.

Compensating transactions: constraints that are easy to miss

• Must be idempotent and retryable: a compensating step in the Saga pattern can be called many times. In the code, release only adds the items back if the reservation has not been returned yet (released=false), so a second call does not inflate stock.
• Not everything can be compensated: an email that was sent or goods that were delivered cannot be withdrawn with one command. The Azure documentation calls the point of no return in a Saga pattern the pivot transaction; the steps after it must be retryable (retried until they succeed) rather than compensated.
• Order the steps sensibly: put the steps most likely to fail early and the steps that cannot be compensated last.
• Compensation can fail too: you need monitored retries and an alert when they exceed a threshold.

Lack of isolation: the Saga pattern exposes intermediate state

In scenario 3, between the successful stock reservation and its compensation, stock was temporarily lower. Another order that reads stock at exactly that moment can be rejected unfairly, which is a form of dirty read. The Azure documentation lists the anomalies of the Saga pattern (lost update, dirty read, fuzzy read) and the countermeasures. The most memorable ones:

• Semantic lock: mark a record with a business status such as PENDING so other sagas know the data is being processed.
• Commutative updates: design updates that do not depend on order (add and subtract quantities instead of overwriting).
• Pessimistic view: reorder the saga so that data updates happen in retryable steps (after the pivot, never compensated), so a compensating step cannot cause a dirty read.
• Reread value: read the value again right before writing, and if it changed, stop the step and run it again.

In the sample code, the stock updates (qty=qty-?) are already commutative, so two concurrent sagas do not overwrite each other; stock appearing temporarily low can still happen.

Outbox in production: polling or transaction log reading

The demo uses a polling publisher: the relay scans the outbox table periodically. This is easy to implement, but it adds a steady query load and a latency equal to the polling interval. The second option is to read the database's transaction log (log tailing, also known as CDC). Debezium provides an Outbox Event Router: a connector captures changes to the outbox table and routes them to Kafka topics. According to the Debezium documentation, the outbox table acts as an INSERT-only queue, with columns such as id (used for deduplication), aggregatetype, aggregateid (the message key, which preserves order within a partition), type and payload.

With either approach, a message can still arrive more than once, so an idempotent consumer is not optional. The outbox also needs a policy for cleaning up sent rows so the table does not grow forever.

When not to use the Saga pattern

• All data that needs immediate consistency lives in one database: merging into one service with a local transaction is far simpler. The Saga pattern is the price of splitting data, not a goal in itself.
• The business operation cannot be compensated: if no step can be undone, the Saga pattern is just a chain of retries; revisit your service boundaries.
• The team cannot monitor it yet: the Saga pattern needs tracing by order id, alerts for stuck steps, and a manual procedure for when compensation fails.

Common mistakes

• Publishing directly inside the business transaction: this is still a dual write; use an outbox or CDC.
• Forgetting the idempotent consumer: the outbox guarantees at-least-once, not exactly-once; duplicates will show up exactly when something goes wrong.
• Not persisting the Saga pattern state: if the orchestrator keeps state only in RAM, a restart loses it.
• No timeout per step: one hung service makes the saga hang forever; combine Retry and Circuit Breaker from the previous article.

Checklist

• Each step of the Saga pattern is a local transaction committed in that service's own database.
• Choose choreography for short flows and orchestration for long ones; store the saga state durably.
• Every step has a compensating action (except the last step and steps after the pivot), and compensation is idempotent and retryable.
• No dual write: every message goes through the outbox in the same transaction as the data.
• Idempotent consumers using a processed_messages table or a natural key, in the same transaction as the side effect.
• Countermeasures for the lack of isolation: semantic lock, commutative updates, reread value.
• Timeouts, alerts, and a manual procedure for when Saga pattern compensation fails.

In short, the Saga pattern replaces a distributed transaction with a chain of compensable steps, while the Transactional Outbox guarantees that each step emits its message if and only if the data was really committed. The Saga pattern and the Outbox go together, and both demand idempotency at every receiving end. All the code is in the code/s6-28 folder and runs on JDK 21 with in-memory H2 2.2.224.

Next article

The Saga pattern and the Outbox only solve consistency between services; inside a single service, you still have to coordinate many concurrent tasks. Continue with Concurrency patterns: Producer-Consumer, thread pools and CompletableFuture.