EoneShop: Microservices, Live

Break it on purpose

On the live page you place an order and watch each event travel through Pub/Sub in real time: order placed, stock reserved, payment charged, order confirmed. Four scenarios are on offer, and each one only ever affects your own order:

• Normal: the order goes through the full flow end to end.
• Card declined: the saga unwinds and gives the reserved stock back.
• Payment too slow: the order times out first, so the late charge is blocked when it finally arrives.
• Catalog is down: Pub/Sub retries the delivery, dead-letters it after 5 attempts, and the order times out.

How it's built

• Outbox pattern plus idempotent consumers, so a retried event never double-applies.
• A saga with one cancel event that every service undoes its own step for, and tombstones so an event that arrives too late can't reserve stock or charge a cancelled order.
• One trace per order across both HTTP and events, via OpenTelemetry into Cloud Trace.
• Private services are only reachable with ID tokens, across six least-privilege service accounts.
• Cloud Build only redeploys the services whose code actually changed.

Seven parts, written up

1. Why I Started With a Monolith
2. When a Function Call Becomes a Network Call
3. Events, the Outbox, and Getting Every Message Twice
4. Undo Buttons, Timeouts, and Breaking Things on Purpose
5. One Trace per Order, and the Bug It Found
6. Leaving the Laptop
7. Watch It Live (and the Bug It Caught)

Bugs it caught

• Windows resolved "localhost" over IPv6 first, adding about 2 seconds per hop, so orders looked stuck for no reason.
• Creating a fresh HTTP client per call cost about 130 ms each; reusing clients dropped a 2.6 second order down to 0.6 seconds.
• A duplicate event could trigger two emails; fixed by claiming the event before doing any work on it.
• Cloud Trace stayed empty because Cloud Run pauses the CPU right after a response is sent; fixed by flushing spans before the last byte goes out.
• Two overlapping timeout sweeps could cancel the same order twice; fixed with a compare-and-set instead of a plain read-then-write.