The coupon that worked 1,204 times
A caching change made coupon limits check a stale count. Every test passed, review approved it, and nothing crashed.
Composite example. The details are combined from common coupon bugs, not one company’s incident.
What happened
A promo code, WELCOME20, was capped at 1,000 uses. To speed up checkout, a deploy started caching coupon lookups for ten minutes. The limit check read uses from that cache.
During a campaign spike the cached count stayed at 961 while real redemptions kept climbing. By the time someone in finance noticed the discount line, the code had been used 1,204 times.
Why every test passed
The unit test checked that a coupon at its limit is rejected. It still was, as long as the cache was fresh. Nothing threw, latency improved, error rates stayed flat. Monitoring had no reason to fire, and the diff looked like a harmless optimization.
The bug only existed in the data: a number that should never be larger than another number, quietly growing past it.
The rule
The fix is not another test. It is a statement about the business that must always hold, checked against live data.
id: coupon-max-uses source: postgres.main query: SELECT c.code, c.max_uses, count(r.id) AS uses FROM coupons c JOIN redemptions r ON r.coupon_id = c.id GROUP BY c.code, c.max_uses entity: row.code violation: row.uses > row.max_uses
Run every five minutes, it reads the real redemption count, not the cached one. The first run after the cap is crossed opens an incident.
What the alert looks like
≠ coupon-max-uses
open 4 minWELCOME20 uses 1,004 / max 1,000
7be20d1 13:31 “cache coupon lookups”
✓ limit check reads couponCache.uses checkout/coupon.go:41
Four redemptions over the cap instead of two hundred and four. The alert names the deploy that shipped just before the first violation, so the fix starts at the right diff.
Takeaways
Limits that money depends on should be checked where the truth lives: in the database, not in the code path that enforces them. Caches, queues and retries are exactly where those code paths drift.
If you have a coupon, quota or seat limit today, the rule above is a good first one to write.