Blog / Incident library 01

The coupon that worked 1,204 times

A caching change made coupon limits check a stale count. Every test passed, review approved it, and nothing crashed.

Invarly teamOct 15, 20268 min readrule pack: coupons

Composite example. The details are combined from common coupon bugs, not one company’s incident.

What happened

A promo code, WELCOME20, was capped at 1,000 uses. To speed up checkout, a deploy started caching coupon lookups for ten minutes. The limit check read uses from that cache.

During a campaign spike the cached count stayed at 961 while real redemptions kept climbing. By the time someone in finance noticed the discount line, the code had been used 1,204 times.

Why every test passed

The unit test checked that a coupon at its limit is rejected. It still was, as long as the cache was fresh. Nothing threw, latency improved, error rates stayed flat. Monitoring had no reason to fire, and the diff looked like a harmless optimization.

The bug only existed in the data: a number that should never be larger than another number, quietly growing past it.

The rule

The fix is not another test. It is a statement about the business that must always hold, checked against live data.

id:        coupon-max-uses
source:    postgres.main
query:     SELECT c.code, c.max_uses, count(r.id) AS uses
           FROM coupons c JOIN redemptions r ON r.coupon_id = c.id
           GROUP BY c.code, c.max_uses
entity:    row.code
violation: row.uses > row.max_uses

Run every five minutes, it reads the real redemption count, not the cached one. The first run after the cap is crossed opens an incident.

What the alert looks like

≠ coupon-max-uses

open 4 min

WELCOME20 uses 1,004 / max 1,000

shipped before first violation

7be20d1 13:31 “cache coupon lookups”

✓ limit check reads couponCache.uses checkout/coupon.go:41

Four redemptions over the cap instead of two hundred and four. The alert names the deploy that shipped just before the first violation, so the fix starts at the right diff.

Takeaways

Limits that money depends on should be checked where the truth lives: in the database, not in the code path that enforces them. Caches, queues and retries are exactly where those code paths drift.

If you have a coupon, quota or seat limit today, the rule above is a good first one to write.

Up next

subscribe below to get them

Write this rule on your own data.

Join waitlist