ResolveOps AI

Postmortem

Structured postmortem for INC-4821 — Payment processing failures after schema migration

SEV1

Payment processing failures after schema migration

INC-4821 · Author Maya Chen · Reviewers Tom Becker, Sam Whitfield · Draft

Postmortem

Summary

A database migration on the payments schema introduced a non-backward-compatible column rename, causing the Payments API to throw on charge creation. Customer-facing checkout was impacted for roughly 35 minutes before mitigation.

What happened

A scheduled migration renamed the `amount_cents` column to `charge_amount_cents` on the payments schema. The migration was applied independently of the application deploy, so the running Payments API continued to query the old column and failed on every charge creation. Customers experienced failed checkouts until a compatibility view was introduced.

Timeline

  1. 13:42PagerDuty alert: payment error rate > 25%
  2. 13:46Incident declared SEV1, commander assigned
  3. 13:53Correlated with payments schema migration
  4. 14:01Status page updated — investigating
  5. 14:08Root cause identified: non-backward-compatible column rename
  6. 14:17Mitigation applied: backfill compatibility view
  7. 14:24Error rate dropping — monitoring
  8. 14:40Status page updated — mitigated
  9. 15:05Follow-up actions logged

Root cause

Non-backward-compatible column rename in payments migration `2026_06_26_rename_charge_cols`. (AI confidence 94%)

Contributing factors

  • Migration deployed independently of the application code that depended on the renamed column.
  • No CI gate to detect destructive or non-backward-compatible schema changes.
  • No synthetic checkout canary running before and after migrations.
  • Expand/contract migration pattern not enforced for the payments schema.

Customer impact

Approximately 38% of checkout attempts failed over a 35-minute window, affecting an estimated 12,400 customers. Refunds and payment-method updates were also impacted. No customer payment data was lost or exposed.

What went well

  • PagerDuty detection fired within 3 minutes of the error spike.
  • Incident command was established quickly and roles were clear.
  • The compatibility-view mitigation restored service in under 40 minutes.

What needs improvement

  • Migrations are not gated by backward-compatibility checks.
  • No synthetic checkout canary surrounding migrations.
  • Schema changes and dependent code deploy on separate, uncoordinated tracks.

Follow-up actions

Add expand/contract migration policy for all schema changes

Tom Becker · due 2026-07-01 · Migration safety

UrgentIn progress

Add CI gate blocking non-backward-compatible column changes

Maya Chen · due 2026-07-03 · Migration safety

UrgentOpen

Add synthetic checkout canary alert pre-migration

Lena Kowalski · due 2026-06-24 · Observability

HighOpen

Remove temporary compatibility view after API redeploy

Tom Becker · due 2026-07-08 · Cleanup

MediumOpen

Structured sections are populated from DynamoDB. Generate a draft to add the AI executive summary.