Payment processing failures after schema migration
A database migration on the payments schema introduced a non-backward-compatible column rename, causing the Payments API to throw on charge creation.
Affected services
Payments API, Database Cluster, Customer Portal, Notification Service
Incident commander
Maya Chen
Started
Jun 26, 01:42 PM UTC
Responders
3 active · 5 engaged
Customer impact: ~38% of checkout attempts failing; 12,400 customers affected
Recommended incident workflow
Move from live response to AI analysis, postmortem generation, and recurrence-prevention actions.
Incident timeline
Detection through follow-up · 9 events
- 13:42DetectionPagerDuty
PagerDuty alert: payment error rate > 25%
High-urgency alert triggered from the payments error-rate monitor. Auto-paged the on-call SRE.
by PagerDuty
- 13:46TriageSlack
Incident declared SEV1, commander assigned
Maya Chen took incident command and opened the #inc-4821 war room in Slack.
by Maya Chen
- 13:53TriageDatadog
Correlated with payments schema migration
Datadog deploy marker shows migration `2026_06_26_rename_charge_cols` ran at 13:39, 3 min before errors began.
by Tom Becker
- 14:01CommsZendesk
Status page updated — investigating
Customer-facing status posted: 'Some customers may experience payment failures. We are investigating.'
by Sam Whitfield
- 14:08Mitigation
Root cause identified: non-backward-compatible column rename
The migration renamed `amount_cents` to `charge_amount_cents` without a compatibility shim; the API still reads the old column.
by Maya Chen
- 14:17Mitigation
Mitigation applied: backfill compatibility view
Created a database view aliasing the renamed column so the deployed API resolves charges again.
by Tom Becker
- 14:24ResolutionDatadog
Error rate dropping — monitoring
Payment error rate fell from 38% to 4% within 6 minutes of the mitigation.
by Maya Chen
- 14:40CommsZendesk
Status page updated — mitigated
Posted: 'A fix has been applied and payment processing is recovering.'
by Sam Whitfield
- 15:05Follow-upJira
Follow-up actions logged
Action items created for migration safety gates and expand/contract rollout policy.
by Maya Chen