AI Analysis

Automated analysis for INC-4821 — Payment processing failures after schema migration

AI-generated incident summary

Generated from structured data

At 13:39 UTC a payments schema migration renamed the `amount_cents` column without a backward-compatible shim. The deployed Payments API continued reading the old column name, causing charge creation to throw and ~38% of checkout attempts to fail. A compatibility view restored service at 14:17, reducing error rate from 38% to 4% within six minutes.

Likely root cause

Non-backward-compatible column rename in payments migration `2026_06_26_rename_charge_cols`.

Confidence

94%

Risk of recurrence

High risk

3 of the last 5 payments incidents share the same migration-safety root cause class.

Contributing factors

Migration deployed independently of the application code that depended on the renamed column.
No CI gate to detect destructive or non-backward-compatible schema changes.
No synthetic checkout canary running before and after migrations.
Expand/contract migration pattern not enforced for the payments schema.

Recommended next actions

1Adopt expand/contract migrations: add new columns, dual-write, then remove old columns in a later release.
2Add a CI gate that blocks column renames/drops without an explicit compatibility plan.
3Run a synthetic checkout canary as a pre-migration and post-migration gate.
4Keep the temporary compatibility view until the API is redeployed against the new column.

Evidence used

The analysis draws on structured incident data stored in Amazon DynamoDB.

Timeline eventDatadog deploy marker at 13:39

External eventPagerDuty alert.triggered 13:42:11

Affected servicePayments API → Database Cluster dependency

Action itemINC-4756 manual migration rollback (recurring pattern)