All incidentsMITIGATING
P0INC-2341fired 3d ago

Checkout 500s spiked — checkout-api

checkout-api · Maya Alves, On-call SRE

Incident graph10 nodes · 10 edges
Summary

42 correlated alerts clustered into a single incident. Top candidate is the deploy of checkout-api@v412 which landed 3m 10s before the first alert. Salvo proposes rollback.

Source alerts · 5
42 fires
  1. ×18
    checkout-api 5xx > 2%
    datadog · 3d ago
    primary signal
    datadog
  2. ×9
    checkout-api p99 latency > 4s
    prometheus · 3d ago
    prometheus
  3. ×7
    postgres replication lag > 3s
    prometheus · 3d ago
    likely downstream
    prometheus
  4. ×4
    PagerDuty: high-sev business-checkout
    datadog · 3d ago
    datadog
  5. ×4
    Sentry: UnhandledRejection spike
    datadog · 3d ago
    datadog
Timeline
6 events
  1. SALVO03:38:50 AM UTC
    checkout-api@v412 deployed
    CI/CD · auto-rollout
  2. SALVO03:42:00 AM UTC
    First 5xx alert fires
    datadog · checkout-api
  3. SALVO03:42:44 AM UTC
    42 alerts clustered into INC-2341
  4. SALVO03:43:10 AM UTC
    Top cause candidate: v412 deploy (92%)
  5. MAYA ALVES03:43:30 AM UTC
    Acknowledged page
  6. MAYA ALVES03:44:15 AM UTC
    Ran rollback step 01