Incident Response Guide

Overview

This guide outlines our incident response procedures for data quality issues, service disruptions, and customer-reported problems.

Incident Classification

Severity Levels

Critical (P1)

Definition: Complete service outage or data corruption affecting multiple customers
Response Time: Immediate
Resolution Target: 3 business days
Examples:
- Prices off by order of magnitude (10x, 100x errors)
- API completely unavailable
- Authentication system failure

Major (P2)

Definition: Significant functionality impaired or incorrect data for specific commodities
Response Time: <30 minutes
Resolution Target: <4 hours
Examples:
- Single commodity feed showing incorrect values
- WebSocket connections dropping frequently
- Rate limiting not functioning correctly

Minor (P3)

Definition: Non-critical issues with workarounds available
Response Time: 3 business days
Resolution Target: <24 hours
Examples:
- Formatting inconsistencies
- Minor documentation errors
- Delayed data updates (within SLA)

Response Procedures

1. Detection & Triage

Monitor automated alerts from PriceValidationService
Review customer reports via [email protected]
Check system health dashboards
Classify incident severity

2. Initial Response

Acknowledge incident to reporting customer
Create incident ticket with:
- Timestamp of detection
- Affected systems/commodities
- Initial impact assessment
- Assigned responder

3. Investigation

Identify root cause
Document affected data ranges
Determine correction approach
Estimate resolution time

4. Resolution

Create data backup before changes
Apply fixes (automated or manual)
Verify corrections across all systems
Test related functionality

5. Communication

Update affected customers
Provide resolution details
Share preventive measures
Document lessons learned

Recent Incident Examples

UK Natural Gas Price Correction (August 2025)

Issue: Values inconsistent due to decimal parsing errors Detection: Customer report Resolution Time: 3 business days Actions Taken:

Corrected 62,343 price records
Implemented enhanced validation
Added unit clarification to documentation Prevention: Real-time validation now active

Escalation Matrix

Severity	Primary Contact	Escalation (30 min)	Executive (1 hour)
Critical	On-call Engineer	Engineering Lead	CTO
Major	Support Team	Senior Engineer	VP Engineering
Minor	Support Team	Engineer on Duty	Team Lead

Customer Communication Templates

Initial Response

Subject: [INCIDENT] Investigating reported issue with [COMMODITY/SERVICE]

We've received your report and are actively investigating the issue with [specific problem].

Incident ID: [XXX-YYYY]
Severity: [Critical/Major/Minor]
Estimated Resolution: [Time]

We'll update you within [30 minutes/1 hour] with our findings.

Resolution Notice

Subject: [RESOLVED] Issue with [COMMODITY/SERVICE] has been corrected

The issue affecting [specific problem] has been resolved.

Summary:
- Root Cause: [Brief explanation]
- Data Affected: [Time range and scope]
- Actions Taken: [Corrections applied]
- Prevention: [Measures implemented]

All systems are now operating normally. Please verify your data and contact us if you notice any remaining issues.