Data Quality and Validation

Overview

OilPriceAPI implements comprehensive data quality measures to ensure accurate and reliable commodity pricing across all data feeds.

Validation System

Real-Time Price Validation

All incoming price data passes through our PriceValidationService which performs:

  • Range Checking: Ensures prices fall within expected bounds for each commodity
  • Variance Detection: Alerts on price changes exceeding 50% from recent averages
  • Unit Consistency: Validates currency and unit combinations
  • Anomaly Detection: Identifies and flags suspicious data patterns

Commodity-Specific Validation Rules

UK Natural Gas (NATURAL_GAS_GBP)

  • Valid Range: 30-200 pence per therm
  • Units: Pence (GBp), not pounds
  • Update Frequency: Every 15 minutes
  • Source: IG Markets

Crude Oil (BRENT, WTI)

  • Valid Range: $20-$200 per barrel
  • Variance Threshold: ±20% daily movement triggers review
  • Update Frequency: Real-time (15-minute delay)

Gold & Precious Metals

  • Valid Range: Market-dependent with historical bounds
  • Decimal Precision: 2 decimal places
  • Update Frequency: Every minute for spot prices

Data Correction Procedures

Automatic Corrections

The system automatically handles:

  • Decimal point misplacements
  • Unit conversion errors
  • Timezone adjustments

Manual Review Triggers

Manual review is triggered for:

  • Price movements >50% from baseline
  • Values outside historical ranges
  • Source feed disruptions
  • Customer-reported anomalies

Historical Data Integrity

Backup Policy

  • All original data preserved before corrections
  • Backup tables maintained for 90 days
  • Audit trail for all modifications

Correction Process

  1. Identify affected time period and commodity
  2. Create backup of original data
  3. Apply validated corrections
  4. Verify consistency across all aggregations
  5. Update documentation and notify affected users

Quality Metrics

Data Availability

  • Target: 99.9% uptime for all feeds
  • Measurement: Monitored every minute
  • Recovery Time: <15 minutes for feed disruptions

Data Accuracy

  • Target: <0.1% error rate
  • Validation: Daily automated checks
  • Review: Weekly manual audits

Reporting Issues

Customer Reports

To report data quality issues:

  1. Email: [email protected]
  2. Include: Commodity code, timestamp, observed value
  3. Response time: Within 3 business days

Issue Resolution Timeline

  • Critical (wrong order of magnitude): 3 business days
  • Major (incorrect values): <4 hours
  • Minor (formatting issues): <24 hours

Recent Improvements

August 2025

  • Enhanced UK Natural Gas validation
  • Added real-time anomaly detection
  • Improved decimal handling for IG Markets feed

July 2025

  • Implemented PriceValidationService
  • Added variance-based alerts
  • Enhanced backup procedures

Best Practices for API Users

  1. Cache Appropriately: Some data updates infrequently (e.g., weekly storage reports)
  2. Validate Units: Always check the currency and unit fields
  3. Handle Null Values: Missing data may occur during feed disruptions
  4. Monitor Changes: Subscribe to webhooks for data corrections
  5. Report Anomalies: Help us maintain quality by reporting suspicious data

Data Sources & Reliability Tiers

Tier 1 - Primary Sources

  • Government agencies (EIA, NOAA)
  • Major exchanges (ICE, CME, NYMEX)
  • Direct market feeds

Tier 2 - Verified Third-Party

  • Licensed data providers
  • Industry aggregators
  • Regional exchanges

Tier 3 - Supplementary

  • News sources for context
  • Alternative data providers
  • Community-reported data

Contact

For data quality concerns or to report issues:

  • Email: [email protected]
  • Response Time: 3 business days
  • Emergency: For critical data issues affecting trading