Webhook Architecture

This document provides a technical deep-dive into OilPriceAPI's webhook delivery infrastructure, designed for developers who need to understand the system internals for reliable integration.

System Overview

OilPriceAPI uses a distributed event-driven architecture with Sidekiq workers for reliable webhook delivery. Events flow from data sources through processing pipelines to customer endpoints.

flowchart LR
    subgraph Event Sources
        PS[Price Scrapers]
        DS[Drilling Sources]
        AS[Alert Service]
    end

    subgraph Event Processing
        WS[WebhookService]
        FIL[Filter Engine]
        SIG[Signature Generator]
    end

    subgraph Delivery Queue
        SQ[(Sidekiq Queue)]
        WDW[WebhookDeliveryWorker]
    end

    subgraph Customer Infrastructure
        E1[Endpoint 1]
        E2[Endpoint 2]
        EN[Endpoint N]
    end

    subgraph Monitoring
        LOG[(Delivery Logs)]
        RETRY[Retry Scheduler]
    end

    PS & DS & AS --> WS
    WS --> FIL
    FIL --> SIG
    SIG --> SQ
    SQ --> WDW
    WDW --> E1 & E2 & EN
    WDW --> LOG
    WDW -.-> |On Failure| RETRY
    RETRY --> SQ

Event Processing Pipeline

Stage 1: Event Generation

Events are generated from multiple sources within the OilPriceAPI system:

flowchart TD
    subgraph Data Sources
        ICE[ICE Exchange Prices]
        CME[CME Futures]
        BH[Baker Hughes Rig Data]
        RRC[State Permit APIs]
    end

    subgraph Workers
        PW[PriceScraperWorker]
        RW[RigCountWorker]
        WPW[WellPermitWorker]
    end

    subgraph Events Created
        E1[price.updated]
        E2[price.significant_change]
        E3[drilling.rig_count.updated]
        E4[drilling.well_permit.new]
    end

    ICE & CME --> PW
    BH --> RW
    RRC --> WPW

    PW --> E1 & E2
    RW --> E3
    WPW --> E4

Stage 2: Webhook Service

The WebhookService orchestrates event distribution:

# Simplified WebhookService logic
class WebhookService
  def self.broadcast(event_type, payload)
    # Find all webhooks subscribed to this event
    webhooks = Webhook.active.subscribed_to(event_type)

    webhooks.each do |webhook|
      # Apply commodity filters
      next unless matches_commodity_filter?(webhook, payload)

      # Apply state filters (for well permits)
      next unless matches_state_filter?(webhook, payload)

      # Queue for delivery
      WebhookDeliveryWorker.perform_async(
        webhook.id,
        event_type,
        payload,
        generate_event_id
      )
    end
  end
end

Stage 3: Filtering

flowchart TD
    EVENT[Incoming Event]

    EVENT --> CF{Commodity Filter?}
    CF -->|No Filter| SF
    CF -->|Has Filter| CFM{Matches?}
    CFM -->|No| DROP1[Drop Event]
    CFM -->|Yes| SF

    SF{State Filter?}
    SF -->|No Filter| QUEUE
    SF -->|Has Filter| SFM{Matches?}
    SFM -->|No| DROP2[Drop Event]
    SFM -->|Yes| QUEUE

    QUEUE[Queue for Delivery]

Stage 4: Signature Generation

Every webhook payload is signed with HMAC-SHA256 for verification:

sequenceDiagram
    participant WS as WebhookService
    participant SIG as SignatureGenerator
    participant Q as Sidekiq Queue

    WS->>SIG: Generate signature(payload, secret)

    Note over SIG: timestamp = Unix epoch seconds
    Note over SIG: message = JSON(payload) + "." + timestamp
    Note over SIG: signature = HMAC-SHA256(message, secret)

    SIG-->>WS: {signature, timestamp}

    WS->>Q: Enqueue {payload, signature, timestamp, headers}

Signature algorithm:

timestamp = current Unix timestamp (seconds)
message = JSON.stringify(payload) + "." + timestamp
signature = HMAC-SHA256(message, webhook_secret).hexdigest()

Stage 5: Delivery

flowchart TD
    JOB[Job Dequeued]

    JOB --> PREP[Prepare Request]
    PREP --> SEND[HTTP POST]

    SEND --> RESP{Response?}
    RESP -->|2xx| SUCCESS[Mark Delivered]
    RESP -->|4xx Client Error| FAIL1[Mark Failed - No Retry]
    RESP -->|5xx Server Error| RETRY1[Schedule Retry]
    RESP -->|Timeout| RETRY2[Schedule Retry]
    RESP -->|Connection Error| RETRY3[Schedule Retry]

    SUCCESS --> LOG1[Log Success]
    FAIL1 --> LOG2[Log Failure]
    RETRY1 & RETRY2 & RETRY3 --> BACKOFF[Calculate Backoff]
    BACKOFF --> REQUEUE[Requeue Job]

Delivery Guarantees

At-Least-Once Delivery

OilPriceAPI guarantees that every event will be delivered at least once to your endpoint. This means:

Successful events: Delivered exactly once (in most cases)
Failed events: Retried up to 5 times
Network issues: May result in duplicate deliveries

Idempotency Required

Your webhook handler must be idempotent. Use the event_id to detect and handle duplicate deliveries.

Retry Schedule

Failed deliveries follow exponential backoff:

Attempt	Delay After Failure	Cumulative Time
1st retry	1 minute	1 minute
2nd retry	5 minutes	6 minutes
3rd retry	15 minutes	21 minutes
4th retry	1 hour	1 hour 21 minutes
5th retry	6 hours	7 hours 21 minutes

gantt
    title Retry Timeline
    dateFormat HH:mm
    axisFormat %H:%M

    section Delivery Attempts
    Initial Attempt      :a1, 00:00, 1m
    1st Retry (1min)    :a2, 00:01, 1m
    2nd Retry (5min)    :a3, 00:06, 1m
    3rd Retry (15min)   :a4, 00:21, 1m
    4th Retry (1hr)     :a5, 01:21, 1m
    5th Retry (6hr)     :a6, 07:21, 1m

Failure Handling

After 5 failed attempts:

Event marked as failed in delivery logs
Webhook endpoint flagged with failure
After 10 consecutive failures: webhook auto-deactivated
Email notification sent to account owner

Event Retention

Event State	Retention Period
Pending	Until delivered or failed
Delivered	30 days
Failed	30 days

Security Model

Signature Verification Flow

sequenceDiagram
    participant OPA as OilPriceAPI
    participant NET as Network
    participant APP as Your Application

    OPA->>OPA: Generate payload
    OPA->>OPA: timestamp = now()
    OPA->>OPA: signature = HMAC(payload.timestamp, secret)

    OPA->>NET: POST /webhook<br/>Body: payload<br/>X-Signature: signature<br/>X-Timestamp: timestamp

    NET->>APP: Forward request

    APP->>APP: Extract signature header
    APP->>APP: Extract timestamp header
    APP->>APP: Check timestamp freshness (< 5 min)

    alt Timestamp too old
        APP-->>OPA: 401 Unauthorized (replay attack)
    else Timestamp valid
        APP->>APP: expected = HMAC(body.timestamp, secret)
        APP->>APP: Compare signatures (constant-time)

        alt Signatures don't match
            APP-->>OPA: 401 Unauthorized
        else Signatures match
            APP->>APP: Process webhook
            APP-->>OPA: 200 OK
        end
    end

Security Headers

Every webhook request includes these headers:

Header	Description	Example
`X-OilPriceAPI-Event`	Event type	`price.updated`
`X-OilPriceAPI-Event-ID`	Unique event identifier	`evt_1a2b3c4d5e`
`X-OilPriceAPI-Signature`	HMAC-SHA256 signature	`a1b2c3d4e5...`
`X-OilPriceAPI-Signature-Timestamp`	Unix timestamp	`1704067200`
`Content-Type`	Always JSON	`application/json`
`User-Agent`	Identifies OilPriceAPI	`OilPriceAPI-Webhook/2.0`

Replay Attack Prevention

The timestamp check prevents replay attacks:

function isValidTimestamp(timestamp) {
  const now = Math.floor(Date.now() / 1000);
  const age = now - parseInt(timestamp);

  // Reject if older than 5 minutes
  return age <= 300;
}

IP Allowlisting

Webhook requests originate from these IP ranges:

159.203.XX.XX/24  (DigitalOcean NYC)
167.71.XX.XX/24   (DigitalOcean NYC)

Dynamic IPs

For the most current IP list, query our status endpoint:

curl https://api.oilpriceapi.com/v1/webhook-ips

Queue Architecture

Sidekiq Configuration

flowchart LR
    subgraph Priority Queues
        Q1[critical<br/>weight: 10]
        Q2[default<br/>weight: 5]
        Q3[webhooks<br/>weight: 3]
        Q4[low<br/>weight: 1]
    end

    subgraph Workers
        W1[Worker Pool<br/>20 threads]
    end

    Q1 & Q2 & Q3 & Q4 --> W1

Queue priorities:

critical: Account operations, billing
default: Price scraping, data processing
webhooks: Webhook delivery (priority 3)
low: Analytics, cleanup tasks

Worker Configuration

# Sidekiq worker for webhook delivery
class WebhookDeliveryWorker
  include Sidekiq::Worker

  sidekiq_options(
    queue: :webhooks,
    retry: 5,
    dead: false,
    backtrace: true
  )

  sidekiq_retry_in do |count|
    case count
    when 0 then 60        # 1 minute
    when 1 then 300       # 5 minutes
    when 2 then 900       # 15 minutes
    when 3 then 3600      # 1 hour
    else 21600            # 6 hours
    end
  end

  def perform(webhook_id, event_type, payload, event_id)
    # Delivery logic...
  end
end

Performance Characteristics

Throughput

Metric	Value	Notes
Events processed/minute	10,000+	Across all customers
Delivery latency (p50)	200ms	From event to delivery
Delivery latency (p95)	800ms	Includes network time
Delivery latency (p99)	2s	Edge cases, retries

Timeout Configuration

Setting	Value
Connection timeout	10 seconds
Read timeout	30 seconds
Total request timeout	30 seconds

Concurrent Delivery

flowchart TD
    subgraph Sidekiq Process
        T1[Thread 1]
        T2[Thread 2]
        T3[Thread 3]
        TN[Thread N...]
    end

    subgraph Customer Endpoints
        E1[Customer A]
        E2[Customer B]
        E3[Customer C]
    end

    T1 --> E1
    T2 --> E2
    T3 --> E1
    TN --> E3

    Note[20 concurrent threads<br/>per Sidekiq process]

Note: Multiple events may be delivered to the same endpoint concurrently. Ensure your endpoint handles concurrent requests.

Monitoring & Debugging

Delivery Status Dashboard

Access your webhook delivery logs via API:

# List recent deliveries
GET /v1/webhooks/{id}/events

# Response
{
  "webhook_events": [
    {
      "id": "we_abc123",
      "event_id": "evt_xyz789",
      "event_type": "price.updated",
      "status": "delivered",
      "attempts": 1,
      "response_status_code": 200,
      "delivery_duration_ms": 245,
      "created_at": "2025-08-03T14:30:00Z"
    },
    {
      "id": "we_def456",
      "event_id": "evt_uvw012",
      "event_type": "drilling.rig_count.updated",
      "status": "failed",
      "attempts": 5,
      "last_error": "Connection refused",
      "created_at": "2025-08-03T14:25:00Z"
    }
  ]
}

Delivery Statuses

Status	Description	Action
`pending`	Queued for delivery	Wait
`delivering`	Currently being sent	Wait
`delivered`	Successfully received (2xx)	None
`failed`	All retries exhausted	Check endpoint
`skipped`	Filtered out by rules	Check filters

Common Issues

Issue	Symptoms	Solution
SSL Certificate Error	All deliveries fail	Ensure valid SSL cert
Timeout	Deliveries fail after 30s	Optimize endpoint response time
401 Unauthorized	Signature verification fails	Check secret configuration
Connection Refused	Cannot reach endpoint	Check firewall/DNS

Debug Mode

Enable detailed logging for a webhook:

PATCH /v1/webhooks/{id}
{
  "debug_mode": true
}

Debug mode captures:

Full request/response headers
Response body (first 1KB)
Timing breakdowns
TLS negotiation details

High Availability

Multi-Region Delivery

flowchart TB
    subgraph Primary Region
        P_SQ[(Sidekiq Queue)]
        P_W1[Worker 1]
        P_W2[Worker 2]
    end

    subgraph Failover Region
        F_SQ[(Sidekiq Queue)]
        F_W1[Worker 1]
        F_W2[Worker 2]
    end

    REDIS[(Redis Cluster)]

    REDIS --> P_SQ & F_SQ
    P_SQ --> P_W1 & P_W2
    F_SQ --> F_W1 & F_W2

Disaster Recovery

Scenario	Recovery Time	Data Loss
Worker failure	Immediate	None (job requeued)
Region failure	< 5 minutes	None (Redis replication)
Redis failure	< 30 seconds	Possible in-flight jobs

Deduplication

To handle potential duplicates from failover:

Every event has a unique event_id
Store processed event_ids for 24 hours
Skip processing if event_id already seen

const processedEvents = new Set(); // Use Redis in production

function processWebhook(event) {
  if (processedEvents.has(event.id)) {
    return; // Already processed
  }

  // Process event...

  processedEvents.add(event.id);
}

Webhook API Reference - Event types, configuration, and examples
Webhook Cookbook - Real-world integration recipes
Security Guide - Best practices for secure integrations