Webhook Architecture

flowchart LR
    subgraph Event Sources
        PS[Price Scrapers]
        DS[Drilling Sources]
        AS[Alert Service]
    end

    subgraph Event Processing
        WS[WebhookService]
        FIL[Filter Engine]
        SIG[Signature Generator]
    end

    subgraph Delivery Queue
        SQ[(Sidekiq Queue)]
        WDW[WebhookDeliveryWorker]
    end

    subgraph Customer Infrastructure
        E1[Endpoint 1]
        E2[Endpoint 2]
        EN[Endpoint N]
    end

    subgraph Monitoring
        LOG[(Delivery Logs)]
        RETRY[Retry Scheduler]
    end

    PS & DS & AS --> WS
    WS --> FIL
    FIL --> SIG
    SIG --> SQ
    SQ --> WDW
    WDW --> E1 & E2 & EN
    WDW --> LOG
    WDW -.-> |On Failure| RETRY
    RETRY --> SQ

flowchart TD
    subgraph Data Sources
        MR[Published Market Reporting]
        PDS[Public Data Sources]
        BH[Baker Hughes Rig Data]
        RRC[State Permit APIs]
    end

    subgraph Workers
        PW[PriceScraperWorker]
        RW[RigCountWorker]
        WPW[WellPermitWorker]
    end

    subgraph Events Created
        E1[price.updated]
        E2[price.significant_change]
        E3[drilling.rig_count.updated]
        E4[drilling.well_permit.new]
    end

    MR & PDS --> PW
    BH --> RW
    RRC --> WPW

    PW --> E1 & E2
    RW --> E3
    WPW --> E4

flowchart TD
    EVENT[Incoming Event]

    EVENT --> CF{Commodity Filter?}
    CF -->|No Filter| SF
    CF -->|Has Filter| CFM{Matches?}
    CFM -->|No| DROP1[Drop Event]
    CFM -->|Yes| SF

    SF{State Filter?}
    SF -->|No Filter| QUEUE
    SF -->|Has Filter| SFM{Matches?}
    SFM -->|No| DROP2[Drop Event]
    SFM -->|Yes| QUEUE

    QUEUE[Queue for Delivery]

sequenceDiagram
    participant WS as WebhookService
    participant SIG as SignatureGenerator
    participant Q as Sidekiq Queue

    WS->>SIG: Generate signature(payload, secret)

    Note over SIG: timestamp = Unix epoch seconds
    Note over SIG: message = JSON(payload) + "." + timestamp
    Note over SIG: signature = HMAC-SHA256(message, secret)

    SIG-->>WS: {signature, timestamp}

    WS->>Q: Enqueue {payload, signature, timestamp, headers}

flowchart TD
    JOB[Job Dequeued]

    JOB --> PREP[Prepare Request]
    PREP --> SEND[HTTP POST]

    SEND --> RESP{Response?}
    RESP -->|2xx| SUCCESS[Mark Delivered]
    RESP -->|4xx Client Error| FAIL1[Mark Failed - No Retry]
    RESP -->|5xx Server Error| RETRY1[Schedule Retry]
    RESP -->|Timeout| RETRY2[Schedule Retry]
    RESP -->|Connection Error| RETRY3[Schedule Retry]

    SUCCESS --> LOG1[Log Success]
    FAIL1 --> LOG2[Log Failure]
    RETRY1 & RETRY2 & RETRY3 --> BACKOFF[Calculate Backoff]
    BACKOFF --> REQUEUE[Requeue Job]

gantt
    title Retry Timeline
    dateFormat HH:mm
    axisFormat %H:%M

    section Delivery Attempts
    Initial Attempt      :a1, 00:00, 1m
    1st Retry (1min)    :a2, 00:01, 1m
    2nd Retry (5min)    :a3, 00:06, 1m
    3rd Retry (15min)   :a4, 00:21, 1m
    4th Retry (1hr)     :a5, 01:21, 1m
    5th Retry (6hr)     :a6, 07:21, 1m

sequenceDiagram
    participant OPA as OilPriceAPI
    participant NET as Network
    participant APP as Your Application

    OPA->>OPA: Generate payload
    OPA->>OPA: timestamp = now()
    OPA->>OPA: signature = HMAC(payload.timestamp, secret)

    OPA->>NET: POST /webhook<br/>Body: payload<br/>X-Signature: signature<br/>X-Timestamp: timestamp

    NET->>APP: Forward request

    APP->>APP: Extract signature header
    APP->>APP: Extract timestamp header
    APP->>APP: Check timestamp freshness (< 5 min)

    alt Timestamp too old
        APP-->>OPA: 401 Unauthorized (replay attack)
    else Timestamp valid
        APP->>APP: expected = HMAC(body.timestamp, secret)
        APP->>APP: Compare signatures (constant-time)

        alt Signatures don't match
            APP-->>OPA: 401 Unauthorized
        else Signatures match
            APP->>APP: Process webhook
            APP-->>OPA: 200 OK
        end
    end

flowchart LR
    subgraph Priority Queues
        Q1[critical<br/>weight: 10]
        Q2[default<br/>weight: 5]
        Q3[webhooks<br/>weight: 3]
        Q4[low<br/>weight: 1]
    end

    subgraph Workers
        W1[Worker Pool<br/>20 threads]
    end

    Q1 & Q2 & Q3 & Q4 --> W1

flowchart TD
    subgraph Sidekiq Process
        T1[Thread 1]
        T2[Thread 2]
        T3[Thread 3]
        TN[Thread N...]
    end

    subgraph Customer Endpoints
        E1[Customer A]
        E2[Customer B]
        E3[Customer C]
    end

    T1 --> E1
    T2 --> E2
    T3 --> E1
    TN --> E3

    Note[20 concurrent threads<br/>per Sidekiq process]

flowchart TB
    subgraph Primary Region
        P_SQ[(Sidekiq Queue)]
        P_W1[Worker 1]
        P_W2[Worker 2]
    end

    subgraph Failover Region
        F_SQ[(Sidekiq Queue)]
        F_W1[Worker 1]
        F_W2[Worker 2]
    end

    REDIS[(Redis Cluster)]

    REDIS --> P_SQ & F_SQ
    P_SQ --> P_W1 & P_W2
    F_SQ --> F_W1 & F_W2