Building a Multi-Exchange WebSocket Ingest Layer for 1,200+ Trading Pairs

Argus needed a unified market data foundation across the major crypto exchanges. Each exchange publishes real-time order book and trade data over WebSocket, but every venue runs a different authentication model, different subscription protocol, and different message envelope. OKX uses a channel-based subscribe/unsubscribe model with token rotation. KuCoin requires a server-issued token before the WebSocket handshake can begin. Binance uses topic strings with a threshold of 1,024 subscriptions per connection. Deribit delivers option chain data as incremental deltas against a running snapshot.

Sourcing data from a single venue was straightforward. Sourcing from ten simultaneously introduced three compounding problems. First, upstream connection instability was unpredictable. KuCoin's infrastructure would close sockets mid-stream; Binance connections would become silent without emitting any close frame. Second, each exchange had a different normalized structure for OHLCV and tick data, requiring per-source deserialization before a shared insert path could be used. Third, ClickHouse insert throughput had a ceiling: bulk inserts are efficient, but per-row async writes at sustained volume would saturate the async buffer if backpressure was not coordinated across all ten feeds.

The broader goal was not just to collect data. The 1,400-feature Argus engine (argus-features, compiled with AVX2 SIMD) and the 5-regime detection pipeline (argus-regime, 8 timeframes) depended on consistent, low-latency bar data. A stale feed silently corrupted downstream signal quality.

---

Ten separate Rust binaries were compiled: argus-ingest-okx, argus-ingest-binance, argus-ingest-kraken, argus-ingest-bybit, argus-ingest-kucoin, argus-ingest-coinbase, argus-ingest-deribit, argus-ingest-gateio, argus-ingest-bitget, and argus-ingest-hyperliquid. Each binary owned exactly one exchange. This decision had three advantages: process-level isolation meant one exchange crash did not bring down the others; per-binary resource profiles could be tuned independently (Binance required 96.7 MB resident due to high pair count; OKX and Kraken ran at 3.7 MB and 4.9 MB respectively); and systemd supervision gave each process an independent restart policy.

Each binary followed the same internal structure. A connection manager task opened the WebSocket, handled the exchange-specific handshake, and emitted raw message bytes into an unbounded Tokio channel. A normalization task consumed from that channel, parsed the exchange-specific JSON envelope, and emitted a typed Bar struct into a second channel. A writer task batched Bar structs and flushed to ClickHouse via HTTP INSERT using the clickhouse-rs crate against port 8123.

ClickHouse schema for argus.bars_1m used Gorilla codec on all price columns, DoubleDelta on the timestamp column, and LowCardinality on the symbol string. ZSTD(3) was applied as the final compression layer. This combination delivered 5 to 10x compression improvement over the previous QuestDB store while keeping p99 query latency in the 20 to 100 ms range versus the prior 100 to 500 ms.

---

Backpressure strategy. The unbounded Tokio channel between the WebSocket receiver and the normalizer acted as a pressure valve during burst ingestion. The normalizer applied a configurable batch window (50 ms default) before forwarding to the writer. The writer blocked on ClickHouse INSERT acknowledgment before accepting the next batch, providing end-to-end backpressure. If the writer fell behind due to ClickHouse load, channel depth increased and the normalizer naturally slowed consumption, which in turn reduced the rate at which the socket was drained. This avoided out-of-memory conditions during exchange reconnect floods where buffered messages could arrive in large bursts.

At-least-once semantics and idempotency. The ingest layer operated under at-least-once delivery. Each Bar struct carried a composite key of (symbol, source, ts). ClickHouse's ReplacingMergeTree engine deduplicated rows with identical keys during background merges. On reconnect after a socket drop, the binary re-subscribed from the beginning of the current minute window, which could produce duplicate bars for already-ingested timestamps. The deduplication merge handled this without operator intervention.

Replay capability. For historical backfill, a separate argus-backfill binary was written using the same normalization pipeline but sourced from 7-exchange REST APIs rather than live sockets. The same Bar struct and INSERT path were reused, meaning the backfill and live ingest wrote to the same table with no schema divergence. This allowed the system to backfill gaps (for example, the 25-day Binance staleness window discovered in the April 25 audit) by replaying the REST endpoint for that period without touching any other code path.

Lag monitoring. A dedicated argus-health binary queried ClickHouse every 60 seconds:

Any source with lag exceeding 300 seconds triggered a structured alert via argus-alert-dispatcher. During normal operation, OKX sustained 4 to 5 minute data age and Kraken sustained 1 minute data age. Kucoin and Coinbase exhibited a recurring pattern: 60-second reconnect loops (visible in logs as ERROR: WebSocket closed by server; INFO: Reconnecting in 60s) that produced zero writes despite active process uptime, indicating the connection manager was restoring the socket but the subscription confirmation was failing silently.

Consumer group model. The live ingest binaries did not use a message queue. They wrote directly to ClickHouse. The downstream regime and feature pipelines polled ClickHouse on their own cadence, making the storage layer the implicit broker. This avoided the operational overhead of a Kafka or Pulsar deployment at the cost of some latency precision. The regime pipeline polled every 30 seconds; the feature engine polled every 60 seconds.

---

- Sustained throughput: OKX feed alone produced approximately 186 rows per minute across 1,200 pairs during active market hours, or roughly 3.1 rows per second. All ten feeds combined delivered an estimated 800 to 1,200 rows per minute during peak overlap of US and Asia trading sessions. - Row accumulation: OKX reached 93.64 million rows in argus.bars_1m from the live feed. Kraken accumulated 1.08 million rows. Total bars_1m table size reached 714.4 million rows including historical backfill. - Process memory: OKX ingest ran at 3.7 MB resident. Kraken at 4.9 MB. Coinbase at 8.2 MB. Binance at 96.7 MB due to subscribing across 1,000+ pairs in a single process, which is the primary candidate for per-pair channel sharding in a future revision. - Lag during incidents: During the Kucoin 14-hour staleness window, the process logged 840 reconnect cycles over the period. Each cycle successfully re-established the WebSocket and subscription confirmation, but a database write timeout in the writer task caused all batches to be silently dropped. The lag monitor fired after 5 minutes and was visible in the health dashboard, but no automated remediation was in place at audit time. - p99 ClickHouse insert latency: Async insert buffer flushed every 10 MB or every 200 ms. Under sustained load, p99 flush acknowledgment from ClickHouse was measured at under 80 ms.

---