Intelligence API: Serving Market Regime, Novelty, and Correlation Data at Scale

The Argus intelligence layer computed continuously against 670 million rows of market data spanning 19,172 symbols. The output needed to reach two consumers: an internal Apex trading system that required regime data within 100 milliseconds, and an external-facing API serving paying Avo subscribers.

The storage problem was this: regime data for 209,033 Redis keys, novelty scores for tens of thousands of symbols, and correlation pairs for thousands of symbol combinations were computed on a rolling basis and stored in Redis. The API layer needed to resolve arbitrary user-supplied symbols to the correct Redis keys, assemble multi-namespace data into a coherent response, and serve it with appropriate tier-based cache TTLs, all without blocking on N+1 key lookups.

A secondary problem was TTL integrity. The argus-regime service wrote regime hashes to Redis via hset_multiple() but was missing the subsequent EXPIRE call. Without explicit TTLs, 209,033 keys accumulated as permanent entries. A pipeline failure would leave stale regime data in Redis indefinitely, silently serving incorrect regimes to Apex without any staleness signal.

The API was built as a Next.js App Router route at /api/intelligence/[symbol]. It accepted any symbol in several formats: bare tickers, exchange-prefixed crypto pairs, and user-facing shorthand. A normalization step translated these to the Redis key format used internally.

The route performed three parallel SCAN operations against Redis, one per intelligence namespace, then batched all key fetches into a single pipeline. This avoided N individual GET calls and limited the number of round trips to two: one SCAN phase and one pipeline fetch.

Tier-based cache TTL derived the Cache-Control header from the user's tier. Free-tier responses received a shorter s-maxage than paid tiers. This was enforced at the CDN layer without additional server-side logic.

Symbol normalization: the normalizeSymbol function translated user-facing symbols to internal Redis key format. BTC-USD and BTC both resolved to BINANCE:BTC-USDT. Equity symbols were passed through unchanged.

Pipeline batching: all resolved keys across all three namespaces were queued in a single Redis pipeline. hgetall for regime keys (stored as hashes), get for novelty and correlation keys (stored as JSON strings). Results were consumed sequentially from the pipeline response array.

The missing EXPIRE call in argus-regime/src/main.rs was identified and patched: a single redis.expire(&redis_key, 86400) call after every redis.hset_multiple(). The remediation of existing keys used a Redis SCAN loop to set a 24-hour TTL on all 209,033 existing permanent keys. Verification confirmed zero keys with a -1 TTL after the remediation.

The route returned 503 if Redis was unavailable. It returned 404 only if all three namespaces returned zero results. Partial responses returned 200 with null for missing fields. This prevented false 404s for symbols that had regime data but were not yet in the novelty pipeline.

The intelligence API served requests in under 120 milliseconds for cache hits and under 800 milliseconds for cold fetches against a fully populated Redis instance. The two-round-trip pattern reduced Redis connections by approximately 90% compared to individual key lookups.

The 209,033 regime keys with enforced 24-hour TTLs self-expired on pipeline failure rather than accumulating stale state. Before the fix, a 2-day pipeline outage would have left 209,033 stale regime classifications in Redis with no indicator. After the fix, the same outage would result in empty cache returns within 24 hours.