Signal-to-Execution Workflow Orchestration via Redis Streams

Argus generated signals continuously across 6 source types. Apex needed to consume those signals, apply a multi-layer risk filter, calculate position size, execute orders on either OKX (crypto, 1,210 pairs) or IBKR (equities), and record the full audit trail. The challenge was not any single step. The challenge was making the entire sequence durable.

Before Redis Streams, the naive approach polled ClickHouse's argus.signals table on a fixed interval. That approach had four problems. First, polling introduced artificial latency: the 60-second poll interval meant a high-conviction regime_breakout signal could sit for up to 59 seconds before Apex noticed it. Second, there was no acknowledgment layer: if Apex crashed mid-execution, the same signal would be processed again on restart with no mechanism to detect the duplicate. Third, the filter logic was scattered across ad hoc conditional blocks rather than a defined pipeline. Fourth, there was no schema contract between the signal producer (Argus) and the signal consumer (Apex), so schema changes silently broke execution.

The desired end state was a workflow with defined stages, durable progress tracking, at-least-once delivery with idempotent execution, and a complete audit trail from signal emission through trade close.

---

The workflow was divided into 7 stages: SIGNAL_RECEIVED, FILTER_CHECK, POSITION_SIZING, EXECUTION_ATTEMPT, EXECUTION_CONFIRMED, TRADE_OPEN, TRADE_CLOSED. Each stage produced a structured log entry with timestamps, allowing any trade to be reconstructed from its full decision chain.

Redis 7.2 Streams provided the transport between Argus and Apex. The stream key was argus:signals:stream. Argus published signals as JSON payloads via XADD. Apex subscribed via XREADGROUP under consumer group apex-consumer. Multiple Apex worker processes (apex-worker-1, apex-worker-2) could read from the same stream; Redis load-balanced delivery across workers within the group. Each entry was acknowledged (XACK) only after successful execution, providing at-least-once delivery. The pending-entries list (PEL) held unacknowledged entries for replay on worker restart.

The signal schema was versioned (field version: "1.0"). Every field was typed and documented. Argus embedded the complete decision context in each signal: confidence score, regime classification, suggested Kelly fraction, max position size in USD, holding period in hours, exit rules in plain English, and backtest statistics. Apex required no further Argus consultation after receiving a signal; all information needed for filtering and sizing was self-contained.

---

Stage 1: Signal Receipt and Validation. Apex worker read from the stream using XREADGROUP ... count 10 block 0. For each signal, it validated required fields, type-checked confidence as a float in [0, 1], and compared signal age against ttl_seconds. Signals older than their TTL were dropped immediately and acknowledged (to clear them from the PEL without processing). Schema validation used a typed deserialization struct; unknown fields were logged but not rejected, enabling forward compatibility with schema version 1.1 additions.

Stage 2: Filter Pipeline. Five filters ran in sequence. Any failure dropped the signal and recorded the drop reason to the filter audit log.

1. Confidence threshold: signals with confidence below 0.65 were dropped. The 40 to 60% drop rate target for this filter was tracked as a metric; sustained drop rates above 80% indicated Argus over-generating low-quality signals. 2. Regime compatibility: trend_following signals in choppy regime were dropped. Mean_reversion signals in trending_up or trending_down were dropped. Macro and smart_money signals were regime-agnostic. 3. Symbol universe check: only symbols in the declared TRADEABLE_SYMBOLS set (OKX crypto pairs and IBKR equity symbols) were accepted. Universe was refreshed daily from broker APIs. 4. Concentration limit: existing position value plus proposed new position could not exceed 5% of portfolio NAV for any single symbol. 5. Margin check (IBKR signals only): required margin for the proposed size could not exceed 80% of available margin.

Signals passing all five filters moved to sizing.

Stage 3: Kelly Position Sizing. Apex started from suggested_kelly embedded in the signal. Two adjustments were applied. If current portfolio volatility (annualized) exceeded 15%, Kelly was reduced by 30%. If current portfolio drawdown exceeded 10%, Kelly was reduced by 50%. Final position size in USD was portfolio_value * adjusted_kelly, capped at max_position_usd from the signal. A separate KellyTracker struct maintained a rolling efficiency ratio (actual win rate divided by expected win rate) across the trailing 20 trades, further scaling the multiplier within a [0.5, 1.5] bound to account for realized performance divergence from backtest projections.

Stage 4: Execution. For OKX signals, a market order was placed via OKX REST API. For IBKR signals, orders were routed through IB Gateway on port 4002 using ib_insync. The idempotent execution guard queried a local executions table for the incoming signal_id before placing any order:

This guard prevented duplicate orders on reconnect-replay scenarios. In the tested crash-restart scenario, 3 signals were replayed from the PEL; all 3 were correctly short-circuited by the idempotency check with zero duplicate orders submitted.

Stage 5 to 7: Confirmation, Monitoring, Close. Fill confirmations from OKX and IBKR were recorded with entry price, quantity, and order ID. Exit rules from the signal (regime exit, hard stop percentage, maximum holding period) ran as a monitoring loop against live price data. On close, trade results were written to ClickHouse with signal_id as foreign key, enabling direct comparison between backtest_stats embedded in the signal and realized outcome.

Backpressure. Each Apex worker processed signals sequentially within a single Tokio task (not parallel). This was intentional: concurrent order placement against the same portfolio would bypass concentration and margin checks, as both checks read current state before the previous order had settled. Sequential processing bounded execution throughput to roughly one execution cycle per 900 ms (measured end-to-end from signal receipt through order fill confirmation at OKX), which was sufficient given typical signal volumes of 50 to 200 per second at the Argus emission layer filtered to 5 to 20 accepted signals per minute after the confidence and regime filters.

---

- End-to-end latency (signal emit to execution): Measured from signal.timestamp to order fill timestamp at OKX. The example trace showed: emit at 21:30:45.123Z, received at 21:30:45.205Z (82 ms transport latency via Redis Streams), execution completed at 21:30:46.050Z (total 927 ms including OKX network round trip and order routing). - Filter drop rate: Confidence filter dropped approximately 35 to 45% of signals during trending regimes when Argus emitted more exploratory low-confidence signals. Regime compatibility filter dropped an additional 12 to 18%. Combined, 50 to 60% of signals were dropped before sizing, which aligned with the design target of 40 to 60%. - Idempotency effectiveness: 3 PEL replays in a 6-hour monitoring window. Zero duplicate orders. Signal_id lookup latency was under 2 ms on the local SQLite executions table. - Kelly efficiency: Over the first 47 closed trades, realized win rate was 0.64 versus backtest expected 0.62, producing a Kelly efficiency multiplier of 1.03. Position sizes were effectively at full suggested_kelly during this period. - P99 Redis Streams latency: Measured across 10,000 consecutive signal publishes and reads. P99 was 8 ms at the Redis layer (excluding Argus processing and OKX network time). - Crash recovery time: After a SIGKILL to apex-worker, PEL contained 47 unacknowledged entries. Worker restarted, drained PEL, and returned to zero lag within 8 seconds.

---