Skip to content

Live Market Pipeline And Processor Machinery

This document describes the runtime path for live market data inside backend-app, from subscription resolution through strategy execution, trade execution, persistence, and server publishing.

For the broader module map and runtime boundaries around this path, see Architecture Overview.

Scope

The pipeline described here spans:

  • startup task ordering in backend-app
  • live feed collection and fan-out
  • Processor, StrategyExecutor, and StrategyExecution
  • trade execution and trade persistence
  • raw market-data persistence through the recorder
  • event publishing to backend-server through DataBridgeLauncher

It is the source-of-truth overview for the processor-side machinery that the recent EMA and volume_breakout runtime tests exercise.

Diagrams

The following diagrams illustrate the processor and strategy pipeline:

Diagram Description
Data Flow Three-tier architecture: Processor → Executor → Execution
Route State Machine Route lifecycle: Active, Stopping, Degraded, Gone, Fail-Fast
Command Loop Single-owner control plane: command sources, channel, snapshot
Mailbox Hierarchy Two-level mailbox architecture and start barrier
Degradation & Fail-Fast Overflow detection, deduplication, escalation decision

High-Level Flow

QuantApp startup
  -> AddNewFeedsTask / AddNewStrategiesTask
  -> LoadInstrumentNamesTask / SetSubscribedInstrumentsTask
  -> StartTradingLeadershipTask
  -> StartDataFlowTask
  -> StartAuthStateTask
  -> StartDataBridgeTask

feedRepository.subscribedInstrumentsFlow()
  -> NormalizedMarketFeed (collects each raw feed exactly once, shareIn)
  -> liveFeed.feed(dataSourceIdentifier, instrumentIds)
  -> merged Flow<FeedData<RawTick>> (datasources derive nothing)
  -> TickNormalizationStage per feed (hydrated per-instrument TickNormalizer)
       -> shared NormalizedFeedBatch (emitted + recorded-late)
            -> Processor (emitted only)
            -> DataRecorder.feedConsumer (emitted + recorded-late)

Processor
  -> routeFeedData() (sequential batch processing)
  -> StrategyMailbox (Channel<MarketData>) per strategy
  -> StrategyExecutor per strategy
       -> InstrumentMailbox (ProcessorMailbox) per instrument
       -> StrategyExecution per strategy+instrument
  -> StrategyEventCoordinator
       -> StrategyOutput / StrategySignalEvent to DataBridge
       -> strategy outputs and signals to DataRecorder
       -> run-bound trade requests to TradeExecutor
  -> tradeEvents

DataRecorder
  -> raw market data to QuestDB
  -> strategy outputs and signals to QuestDB
  -> trade events to OrdersRepository

DataBridgeLauncher
  -> strategy outputs
  -> strategy signals
  -> market feed events
  -> trade events
  -> open positions
  -> readiness and errors

AuthStateCoordinator
  -> RecoveryManager.recoveryFlow().recoverable() for auth prompts
  -> clearLogicalRecovery() for obsolete prompts
  -> ResumableDataSource.resumeIfStopped() on OAuth Valid

ProvisionedTokenStateCoordinator
  -> ResumableDataSource.resumeIfStopped() on UPSTOX_ANALYTICS Valid

Startup Sequence

QuantApp groups StartupTasks by phase and runs one phase at a time. Tasks inside the same phase run concurrently.

The current startup order is:

  1. AddNewFeedsTask and AddNewStrategiesTask
  2. LoadInstrumentNamesTask and SetSubscribedInstrumentsTask
  3. StartTradingLeadershipTask
  4. StartDataFlowTask and RelaySubscriptionUpdatesTask
  5. StartAuthStateTask
  6. StartDataBridgeTask

Why this order matters:

  • feeds and strategies must exist before subscription resolution starts
  • datasource subscription state must be restored before live streaming begins
  • trading leadership starts before strategy signals can reach TradeExecutor; a standby replica continues processing and recording data but cannot submit broker orders
  • the live feed must exist before the server-forwarding bridge starts collecting from it
  • token-state side effects (recovery publication, obsolete-prompt cleanup, datasource resume) publish through RecoveryManager.errorFlow, a replaying StateFlow, so they are retained until the data bridge subscribes — StartAuthStateTask and StartDataBridgeTask have no hard ordering dependency between them
  • the subscription relay (RelaySubscriptionUpdatesTask, same phase as StartDataFlowTask) starts after the startup push has applied the restored set, so its first emission re-pushes the same set idempotently; each feed's runtime push waits for that feed's current pipeline incarnation to hydrate and attach its raw-stream collector carrying exactly the emission's per-feed instrument set (PipelineLiveGate.awaitFeedLive — set equality, not coverage: the relay and the pipeline are independent collectors of the subscription flow, so a stale mark left over from the previous emission can never authorize a push for a different set, including the stale-superset case of a removal push followed by a re-add) before the mutated set reaches the datasource; an empty or absent feed's push awaits the opposite signal — its live mark disappearing at the pipeline's emission clear. The gate is best-effort only in the sub-millisecond window around the attach itself (the mark is published just before the datasource subscription and is not revoked on a mid-stream recoverable retry); the multi-second hydration and emission-swap windows are closed structurally, because the pipeline collects the RAW instrument-ID flow (FeedRepository.subscribedInstrumentIdsFlow) and resolves the per-feed grouping inside each emission block after a synchronous live-mark clear — a mutation cancels the previous emission's incarnations before any data-store read can suspend, so a live mark can only ever be published by the current emission's incarnation

Token Broker Connection

The tokenBroker() Ktor module runs before core() and establishes the kRPC connection to backend-token-broker. Both kRPC clients are wrapped in ResilientTokenBrokerService and ResilientProvisionedTokenBrokerService, which provide automatic reconnect with exponential backoff and shared state flows per broker. If the token broker is unavailable at startup, the resilient decorators retry the connection rather than crashing the pod — once the broker becomes reachable, the token-state stream reconnects and the app recovers without a restart.

UpstoxApiDecorator uses awaitValidToken() to suspend HTTP requests when the token state is AwaitingAuth. AuthStateCoordinator collects recoveryFlow(broker) wrapped in RecoveryManager.recoverable() to surface auth prompts in the UI, watches tokenStateUpdates(broker) to clear obsolete prompts via clearLogicalRecovery(), and resumes datasources via ResumableDataSource.resumeIfStopped() when the OAuth token transitions back to Valid (either through user auth or out-of-band refresh).

The market data feed and historical candle data authenticate with the provisioned UPSTOX_ANALYTICS token instead of OAuth. ProvisionedTokenStateCoordinator (started by the same StartAuthStateTask) resumes the feed datasource via ResumableDataSource.resumeIfStopped() when UPSTOX_ANALYTICS transitions back to Valid after a Secret rotation — the analytics token has no OAuth flow, so while it is Failed the feed reports terminal DEGRADED until the operator rotates the Secret.

Feed Reception And Fan-Out

StartDataFlowTask.startFeed() is the main entry point for live market data inside backend-app.

It wires every FeedConsumer to the shared NormalizedMarketFeed.batches flow through per-consumer mailboxes: each normalized batch routes its emitted ticks to every consumer and its recorded-late ticks only to consumers that opt in (FeedConsumer.recordsLateFeedData).

NormalizedMarketFeed is the single, hot, normalized market stream:

  • it watches feedRepository.subscribedInstrumentIdsFlow() and resolves the raw instrument IDs to per-feed groups inside the latest block (resolveSubscribedInstrumentIds), mapping each group to a DataSourceIdentifier (the raw IDs come from the operator-intent subscriptions file on the shared volume; the per-feed resolution goes through DataStore)
  • each raw feed flow is wrapped with RecoveryManager.recoverable() and collected exactly once
  • every batch is normalized through the per-feed TickNormalizationStage (see below)
  • the result is shared (shareIn) and fanned out — downstream consumers never collect the raw feeds
  • a configured feed whose name resolves to no registered DataSourceIdentifier is a configuration error: at startup it fails StartDataFlowTask's hydration-gate await (awaitFirstHydrationPass completes exceptionally, terminating the process — the fail-fast / no-partial-readiness contract); mid-run it is dropped with an error log while valid feeds keep flowing, and RequiredFeedReadinessReporter blocks readiness on it — the shared stream is never killed mid-run, since that would take every valid feed down with it
  • a terminal pipeline failure in one feed (an exception that escapes its recovery boundary — a normalizer defect, a non-recoverable datasource failure) stops only that feed: it is logged, marked DEGRADED in the MarketFeedPipelines readiness component, and the feed restarts at the next subscription emission, while every other feed keeps flowing through the shared stream. JVM Errors are never isolated — they fail the whole stream — and a CancellationException thrown while the collecting scope is alive (e.g. an escaping timeout) is treated as a terminal failure of that feed, not a silent end. A persistently failing hydration read stays inside the stage's retry loop — marking the feed DEGRADED on every retry — and keeps the first hydration pass from settling, so the startup timeout — not a shared-stream death — terminates the process; a terminal failure's mark survives a subscription emission until the replacement incarnation demonstrably re-attaches (a feed absent from an emission has its mark cleared deliberately), so a replacement whose hydration keeps failing cannot silently erase an established failure
  • the whole stream is self-healing: a failure of the subscription read itself, or a defect in the emission planning above the per-feed boundary, marks every market-data feed DEGRADED and restarts the entire pipeline after a backoff — the Ktor application scope supervises its children, so such a failure would not terminate the process, and a dead unmarked stream would leave every delivery path stopped while the pod appears healthy. Health concerns are tracked independently per feed (pipeline vs. subscription push vs. subscription relay), so one concern's recovery never clears another concern's still-live failure

The two main consumers are:

  • Processor, which turns emitted ticks into outputs, signals, and trades
  • DataRecorder.feedConsumer, which persists emitted and recorded-late ticks to QuestDB (the only consumer that opts into late ticks — raw coverage for dual-replica union)

DataSourceFactory.feedStream(...) resolves the concrete datasource, optionally narrows it with the requested instrument list, and then collects the datasource stream — NormalizedMarketFeed calls it after resolving instruments through FeedRepository.

The main datasource implementations are currently UpstoxDataSource and BitFlyerDataSource. SetSubscribedInstrumentsTask pushes the restored instrument set into each datasource once at startup via DataSourceFactory.setSubscribedInstruments(...); a rejected push fails the process (fail-fast). Afterwards RelaySubscriptionUpdatesTask — the runtime half of subscription application — collects feedRepository.subscribedInstrumentsFlow() and re-pushes the full set to every market-data feed on each mutation, parallel per feed: a push that keeps failing is retried with backoff and degrades the feed's MarketFeedPipelines health mark (never a process kill — the pod keeps serving every feed whose push succeeded), a newer emission cancels the previous push cycle mid-retry, and a failure of the subscription flow itself is retried the same way with every feed marked degraded (the application scope would not terminate the process on a dead relay). Upstox applies a pushed set as a diff on its active broker connection — SUB for added keys and UNSUB for removed keys, and a fresh connection replays the current set as a full SUB — so a removal actually releases broker subscription capacity (BitFlyer still re-subscribes the full set per push; its websocket protocol has no UNSUB request type, tracked by its existing TODO). On the server side, unsubscribing an instrument that currently has a live strategy assignment, an open position, or a live/draining route is rejected at the mutation surface — the server reads current assignments and open positions fresh from the shared store (the authoritative cross-replica floor), reads the route-state registry fresh from the shared volume (every replica's live routed instruments and draining routes, each published to runtime-records/<pod>/route-state.json and discounted once its freshness lease expires — the protection for a peer replica still holding a LIVE route whose assignment was just removed, and for a signal-level drain whose order was never placed or failed; both exist only in the hosting replica's process-local state, which no shared store holds and only that replica can publish; the registry leg is only as current as the writer's last successful publication — a contained write keeps the route running and the writer DEGRADED while the file still lacks the new route, briefly reopening this window until the next publication converges, and a sustained failure expires the writer's lease entirely), and queries the app tier's AppDataBridge.getRoutedInstruments() (the queried replica's assignment view ∪ routing index — a supplementary input that can only make the gate stricter), failing the removal with a typed input error — because the route would otherwise keep expecting ticks that never arrive (silent execution starvation); additions skip all four protection reads so the mutation keeps working while the app tier is unreachable, and the Processor's component health degrades on any routed instrument that nonetheless lacks a subscription (never-subscribed assignment or gate bypass), or when the subscription state itself cannot be read.

Operator-intent propagation. The subscription set lives in one place: the operator-intent/subscriptions.json file on the shared storage volume. The server tier is its single writer — every subscribedInstruments mutation rewrites the file wholesale with a temp-file-and-atomic-rename, and a failed or timed-out write fails the mutation (fail-loud; the file has no other source of truth); the single-writer invariant is upheld by the server deployment's Recreate strategy (one server pod at a time, closing the sustained two-writer window a RollingUpdate surge would open; the residual deploy-time windows — the terminating old pod's in-flight write — trade a transiently stale propagation file for the strong invariant, accepted because the durable stores stay authoritative), and within the process each repository serializes its mutations with a mutex so the file's publish order matches the durable store's persist order. The app replicas read it through a 5-second poll that reads and decodes the file fresh every tick (FileOperatorIntentSource, driven by TaskScheduler) — content-based change detection, so a replacement with equal length and a preserved timestamp still propagates — exposed as FeedRepository's subscription flows — cross-process propagation the previous KVStore-backed flows never provided (androidx DataStore is single-process by design; its JVM updateNotifications never emits, so the old flows were boot-time snapshots that never saw a mutation from another pod). The flows are conflated state: a slow collector — e.g. the relay's per-feed resolution read suspended on the data store — never holds back newer intent, and an unchanged re-read does not re-emit. The same channel carries strategy membership: strategy-instruments.json — written after every strategyInstruments mutation alongside the Gel persist, which remains the durable authority — is applied by SyncStrategyMembershipTask, which first refreshes the trade-setting view from the store and then the membership through StrategyRepository.refresh(...) (retried with backoff until it applies — a skipped removal would leave the old assignment executing after the store recovers; a newer emission cancels the in-flight application including its retry loop, and the replacement applies only after the previous attempt is cancelled and joined (it runs to completion, non-suspending tail included) — so the newest snapshot always wins the applied state), so a replica's process-local membership snapshot no longer goes stale when assignments change while it runs; the executor reads the trade-setting state per request, so a newly assigned pair's routes start trading with the settings the mutation persisted (a collection-start snapshot would drop every trade request with "No trade setting found"). Absent-file semantics differ by domain: subscriptions treat an absent file as a defined empty set (the file is the set's only store), while strategy membership keeps its Gel-seeded view and only surfaces the absence; a present-but-unreadable or undecodable file — including a non-regular path such as a directory left by a bad manual migration, or a read failure e.g. from a lost directory-traverse permission — keeps the last successfully-read value in both flows (never silently emptying the subscriptions) and degrades the OperatorIntent readiness component (never blocks readiness); the feeds readiness reporter consumes the last applied view rather than a fresh file read, so an unreadable file degrades health instead of throwing out of the readiness calculation.

DataSourceFactory receives BrokerApiResolver rather than token broker services directly — it has no auth knowledge. UpstoxApiDecorator (constructed by the resolver) creates three HttpClient.withConfig instances: one for live trading (positions, holdings, live HFT order placement) that suspends on OAuthTokenBrokerService.awaitValidToken(), one for the market data feed and historical candle data that suspends on ProvisionedTokenBrokerService.awaitValidToken(UPSTOX_ANALYTICS), and one for sandbox HFT order placement that suspends on ProvisionedTokenBrokerService.awaitValidToken(UPSTOX_SANDBOX). All three clients use the shared UpstoxErrorParser for 401 classification — it reads the raw response body from the SavedHttpResponse cache and classifies UDAPI100050 as a token error, triggering invalidation of the appropriate token type.

Datasource Health

DataSource exposes both isReady(): Boolean (binary readiness) and health(): DataSourceHealth (structured health with HealthStatus and optional message). DataSourceHealth carries a HealthStatus enum — HEALTHY, DEGRADED, BLOCKED — where DEGRADED means the datasource is partially functional (allows readiness) and BLOCKED means it is non-functional (fails readiness).

UpstoxDataSource overrides health() to report DEGRADED with a terminalFailureMessage when a permanent token failure occurs (TokenUnavailableException from OAuthTokenState.Failed). This prevents infinite re-auth loops for configuration errors while still allowing the pod to report readiness — the feed is degraded, not dead. The datasource does not observe token state itself; AuthStateCoordinator triggers recovery externally via ResumableDataSource.resumeIfStopped() when the token becomes Valid again. UpstoxDataSource clears the terminal failure as soon as the upstream reconnects (when isReadyState becomes true), not only on tick emission — so even a quiet market after auth recovery reports HEALTHY.

Tick Normalization (RawTick → MarketData)

Datasources emit raw broker facts only — UpstoxDataSource and BitFlyerDataSource derive nothing (no deltas, no durations, no day-boundary heuristics). Feed frames are mapped to RawTick through a shared broker-frame mapper and normalized centrally by the pure TickNormalizer policy from backend-normalizer — one policy shared by the live path, the backfill folds, and reconciliation.

Upstox frame mapping. UpstoxFrameMapper (in backend-broker/upstox/websocket, beside the frame vocabulary) maps MarketFullFeed frames directly to RawTick for the already-resolved instrument: ltt → time, ltp → ltp, and vtt → seq (the cumulative volume-traded-today counter doubles as the ordering discriminator and the volume fact). Envelope validation is reject-and-count — missing ltpc, non-finite ltp, or ltt <= 0 map to Invalid, are logged and counted, and never kill the feed. A frame with vtt = null (LTPC-mode/index variants) maps with vtt = 0, preserving index-recording behavior.

BitFlyer frame mapping. BitFlyerDataSource emits RawTick with seq = the per-product tick_id and volumeFact = the rolling 24h cumulative. Accepted tradeoff: normalized BitFlyer volume is a tick-count delta (monotonic, never negative), not trade volume — true volume bars for generic sources are deferred per the design. Because of that, volume-frame strategies (volumeFrameStrategyNames, v1: volume_breakout) only accept instruments on feeds whose seq is the broker's per-session cumulative volume (Feed.carriesSessionVolumeFact, v1: Upstox). The scope is enforced twice: StrategyRepository.strategyInstruments rejects the combination at configuration time (surfaced on the Strategies screen's save-error banner), and the strategy initializer fails the instrument at runtime through the per-instrument teardown path (no route, DEGRADED component health, re-initialization only after the instrument is removed and re-added) — so pre-existing configuration cannot silently produce tick-count bars either. Recording, charting, and EMA strategies are unaffected.

Normalizer policy (rule v1). TickNormalizer is a pure per-instrument state machine:

  • Session identity comes from the versioned SessionCalendar (sessionContaining), never from gap heuristics. The first accepted tick of each session seeds: volume = 0, durationSinceLast = 0, baseline = seq (always-seed — an unobserved session open can never attribute the morning's unseen volume to the reconnection price).
  • Ordering gate: a tick is accepted only when its (ts, seq) is strictly greater than the preceding accepted tuple; sub-millisecond coalescing (same ts, higher seq) is accepted.
  • Effective volume = seq − heldBaseline on advance; a vtt regression with timestamp advance yields 0 and keeps the baseline (100 → 90 → 105 ⇒ deltas 0, then 5).
  • Duplicates: an exact duplicate of the immediately preceding accepted tuple (same key, same ltp) is dropped and counted; a same-key/different-ltp duplicate is dropped and counted as a conflict.
  • Hydration: on restart or mid-run enrollment the stage reads getHydrationMark once per instrument (the combined high-water tuple plus session-max baseline read) and constructs TickNormalizer.hydrated(...), so the first live tick after a restart carries a real delta and a real duration instead of re-seeding. Membership is reconciled on every subscription change: a departed instrument's normalizer state is evicted, so a re-added instrument hydrates fresh from the recorded mark rather than resuming a baseline the peer replica has since advanced; instruments that stay subscribed keep their live state, which can be ahead of the recorded mark (ingest flush lag). Backstop-created state (a straggler frame for a departed instrument, or an unsolicited frame) is marked provisional and discarded by the next enrollment, which re-reads fresh — the peer may have recorded past the straggler-time mark.

Record/emission split. The stage produces two streams per batch:

  • emitted — strictly-ordered accepted ticks with real derived fields; strategy consumers and the recorder both see these
  • recorded-late — distinct-key late tuples and out-of-session ticks (pre-open/post-close prints), recorded with synthetic derived fields (volume = 0, durationSinceLast = 0) for dual-replica union coverage, never emitted to strategies. In/out-of-session classification stays derivable at read time from ts plus the versioned calendar; the recording policy itself is tracked for reexamination.

Drops (EXACT_DUPLICATE, DUPLICATE_LTP_CONFLICT) are counted and logged — feed health never reacts to them.

QuestDB dedup key. A broker-provided monotonic discriminator is stored in the seq column alongside ts (ltt) and included in the dedup key: DEDUP UPSERT KEYS(ts, instrument_id, seq). For Upstox this is the cumulative vtt (volume traded today); for BitFlyer it is tick_id (a per-product sequence). The ordering gate guarantees the pair (ltt, seq) is unique per emitted tick, keeping same-millisecond trades distinct in storage rather than collapsing them via last-write-wins, and keeping re-ingest from the raw store idempotent (the same raw frame always produces the same (ltt, seq) key). The derived columns (volume, duration_since_last) are last-write-wins and non-authoritative — canonical consumers (backfill folds, frame totals via max(seq), the reconciliation replay) always recompute from ts/seq.

Raw-vs-QuestDB Reconciliation

backend-reconciliation is an after-the-fact cross-check, not part of the live path. It replays raw frames captured in the NATS raw store over a time window, decodes each to UpstoxMarketDataFeed, maps frames through the shared UpstoxFrameMapper, and folds each instrument's deduplicated union through the same TickNormalizer policy the live path runs (epoch-reset state, no hydration) — so ticks the live path dropped never appear as false "missing in QuestDB". The raw read is session-anchored: storage-time from the window's session open through window end plus a publish-lag margin, with the exchange-time predicate doing the filtering. It then diffs the passthrough fields ltt (QuestDB ts) and ltp against the market_data table. The join key is instrumentId + ltt + seq — matching the QuestDB dedup key — so same-millisecond trades are matched individually. The QuestDB read-back window is padded ±5s to tolerate storage-time-vs-exchange-time skew at the edges.

Union dedupe and conflicts. Both replicas capture the same trade: identical copies on (instrumentId, ts, seq) collapse to one expected tick. Same-key copies with different ltp make the price indeterminate — reported as INDETERMINATE_CONFLICT (counted and sampled, never resolved to an arbitrary expected price); a stored row carrying a conflicted key counts as conflicted storage.

Completeness. A window whose tail is more recent than a provisional 10-minute capture-lag bound is INDETERMINATE (raw frames may still be in flight); older windows are LIKELY_COMPLETE. COMPLETE is reserved for a durable per-source publish watermark (deferred).

Raw frame provenance. Each stored raw frame carries the publishing source's identity in a NATS source message header (RawMarketData.source). backend-app runs 2 replicas that both receive the same Upstox frame; content-hash message-ID dedup collapses the two publishes to one stored message, so the stored source is the winning source's id (≈random), not a complete list of sources that saw the frame. The header is additive — messages published before this change read back as source = "unknown" — so it is a lightweight audit tag, not a guarantee of provenance completeness. The publisher resolves the source id from the HOSTNAME environment variable, which k8s sets to the pod name.

The derived fields volume and duration_since_last are intentionally not compared — they are non-authoritative by contract and recomputed by the shared policy replay. Reconciliation remains Upstox-only (BitFlyer streams JSON and is not captured as raw bytes).

The logic is wired in backend-app (both RawDataRetriever and HistoricalDataProvider are already in scope there) and exposed on-demand through the AppDataBridge kRPC — folded as reconcileReport (summary counts + ≤50 sample diffs) and reconcileDiffs (full diff stream). backend-server proxies these as POST /reconcile and GET /reconcile/diffs; see operations.md for request shapes and the SSE retry caveat (QAPP-109).

Processor Contract

Processor.consume() is the orchestration boundary between live feed input and strategy execution.

At a high level it:

  • runs under a coroutineScope with a CompletableDeferred<Nothing> fatal failure signal
  • processes feed batches sequentially through routeFeedData()
  • watches strategyRepository.strategyInstrumentIds and enqueues payload-free RouteCommand.TopologyDirty wakes
  • runs a single-owner RouteCommandLoop that handles all route mutations (reconciliation and degradation)
  • launches route-side jobs in a supervised child scope (routeSideScope)
  • starts one route-local StrategyEventCoordinator collector for each strategy output and signal stream
flowchart TD
    FD["FeedData&lt;MarketData&gt;"]

    subgraph processor ["Processor (backend-app)"]
        FR["FeedRouter — routeFeedData()\nreads routesByInstrument snapshot\ntrySend to strategy mailboxes\noverflow → requestDegrade()"]
        RCL["RouteCommandLoop\nsole writer of routingSnapshot\nTopologyDirty · TargetsDirty · Degrade · TeardownStrategy"]
        RL["ReconcileLoop\nstrategyInstrumentIds → TopologyDirty"]
        FW["FatalWatcher\nfatalFailure → coroutineScope fails → pod restart"]
        RS["RoutingSnapshot (@Volatile)\nroutesByInstrument · routesByStrategy\nStrategyRoute: routeInstanceId · strategy · mailbox · executor · routeJobs · state"]
    end

    SM["Strategy Mailbox\nChannel&lt;MarketData&gt; · cap 1024\nProcessor-owned"]

    subgraph executor ["StrategyExecutor (backend-processor) — one per strategy route"]
        ER["Router coroutine\nmailbox.receiveAsFlow() → deliver() trySend"]
        subgraph routes ["Instrument Routes"]
            IR["InstrumentRoute\nProcessorMailbox (Channel, cap 1024)\nStrategyExecution (runningFold)\nBridge jobs: output → _strategyOutput · signals → _signals"]
            IR2["InstrumentRoute … (one per instrument)"]
        end
    end

    subgraph execution ["StrategyExecution (backend-strategy)"]
        MDF["marketDataFlow (ProcessorMailbox.flow)"] --> RF["runningFold (accumulate state)"] --> SO["strategyOutput (MutableSharedFlow)"]
        RF --> SS["strategySignals"]
    end

    FD --> FR
    RL -->|"TopologyDirty"| RCL
    FR -->|"Degrade"| RCL
    FR -.->|"reads"| RS
    RCL -.->|"writes"| RS
    RS --> SM
    SM -->|"receiveAsFlow"| ER
    ER --> IR
    ER --> IR2
    IR --> MDF
    IR -.->|"onRouteOverflow\nexecutionFailed"| FR

    classDef feed fill:#e3f2fd,stroke:#1565c0,color:#0d47a1
    classDef processorNode fill:#e8eaf6,stroke:#283593,color:#1a237e
    classDef fatal fill:#ffebee,stroke:#c62828,color:#b71c1c
    classDef mailbox fill:#fff3e0,stroke:#e65100,color:#bf360c
    classDef executorNode fill:#f3e5f5,stroke:#6a1b9a,color:#4a148c
    classDef executionNode fill:#e8f5e9,stroke:#2e7d32,color:#1b5e20

    class FD feed
    class FR,RCL,RL,RS processorNode
    class FW fatal
    class SM mailbox
    class ER,IR,IR2 executorNode
    class MDF,RF,SO,SS executionNode

    style processor fill:#e8eaf6,stroke:#283593,color:#1a237e
    style executor fill:#f3e5f5,stroke:#6a1b9a,color:#4a148c
    style execution fill:#e8f5e9,stroke:#2e7d32,color:#1b5e20
    style routes fill:#ede7f6,stroke:#7b1fa2,color:#4a148c

Processor-Owned Routing

The processor maintains a single @Volatile routing snapshot that is the source of truth for all routing decisions:

RoutingSnapshot
  ├── routesByInstrument: ImmutableMap<InstrumentId, ImmutableSet<StrategyId>>  (Eclipse Collections)
  └── routesByStrategy: ImmutableMap<StrategyId, StrategyRoute>  (Eclipse Collections)

StrategyRoute
  ├── strategy: Strategy
  ├── mailbox: Channel<MarketData>       (strategy-level mailbox)
  ├── executor: StrategyExecutor
  ├── routeJobs: RouteJobs              (signalsJob, outputJob, tradeJob, stoppingWatcherJob?)
  └── state: RouteState (Active / Degraded / Stopping)

RouteState
  ├── Active
  ├── Degraded(cause, degradedAt, hasOpenPositions)
  └── Stopping(retainedInstrumentIds)   (strategy removed but positions still open)

Single-owner mutation: Only the RouteCommandLoop coroutine mutates routingSnapshot. The FeedRouter and ReconcileLoop coroutines only read it or enqueue commands. The @Volatile annotation ensures visibility across the writer/reader boundary. The routesByInstrument and routesByStrategy maps use Eclipse Collections' ImmutableMap, which provides structural immutability — the maps cannot be modified after construction, and every mutation produces a new ImmutableMap instance via newWithKeyValue() / newWithoutKey().

Route commands are sent through an unlimited channel:

private val routeCommands = Channel<RouteCommand>(Channel.UNLIMITED)

This is a control queue, not a back-pressure boundary — it must never overflow.

Feed Routing

The FeedRouter coroutine processes feed data sequentially:

  1. For each MarketData in a FeedData batch, look up the instrument in routesByInstrument
  2. For each strategy mapped to that instrument, trySend into the strategy's mailbox
  3. If trySend fails (mailbox full), call requestDegrade(strategyId, routeInstanceId, cause) — this deduplicates against pendingDegradation by route incarnation before enqueuing a RouteCommand.Degrade command
  4. Skip routes that are in Degraded state (both Active and Stopping routes receive ticks)

This replaces the previous flatMapMerge { it.data.asFlow() } approach. The key property is that trySend is non-suspending: a slow consumer never blocks the router — it degrades the affected route instead.

Pending degradation deduplication: Because the route is not marked Degraded until the command loop processes the Degrade command, a naive implementation would enqueue a new Degrade command on every failed trySend, building an unbounded control-plane backlog. This applies to both strategy-mailbox overflow (detected in routeFeedData()) and instrument-mailbox overflow (detected in the executor's router and signaled via onRouteOverflow).

To prevent this, the processor centralizes all degrade requests through a requestDegrade(strategyId, routeInstanceId, cause) helper. This helper uses a ConcurrentHashMap.newKeySet<Long>()-backed pendingDegradation set keyed by route instance ID (not strategy ID) for atomic check-and-add: add() returns false if the route incarnation is already pending, so only the first overflow for a given route incarnation enqueues a Degrade command. Keying by route instance ID prevents a stale degrade from an old route incarnation from degrading a newly recreated route for the same strategy. The command loop removes the route instance from pendingDegradation when processing Degrade, TeardownStrategy, or immediate teardown in TopologyDirty. If a Degrade command arrives for a route whose routeInstanceId no longer matches (the route was recreated), the command is silently discarded.

flowchart LR
    subgraph sources ["Command Sources"]
        RL["ReconcileLoop\nstrategyInstrumentIds watcher\n→ TopologyDirty (payload-free)"]
        FR["FeedRouter\nstrategy mailbox overflow\n→ Degrade(strategyId, routeInstanceId)"]
        SE["StrategyExecutor\nonRouteOverflow → Degrade\nonTargetsDirty → TargetsDirty\nexecutionFailed → InstrumentExecutionFailed"]
        SW["StoppingWatcher\nhasOpenPositions filter\n→ TeardownStrategy(strategyId)"]
        RJ["Route-Side Jobs\nsignals/output/trade failure\n→ Degrade(strategyId, routeInstanceId)"]
    end

    CH["routeCommands\nChannel.UNLIMITED\nTopologyDirty · TargetsDirty · Degrade · TeardownStrategy"]

    subgraph loop ["RouteCommandLoop (sole writer)"]
        RC["TopologyDirty\nRead `.value` at consumption\nRemove / Reactivate / Add routes\nRebuild instrument index from views"]
        DG["Degrade\nMark Degraded · Close mailbox\nCancel jobs + executor\nRebuild index · Check open positions"]
        TD["TeardownStrategy\nClose mailbox · Cancel jobs\nCancel executor · Remove route"]
        IR["TargetsDirty\nRebuild instrument index\nfrom the routes' delivery-target views"]
    end

    RS["RoutingSnapshot\n(@Volatile)"]

    RL --> CH
    FR --> CH
    SE --> CH
    SW --> CH
    RJ --> CH
    CH --> RC
    CH --> DG
    CH --> TD
    CH --> IR
    loop --> RS

    classDef source fill:#e3f2fd,stroke:#1565c0,color:#0d47a1
    classDef channel fill:#fff3e0,stroke:#e65100,color:#bf360c
    classDef handler fill:#f3e5f5,stroke:#6a1b9a,color:#4a148c
    classDef snapshot fill:#e8f5e9,stroke:#2e7d32,color:#1b5e20

    class RL,FR,SE,SW,RJ source
    class CH channel
    class RC,DG,TD,IR handler
    class RS snapshot

    style sources fill:#e3f2fd,stroke:#1565c0,color:#0d47a1
    style loop fill:#f3e5f5,stroke:#6a1b9a,color:#4a148c

Route Command Loop

The RouteCommandLoop coroutine is the single owner of route state mutations:

for (command in routeCommands) {
    routingSnapshot = when (command) {
        RouteCommand.TopologyDirty       -> reconcileRoutes(current, strategyInstrumentIds.value (read once at consumption), routeSideScope)
        RouteCommand.Degrade            -> degradeRoute(current, strategyId, routeInstanceId, cause, fatalFailure)
        RouteCommand.TeardownStrategy   -> teardownStrategy(current, strategyId)
    }
}

On RouteCommand.TopologyDirty:

The wake is payload-free: the handler reads the repository's current strategyInstrumentIds.value exactly once at consumption and wires that. The former payload-carrying Reconcile captured its snapshot at emission receipt, so a delayed command could wire a superseded view; with no payload there is nothing to be stale — a delayed wake and a fresh one are indistinguishable. Boundary (stated non-goal): consumption-time reads remove the delayed-payload race; they do NOT make a suspending topology reconciliation transactional — membership can advance during a suspending executor build, and a later wake converges.

  1. Removal: strategies in the current snapshot but absent from the desired repository state:
  2. If the executor has open positions → transition to Stopping(retainedInstrumentIds), launch a StoppingWatcher that posts TeardownStrategy when positions flatten
  3. If no open positions → immediate teardown: close mailbox, cancel jobs, cancel executor, remove from snapshot
  4. Reactivation: Stopping routes whose strategies reappear in desired are transitioned back to Active (the stopping watcher is cancelled)
  5. Addition: strategies in the desired state but absent from the current snapshot get a new strategy mailbox, executor, and route-side jobs
  6. Instrument index rebuild: routesByInstrument is rebuilt solely from the routes' pulled delivery-target views (executor.deliveryTargetIds — the coordinator's current authoritative desired set ∪ its routable records). The coordinator fires the payload-free onTargetsDirty wake after every inbox event, so the index is always rebuilt after the executor's state has changed. Degraded routes are excluded; a Stopping route's authoritative desired is empty, so its view reduces to its retained records without a special case.

On RouteCommand.Degrade:

  1. Mark the route as Degraded(cause, hasOpenPositions) (works for both Active and Stopping routes — a Stopping route that overflows with open positions must still reach the fail-fast path)
  2. Close the strategy mailbox (terminates the executor's input)
  3. Cancel the route-side jobs (signals, output, trade, stoppingWatcher)
  4. Cancel the executor (executor.cancel())
  5. Rebuild the instrument index using rebuildRoutesByInstrumentFromViews(routesByStrategy)
  6. If the degraded route has open positions → failProcessor() completes the fatal failure signal

On RouteCommand.TeardownStrategy:

  1. Close the strategy mailbox
  2. Cancel the route-side jobs (signals, output, trade, stoppingWatcher)
  3. Cancel the executor
  4. Remove the route from the snapshot
  5. Rebuild the instrument index

On RouteCommand.TargetsDirty:

  1. Rebuild the instrument routing index from the routes' delivery-target views — the executor's state has changed (a route published, purged or failed), so its view is now up-to-date
  2. No changes to routesByStrategy — this is a pure index refresh

The wake is payload-free and fired by the coordinator after EVERY inbox event: route addition/removal does not cross the executor boundary as commands; the processor's routing index is a pull-derivation of the coordinators' delivery-target views, so a record's appearance in or disappearance from the views is the entire routing consequence. What still crosses the boundary is exactly the failure fact, enqueued through the executor's RouteLifecycleEventSink : the sink is invoked by the route coordinator's serialized event loop while holding its routingLock, in the same critical section that performed the teardown mutation, so the failure is always observed after the state change it describes. A fatal teardown removes the route as coordinator-internal record mutation in the same critical section that enqueues the failure — the former removal-before-failure event ordering is now mutation order inside the coordinator. Without the wake, the processor's routing index would be derived from a stale snapshot of executor state, causing ticks to be dropped (for additions) or routed to purged routes (for removals); a causal convergence regression and the processor-level model suite pin the wake as the sole load-bearing rebuild path (a mutation proof disabled it and failed exactly the convergence assertions).

Batch Ordering

There are two separate ordering rules:

  • Batches are processed sequentially because the FeedRouter coroutine collects the feed flow in order.
  • Ordering across instruments inside the same batch is not guaranteed, because routeFeedData() iterates over feedData.data and routes each instrument independently.

That means:

  • a later batch cannot overtake an earlier batch
  • two instruments inside one batch may be observed in either order

The event-contract tests in backend-app deliberately assert the parts of ordering that are guaranteed and avoid asserting the parts that are not.

StrategyExecutor

Each StrategyExecutor owns one strategy and manages one StrategyExecution<T> per active instrument.

Its responsibilities are:

  • receive all market data for its strategy through a single Channel<MarketData> (the strategy mailbox)
  • fan out internally to per-instrument ProcessorMailbox instances (bounded Channel, capacity 1024)
  • signal overflow via onRouteOverflow — the processor decides what to do (degrade the route)
  • expose hasOpenPositions(): Boolean so the processor can decide escalation severity
  • expose hasOpenPositionsFlow: Flow<Boolean> for reactive position monitoring (used by StoppingWatcher)
  • expose cancel() to stop all executor-owned coroutines when the route is torn down
  • expose start() to begin the router and instrument reconciliation loop after the processor has subscribed to outputs/signals

Strategy Mailbox Contract

The executor's public contract is:

StrategyExecutor(
    strategyMailbox: Channel<MarketData>,
    membership: StateFlow<List<StrategyInstrumentIds>>,
    resolveFeedContext: suspend (InstrumentId) -> InstrumentFeedContext?,
    historicalDataProvider: HistoricalDataProvider,
    onRouteOverflow: (InstrumentId, Throwable) -> Unit,
    lifecycleSink: RouteLifecycleEventSink,
)

: the coordinator — not the worker — is the sole per-strategy consumer of the raw repository StateFlow; membership crosses as a payload-free MembershipDirty wake and the authoritative .value (filtered to this strategy) is read at wake consumption, so a wake enqueued before the repository advanced can never reconcile a stale view (the hazard is structurally absent). "Consumed once" is scoped to per-strategy membership — the processor keeps its own separate repository collector for strategy-topology reconciliation — a payload-free TopologyDirty wake on the same command channel, reading .value at consumption. Feed-context resolution (the suspending feed-store lookup) runs on the worker's slow initialization lane, after the InitStarted handshake; a null result (instrument desired but no feed row) takes the benign terminal FeedUnresolvable phase — no route, no health degradation — and a fresh resolution happens only after a leave-and-re-add.

The exception is deliberate and fully terminal: the instrument is unroutable, emits an actionable warn (naming the instrument, its strategy, and the removal/re-add remedy), and is retried only after it leaves the desired set and is re-added — no automatic retry, since a missing feed row is a configuration gap, not a transient pipeline failure. It informs through logs only today: derived health stays HEALTHY and readiness stays true (the Failed ∧ desired ⇒ DEGRADED contract deliberately excludes FeedUnresolvable); surfacing it through an explicit UI/health projection is a recorded follow-up. Resolution itself is per-instrument — one feed-store lookup per instrument on the sequential slow lane, in exchange for loud per-instrument failure isolation (a resolver exception fails exactly one instrument) — a recorded trade-off to measure or batch if startup-scale tests expose it.

The processor's routing index is a pull-derivation of the coordinator's delivery-target view, and initialization is gated on publication. Three primitives: (1) deliveryTargetIds() — the coordinator's lock-protected pull view (the authoritative desired set ∪ the routable records, composed wholesale), read at pull so it can never serve a stale snapshot; the TargetsDirty command's handler rebuilds the instrument index from the views alone (payload-free — the class is structurally absent, the command carries no route reference). (2) onTargetsDirty — a separate callback, deliberately NOT a method on the sink (whose calls are lock-linearized): the wake is payload-free and coalescible and fires strictly after routingLock is released, after every inbox event. (3) The per-init acknowledgement: InitStarted's proceedAck is PARKED — the coordinator registers it under the lock (validating the token, installing the ingress, marking the record Initializing), releases the lock, fires the dirty wake, and the worker waits; only the processor's centralized publishRoutingSnapshot (assign → retain the exact per-route target sets → onTargetsPublished callback even on a no-op publication) completes the ack — route-qualified against the retained target sets, never a re-pull — strictly outside the lock. Backfill, and its high-water read, begin only after publication, so a tick dropped in the publication-lag window predates the read and is backfill-recoverable. The callback is completion-only: absence from a publication is not supersession — a publication's content can predate the park by construction (a topology reconciliation reads membership, then suspends on an executor build while membership advances, and its snapshot-exact targets are deliberately not re-pulled), and a no-op publication changes nothing — so a parked ack stays parked until a containing publication arrives. A parked ack is cancelled only by an explicit membership-invalid transition: the park-time validation (the repository advanced past the sweep's dispatch) or the membership-wake re-check. Both normalize the record under the routing lock first — reverted to Buffering, the registration's drain returned FIFO, the ingress reference cleared — and cancel the deferred strictly after it, so the worker's closed ingress can never receive a live tick; the sweep continues inline and the next membership pass purges or re-dispatches the reverted record.

The processor provides the strategy mailbox, the overflow callback, and the route-lifecycle event sink. The sink's contract: invoked by the route coordinator's serialized event loop under its routingLock in the same critical section as the routing mutation the fact describes — the boundary is the per-instrument failure fact and the drain lifecycle facts (drainRetained on entering DrainingForOpenPosition, drainCleared on every exit: position flat, re-add heal, or the draining record failing) — and contractually non-suspending, non-blocking, non-throwing and non-re-entering — the processor's implementation only enqueues into its unlimited command channel. A drain fact carries a publication ack: the coordinator awaits it AFTER releasing routingLock, and the processor's command loop completes it once the fact is processed — applied to the registry and either durably written or contained (a failed or timed-out write keeps the applied in-memory set, degrades the Processor's health, and retries on the next transition or heartbeat, converging after a successful rewrite) — or skipped for a stale route, in which case nothing is applied. Ack completion is completion of processing, not proof of durable peer visibility: under a contained write the registry file still predates the transition until a later successful rewrite, so the server's unsubscribe gate can read a pre-transition file in that window — the accepted visibility gap, carried by the degraded health signal. The ownership is split: the per-strategy RouteCoordinator is the sole owner of the logical route records — each (strategy, instrument) is one typed record carrying a RouteToken (the processor's route-instance id plus a coordinator-local record generation allocated under routingLock), membership intent, a sealed RoutePhase (Buffering, Initializing, Active, AwaitingHistory, FeedUnresolvable, Failed, DrainingForOpenPosition) and retry intent — while the executor is reduced to the worker: it owns each ingress's lifecycle (creation, buffering, handoff, closing) and reports results back with the token through the acknowledged handshake protocol (InitStarted/ InitReady/ ExecutionReady/ AwaitingHistory/ Failed/ PositionFlat events on a non-dropping, unbounded inbox consumed by the coordinator's single loop, which serializes only state transitions — backfill and initialization run outside it). The worker must not begin slow work until its InitStarted acknowledgement completes (the coordinator publishes the buffering ingress, drains the record's buffered ticks into it and flips the record to Initializing first — making the publish-before-start ordering structural), and supersession or loop shutdown cancels the acknowledgement so the worker fails fast. Every asynchronous completion validates its token against the record before acting — a completion whose token is stale (its record was replaced by a removal/re-enrollment) is ignored. Pre-publication ticks are buffered in the record's own storage under the coordinator's aggregate bound (262,144 across all buffering records, loud overflow), and the record's pending ingress (65,536) stays per-instrument; the buffering→active ingress handoff is the acknowledged InitReady flip barrier (the router prefers the buffering ingress while registered, and the worker closes it only after the flip — FIFO holds across the switch). The execution view is single-writer: only the coordinator's loop publishes it (a plain immutable value per transition). Membership arrives as a payload-free wake — the coordinator itself is the sole collector of the raw repository StateFlow and reads the authoritative .value (filtered to its strategy) at wake consumption; the worker holds no membership mirror at all, so a stale payload can never be processed (the hazard is structurally absent). The processor does not own route state — it signals overflow conditions, receives the per-instrument failure fact through the sink, and rebuilds its routing index from the executors' delivery-target views on the post-event targets-dirty wake (route addition/removal do not cross the boundary), ensuring ticks are only routed to instruments the executor is ready to receive.

The coordinator's stable record stores a single sealed phase/resource state, not independent nullable ingress and execution fields. Initializing requires buffering ingress. Both Active and DrainingForOpenPosition require an execution plus either buffering or steady delivery: execution publication precedes the fold's InitReady handoff, so an Active route does not imply steady delivery. Drain, heal, and handoff replace the whole state under routingLock, preserving the record identity and token; lock-free readers capture the volatile state once. Each drain transition publishes its fact through the lifecycle sink into the shared-volume route-state registry (see the feeds section): entry retains the (strategy, instrument) pair, every exit clears it, and a full route teardown clears that strategy's drains — a cancelled executor's drains can never converge by ticks, and in the starved case (no persisted position) nothing else would ever clear them. The public RoutePhase remains a resource-free projection for health/readiness. Local ticks belong only to LocalBuffer; the exceptional registration-overflow state retains its ingress and still projects to Buffering until executor degradation. deliverTick exhaustively dispatches these states, with explicit drops only for awaiting-history, feed-unresolvable, and failed records; it needs neither nullable delivery fallbacks nor non-null assertions. Token/source-state checks, mailbox liveness, locking, and acknowledgment ordering remain runtime protocol responsibilities.

The executor is constructed in a stopped state — no coroutines are launched until start() is called. The processor subscribes to executor.strategyOutput and executor.signals before calling executor.start(), ensuring no emissions are lost when instrument routes are added and their executions start producing output.

Internal Routing

The executor runs a single router coroutine (launched by start()) that:

  1. Collects from strategyMailbox.receiveAsFlow()
  2. Delegates the per-tick decision to the coordinator's deliverTick(marketData) — the entire routing decision (which ingress, buffering storage, or drop) runs inside the coordinator under its routingLock, together with all routing mutations
  3. If deliverTick reports overflow, calls onRouteOverflow(instrumentId, cause)

Per-Route Bridge Jobs

Each instrument route has two bridge jobs that forward the execution's output and signals to the executor's shared flows:

  1. outputBridgeJob: collects from execution.strategyOutput and emits to _strategyOutput
  2. signalBridgeJob: collects from execution.strategySignals and emits to _signals

These bridge jobs replace the previous executor-wide flatMapLatest { merge() } fan-in. The per-route approach eliminates the re-subscription gap: when a new instrument route is added, the bridge jobs are launched with CoroutineStart.UNDISPATCHED before execution.start() is called. UNDISPATCHED ensures the coroutine executes synchronously up to its first suspension point (the collect(...) call), so the subscription is established before start() returns. When the first tick arrives and the execution produces its first output/signal, the bridge jobs are already subscribed. No emissions are lost.

When an instrument route is removed, the bridge jobs are cancelled before the execution is cancelled, ensuring clean teardown.

Runtime Lifecycle Behavior

The runtime-lifecycle tests currently document an important behavior contract:

  • removing an instrument or strategy does not immediately kill an execution if that execution still owns an open position
  • the retained execution stays alive long enough to emit the managed exit or reversal
  • re-adding before cleanup preserves the existing retained execution
  • re-adding after cleanup creates a fresh execution and re-runs initialization/backfill
  • these drain/removal transitions apply immediately on every membership change, even while another instrument's initialization (feed-context resolution or backfill) is still in flight: the coordinator's serialized sweeps wait for the in-flight initialization's terminal event, but every membership wake runs the retained-routes pass against the same authoritative snapshot the wake captured right away — so a draining route's watcher is armed and a flat removed route is torn down without waiting behind an unrelated, possibly hung, initialization
  • every drain transition also publishes its fact cross-replica through the lifecycle sink (drainRetained on entry, drainCleared on every exit — position flat, re-add heal, or the draining record failing; the route-command loop applies it to the per-writer registry file with the publication ack awaited after the routing lock), and a full route teardown clears that strategy's drains — a cancelled executor's drains can never converge by ticks, and in the starved case (no persisted position) nothing else would ever clear them. The registry file — the replica's routing-index keys (converged after every snapshot publication: written only when the set changed or a pending write needs converging, so a peer replica still holding a LIVE route is protected too) plus its drain set — is rewritten atomically on every routed-set or drain transition and on a 60-second heartbeat (the server-side lease window is 3× that, so a crashed pod's routes and drains stop blocking unsubscription within the window; a replaced or scaled-down pod's old file likewise stops blocking within the window — fail-safe over-blocking after every rollout, accepted); a contained write keeps the applied in-memory sets, degrades the Processor's component health until the next rewrite converges, and never fails the process

This behavior is intentional and is one of the main reasons the processor runtime-lifecycle suite exists.

Executor Lifecycle

The executor owns its CoroutineScope (created with SupervisorJob as a child of the calling coroutine's job). All internal coroutines (router, the worker's two command runners — the sequential initialization lane and the immediate fast lane, the coordinator's event loop and its sole membership wake collector, per-route bridge jobs) are launched in this scope.

The executor is constructed in a stopped state. The processor must call start after subscribing to executor.strategyOutput and executor.signals (via launchRouteJobs). The processor's route-side collector jobs (signals, output, trade) are launched with CoroutineStart.UNDISPATCHED, ensuring their subscriptions are established synchronously before start is called. start launches the coordinator's serialized event loop and its sole membership wake collector, the router coroutine, and the worker's two command runners — the sequential initialization lane and the immediate fast lane (watcher arming and transport cleanup never queue behind a slow initialization) — which together trigger the first sweep and begin routing ticks.

Calling cancel() on the executor cancels its scope, which stops all internal coroutines. The processor calls executor.cancel() in the immediate teardown path, the deferred teardown path (TeardownStrategy), and the degradation path.

StrategyExecution

StrategyExecution<T> is the concrete state machine for one strategy on one instrument.

It:

  • pre-fills historical state through HistoricalDataProvider
  • runs a runningFold over live MarketData
  • emits one StrategyOutput for each accepted live tick
  • derives long and short signal streams from state transitions

In production, HistoricalDataProvider is backed by Recorder.historicalDataProvider, so strategy backfill reads from QuestDB.

StrategyExecution is constructed in a stopped state — the runningFold and signal collection coroutines are not launched until start() is called. The per-route bridge jobs in StrategyExecutor are launched with CoroutineStart.UNDISPATCHED before start() is called, ensuring their subscriptions are established synchronously. This makes the start() barrier deterministic: no early emissions can be lost because the subscribers are guaranteed to be attached before any data flows.

flowchart LR
    subgraph overflow ["Overflow Sources"]
        A["Strategy Mailbox Overflow\nrouteFeedData() → trySend fails\nChannel&lt;MarketData&gt; · cap 1024"]
        B["Instrument Mailbox Overflow\nexecutor router → deliver() fails\nProcessorMailbox · cap 1024"]
        C["Route-Side Job Failure\nsignals/output/trade collector\nthrows non-CancellationException"]
    end

    subgraph dedup ["Dedup Gate"]
        RD["requestDegrade()\npendingDegradation: ConcurrentHashMap.newKeySet()\nkeyed by routeInstanceId\nAt most 1 pending Degrade per route incarnation\nStale incarnations automatically rejected"]
    end

    subgraph cmdloop ["Command Loop"]
        DR["degradeRoute()\n① Verify routeInstanceId matches\n② Mark route state = Degraded\n③ Close strategy mailbox\n④ Cancel route-side jobs\n⑤ Cancel executor\n⑥ Rebuild routesByInstrument\n⑦ Remove from pendingDegradation\n⑧ Check hasOpenPositions()"]
    end

    A --> RD
    B --> RD
    C --> RD
    RD -->|"RouteCommand.Degrade"| DR
    DR -->|"!hasOpenPositions"| OK["No Open Positions\nRoute removed from routingSnapshot\nSibling strategies continue\nProcessor remains healthy"]
    DR -->|"hasOpenPositions"| FF["Open Positions Present\nfailProcessor() → fatalFailure\nProcess terminates\nKubernetes restarts pod"]

    classDef overflow fill:#ffebee,stroke:#c62828,color:#b71c1c
    classDef dedup fill:#fff3e0,stroke:#e65100,color:#bf360c
    classDef cmd fill:#f3e5f5,stroke:#6a1b9a,color:#4a148c
    classDef ok fill:#e8f5e9,stroke:#2e7d32,color:#1b5e20
    classDef fail fill:#ffcdd2,stroke:#c62828,color:#b71c1c

    class A,B,C overflow
    class RD dedup
    class DR cmd
    class OK ok
    class FF fail

    style overflow fill:#ffebee,stroke:#c62828,color:#b71c1c
    style dedup fill:#fff3e0,stroke:#e65100,color:#bf360c
    style cmdloop fill:#f3e5f5,stroke:#6a1b9a,color:#4a148c

Degradation And Fail-Fast

The degradation path is:

  1. Overflow detected → requestDegrade(strategyId, routeInstanceId, cause) — atomically checks pendingDegradation (keyed by route instance ID) and enqueues RouteCommand.Degrade only if not already pending for that route incarnation. This covers both strategy-mailbox overflow (in routeFeedData()) and instrument-mailbox overflow (via onRouteOverflow callback from the executor).
  2. Command processed on the single-owner RouteCommandLoop — pendingDegradation is cleared. If the route's routeInstanceId no longer matches the command's (the route was recreated), the stale command is silently discarded:
  3. Route marked Degraded(cause, hasOpenPositions)
  4. Strategy mailbox closed (terminates the executor's input)
  5. Route-side jobs (signals, output, trade) cancelled
  6. Executor cancelled (executor.cancel())
  7. Escalation decision:
  8. Open positions → failProcessor() completes the fatalFailure signal → ProcessorFatalWatcher propagates the exception → coroutineScope fails → Kubernetes detects failure → pod restart (fail-fast)
  9. No open positions → route is removed from active routing, processor stays healthy for sibling strategies

This means one degraded strategy does not stall sibling strategies unless it has open positions that require the processor to fail-fast.

Strategy Removal With Open Positions

When a strategy is removed from the repository while its executor has open positions:

  1. The route transitions to Stopping(retainedInstrumentIds) instead of being torn down immediately (routes already in Stopping are skipped to prevent duplicate watchers)
  2. The strategy mailbox stays open — the executor continues receiving ticks for retained instruments
  3. A StoppingWatcher coroutine monitors executor.hasOpenPositionsFlow and posts RouteCommand.TeardownStrategy when positions flatten. The watcher double-checks executor.hasOpenPositions() before posting to guard against premature false emissions from hasOpenPositionsFlow before executor initialization
  4. routeFeedData() continues routing to Stopping routes (only Degraded routes are skipped)
  5. The instrument index uses executor.currentInstrumentIds for both Active and Stopping routes (the index is rebuilt from the routes' delivery-target views on the post-event targets-dirty wake —: route addition/removal no longer cross the boundary as commands)
  6. When a retained instrument's position flattens, the worker's watcher reports a flat level-set with the record's token, the coordinator applies the explicit removal transition (the per-route bridge jobs are cancelled, the execution is cancelled), and the post-event targets-dirty wake — fired after the inbox event, outside the lock — rebuilds the routing index to remove the flattened instrument
  7. When a new instrument route is published, per-route bridge jobs are launched before execution.start, and execution.start itself is guarded by an irreversible per-transport start/cleanup gate: a current-token fast-lane cleanup can run while the initialization is suspended at its publication ack — both operations target the same valid token — and once cleanup has claimed the transport, the post-ack start is skipped on the cancelled execution instead of invoked. The routing index learns of the new instrument through the delivery-target views: the post-event targets-dirty wake (fired after the publication's inbox event, outside the lock) rebuilds it to include the new instrument. The execution's fatal-error handler is likewise bound to the route it was published under: if a removal/re-enrollment has replaced the route by the time the failing pipeline's callback runs, the callback is stale (the originating execution is gone and its failure is moot for the fresh canonical rebuild) and is ignored with a warning instead of tearing down the replacement. 's critical-section discipline makes the orderings structural — a teardown's route removal and its failure fact are one coordinator-internal mutation followed by the enqueued failure — so the executor-local incarnation machinery and the processor's mirrored incarnation guards are deleted; the failure-fact order regression pins what still crosses at the executor level. What remains necessary is the route-instance gate: the failure command carries the owning strategy route's routeInstanceId and is ignored when that route is no longer current, because executor cancellation does not join and a delayed failure from a dead executor can otherwise land after its replacement's first publication.
  8. When TeardownStrategy is processed: mailbox closed, jobs cancelled, executor cancelled, route removed
  9. If a Stopping route's mailbox overflows, degradeRoute() handles it — if it has open positions, fail-fast is triggered

If a Stopping route's strategy reappears in the repository (re-added), the route transitions back to Active and the stopping watcher is cancelled.

Route-side jobs are launched in a supervised child scope (routeSideScope) with inline try/catch: if a signals, output, or trade job fails with a real exception (not CancellationException), it enqueues a RouteCommand.Degrade for that strategy.

stateDiagram-v2
    [*] --> Active

    Active --> Stopping : removed from repo + hasOpenPositions
    Stopping --> Active : strategy reappears in repo
    Active --> Degraded : mailbox overflow / route-side job failure
    Active --> Gone : removed from repo + !hasOpenPositions
    Stopping --> Degraded : overflow while stopping
    Stopping --> Gone : TeardownStrategy (positions flattened)
    Degraded --> Gone : reconciliation removes route
    Degraded --> FailFast : hasOpenPositions

    classDef active fill:#c8e6c9,stroke:#2e7d32,color:#1b5e20
    classDef stopping fill:#ffe0b2,stroke:#e65100,color:#bf360c
    classDef degraded fill:#ffccbc,stroke:#d84315,color:#bf360c
    classDef gone fill:#eceff1,stroke:#546e7a,color:#37474f
    classDef failfast fill:#ffcdd2,stroke:#c62828,color:#b71c1c

    class Active active
    class Stopping stopping
    class Degraded degraded
    class Gone gone
    class FailFast failfast

Readiness And Health

The processor's readiness and health are route-aware:

Condition isReady() componentHealth()
No core jobs false INACTIVE
Any core job not active (fatalWatcher, routeCommandLoop, feedRouter, reconcileLoop) false BLOCKED
Degraded route with open positions false BLOCKED
Degraded route without open positions true DEGRADED
Stopping route (positions winding down) true DEGRADED
All routes active true HEALTHY

Readiness reflects processor-core health, not every route-side job. A degraded route without open positions does not make the processor unready. A stopping route (strategy removed but positions still open) does not make the processor unready — it is still functioning correctly, just winding down.

Failed instruments (the per-instrument teardown path) surface as DEGRADED health naming the failed (strategy, instrument) pairs — derived at projection time from the executors' tokenized route-phase records: each executor exposes a locked instrumentRouteStates snapshot (every record's RouteToken, membership intent, and RoutePhase), and componentHealth projects the Failed records of all non-Degraded routes through a live-membership filter that reads the repository flow at projection time. Stale executor views are rejected by token: a state contributes only when its RouteToken's route instance matches the CURRENT routing snapshot's route for the strategy — read after the states — so a health poll preempted through a strategy remove/re-add cannot project the old executor's retained Failed record against the recreated route. A failed pair therefore stops surfacing when the instrument re-initializes under the same strategy (the record leaves Failed at publication), when the operator removes it from the strategy's desired membership (the filter excludes it — this covers the convergence window where the executor's record is still Failed and no route exists to remove — deleted the route-removed boundary event regardless, since a failed instrument has no route — and also a failure command arriving after its instrument's removal, e.g. a slow backfill failing post-removal;), or when the strategy route is degraded or torn down (Degraded routes are excluded from the scan — their health is carried by the route-state branches above; a Stopping route's strategy is absent from live desired membership, so the filter excludes its records anyway). DrainingForOpenPosition is deliberately not a failure. FeedUnresolvable is deliberately excluded from the projection as well — an unresolvable feed is a configuration gap, not a pipeline failure, so it never degrades health or readiness; it informs through the warn log only, with an explicit UI/health projection as a recorded follow-up. There is no mirrored failure registry to go stale: the stale-payload clear hazard is structural under derivation, and the route-wiring stale-payload hazard no longer exists — the topology wake carries no payload and reads the repository's current .value at consumption (the class, deleted at rather than defended). A Failed phase affects component health only — never isReady. Under the event orderings are structural — a route-added event can never arrive after a failure of the route it announces, and a removal can never arrive after a newer route's failure — so no incarnation bookkeeping is needed to reject stale announcements or stale failures; the only remaining staleness gate is route-instance identity: every lifecycle command (added, removed, failed) carries the owning strategy route's instance id and is ignored when that route is no longer current, because executor cancellation does not join and a delayed command from a dead executor can otherwise land after its replacement's first announcement. For a stale-instance failure command the open-position fail-fast escalation deliberately still fires: a failed execution's open position is unmanaged regardless of which route or executor owned it, so a conservatively-stale hasOpenPosition flag fails the processor rather than risking a zombie position. That escalation (and the future durable failure record) is transition-sourced — it fires from the command-loop handler upon the lossless, lock-ordered event, never from the conflatable health projection.

flowchart TD
    subgraph hierarchy ["Mailbox Hierarchy"]
        FD["FeedData&lt;MarketData&gt;"] --> PR["Processor · routeFeedData()"] -->|"trySend"| SM["Strategy Mailbox\nChannel&lt;MarketData&gt; · cap 1024\nowned by Processor (backend-app)"]
        SM -->|"receiveAsFlow"| ER["StrategyExecutor · Router coroutine"] -->|"deliver() trySend"| IM["Instrument Mailbox\nProcessorMailbox · Channel · cap 1024\nowned by StrategyExecutor (backend-processor)"] -->|"mailbox.flow"| SX["StrategyExecution\nrunningFold → strategyOutput · signals"]
    end

    SM -.->|"overflow"| D1["requestDegrade()"]
    IM -.->|"overflow"| D2["onRouteOverflow()"]

    classDef data fill:#e3f2fd,stroke:#1565c0,color:#0d47a1
    classDef processor fill:#e8eaf6,stroke:#283593,color:#1a237e
    classDef mailbox fill:#fff3e0,stroke:#e65100,color:#bf360c
    classDef executor fill:#f3e5f5,stroke:#6a1b9a,color:#4a148c
    classDef execution fill:#e8f5e9,stroke:#2e7d32,color:#1b5e20
    classDef overflow fill:#ffebee,stroke:#c62828,color:#b71c1c

    class FD data
    class PR processor
    class SM,IM mailbox
    class ER executor
    class SX execution
    class D1,D2 overflow

    style hierarchy fill:#fafafa,stroke:#9e9e9e,color:#424242
flowchart TD
 S1["① Processor · launchRouteJobs\nsignalsJob · outputJob · tradeJob\nlaunched with CoroutineStart.UNDISPATCHED\n→ collect reached synchronously"] --> S2["② strategyExecutor.start\nLaunches coordinator loop (sole membership wake collector) +\nrouter + worker init and fast command lanes\nProcessor already subscribed to outputs/signals"] --> S3["③ Coordinator sweep → worker init, per instrument\na. InitStarted(token, ingress) acknowledged\n (ingress published, buffered ticks drained)\nb. Construct StrategyExecution (stopped)\nc. Launch bridge jobs (UNDISPATCHED) → subscribe synchronously\nd. ExecutionReady acknowledged → route published \ne. execution.start → pipelines launch after subscribers attached"] --> S4["④ Failure fact → RouteCommand.InstrumentExecutionFailed\nenqueued under routingLock with the mutation \nPost-event targets-dirty wake (stage c3) rebuilds\nroutesByInstrument from the routes' delivery-target views"]

    classDef step1 fill:#e3f2fd,stroke:#1565c0,color:#0d47a1
    classDef step2 fill:#e8eaf6,stroke:#283593,color:#1a237e
    classDef step3 fill:#f3e5f5,stroke:#6a1b9a,color:#4a148c
    classDef step4 fill:#e8f5e9,stroke:#2e7d32,color:#1b5e20

    class S1 step1
    class S2 step2
    class S3 step3
    class S4 step4

Mailbox Hierarchy

The routing model uses a two-level mailbox architecture:

Level Type Capacity Owner Location
Strategy mailbox Channel<MarketData> 1024 Processor backend-app (raw Channel)
Instrument mailbox ProcessorMailbox (wraps Channel<MarketData>) 1024 StrategyExecutor backend-processor (internal)

The strategy mailbox is a raw Channel in backend-app because ProcessorMailbox is internal to backend-processor. The executor's instrument mailboxes use ProcessorMailbox since they're within the same module.

Both levels use trySend for non-suspending delivery. Overflow is detected immediately via ChannelResult.isFailure, not by suspending the sender. This means a slow consumer never blocks the router — it degrades the affected route instead.

Timing And Event Contracts

There are three time concepts that matter across the pipeline:

1. StrategyOutput.time

This is the logical strategy time:

  • for EMA, it is the active strategy bucket time
  • for volume_breakout, it is the synthetic volume-bar time

2. StrategyOutput.occurrenceTime

This is the live tick time that caused the output.

This field is the canonical way to recover the real market event time that produced an output.

3. StrategySignalEvent.time

Signals currently use the app's emission time, not the source tick time.

The processor-path tests assert this as an emission window:

  • signal time must fall between "emission started" and "emission observed"

Signals link back to outputs through strategyOutputId, so downstream code can recover the source tick context from the linked StrategyOutput.occurrenceTime.

Strategy-Specific Runtime Semantics

Two strategy timing families currently matter at the processor boundary.

EMA

ema is live and intra-bucket:

  • the active EMA bucket is keyed from the incoming live tick time
  • entries can happen before the next timeframe rollover tick arrives
  • reversals are expected to emit exit first and opposite entry second

Volume Breakout

volume_breakout is close-gated:

  • partial live volume bars may update the latest StrategyOutput
  • entry and exit signals emit only when the current synthetic volume bar closes
  • delayed re-entry and managed exits are part of the contract

The strategy runs on canonical volume bars: the frame length F is derived per session — rule v1: F = max(quantum, roundHalfUp(totalVtt / scheduledTradingMinutes, 18)) over the last 5 eligible sessions — and every session switch resets the builder with the new frame, dropping the overnight partial. Initialization backfills from QuestDB via a streaming session walk - sessions visited newest-to-oldest, every read a single session window bounded by the recorded high-water mark, each frame derived from its support block's per-session aggregates, retention staged under the caller's real bar bound (a retention-breaching tick stages nothing), and termination at the data floor — the first session entirely older than the oldest recorded row, with the walk never stopping short of the oldest in-session row's session; an instrument with no eligible history enters the awaiting-history state and retries at the next session open. Frames come from an app-owned SessionFrameProvider: the strategy can only obtain frame sources through the factory handed down from the app wiring — it constructs none itself. The production implementation (CachedSessionFrameProvider, backend-app) precomputes the next session's frame in the overnight gap (30 minutes before open, provisional) with one job per desired volume-frame instrument, so the first new-session tick switches atomically on a cached frame with no lookback reads on the strategy mailbox path; a cache miss falls back to the uncached provider-backed lookup under the execution's availability retry. Job removal drops the job and its cached frame, and a calendar change restarts the job and drops the old calendar's frame; since cancellation is cooperative, a job whose derivation is in flight when it is removed or replaced commits only while still the instrument's registered job (identity-checked atomically against reconcile's cancel-and-clear), so a stale frame can neither be restored after removal nor clobber its replacement's fresh frame. Each frame selection writes an append-only frame_selected audit record with full provenance (writerId, executionRole, tradingRunId, trigger). Provenance is atomic: the run id and role are captured once per tick — a single read of the leadership flow, before any retrying write — so a leadership transition during a retried audit write can neither tear a record (run id from one moment, role from another) nor re-attribute it to a run other than the one the tick observed; a mid-retry flap is audited as a promotion at the next tick. A single tick whose delta would materialize more bars than the retention window is a defined breach: record the tick, discard the partial, reseed the phase at vtt mod F, keep the price.

Trade Execution And Trade Persistence

Trade execution is downstream of signals, not outputs.

The runtime flow is:

  1. The route-local StrategyEventCoordinator is the sole collector of each StrategyExecutor signal stream. It briefly enters the process-wide TradingLeadership fence and captures the active trading-run UUID, if any.
  2. It immediately enqueues the signal with that immutable provenance for DataRecorder, publishes the raw signal to the processor/DataBridge stream, and—only when a run was captured—places a TradeExecutionRequest on the route's bounded trade mailbox.
  3. TradeExecutor consumes those requests using the strategy-specific trade settings selected from ordersRepository.tradeSettingUpdates. It enters the fence again before calling the broker and requires the request's run UUID to equal the active run.
  4. On a failed renewal or ownership change, the adapter clears the fence before failing its coroutine, so the former leader process terminates and Kubernetes restarts it. A request captured under the former run can be recorded, but cannot submit an order.
  5. TradeExecutor calls the broker and emits tradeEvents.
  6. DataRecorder.tradeConsumer persists those trade events through OrdersRepository.saveOrder(...).

Important behavioral contract:

  • outputs and signals can still exist even when a trade is not executed
  • missing trade settings or broker failures should not suppress outputs or signals
  • those situations only affect trade execution and trade-event emission

The resilience tests in backend-app cover those cases explicitly.

Recorder Path

DataRecorder.feedConsumer is the raw market-data persistence path. It is the only feed consumer that opts into recorded-late ticks (recordsLateFeedData): emitted ticks persist with their real derived fields; recorded-late ticks (late tuples, out-of-session prints) persist with synthetic volume = 0 / duration_since_last = 0 for dual-replica union coverage — the derived columns are non-authoritative and recomputed by every canonical reader.

StrategyEventCoordinator passes strategy outputs and signals to DataRecorder with the trading run captured at route admission. DataRecorder enqueues them on the same single ILP/HTTP writer. They persist complete JSON payloads for replay/audit and store queryable IDs, strategy/instrument IDs, event timestamps, and provenance (writer_id, execution role, nullable trading-run ID). The output table also stores occurrence_time; the signal table stores signal_id and strategy_output_id.

It:

  • flattens the feed stream with flatMapMerge
  • converts it to a ReceiveChannel
  • passes the channel to Recorder.recordMarketData()

Ingester then writes QuestDB rows with:

  • instrument_id
  • duration_since_last
  • ltp
  • volume
  • ts

The recorder also appends frame_selected audit records via the same ILP writer with an immediate flush — written at strategy initialization, every session switch, and leadership promotion, each carrying writerId, executionRole, tradingRunId, the exact frame as a Decimal-literal string, derivation-rule and calendar versions, and the per-session totals that produced it. The promotion trigger is fold-driven: when a promoted leader folds its next in-session tick under a new trading run, a LEADERSHIP_PROMOTION record audits the frame it adopted (selected earlier, possibly as standby). A promotion observed on a tick that itself switched sessions writes only the session-switch record, and a pod that was already leader at initialization writes no adoption record — its initialization records already carry the LEADER role.

Trade persistence is separate from QuestDB ingestion. Trade events are stored through OrdersRepository in the main application datastore.

Server Publishing Through DataBridgeLauncher

DataBridgeLauncher.launch() starts one coroutine per event family and forwards app state to backend-server.

The main forwarded streams are:

  • strategy signal events
  • strategy output events
  • market feed events
  • trade events
  • open positions
  • readiness
  • error snapshots
  • recovery events

Three details matter operationally:

  • DataBridgeLauncher does not reuse the shared feed flow from StartDataFlowTask; it opens fresh liveFeed.feed(...) collectors for market-feed forwarding
  • readiness published to the server is the diagnostic health set, not the pod-readiness set
  • DataBridgeLauncher does not keep a top-level eager kRPC client; launch() starts one lazy background supervisor job that repeatedly creates a fresh session with Arrow retry, marks readiness from the active session, and cancels cleanly on ApplicationStopPreparing. All bridge traffic goes to ApplicationConfig.INTERNAL_EVENTS_HOST/PORT/PATH.
  • AuthStateCoordinator runs alongside DataBridgeLauncher at app-lifetime scope, started by StartAuthStateTask. It collects recoveryFlow().recoverable() for auth prompt publication, watches tokenStateUpdates() for obsolete-prompt cleanup, and resumes datasources when the OAuth token recovers.
  • ProvisionedTokenStateCoordinator is started by the same StartAuthStateTask. It resumes the market-feed datasource via ResumableDataSource.resumeIfStopped() when UPSTOX_ANALYTICS transitions to Valid after a Secret rotation (the analytics token powers the market feed and historical candle data).

Current Guarantees And Non-Guarantees

Guaranteed:

  • feed batches are processed sequentially
  • EMA remains live and intra-bucket
  • volume_breakout remains close-gated
  • reversal-capable strategies emit exit before opposite entry
  • open-position removals retain executions until managed cleanup completes
  • a slow consumer never blocks the router (non-suspending trySend delivery)
  • route degradation with open positions triggers processor fail-fast
  • UpstoxDataSource detects day boundaries via vtt-decrease or time-delta heuristic, preventing negative volume deltas at session open and preserving opening auction volume
  • DEGRADED datasources do not flip pod readiness red — HealthStatus.DEGRADED.allowsReadiness returns true, so a feed with a terminal token failure reports degraded health without blocking the readiness endpoint

Not guaranteed:

  • deterministic ordering across instruments inside the same batch
  • reuse of the same live-feed collector between local processing and bridge publishing
  • correct day-boundary detection for non-equity Upstox segments (NSE_COM, MCX_FO) whose inter-session gaps may be shorter than the 12-hour threshold (QAPP-113)

Token Auth And Recovery Flow

When the token broker transitions a broker's state to AwaitingAuth, the recovery pipeline activates:

  1. OAuthTokenBrokerService.recoveryFlow(AuthBroker.UPSTOX) emits RecoverableErrorException, collected by AuthStateCoordinator via RecoveryManager.recoverable()
  2. RecoveryManager publishes the error to errorFlow, surfaced by DataBridge to backend-server
  3. The UI displays the auth URL and accepts the auth code
  4. The code flows back through RecoveryRouter → DataBridge.recoveryEvents() → RecoveryManager.recover() → TokenRecoveryAction.recover() → OAuthTokenBrokerService.submitAuthCode()
  5. The token broker exchanges the code for an AccessToken, transitions state to Valid
  6. AuthStateCoordinator detects the Valid transition and calls ResumableDataSource.resumeIfStopped(); awaitValidToken() also resumes in UpstoxApiDecorator, and the market data feed reconnects

RecoveryManager coalesces by recoveryUrl — concurrent AwaitingAuth emissions for the same broker share one recoveryToken UUID and one published error. The recover() method removes only the resolved error and preserves all other active recoverable errors across all recovery keys. Out-of-band transitions (token becomes Valid without explicit recovery) are handled by clearLogicalRecovery(), which removes the stale prompt, resumes suspended .recoverable() collectors, and cleans up the logical recovery entry.

Terminal Token Failure

Permanent OAuth errors (e.g., UDAPI100069 — wrong client ID/secret) throw NonRecoverableTokenException from UpstoxTokenExchangeStrategy.exchange(). The TokenManager surfaces this as OAuthTokenState.Failed, and awaitValidToken() throws TokenUnavailableException. UpstoxDataSource catches this in its outer .catch handler, sets a terminalFailureMessage (reporting DEGRADED health), and does not rethrow — this contains the failure and prevents infinite retry loops. The terminalFailureMessage is cleared only after the datasource successfully receives and maps upstream data following recovery.

Blackout Period

UpstoxTokenExchangeStrategy enforces a blackout period (3:30 AM–8:00 AM IST) via isTokenRequestAllowed(). During this window, submitAuthCode() throws TokenBlackoutException. Tokens expire at 3:30 AM IST daily; the expiry timer in TokenManager proactively transitions Valid → AwaitingAuth without waiting for a 401.

Provisioned Token Flow

Provisioned tokens — UPSTOX_SANDBOX for sandbox HFT order placement and UPSTOX_ANALYTICS for the market data feed and historical candle data — follow a different lifecycle than OAuth tokens:

  • Tokens are mounted from a Kubernetes Secret as JSON files at /var/run/secrets/token-broker/provisioned
  • ProvisionedTokenManager loads and validates the files at startup; if the file is missing, expired, or unparseable, the manager enters Failed and starts a 30-second poll loop
  • On 401, UpstoxApiDecorator's sandbox client calls ProvisionedTokenBrokerService.invalidateToken(UPSTOX_SANDBOX) which re-reads the file; if the fingerprint is unchanged, it transitions to Failed and the poller watches for Secret rotation
  • The analytics client follows the same pattern: on 401 it calls invalidateToken(UPSTOX_ANALYTICS). While the handle is Failed, awaitValidToken() throws TokenUnavailableException, surfacing as terminal DEGRADED in UpstoxDataSource; ProvisionedTokenStateCoordinator resumes the feed when the handle returns to Valid after the Secret is rotated
  • When the operator rotates the Secret, the kubelet refreshes the mounted files and the poller recovers on its next cycle — no pod restart is required
  • ProvisionedTokenManager uses a mutex-owned state machine: becomeValidUnderLock cancels any running poller and old expiry timer, publishes Valid, and starts a new expiry timer; becomeFailedUnderLock cancels the expiry timer, publishes Failed, and ensures exactly one poller is running
  • The expiry timer reacquires the lock and checks token identity before transitioning to Failed — stale timers from old tokens are silently ignored
  • Transient I/O errors (file temporarily unreadable) clear lastFingerprint to null, so the poller will detect when the same bytes reappear after the I/O issue resolves

Feed And Repository Separation

FeedRepository is storage/KV-only — it persists FeedIdentifier rows and subscribed-instrument KV state, with no dependency on backend-datasources. The live datasource runtime is owned directly by its consumers: NormalizedMarketFeed builds raw-feed streams through DataSourceFactory.feedStream(...), and SetSubscribedInstrumentsTask pushes the subscribed-instrument set through DataSourceFactory.setSubscribedInstruments(...). AddNewFeedsTask passes the dataSources list from backend-datasources to FeedRepository.addMissingFeeds() for diff-based feed creation.

backend-server and backend-repository do not depend on backend-datasources — the live feed runtime is scoped to backend-app only, consistent with the app/server separation invariant.

Public Instruments

Broker.publicInstruments(action) is a top-level extension in backend-broker that fetches instrument catalogs without authentication. UpstoxPublicInstrumentCatalog uses a plain HttpClient (no auth plugins) to fetch from the assets.upstox.com CDN. This is used by backend-sync's LiveInstrumentFeedRefresher instead of DataSourceFactory, since backend-sync does not have OAuthTokenBrokerService connections.

Code Map

If you need to trace the machinery in code, start here:

  • backend-app: startup tasks, feed fan-out, processor wiring, data bridge launcher, token broker connection, NormalizedMarketFeed (live datasource stream collection through DataSourceFactory), AuthStateCoordinator (OAuth auth recovery → datasource resume), ProvisionedTokenStateCoordinator (analytics-token recovery → market-feed resume), ResilientTokenBrokerService + ResilientProvisionedTokenBrokerService (kRPC reconnect + shared state flows)
  • backend-app (Processor.kt): RoutingSnapshot, StrategyRoute (with routeInstanceId), RouteCommand (TopologyDirty/TargetsDirty/Degrade/TeardownStrategy), RouteState (Active/Degraded/Stopping), RouteJobs, CoreJobs (private types), feed routing, route command loop, reconciliation, degradation, stopping lifecycle, and fail-fast logic
  • backend-token-broker: TokenManager (token state machine, expiry timer, blackout, write-through persistence), OAuthTokenBrokerServiceImpl (kRPC service), ProvisionedTokenManager (mutex-owned state machine with @Volatile lastFingerprint, file-backed polling recovery, expiry timer with identity check), ProvisionedTokenBrokerServiceImpl (kRPC service), UpstoxTokenExchangeStrategy (OAuth token exchange, error classification), AuthBroker (type-safe key for auth-requiring brokers), ProvisionedTokenHandle (type-safe key for provisioned-token handles)
  • backend-broker: BrokerApiResolver (compile-time-safe Broker→BrokerApi mapping, requires both OAuthTokenBrokerService and ProvisionedTokenBrokerService), UpstoxApiDecorator (three HttpClient.withConfig instances: OAuth for live trading, UPSTOX_ANALYTICS for market feed + historical, UPSTOX_SANDBOX for sandbox HFT; UpstoxErrorParser for shared 401 classification), UpstoxPublicInstrumentCatalog (auth-free instrument fetch), Broker.publicInstruments() extension
  • backend-processor: StrategyExecutor (with start(), per-route bridge jobs, InstrumentRoute), ProcessorMailbox (internal)
  • backend-strategy: StrategyExecution (with start()), EmaStrategyExecution, VolumeBreakoutStrategyExecution
  • backend-trade-executor: TradeExecutor (receives BrokerApiResolver)
  • backend-datasources: UpstoxDataSource (tick-to-MarketData conversion, vtt delta, day-boundary detection, DEGRADED health on terminal token failure, ResumableDataSource), BitFlyerDataSource, DataSourceFactory (receives BrokerApiResolver)
  • backend-repository: FeedRepository (storage/KV-only, no datasource dependency)
  • backend-recovery: RecoveryManager (coalescing by recoveryUrl, clearLogicalRecovery() for out-of-band transitions, targeted recover() that preserves unrelated errors)
  • backend-recorder: DataRecorder, Recorder, Ingester