Live Market Pipeline And Processor Machinery
This document describes the runtime path for live market data inside backend-app, from subscription resolution through strategy execution, trade execution, persistence, and server publishing.
For the broader module map and runtime boundaries around this path, see Architecture Overview.
Scope
The pipeline described here spans:
- startup task ordering in
backend-app - live feed collection and fan-out
Processor,StrategyExecutor, andStrategyExecution- trade execution and trade persistence
- raw market-data persistence through the recorder
- event publishing to
backend-serverthroughDataBridgeLauncher
It is the source-of-truth overview for the processor-side machinery that the recent EMA and volume_breakout runtime tests exercise.
Diagrams
The following diagrams illustrate the processor and strategy pipeline:
| Diagram | Description |
|---|---|
| Data Flow | Three-tier architecture: Processor → Executor → Execution |
| Route State Machine | Route lifecycle: Active, Stopping, Degraded, Gone, Fail-Fast |
| Command Loop | Single-owner control plane: command sources, channel, snapshot |
| Mailbox Hierarchy | Two-level mailbox architecture and start barrier |
| Degradation & Fail-Fast | Overflow detection, deduplication, escalation decision |
High-Level Flow
QuantApp startup
-> AddNewFeedsTask / AddNewStrategiesTask
-> LoadInstrumentNamesTask / SetSubscribedInstrumentsTask
-> StartTradingLeadershipTask
-> StartDataFlowTask
-> StartAuthStateTask
-> StartDataBridgeTask
feedRepository.subscribedInstrumentsFlow()
-> NormalizedMarketFeed (collects each raw feed exactly once, shareIn)
-> liveFeed.feed(dataSourceIdentifier, instrumentIds)
-> merged Flow<FeedData<RawTick>> (datasources derive nothing)
-> TickNormalizationStage per feed (hydrated per-instrument TickNormalizer)
-> shared NormalizedFeedBatch (emitted + recorded-late)
-> Processor (emitted only)
-> DataRecorder.feedConsumer (emitted + recorded-late)
Processor
-> routeFeedData() (sequential batch processing)
-> StrategyMailbox (Channel<MarketData>) per strategy
-> StrategyExecutor per strategy
-> InstrumentMailbox (ProcessorMailbox) per instrument
-> StrategyExecution per strategy+instrument
-> StrategyEventCoordinator
-> StrategyOutput / StrategySignalEvent to DataBridge
-> strategy outputs and signals to DataRecorder
-> run-bound trade requests to TradeExecutor
-> tradeEvents
DataRecorder
-> raw market data to QuestDB
-> strategy outputs and signals to QuestDB
-> trade events to OrdersRepository
DataBridgeLauncher
-> strategy outputs
-> strategy signals
-> market feed events
-> trade events
-> open positions
-> readiness and errors
AuthStateCoordinator
-> RecoveryManager.recoveryFlow().recoverable() for auth prompts
-> clearLogicalRecovery() for obsolete prompts
-> ResumableDataSource.resumeIfStopped() on OAuth Valid
ProvisionedTokenStateCoordinator
-> ResumableDataSource.resumeIfStopped() on UPSTOX_ANALYTICS Valid
Startup Sequence
QuantApp groups StartupTasks by phase and runs one phase at a time. Tasks inside the same phase run concurrently.
The current startup order is:
AddNewFeedsTaskandAddNewStrategiesTaskLoadInstrumentNamesTaskandSetSubscribedInstrumentsTaskStartTradingLeadershipTaskStartDataFlowTaskandRelaySubscriptionUpdatesTaskStartAuthStateTaskStartDataBridgeTask
Why this order matters:
- feeds and strategies must exist before subscription resolution starts
- datasource subscription state must be restored before live streaming begins
- trading leadership starts before strategy signals can reach
TradeExecutor; a standby replica continues processing and recording data but cannot submit broker orders - the live feed must exist before the server-forwarding bridge starts collecting from it
- token-state side effects (recovery publication, obsolete-prompt cleanup, datasource resume) publish through
RecoveryManager.errorFlow, a replayingStateFlow, so they are retained until the data bridge subscribes —StartAuthStateTaskandStartDataBridgeTaskhave no hard ordering dependency between them - the subscription relay (
RelaySubscriptionUpdatesTask, same phase asStartDataFlowTask) starts after the startup push has applied the restored set, so its first emission re-pushes the same set idempotently; each feed's runtime push waits for that feed's current pipeline incarnation to hydrate and attach its raw-stream collector carrying exactly the emission's per-feed instrument set (PipelineLiveGate.awaitFeedLive— set equality, not coverage: the relay and the pipeline are independent collectors of the subscription flow, so a stale mark left over from the previous emission can never authorize a push for a different set, including the stale-superset case of a removal push followed by a re-add) before the mutated set reaches the datasource; an empty or absent feed's push awaits the opposite signal — its live mark disappearing at the pipeline's emission clear. The gate is best-effort only in the sub-millisecond window around the attach itself (the mark is published just before the datasource subscription and is not revoked on a mid-stream recoverable retry); the multi-second hydration and emission-swap windows are closed structurally, because the pipeline collects the RAW instrument-ID flow (FeedRepository.subscribedInstrumentIdsFlow) and resolves the per-feed grouping inside each emission block after a synchronous live-mark clear — a mutation cancels the previous emission's incarnations before any data-store read can suspend, so a live mark can only ever be published by the current emission's incarnation
Token Broker Connection
The tokenBroker() Ktor module runs before core() and establishes the kRPC connection to backend-token-broker. Both kRPC clients are wrapped in ResilientTokenBrokerService and ResilientProvisionedTokenBrokerService, which provide automatic reconnect with exponential backoff and shared state flows per broker. If the token broker is unavailable at startup, the resilient decorators retry the connection rather than crashing the pod — once the broker becomes reachable, the token-state stream reconnects and the app recovers without a restart.
UpstoxApiDecorator uses awaitValidToken() to suspend HTTP requests when the token state is AwaitingAuth. AuthStateCoordinator collects recoveryFlow(broker) wrapped in RecoveryManager.recoverable() to surface auth prompts in the UI, watches tokenStateUpdates(broker) to clear obsolete prompts via clearLogicalRecovery(), and resumes datasources via ResumableDataSource.resumeIfStopped() when the OAuth token transitions back to Valid (either through user auth or out-of-band refresh).
The market data feed and historical candle data authenticate with the provisioned UPSTOX_ANALYTICS token instead of OAuth. ProvisionedTokenStateCoordinator (started by the same StartAuthStateTask) resumes the feed datasource via ResumableDataSource.resumeIfStopped() when UPSTOX_ANALYTICS transitions back to Valid after a Secret rotation — the analytics token has no OAuth flow, so while it is Failed the feed reports terminal DEGRADED until the operator rotates the Secret.
Feed Reception And Fan-Out
StartDataFlowTask.startFeed() is the main entry point for live market data inside backend-app.
It wires every FeedConsumer to the shared NormalizedMarketFeed.batches flow through per-consumer mailboxes: each normalized batch routes its emitted ticks to every consumer and its recorded-late ticks only to consumers that opt in (FeedConsumer.recordsLateFeedData).
NormalizedMarketFeed is the single, hot, normalized market stream:
- it watches
feedRepository.subscribedInstrumentIdsFlow()and resolves the raw instrument IDs to per-feed groups inside the latest block (resolveSubscribedInstrumentIds), mapping each group to aDataSourceIdentifier(the raw IDs come from the operator-intent subscriptions file on the shared volume; the per-feed resolution goes throughDataStore) - each raw feed flow is wrapped with
RecoveryManager.recoverable()and collected exactly once - every batch is normalized through the per-feed
TickNormalizationStage(see below) - the result is shared (
shareIn) and fanned out — downstream consumers never collect the raw feeds - a configured feed whose name resolves to no registered
DataSourceIdentifieris a configuration error: at startup it failsStartDataFlowTask's hydration-gate await (awaitFirstHydrationPasscompletes exceptionally, terminating the process — the fail-fast / no-partial-readiness contract); mid-run it is dropped with an error log while valid feeds keep flowing, andRequiredFeedReadinessReporterblocks readiness on it — the shared stream is never killed mid-run, since that would take every valid feed down with it - a terminal pipeline failure in one feed (an exception that escapes its recovery boundary — a normalizer defect, a non-recoverable datasource failure) stops only that feed: it is logged, marked DEGRADED in the
MarketFeedPipelinesreadiness component, and the feed restarts at the next subscription emission, while every other feed keeps flowing through the shared stream. JVMErrors are never isolated — they fail the whole stream — and aCancellationExceptionthrown while the collecting scope is alive (e.g. an escaping timeout) is treated as a terminal failure of that feed, not a silent end. A persistently failing hydration read stays inside the stage's retry loop — marking the feed DEGRADED on every retry — and keeps the first hydration pass from settling, so the startup timeout — not a shared-stream death — terminates the process; a terminal failure's mark survives a subscription emission until the replacement incarnation demonstrably re-attaches (a feed absent from an emission has its mark cleared deliberately), so a replacement whose hydration keeps failing cannot silently erase an established failure - the whole stream is self-healing: a failure of the subscription read itself, or a defect in the emission planning above the per-feed boundary, marks every market-data feed DEGRADED and restarts the entire pipeline after a backoff — the Ktor application scope supervises its children, so such a failure would not terminate the process, and a dead unmarked stream would leave every delivery path stopped while the pod appears healthy. Health concerns are tracked independently per feed (pipeline vs. subscription push vs. subscription relay), so one concern's recovery never clears another concern's still-live failure
The two main consumers are:
Processor, which turns emitted ticks into outputs, signals, and tradesDataRecorder.feedConsumer, which persists emitted and recorded-late ticks to QuestDB (the only consumer that opts into late ticks — raw coverage for dual-replica union)
DataSourceFactory.feedStream(...) resolves the concrete datasource, optionally narrows it with the requested instrument list, and then collects the datasource stream — NormalizedMarketFeed calls it after resolving instruments through FeedRepository.
The main datasource implementations are currently UpstoxDataSource and BitFlyerDataSource. SetSubscribedInstrumentsTask pushes the restored instrument set into each datasource once at startup via DataSourceFactory.setSubscribedInstruments(...); a rejected push fails the process (fail-fast). Afterwards RelaySubscriptionUpdatesTask — the runtime half of subscription application — collects feedRepository.subscribedInstrumentsFlow() and re-pushes the full set to every market-data feed on each mutation, parallel per feed: a push that keeps failing is retried with backoff and degrades the feed's MarketFeedPipelines health mark (never a process kill — the pod keeps serving every feed whose push succeeded), a newer emission cancels the previous push cycle mid-retry, and a failure of the subscription flow itself is retried the same way with every feed marked degraded (the application scope would not terminate the process on a dead relay). Upstox applies a pushed set as a diff on its active broker connection — SUB for added keys and UNSUB for removed keys, and a fresh connection replays the current set as a full SUB — so a removal actually releases broker subscription capacity (BitFlyer still re-subscribes the full set per push; its websocket protocol has no UNSUB request type, tracked by its existing TODO). On the server side, unsubscribing an instrument that currently has a live strategy assignment, an open position, or a live/draining route is rejected at the mutation surface — the server reads current assignments and open positions fresh from the shared store (the authoritative cross-replica floor), reads the route-state registry fresh from the shared volume (every replica's live routed instruments and draining routes, each published to runtime-records/<pod>/route-state.json and discounted once its freshness lease expires — the protection for a peer replica still holding a LIVE route whose assignment was just removed, and for a signal-level drain whose order was never placed or failed; both exist only in the hosting replica's process-local state, which no shared store holds and only that replica can publish; the registry leg is only as current as the writer's last successful publication — a contained write keeps the route running and the writer DEGRADED while the file still lacks the new route, briefly reopening this window until the next publication converges, and a sustained failure expires the writer's lease entirely), and queries the app tier's AppDataBridge.getRoutedInstruments() (the queried replica's assignment view ∪ routing index — a supplementary input that can only make the gate stricter), failing the removal with a typed input error — because the route would otherwise keep expecting ticks that never arrive (silent execution starvation); additions skip all four protection reads so the mutation keeps working while the app tier is unreachable, and the Processor's component health degrades on any routed instrument that nonetheless lacks a subscription (never-subscribed assignment or gate bypass), or when the subscription state itself cannot be read.
Operator-intent propagation. The subscription set lives in one place: the operator-intent/subscriptions.json file on the shared storage volume. The server tier is its single writer — every subscribedInstruments mutation rewrites the file wholesale with a temp-file-and-atomic-rename, and a failed or timed-out write fails the mutation (fail-loud; the file has no other source of truth); the single-writer invariant is upheld by the server deployment's Recreate strategy (one server pod at a time, closing the sustained two-writer window a RollingUpdate surge would open; the residual deploy-time windows — the terminating old pod's in-flight write — trade a transiently stale propagation file for the strong invariant, accepted because the durable stores stay authoritative), and within the process each repository serializes its mutations with a mutex so the file's publish order matches the durable store's persist order. The app replicas read it through a 5-second poll that reads and decodes the file fresh every tick (FileOperatorIntentSource, driven by TaskScheduler) — content-based change detection, so a replacement with equal length and a preserved timestamp still propagates — exposed as FeedRepository's subscription flows — cross-process propagation the previous KVStore-backed flows never provided (androidx DataStore is single-process by design; its JVM updateNotifications never emits, so the old flows were boot-time snapshots that never saw a mutation from another pod). The flows are conflated state: a slow collector — e.g. the relay's per-feed resolution read suspended on the data store — never holds back newer intent, and an unchanged re-read does not re-emit. The same channel carries strategy membership: strategy-instruments.json — written after every strategyInstruments mutation alongside the Gel persist, which remains the durable authority — is applied by SyncStrategyMembershipTask, which first refreshes the trade-setting view from the store and then the membership through StrategyRepository.refresh(...) (retried with backoff until it applies — a skipped removal would leave the old assignment executing after the store recovers; a newer emission cancels the in-flight application including its retry loop, and the replacement applies only after the previous attempt is cancelled and joined (it runs to completion, non-suspending tail included) — so the newest snapshot always wins the applied state), so a replica's process-local membership snapshot no longer goes stale when assignments change while it runs; the executor reads the trade-setting state per request, so a newly assigned pair's routes start trading with the settings the mutation persisted (a collection-start snapshot would drop every trade request with "No trade setting found"). Absent-file semantics differ by domain: subscriptions treat an absent file as a defined empty set (the file is the set's only store), while strategy membership keeps its Gel-seeded view and only surfaces the absence; a present-but-unreadable or undecodable file — including a non-regular path such as a directory left by a bad manual migration, or a read failure e.g. from a lost directory-traverse permission — keeps the last successfully-read value in both flows (never silently emptying the subscriptions) and degrades the OperatorIntent readiness component (never blocks readiness); the feeds readiness reporter consumes the last applied view rather than a fresh file read, so an unreadable file degrades health instead of throwing out of the readiness calculation.
DataSourceFactory receives BrokerApiResolver rather than token broker services directly — it has no auth knowledge. UpstoxApiDecorator (constructed by the resolver) creates three HttpClient.withConfig instances: one for live trading (positions, holdings, live HFT order placement) that suspends on OAuthTokenBrokerService.awaitValidToken(), one for the market data feed and historical candle data that suspends on ProvisionedTokenBrokerService.awaitValidToken(UPSTOX_ANALYTICS), and one for sandbox HFT order placement that suspends on ProvisionedTokenBrokerService.awaitValidToken(UPSTOX_SANDBOX). All three clients use the shared UpstoxErrorParser for 401 classification — it reads the raw response body from the SavedHttpResponse cache and classifies UDAPI100050 as a token error, triggering invalidation of the appropriate token type.
Datasource Health
DataSource exposes both isReady(): Boolean (binary readiness) and health(): DataSourceHealth (structured health with HealthStatus and optional message). DataSourceHealth carries a HealthStatus enum — HEALTHY, DEGRADED, BLOCKED — where DEGRADED means the datasource is partially functional (allows readiness) and BLOCKED means it is non-functional (fails readiness).
UpstoxDataSource overrides health() to report DEGRADED with a terminalFailureMessage when a permanent token failure occurs (TokenUnavailableException from OAuthTokenState.Failed). This prevents infinite re-auth loops for configuration errors while still allowing the pod to report readiness — the feed is degraded, not dead. The datasource does not observe token state itself; AuthStateCoordinator triggers recovery externally via ResumableDataSource.resumeIfStopped() when the token becomes Valid again. UpstoxDataSource clears the terminal failure as soon as the upstream reconnects (when isReadyState becomes true), not only on tick emission — so even a quiet market after auth recovery reports HEALTHY.
Tick Normalization (RawTick → MarketData)
Datasources emit raw broker facts only — UpstoxDataSource and BitFlyerDataSource derive nothing (no deltas, no durations, no day-boundary heuristics). Feed frames are mapped to RawTick through a shared broker-frame mapper and normalized centrally by the pure TickNormalizer policy from backend-normalizer — one policy shared by the live path, the backfill folds, and reconciliation.
Upstox frame mapping. UpstoxFrameMapper (in backend-broker/upstox/websocket, beside the frame vocabulary) maps MarketFullFeed frames directly to RawTick for the already-resolved instrument: ltt → time, ltp → ltp, and vtt → seq (the cumulative volume-traded-today counter doubles as the ordering discriminator and the volume fact). Envelope validation is reject-and-count — missing ltpc, non-finite ltp, or ltt <= 0 map to Invalid, are logged and counted, and never kill the feed. A frame with vtt = null (LTPC-mode/index variants) maps with vtt = 0, preserving index-recording behavior.
BitFlyer frame mapping. BitFlyerDataSource emits RawTick with seq = the per-product tick_id and volumeFact = the rolling 24h cumulative. Accepted tradeoff: normalized BitFlyer volume is a tick-count delta (monotonic, never negative), not trade volume — true volume bars for generic sources are deferred per the design. Because of that, volume-frame strategies (volumeFrameStrategyNames, v1: volume_breakout) only accept instruments on feeds whose seq is the broker's per-session cumulative volume (Feed.carriesSessionVolumeFact, v1: Upstox). The scope is enforced twice: StrategyRepository.strategyInstruments rejects the combination at configuration time (surfaced on the Strategies screen's save-error banner), and the strategy initializer fails the instrument at runtime through the per-instrument teardown path (no route, DEGRADED component health, re-initialization only after the instrument is removed and re-added) — so pre-existing configuration cannot silently produce tick-count bars either. Recording, charting, and EMA strategies are unaffected.
Normalizer policy (rule v1). TickNormalizer is a pure per-instrument state machine:
- Session identity comes from the versioned
SessionCalendar(sessionContaining), never from gap heuristics. The first accepted tick of each session seeds:volume = 0,durationSinceLast = 0, baseline =seq(always-seed — an unobserved session open can never attribute the morning's unseen volume to the reconnection price). - Ordering gate: a tick is accepted only when its
(ts, seq)is strictly greater than the preceding accepted tuple; sub-millisecond coalescing (samets, higherseq) is accepted. - Effective volume =
seq − heldBaselineon advance; avttregression with timestamp advance yields0and keeps the baseline (100 → 90 → 105⇒ deltas0, then5). - Duplicates: an exact duplicate of the immediately preceding accepted tuple (same key, same
ltp) is dropped and counted; a same-key/different-ltpduplicate is dropped and counted as a conflict. - Hydration: on restart or mid-run enrollment the stage reads
getHydrationMarkonce per instrument (the combined high-water tuple plus session-max baseline read) and constructsTickNormalizer.hydrated(...), so the first live tick after a restart carries a real delta and a real duration instead of re-seeding. Membership is reconciled on every subscription change: a departed instrument's normalizer state is evicted, so a re-added instrument hydrates fresh from the recorded mark rather than resuming a baseline the peer replica has since advanced; instruments that stay subscribed keep their live state, which can be ahead of the recorded mark (ingest flush lag). Backstop-created state (a straggler frame for a departed instrument, or an unsolicited frame) is marked provisional and discarded by the next enrollment, which re-reads fresh — the peer may have recorded past the straggler-time mark.
Record/emission split. The stage produces two streams per batch:
- emitted — strictly-ordered accepted ticks with real derived fields; strategy consumers and the recorder both see these
- recorded-late — distinct-key late tuples and out-of-session ticks (pre-open/post-close prints), recorded with synthetic derived fields (
volume = 0,durationSinceLast = 0) for dual-replica union coverage, never emitted to strategies. In/out-of-session classification stays derivable at read time fromtsplus the versioned calendar; the recording policy itself is tracked for reexamination.
Drops (EXACT_DUPLICATE, DUPLICATE_LTP_CONFLICT) are counted and logged — feed health never reacts to them.
QuestDB dedup key. A broker-provided monotonic discriminator is stored in the seq column alongside ts (ltt) and included in the dedup key: DEDUP UPSERT KEYS(ts, instrument_id, seq). For Upstox this is the cumulative vtt (volume traded today); for BitFlyer it is tick_id (a per-product sequence). The ordering gate guarantees the pair (ltt, seq) is unique per emitted tick, keeping same-millisecond trades distinct in storage rather than collapsing them via last-write-wins, and keeping re-ingest from the raw store idempotent (the same raw frame always produces the same (ltt, seq) key). The derived columns (volume, duration_since_last) are last-write-wins and non-authoritative — canonical consumers (backfill folds, frame totals via max(seq), the reconciliation replay) always recompute from ts/seq.
Raw-vs-QuestDB Reconciliation
backend-reconciliation is an after-the-fact cross-check, not part of the live path. It replays raw frames captured in the NATS raw store over a time window, decodes each to UpstoxMarketDataFeed, maps frames through the shared UpstoxFrameMapper, and folds each instrument's deduplicated union through the same TickNormalizer policy the live path runs (epoch-reset state, no hydration) — so ticks the live path dropped never appear as false "missing in QuestDB". The raw read is session-anchored: storage-time from the window's session open through window end plus a publish-lag margin, with the exchange-time predicate doing the filtering. It then diffs the passthrough fields ltt (QuestDB ts) and ltp against the market_data table. The join key is instrumentId + ltt + seq — matching the QuestDB dedup key — so same-millisecond trades are matched individually. The QuestDB read-back window is padded ±5s to tolerate storage-time-vs-exchange-time skew at the edges.
Union dedupe and conflicts. Both replicas capture the same trade: identical copies on (instrumentId, ts, seq) collapse to one expected tick. Same-key copies with different ltp make the price indeterminate — reported as INDETERMINATE_CONFLICT (counted and sampled, never resolved to an arbitrary expected price); a stored row carrying a conflicted key counts as conflicted storage.
Completeness. A window whose tail is more recent than a provisional 10-minute capture-lag bound is INDETERMINATE (raw frames may still be in flight); older windows are LIKELY_COMPLETE. COMPLETE is reserved for a durable per-source publish watermark (deferred).
Raw frame provenance. Each stored raw frame carries the publishing source's identity in a NATS source message header (RawMarketData.source). backend-app runs 2 replicas that both receive the same Upstox frame; content-hash message-ID dedup collapses the two publishes to one stored message, so the stored source is the winning source's id (≈random), not a complete list of sources that saw the frame. The header is additive — messages published before this change read back as source = "unknown" — so it is a lightweight audit tag, not a guarantee of provenance completeness. The publisher resolves the source id from the HOSTNAME environment variable, which k8s sets to the pod name.
The derived fields volume and duration_since_last are intentionally not compared — they are non-authoritative by contract and recomputed by the shared policy replay. Reconciliation remains Upstox-only (BitFlyer streams JSON and is not captured as raw bytes).
The logic is wired in backend-app (both RawDataRetriever and HistoricalDataProvider are already in scope there) and exposed on-demand through the AppDataBridge kRPC — folded as reconcileReport (summary counts + ≤50 sample diffs) and reconcileDiffs (full diff stream). backend-server proxies these as POST /reconcile and GET /reconcile/diffs; see operations.md for request shapes and the SSE retry caveat (QAPP-109).
Processor Contract
Processor.consume() is the orchestration boundary between live feed input and strategy execution.
At a high level it:
- runs under a
coroutineScopewith aCompletableDeferred<Nothing>fatal failure signal - processes feed batches sequentially through
routeFeedData() - watches
strategyRepository.strategyInstrumentIdsand enqueues payload-freeRouteCommand.TopologyDirtywakes - runs a single-owner
RouteCommandLoopthat handles all route mutations (reconciliation and degradation) - launches route-side jobs in a supervised child scope (
routeSideScope) - starts one route-local
StrategyEventCoordinatorcollector for each strategy output and signal stream
flowchart TD
FD["FeedData<MarketData>"]
subgraph processor ["Processor (backend-app)"]
FR["FeedRouter — routeFeedData()\nreads routesByInstrument snapshot\ntrySend to strategy mailboxes\noverflow → requestDegrade()"]
RCL["RouteCommandLoop\nsole writer of routingSnapshot\nTopologyDirty · TargetsDirty · Degrade · TeardownStrategy"]
RL["ReconcileLoop\nstrategyInstrumentIds → TopologyDirty"]
FW["FatalWatcher\nfatalFailure → coroutineScope fails → pod restart"]
RS["RoutingSnapshot (@Volatile)\nroutesByInstrument · routesByStrategy\nStrategyRoute: routeInstanceId · strategy · mailbox · executor · routeJobs · state"]
end
SM["Strategy Mailbox\nChannel<MarketData> · cap 1024\nProcessor-owned"]
subgraph executor ["StrategyExecutor (backend-processor) — one per strategy route"]
ER["Router coroutine\nmailbox.receiveAsFlow() → deliver() trySend"]
subgraph routes ["Instrument Routes"]
IR["InstrumentRoute\nProcessorMailbox (Channel, cap 1024)\nStrategyExecution (runningFold)\nBridge jobs: output → _strategyOutput · signals → _signals"]
IR2["InstrumentRoute … (one per instrument)"]
end
end
subgraph execution ["StrategyExecution (backend-strategy)"]
MDF["marketDataFlow (ProcessorMailbox.flow)"] --> RF["runningFold (accumulate state)"] --> SO["strategyOutput (MutableSharedFlow)"]
RF --> SS["strategySignals"]
end
FD --> FR
RL -->|"TopologyDirty"| RCL
FR -->|"Degrade"| RCL
FR -.->|"reads"| RS
RCL -.->|"writes"| RS
RS --> SM
SM -->|"receiveAsFlow"| ER
ER --> IR
ER --> IR2
IR --> MDF
IR -.->|"onRouteOverflow\nexecutionFailed"| FR
classDef feed fill:#e3f2fd,stroke:#1565c0,color:#0d47a1
classDef processorNode fill:#e8eaf6,stroke:#283593,color:#1a237e
classDef fatal fill:#ffebee,stroke:#c62828,color:#b71c1c
classDef mailbox fill:#fff3e0,stroke:#e65100,color:#bf360c
classDef executorNode fill:#f3e5f5,stroke:#6a1b9a,color:#4a148c
classDef executionNode fill:#e8f5e9,stroke:#2e7d32,color:#1b5e20
class FD feed
class FR,RCL,RL,RS processorNode
class FW fatal
class SM mailbox
class ER,IR,IR2 executorNode
class MDF,RF,SO,SS executionNode
style processor fill:#e8eaf6,stroke:#283593,color:#1a237e
style executor fill:#f3e5f5,stroke:#6a1b9a,color:#4a148c
style execution fill:#e8f5e9,stroke:#2e7d32,color:#1b5e20
style routes fill:#ede7f6,stroke:#7b1fa2,color:#4a148c
Processor-Owned Routing
The processor maintains a single @Volatile routing snapshot that is the source of truth for all routing decisions:
RoutingSnapshot
├── routesByInstrument: ImmutableMap<InstrumentId, ImmutableSet<StrategyId>> (Eclipse Collections)
└── routesByStrategy: ImmutableMap<StrategyId, StrategyRoute> (Eclipse Collections)
StrategyRoute
├── strategy: Strategy
├── mailbox: Channel<MarketData> (strategy-level mailbox)
├── executor: StrategyExecutor
├── routeJobs: RouteJobs (signalsJob, outputJob, tradeJob, stoppingWatcherJob?)
└── state: RouteState (Active / Degraded / Stopping)
RouteState
├── Active
├── Degraded(cause, degradedAt, hasOpenPositions)
└── Stopping(retainedInstrumentIds) (strategy removed but positions still open)
Single-owner mutation: Only the RouteCommandLoop coroutine mutates routingSnapshot. The FeedRouter and ReconcileLoop coroutines only read it or enqueue commands. The @Volatile annotation ensures visibility across the writer/reader boundary. The routesByInstrument and routesByStrategy maps use Eclipse Collections' ImmutableMap, which provides structural immutability — the maps cannot be modified after construction, and every mutation produces a new ImmutableMap instance via newWithKeyValue() / newWithoutKey().
Route commands are sent through an unlimited channel:
private val routeCommands = Channel<RouteCommand>(Channel.UNLIMITED)
This is a control queue, not a back-pressure boundary — it must never overflow.
Feed Routing
The FeedRouter coroutine processes feed data sequentially:
- For each
MarketDatain aFeedDatabatch, look up the instrument inroutesByInstrument - For each strategy mapped to that instrument,
trySendinto the strategy's mailbox - If
trySendfails (mailbox full), callrequestDegrade(strategyId, routeInstanceId, cause)— this deduplicates againstpendingDegradationby route incarnation before enqueuing aRouteCommand.Degradecommand - Skip routes that are in
Degradedstate (bothActiveandStoppingroutes receive ticks)
This replaces the previous flatMapMerge { it.data.asFlow() } approach. The key property is that trySend is non-suspending: a slow consumer never blocks the router — it degrades the affected route instead.
Pending degradation deduplication: Because the route is not marked Degraded until the command loop processes the Degrade command, a naive implementation would enqueue a new Degrade command on every failed trySend, building an unbounded control-plane backlog. This applies to both strategy-mailbox overflow (detected in routeFeedData()) and instrument-mailbox overflow (detected in the executor's router and signaled via onRouteOverflow).
To prevent this, the processor centralizes all degrade requests through a requestDegrade(strategyId, routeInstanceId, cause) helper. This helper uses a ConcurrentHashMap.newKeySet<Long>()-backed pendingDegradation set keyed by route instance ID (not strategy ID) for atomic check-and-add: add() returns false if the route incarnation is already pending, so only the first overflow for a given route incarnation enqueues a Degrade command. Keying by route instance ID prevents a stale degrade from an old route incarnation from degrading a newly recreated route for the same strategy. The command loop removes the route instance from pendingDegradation when processing Degrade, TeardownStrategy, or immediate teardown in TopologyDirty. If a Degrade command arrives for a route whose routeInstanceId no longer matches (the route was recreated), the command is silently discarded.
flowchart LR
subgraph sources ["Command Sources"]
RL["ReconcileLoop\nstrategyInstrumentIds watcher\n→ TopologyDirty (payload-free)"]
FR["FeedRouter\nstrategy mailbox overflow\n→ Degrade(strategyId, routeInstanceId)"]
SE["StrategyExecutor\nonRouteOverflow → Degrade\nonTargetsDirty → TargetsDirty\nexecutionFailed → InstrumentExecutionFailed"]
SW["StoppingWatcher\nhasOpenPositions filter\n→ TeardownStrategy(strategyId)"]
RJ["Route-Side Jobs\nsignals/output/trade failure\n→ Degrade(strategyId, routeInstanceId)"]
end
CH["routeCommands\nChannel.UNLIMITED\nTopologyDirty · TargetsDirty · Degrade · TeardownStrategy"]
subgraph loop ["RouteCommandLoop (sole writer)"]
RC["TopologyDirty\nRead `.value` at consumption\nRemove / Reactivate / Add routes\nRebuild instrument index from views"]
DG["Degrade\nMark Degraded · Close mailbox\nCancel jobs + executor\nRebuild index · Check open positions"]
TD["TeardownStrategy\nClose mailbox · Cancel jobs\nCancel executor · Remove route"]
IR["TargetsDirty\nRebuild instrument index\nfrom the routes' delivery-target views"]
end
RS["RoutingSnapshot\n(@Volatile)"]
RL --> CH
FR --> CH
SE --> CH
SW --> CH
RJ --> CH
CH --> RC
CH --> DG
CH --> TD
CH --> IR
loop --> RS
classDef source fill:#e3f2fd,stroke:#1565c0,color:#0d47a1
classDef channel fill:#fff3e0,stroke:#e65100,color:#bf360c
classDef handler fill:#f3e5f5,stroke:#6a1b9a,color:#4a148c
classDef snapshot fill:#e8f5e9,stroke:#2e7d32,color:#1b5e20
class RL,FR,SE,SW,RJ source
class CH channel
class RC,DG,TD,IR handler
class RS snapshot
style sources fill:#e3f2fd,stroke:#1565c0,color:#0d47a1
style loop fill:#f3e5f5,stroke:#6a1b9a,color:#4a148c
Route Command Loop
The RouteCommandLoop coroutine is the single owner of route state mutations:
for (command in routeCommands) {
routingSnapshot = when (command) {
RouteCommand.TopologyDirty -> reconcileRoutes(current, strategyInstrumentIds.value (read once at consumption), routeSideScope)
RouteCommand.Degrade -> degradeRoute(current, strategyId, routeInstanceId, cause, fatalFailure)
RouteCommand.TeardownStrategy -> teardownStrategy(current, strategyId)
}
}
On RouteCommand.TopologyDirty:
The wake is payload-free: the handler reads the repository's current strategyInstrumentIds.value exactly once at consumption and wires that. The former payload-carrying Reconcile captured its snapshot at emission receipt, so a delayed command could wire a superseded view; with no payload there is nothing to be stale — a delayed wake and a fresh one are indistinguishable. Boundary (stated non-goal): consumption-time reads remove the delayed-payload race; they do NOT make a suspending topology reconciliation transactional — membership can advance during a suspending executor build, and a later wake converges.
- Removal: strategies in the current snapshot but absent from the desired repository state:
- If the executor has open positions → transition to
Stopping(retainedInstrumentIds), launch aStoppingWatcherthat postsTeardownStrategywhen positions flatten - If no open positions → immediate teardown: close mailbox, cancel jobs, cancel executor, remove from snapshot
- Reactivation:
Stoppingroutes whose strategies reappear indesiredare transitioned back toActive(the stopping watcher is cancelled) - Addition: strategies in the desired state but absent from the current snapshot get a new strategy mailbox, executor, and route-side jobs
- Instrument index rebuild:
routesByInstrumentis rebuilt solely from the routes' pulled delivery-target views (executor.deliveryTargetIds— the coordinator's current authoritative desired set ∪ its routable records). The coordinator fires the payload-freeonTargetsDirtywake after every inbox event, so the index is always rebuilt after the executor's state has changed.Degradedroutes are excluded; aStoppingroute's authoritative desired is empty, so its view reduces to its retained records without a special case.
On RouteCommand.Degrade:
- Mark the route as
Degraded(cause, hasOpenPositions)(works for bothActiveandStoppingroutes — aStoppingroute that overflows with open positions must still reach the fail-fast path) - Close the strategy mailbox (terminates the executor's input)
- Cancel the route-side jobs (signals, output, trade, stoppingWatcher)
- Cancel the executor (
executor.cancel()) - Rebuild the instrument index using
rebuildRoutesByInstrumentFromViews(routesByStrategy) - If the degraded route has open positions →
failProcessor()completes the fatal failure signal
On RouteCommand.TeardownStrategy:
- Close the strategy mailbox
- Cancel the route-side jobs (signals, output, trade, stoppingWatcher)
- Cancel the executor
- Remove the route from the snapshot
- Rebuild the instrument index
On RouteCommand.TargetsDirty:
- Rebuild the instrument routing index from the routes' delivery-target views — the executor's state has changed (a route published, purged or failed), so its view is now up-to-date
- No changes to
routesByStrategy— this is a pure index refresh
The wake is payload-free and fired by the coordinator after EVERY inbox event: route addition/removal does not cross the executor boundary as commands; the processor's routing index is a pull-derivation of the coordinators' delivery-target views, so a record's appearance in or disappearance from the views is the entire routing consequence. What still crosses the boundary is exactly the failure fact, enqueued through the executor's RouteLifecycleEventSink : the sink is invoked by the route coordinator's serialized event loop while holding its routingLock, in the same critical section that performed the teardown mutation, so the failure is always observed after the state change it describes. A fatal teardown removes the route as coordinator-internal record mutation in the same critical section that enqueues the failure — the former removal-before-failure event ordering is now mutation order inside the coordinator. Without the wake, the processor's routing index would be derived from a stale snapshot of executor state, causing ticks to be dropped (for additions) or routed to purged routes (for removals); a causal convergence regression and the processor-level model suite pin the wake as the sole load-bearing rebuild path (a mutation proof disabled it and failed exactly the convergence assertions).
Batch Ordering
There are two separate ordering rules:
- Batches are processed sequentially because the
FeedRoutercoroutine collects the feed flow in order. - Ordering across instruments inside the same batch is not guaranteed, because
routeFeedData()iterates overfeedData.dataand routes each instrument independently.
That means:
- a later batch cannot overtake an earlier batch
- two instruments inside one batch may be observed in either order
The event-contract tests in backend-app deliberately assert the parts of ordering that are guaranteed and avoid asserting the parts that are not.
StrategyExecutor
Each StrategyExecutor owns one strategy and manages one StrategyExecution<T> per active instrument.
Its responsibilities are:
- receive all market data for its strategy through a single
Channel<MarketData>(the strategy mailbox) - fan out internally to per-instrument
ProcessorMailboxinstances (boundedChannel, capacity 1024) - signal overflow via
onRouteOverflow— the processor decides what to do (degrade the route) - expose
hasOpenPositions(): Booleanso the processor can decide escalation severity - expose
hasOpenPositionsFlow: Flow<Boolean>for reactive position monitoring (used byStoppingWatcher) - expose
cancel()to stop all executor-owned coroutines when the route is torn down - expose
start()to begin the router and instrument reconciliation loop after the processor has subscribed to outputs/signals
Strategy Mailbox Contract
The executor's public contract is:
StrategyExecutor(
strategyMailbox: Channel<MarketData>,
membership: StateFlow<List<StrategyInstrumentIds>>,
resolveFeedContext: suspend (InstrumentId) -> InstrumentFeedContext?,
historicalDataProvider: HistoricalDataProvider,
onRouteOverflow: (InstrumentId, Throwable) -> Unit,
lifecycleSink: RouteLifecycleEventSink,
)
: the coordinator — not the worker — is the sole per-strategy consumer
of the raw repository StateFlow; membership crosses as a payload-free
MembershipDirty wake and the authoritative .value (filtered to this
strategy) is read at wake consumption, so a wake enqueued before the
repository advanced can never reconcile a stale view (the hazard is
structurally absent). "Consumed once" is scoped to per-strategy membership —
the processor keeps its own separate repository collector for
strategy-topology reconciliation — a payload-free TopologyDirty wake on the
same command channel, reading .value at consumption. Feed-context resolution (the suspending
feed-store lookup) runs on the worker's slow initialization lane, after the
InitStarted handshake; a null result (instrument desired but no feed row)
takes the benign terminal FeedUnresolvable phase — no route, no health
degradation — and a fresh resolution happens only after a leave-and-re-add.
The exception is deliberate and fully terminal: the instrument is
unroutable, emits an actionable warn (naming the instrument, its strategy,
and the removal/re-add remedy), and is retried only after it leaves the
desired set and is re-added — no automatic retry, since a missing feed row
is a configuration gap, not a transient pipeline failure. It informs
through logs only today: derived health stays HEALTHY and readiness stays
true (the Failed ∧ desired ⇒ DEGRADED contract deliberately
excludes FeedUnresolvable); surfacing it through an explicit UI/health
projection is a recorded follow-up. Resolution itself is
per-instrument — one feed-store lookup per instrument on the sequential
slow lane, in exchange for loud per-instrument failure isolation (a
resolver exception fails exactly one instrument) — a recorded trade-off to
measure or batch if startup-scale tests expose it.
The processor's routing index is a pull-derivation of the
coordinator's delivery-target view, and initialization is gated on
publication. Three primitives: (1) deliveryTargetIds() — the coordinator's
lock-protected pull view (the authoritative desired set ∪ the routable
records, composed wholesale), read at pull so it can never serve a stale
snapshot; the TargetsDirty command's handler rebuilds the instrument index
from the views alone (payload-free — the class is structurally
absent, the command carries no route reference). (2) onTargetsDirty — a
separate callback, deliberately NOT a method on the sink (whose calls are
lock-linearized): the wake is payload-free and coalescible and fires strictly
after routingLock is released, after every inbox event. (3) The per-init
acknowledgement: InitStarted's proceedAck is PARKED — the
coordinator registers it under the lock (validating the token, installing the
ingress, marking the record Initializing), releases the lock, fires the
dirty wake, and the worker waits; only the processor's centralized
publishRoutingSnapshot (assign → retain the exact per-route target sets →
onTargetsPublished callback even on a no-op publication) completes the ack
— route-qualified against the retained target sets, never a re-pull —
strictly outside the lock. Backfill, and its
high-water read, begin only after publication, so a tick dropped in the
publication-lag window predates the read and is backfill-recoverable. The
callback is completion-only: absence from a publication is not
supersession — a publication's content can predate the park by construction
(a topology reconciliation reads membership, then suspends on an executor
build while membership advances, and its snapshot-exact targets are
deliberately not re-pulled), and a no-op publication changes nothing — so a
parked ack stays parked until a containing publication arrives. A parked ack is cancelled
only by an explicit membership-invalid transition: the park-time
validation (the repository advanced past the sweep's dispatch) or the
membership-wake re-check. Both normalize the record under the routing lock
first — reverted to Buffering, the registration's drain returned FIFO, the
ingress reference cleared — and cancel the deferred strictly after it, so
the worker's closed ingress can never receive a live tick; the sweep
continues inline and the next membership pass purges or re-dispatches the
reverted record.
The processor provides the strategy mailbox, the overflow callback, and the route-lifecycle event sink. The sink's contract: invoked by the route coordinator's serialized event loop under its routingLock in the same critical section as the routing mutation the fact describes — the boundary is the per-instrument failure fact and the drain lifecycle facts (drainRetained on entering DrainingForOpenPosition, drainCleared on every exit: position flat, re-add heal, or the draining record failing) — and contractually non-suspending, non-blocking, non-throwing and non-re-entering — the processor's implementation only enqueues into its unlimited command channel. A drain fact carries a publication ack: the coordinator awaits it AFTER releasing routingLock, and the processor's command loop completes it once the fact is processed — applied to the registry and either durably written or contained (a failed or timed-out write keeps the applied in-memory set, degrades the Processor's health, and retries on the next transition or heartbeat, converging after a successful rewrite) — or skipped for a stale route, in which case nothing is applied. Ack completion is completion of processing, not proof of durable peer visibility: under a contained write the registry file still predates the transition until a later successful rewrite, so the server's unsubscribe gate can read a pre-transition file in that window — the accepted visibility gap, carried by the degraded health signal. The ownership is split: the per-strategy RouteCoordinator is the sole owner of the logical route records — each (strategy, instrument) is one typed record carrying a RouteToken (the processor's route-instance id plus a coordinator-local record generation allocated under routingLock), membership intent, a sealed RoutePhase (Buffering, Initializing, Active, AwaitingHistory, FeedUnresolvable, Failed, DrainingForOpenPosition) and retry intent — while the executor is reduced to the worker: it owns each ingress's lifecycle (creation, buffering, handoff, closing) and reports results back with the token through the acknowledged handshake protocol (InitStarted/ InitReady/ ExecutionReady/ AwaitingHistory/ Failed/ PositionFlat events on a non-dropping, unbounded inbox consumed by the coordinator's single loop, which serializes only state transitions — backfill and initialization run outside it). The worker must not begin slow work until its InitStarted acknowledgement completes (the coordinator publishes the buffering ingress, drains the record's buffered ticks into it and flips the record to Initializing first — making the publish-before-start ordering structural), and supersession or loop shutdown cancels the acknowledgement so the worker fails fast. Every asynchronous completion validates its token against the record before acting — a completion whose token is stale (its record was replaced by a removal/re-enrollment) is ignored. Pre-publication ticks are buffered in the record's own storage under the coordinator's aggregate bound (262,144 across all buffering records, loud overflow), and the record's pending ingress (65,536) stays per-instrument; the buffering→active ingress handoff is the acknowledged InitReady flip barrier (the router prefers the buffering ingress while registered, and the worker closes it only after the flip — FIFO holds across the switch). The execution view is single-writer: only the coordinator's loop publishes it (a plain immutable value per transition). Membership arrives as a payload-free wake — the coordinator itself is the sole collector of the raw repository StateFlow and reads the authoritative .value (filtered to its strategy) at wake consumption; the worker holds no membership mirror at all, so a stale payload can never be processed (the hazard is structurally absent). The processor does not own route state — it signals overflow conditions, receives the per-instrument failure fact through the sink, and rebuilds its routing index from the executors' delivery-target views on the post-event targets-dirty wake (route addition/removal do not cross the boundary), ensuring ticks are only routed to instruments the executor is ready to receive.
The coordinator's stable record stores a single sealed phase/resource state, not independent nullable ingress and execution fields. Initializing requires buffering ingress. Both Active and DrainingForOpenPosition require an execution plus either buffering or steady delivery: execution publication precedes the fold's InitReady handoff, so an Active route does not imply steady delivery. Drain, heal, and handoff replace the whole state under routingLock, preserving the record identity and token; lock-free readers capture the volatile state once. Each drain transition publishes its fact through the lifecycle sink into the shared-volume route-state registry (see the feeds section): entry retains the (strategy, instrument) pair, every exit clears it, and a full route teardown clears that strategy's drains — a cancelled executor's drains can never converge by ticks, and in the starved case (no persisted position) nothing else would ever clear them. The public RoutePhase remains a resource-free projection for health/readiness. Local ticks belong only to LocalBuffer; the exceptional registration-overflow state retains its ingress and still projects to Buffering until executor degradation. deliverTick exhaustively dispatches these states, with explicit drops only for awaiting-history, feed-unresolvable, and failed records; it needs neither nullable delivery fallbacks nor non-null assertions. Token/source-state checks, mailbox liveness, locking, and acknowledgment ordering remain runtime protocol responsibilities.
The executor is constructed in a stopped state — no coroutines are launched until start() is called. The processor subscribes to executor.strategyOutput and executor.signals before calling executor.start(), ensuring no emissions are lost when instrument routes are added and their executions start producing output.
Internal Routing
The executor runs a single router coroutine (launched by start()) that:
- Collects from
strategyMailbox.receiveAsFlow() - Delegates the per-tick decision to the coordinator's
deliverTick(marketData)— the entire routing decision (which ingress, buffering storage, or drop) runs inside the coordinator under itsroutingLock, together with all routing mutations - If
deliverTickreports overflow, callsonRouteOverflow(instrumentId, cause)
Per-Route Bridge Jobs
Each instrument route has two bridge jobs that forward the execution's output and signals to the executor's shared flows:
outputBridgeJob: collects fromexecution.strategyOutputand emits to_strategyOutputsignalBridgeJob: collects fromexecution.strategySignalsand emits to_signals
These bridge jobs replace the previous executor-wide flatMapLatest { merge() } fan-in. The per-route approach eliminates the re-subscription gap: when a new instrument route is added, the bridge jobs are launched with CoroutineStart.UNDISPATCHED before execution.start() is called. UNDISPATCHED ensures the coroutine executes synchronously up to its first suspension point (the collect(...) call), so the subscription is established before start() returns. When the first tick arrives and the execution produces its first output/signal, the bridge jobs are already subscribed. No emissions are lost.
When an instrument route is removed, the bridge jobs are cancelled before the execution is cancelled, ensuring clean teardown.
Runtime Lifecycle Behavior
The runtime-lifecycle tests currently document an important behavior contract:
- removing an instrument or strategy does not immediately kill an execution if that execution still owns an open position
- the retained execution stays alive long enough to emit the managed exit or reversal
- re-adding before cleanup preserves the existing retained execution
- re-adding after cleanup creates a fresh execution and re-runs initialization/backfill
- these drain/removal transitions apply immediately on every membership change, even while another instrument's initialization (feed-context resolution or backfill) is still in flight: the coordinator's serialized sweeps wait for the in-flight initialization's terminal event, but every membership wake runs the retained-routes pass against the same authoritative snapshot the wake captured right away — so a draining route's watcher is armed and a flat removed route is torn down without waiting behind an unrelated, possibly hung, initialization
- every drain transition also publishes its fact cross-replica through the lifecycle sink (
drainRetainedon entry,drainClearedon every exit — position flat, re-add heal, or the draining record failing; the route-command loop applies it to the per-writer registry file with the publication ack awaited after the routing lock), and a full route teardown clears that strategy's drains — a cancelled executor's drains can never converge by ticks, and in the starved case (no persisted position) nothing else would ever clear them. The registry file — the replica's routing-index keys (converged after every snapshot publication: written only when the set changed or a pending write needs converging, so a peer replica still holding a LIVE route is protected too) plus its drain set — is rewritten atomically on every routed-set or drain transition and on a 60-second heartbeat (the server-side lease window is 3× that, so a crashed pod's routes and drains stop blocking unsubscription within the window; a replaced or scaled-down pod's old file likewise stops blocking within the window — fail-safe over-blocking after every rollout, accepted); a contained write keeps the applied in-memory sets, degrades the Processor's component health until the next rewrite converges, and never fails the process
This behavior is intentional and is one of the main reasons the processor runtime-lifecycle suite exists.
Executor Lifecycle
The executor owns its CoroutineScope (created with SupervisorJob as a child of the calling coroutine's job). All internal coroutines (router, the worker's two command runners — the sequential initialization lane and the immediate fast lane, the coordinator's event loop and its sole membership wake collector, per-route bridge jobs) are launched in this scope.
The executor is constructed in a stopped state. The processor must call start after subscribing to executor.strategyOutput and executor.signals (via launchRouteJobs). The processor's route-side collector jobs (signals, output, trade) are launched with CoroutineStart.UNDISPATCHED, ensuring their subscriptions are established synchronously before start is called. start launches the coordinator's serialized event loop and its sole membership wake collector, the router coroutine, and the worker's two command runners — the sequential initialization lane and the immediate fast lane (watcher arming and transport cleanup never queue behind a slow initialization) — which together trigger the first sweep and begin routing ticks.
Calling cancel() on the executor cancels its scope, which stops all internal coroutines. The processor calls executor.cancel() in the immediate teardown path, the deferred teardown path (TeardownStrategy), and the degradation path.
StrategyExecution
StrategyExecution<T> is the concrete state machine for one strategy on one instrument.
It:
- pre-fills historical state through
HistoricalDataProvider - runs a
runningFoldover liveMarketData - emits one
StrategyOutputfor each accepted live tick - derives long and short signal streams from state transitions
In production, HistoricalDataProvider is backed by Recorder.historicalDataProvider, so strategy backfill reads from QuestDB.
StrategyExecution is constructed in a stopped state — the runningFold and signal collection coroutines are not launched until start() is called. The per-route bridge jobs in StrategyExecutor are launched with CoroutineStart.UNDISPATCHED before start() is called, ensuring their subscriptions are established synchronously. This makes the start() barrier deterministic: no early emissions can be lost because the subscribers are guaranteed to be attached before any data flows.
flowchart LR
subgraph overflow ["Overflow Sources"]
A["Strategy Mailbox Overflow\nrouteFeedData() → trySend fails\nChannel<MarketData> · cap 1024"]
B["Instrument Mailbox Overflow\nexecutor router → deliver() fails\nProcessorMailbox · cap 1024"]
C["Route-Side Job Failure\nsignals/output/trade collector\nthrows non-CancellationException"]
end
subgraph dedup ["Dedup Gate"]
RD["requestDegrade()\npendingDegradation: ConcurrentHashMap.newKeySet()\nkeyed by routeInstanceId\nAt most 1 pending Degrade per route incarnation\nStale incarnations automatically rejected"]
end
subgraph cmdloop ["Command Loop"]
DR["degradeRoute()\n① Verify routeInstanceId matches\n② Mark route state = Degraded\n③ Close strategy mailbox\n④ Cancel route-side jobs\n⑤ Cancel executor\n⑥ Rebuild routesByInstrument\n⑦ Remove from pendingDegradation\n⑧ Check hasOpenPositions()"]
end
A --> RD
B --> RD
C --> RD
RD -->|"RouteCommand.Degrade"| DR
DR -->|"!hasOpenPositions"| OK["No Open Positions\nRoute removed from routingSnapshot\nSibling strategies continue\nProcessor remains healthy"]
DR -->|"hasOpenPositions"| FF["Open Positions Present\nfailProcessor() → fatalFailure\nProcess terminates\nKubernetes restarts pod"]
classDef overflow fill:#ffebee,stroke:#c62828,color:#b71c1c
classDef dedup fill:#fff3e0,stroke:#e65100,color:#bf360c
classDef cmd fill:#f3e5f5,stroke:#6a1b9a,color:#4a148c
classDef ok fill:#e8f5e9,stroke:#2e7d32,color:#1b5e20
classDef fail fill:#ffcdd2,stroke:#c62828,color:#b71c1c
class A,B,C overflow
class RD dedup
class DR cmd
class OK ok
class FF fail
style overflow fill:#ffebee,stroke:#c62828,color:#b71c1c
style dedup fill:#fff3e0,stroke:#e65100,color:#bf360c
style cmdloop fill:#f3e5f5,stroke:#6a1b9a,color:#4a148c
Degradation And Fail-Fast
The degradation path is:
- Overflow detected →
requestDegrade(strategyId, routeInstanceId, cause)— atomically checkspendingDegradation(keyed by route instance ID) and enqueuesRouteCommand.Degradeonly if not already pending for that route incarnation. This covers both strategy-mailbox overflow (inrouteFeedData()) and instrument-mailbox overflow (viaonRouteOverflowcallback from the executor). - Command processed on the single-owner
RouteCommandLoop—pendingDegradationis cleared. If the route'srouteInstanceIdno longer matches the command's (the route was recreated), the stale command is silently discarded: - Route marked
Degraded(cause, hasOpenPositions) - Strategy mailbox closed (terminates the executor's input)
- Route-side jobs (signals, output, trade) cancelled
- Executor cancelled (
executor.cancel()) - Escalation decision:
- Open positions →
failProcessor()completes thefatalFailuresignal →ProcessorFatalWatcherpropagates the exception →coroutineScopefails → Kubernetes detects failure → pod restart (fail-fast) - No open positions → route is removed from active routing, processor stays healthy for sibling strategies
This means one degraded strategy does not stall sibling strategies unless it has open positions that require the processor to fail-fast.
Strategy Removal With Open Positions
When a strategy is removed from the repository while its executor has open positions:
- The route transitions to
Stopping(retainedInstrumentIds)instead of being torn down immediately (routes already inStoppingare skipped to prevent duplicate watchers) - The strategy mailbox stays open — the executor continues receiving ticks for retained instruments
- A
StoppingWatchercoroutine monitorsexecutor.hasOpenPositionsFlowand postsRouteCommand.TeardownStrategywhen positions flatten. The watcher double-checksexecutor.hasOpenPositions()before posting to guard against prematurefalseemissions fromhasOpenPositionsFlowbefore executor initialization routeFeedData()continues routing toStoppingroutes (onlyDegradedroutes are skipped)- The instrument index uses
executor.currentInstrumentIdsfor bothActiveandStoppingroutes (the index is rebuilt from the routes' delivery-target views on the post-event targets-dirty wake —: route addition/removal no longer cross the boundary as commands) - When a retained instrument's position flattens, the worker's watcher reports a flat level-set with the record's token, the coordinator applies the explicit removal transition (the per-route bridge jobs are cancelled, the execution is cancelled), and the post-event targets-dirty wake — fired after the inbox event, outside the lock — rebuilds the routing index to remove the flattened instrument
- When a new instrument route is published, per-route bridge jobs are launched before
execution.start, andexecution.startitself is guarded by an irreversible per-transport start/cleanup gate: a current-token fast-lane cleanup can run while the initialization is suspended at its publication ack — both operations target the same valid token — and once cleanup has claimed the transport, the post-ack start is skipped on the cancelled execution instead of invoked. The routing index learns of the new instrument through the delivery-target views: the post-event targets-dirty wake (fired after the publication's inbox event, outside the lock) rebuilds it to include the new instrument. The execution's fatal-error handler is likewise bound to the route it was published under: if a removal/re-enrollment has replaced the route by the time the failing pipeline's callback runs, the callback is stale (the originating execution is gone and its failure is moot for the fresh canonical rebuild) and is ignored with a warning instead of tearing down the replacement. 's critical-section discipline makes the orderings structural — a teardown's route removal and its failure fact are one coordinator-internal mutation followed by the enqueued failure — so the executor-local incarnation machinery and the processor's mirrored incarnation guards are deleted; the failure-fact order regression pins what still crosses at the executor level. What remains necessary is the route-instance gate: the failure command carries the owning strategy route'srouteInstanceIdand is ignored when that route is no longer current, because executor cancellation does not join and a delayed failure from a dead executor can otherwise land after its replacement's first publication. - When
TeardownStrategyis processed: mailbox closed, jobs cancelled, executor cancelled, route removed - If a
Stoppingroute's mailbox overflows,degradeRoute()handles it — if it has open positions, fail-fast is triggered
If a Stopping route's strategy reappears in the repository (re-added), the route transitions back to Active and the stopping watcher is cancelled.
Route-side jobs are launched in a supervised child scope (routeSideScope) with inline try/catch: if a signals, output, or trade job fails with a real exception (not CancellationException), it enqueues a RouteCommand.Degrade for that strategy.
stateDiagram-v2
[*] --> Active
Active --> Stopping : removed from repo + hasOpenPositions
Stopping --> Active : strategy reappears in repo
Active --> Degraded : mailbox overflow / route-side job failure
Active --> Gone : removed from repo + !hasOpenPositions
Stopping --> Degraded : overflow while stopping
Stopping --> Gone : TeardownStrategy (positions flattened)
Degraded --> Gone : reconciliation removes route
Degraded --> FailFast : hasOpenPositions
classDef active fill:#c8e6c9,stroke:#2e7d32,color:#1b5e20
classDef stopping fill:#ffe0b2,stroke:#e65100,color:#bf360c
classDef degraded fill:#ffccbc,stroke:#d84315,color:#bf360c
classDef gone fill:#eceff1,stroke:#546e7a,color:#37474f
classDef failfast fill:#ffcdd2,stroke:#c62828,color:#b71c1c
class Active active
class Stopping stopping
class Degraded degraded
class Gone gone
class FailFast failfast
Readiness And Health
The processor's readiness and health are route-aware:
| Condition | isReady() |
componentHealth() |
|---|---|---|
| No core jobs | false |
INACTIVE |
| Any core job not active (fatalWatcher, routeCommandLoop, feedRouter, reconcileLoop) | false |
BLOCKED |
| Degraded route with open positions | false |
BLOCKED |
| Degraded route without open positions | true |
DEGRADED |
| Stopping route (positions winding down) | true |
DEGRADED |
| All routes active | true |
HEALTHY |
Readiness reflects processor-core health, not every route-side job. A degraded route without open positions does not make the processor unready. A stopping route (strategy removed but positions still open) does not make the processor unready — it is still functioning correctly, just winding down.
Failed instruments (the per-instrument teardown path) surface as DEGRADED health naming the failed (strategy, instrument) pairs — derived at projection time from the executors' tokenized route-phase records: each executor exposes a locked instrumentRouteStates snapshot (every record's RouteToken, membership intent, and RoutePhase), and componentHealth projects the Failed records of all non-Degraded routes through a live-membership filter that reads the repository flow at projection time. Stale executor views are rejected by token: a state contributes only when its RouteToken's route instance matches the CURRENT routing snapshot's route for the strategy — read after the states — so a health poll preempted through a strategy remove/re-add cannot project the old executor's retained Failed record against the recreated route. A failed pair therefore stops surfacing when the instrument re-initializes under the same strategy (the record leaves Failed at publication), when the operator removes it from the strategy's desired membership (the filter excludes it — this covers the convergence window where the executor's record is still Failed and no route exists to remove — deleted the route-removed boundary event regardless, since a failed instrument has no route — and also a failure command arriving after its instrument's removal, e.g. a slow backfill failing post-removal;), or when the strategy route is degraded or torn down (Degraded routes are excluded from the scan — their health is carried by the route-state branches above; a Stopping route's strategy is absent from live desired membership, so the filter excludes its records anyway). DrainingForOpenPosition is deliberately not a failure. FeedUnresolvable is deliberately excluded from the projection as well — an unresolvable feed is a configuration gap, not a pipeline failure, so it never degrades health or readiness; it informs through the warn log only, with an explicit UI/health projection as a recorded follow-up. There is no mirrored failure registry to go stale: the stale-payload clear hazard is structural under derivation, and the route-wiring stale-payload hazard no longer exists — the topology wake carries no payload and reads the repository's current .value at consumption (the class, deleted at rather than defended). A Failed phase affects component health only — never isReady. Under the event orderings are structural — a route-added event can never arrive after a failure of the route it announces, and a removal can never arrive after a newer route's failure — so no incarnation bookkeeping is needed to reject stale announcements or stale failures; the only remaining staleness gate is route-instance identity: every lifecycle command (added, removed, failed) carries the owning strategy route's instance id and is ignored when that route is no longer current, because executor cancellation does not join and a delayed command from a dead executor can otherwise land after its replacement's first announcement. For a stale-instance failure command the open-position fail-fast escalation deliberately still fires: a failed execution's open position is unmanaged regardless of which route or executor owned it, so a conservatively-stale hasOpenPosition flag fails the processor rather than risking a zombie position. That escalation (and the future durable failure record) is transition-sourced — it fires from the command-loop handler upon the lossless, lock-ordered event, never from the conflatable health projection.
flowchart TD
subgraph hierarchy ["Mailbox Hierarchy"]
FD["FeedData<MarketData>"] --> PR["Processor · routeFeedData()"] -->|"trySend"| SM["Strategy Mailbox\nChannel<MarketData> · cap 1024\nowned by Processor (backend-app)"]
SM -->|"receiveAsFlow"| ER["StrategyExecutor · Router coroutine"] -->|"deliver() trySend"| IM["Instrument Mailbox\nProcessorMailbox · Channel · cap 1024\nowned by StrategyExecutor (backend-processor)"] -->|"mailbox.flow"| SX["StrategyExecution\nrunningFold → strategyOutput · signals"]
end
SM -.->|"overflow"| D1["requestDegrade()"]
IM -.->|"overflow"| D2["onRouteOverflow()"]
classDef data fill:#e3f2fd,stroke:#1565c0,color:#0d47a1
classDef processor fill:#e8eaf6,stroke:#283593,color:#1a237e
classDef mailbox fill:#fff3e0,stroke:#e65100,color:#bf360c
classDef executor fill:#f3e5f5,stroke:#6a1b9a,color:#4a148c
classDef execution fill:#e8f5e9,stroke:#2e7d32,color:#1b5e20
classDef overflow fill:#ffebee,stroke:#c62828,color:#b71c1c
class FD data
class PR processor
class SM,IM mailbox
class ER executor
class SX execution
class D1,D2 overflow
style hierarchy fill:#fafafa,stroke:#9e9e9e,color:#424242
flowchart TD
S1["① Processor · launchRouteJobs\nsignalsJob · outputJob · tradeJob\nlaunched with CoroutineStart.UNDISPATCHED\n→ collect reached synchronously"] --> S2["② strategyExecutor.start\nLaunches coordinator loop (sole membership wake collector) +\nrouter + worker init and fast command lanes\nProcessor already subscribed to outputs/signals"] --> S3["③ Coordinator sweep → worker init, per instrument\na. InitStarted(token, ingress) acknowledged\n (ingress published, buffered ticks drained)\nb. Construct StrategyExecution (stopped)\nc. Launch bridge jobs (UNDISPATCHED) → subscribe synchronously\nd. ExecutionReady acknowledged → route published \ne. execution.start → pipelines launch after subscribers attached"] --> S4["④ Failure fact → RouteCommand.InstrumentExecutionFailed\nenqueued under routingLock with the mutation \nPost-event targets-dirty wake (stage c3) rebuilds\nroutesByInstrument from the routes' delivery-target views"]
classDef step1 fill:#e3f2fd,stroke:#1565c0,color:#0d47a1
classDef step2 fill:#e8eaf6,stroke:#283593,color:#1a237e
classDef step3 fill:#f3e5f5,stroke:#6a1b9a,color:#4a148c
classDef step4 fill:#e8f5e9,stroke:#2e7d32,color:#1b5e20
class S1 step1
class S2 step2
class S3 step3
class S4 step4
Mailbox Hierarchy
The routing model uses a two-level mailbox architecture:
| Level | Type | Capacity | Owner | Location |
|---|---|---|---|---|
| Strategy mailbox | Channel<MarketData> |
1024 | Processor | backend-app (raw Channel) |
| Instrument mailbox | ProcessorMailbox (wraps Channel<MarketData>) |
1024 | StrategyExecutor | backend-processor (internal) |
The strategy mailbox is a raw Channel in backend-app because ProcessorMailbox is internal to backend-processor. The executor's instrument mailboxes use ProcessorMailbox since they're within the same module.
Both levels use trySend for non-suspending delivery. Overflow is detected immediately via ChannelResult.isFailure, not by suspending the sender. This means a slow consumer never blocks the router — it degrades the affected route instead.
Timing And Event Contracts
There are three time concepts that matter across the pipeline:
1. StrategyOutput.time
This is the logical strategy time:
- for EMA, it is the active strategy bucket time
- for
volume_breakout, it is the synthetic volume-bar time
2. StrategyOutput.occurrenceTime
This is the live tick time that caused the output.
This field is the canonical way to recover the real market event time that produced an output.
3. StrategySignalEvent.time
Signals currently use the app's emission time, not the source tick time.
The processor-path tests assert this as an emission window:
- signal time must fall between "emission started" and "emission observed"
Signals link back to outputs through strategyOutputId, so downstream code can recover the source tick context from the linked StrategyOutput.occurrenceTime.
Strategy-Specific Runtime Semantics
Two strategy timing families currently matter at the processor boundary.
EMA
ema is live and intra-bucket:
- the active EMA bucket is keyed from the incoming live tick time
- entries can happen before the next timeframe rollover tick arrives
- reversals are expected to emit exit first and opposite entry second
Volume Breakout
volume_breakout is close-gated:
- partial live volume bars may update the latest
StrategyOutput - entry and exit signals emit only when the current synthetic volume bar closes
- delayed re-entry and managed exits are part of the contract
The strategy runs on canonical volume bars: the frame length F is derived per session — rule v1: F = max(quantum, roundHalfUp(totalVtt / scheduledTradingMinutes, 18)) over the last 5 eligible sessions — and every session switch resets the builder with the new frame, dropping the overnight partial. Initialization backfills from QuestDB via a streaming session walk - sessions visited newest-to-oldest, every read a single session window bounded by the recorded high-water mark, each frame derived from its support block's per-session aggregates, retention staged under the caller's real bar bound (a retention-breaching tick stages nothing), and termination at the data floor — the first session entirely older than the oldest recorded row, with the walk never stopping short of the oldest in-session row's session; an instrument with no eligible history enters the awaiting-history state and retries at the next session open. Frames come from an app-owned SessionFrameProvider: the strategy can only obtain frame sources through the factory handed down from the app wiring — it constructs none itself. The production implementation (CachedSessionFrameProvider, backend-app) precomputes the next session's frame in the overnight gap (30 minutes before open, provisional) with one job per desired volume-frame instrument, so the first new-session tick switches atomically on a cached frame with no lookback reads on the strategy mailbox path; a cache miss falls back to the uncached provider-backed lookup under the execution's availability retry. Job removal drops the job and its cached frame, and a calendar change restarts the job and drops the old calendar's frame; since cancellation is cooperative, a job whose derivation is in flight when it is removed or replaced commits only while still the instrument's registered job (identity-checked atomically against reconcile's cancel-and-clear), so a stale frame can neither be restored after removal nor clobber its replacement's fresh frame. Each frame selection writes an append-only frame_selected audit record with full provenance (writerId, executionRole, tradingRunId, trigger). Provenance is atomic: the run id and role are captured once per tick — a single read of the leadership flow, before any retrying write — so a leadership transition during a retried audit write can neither tear a record (run id from one moment, role from another) nor re-attribute it to a run other than the one the tick observed; a mid-retry flap is audited as a promotion at the next tick. A single tick whose delta would materialize more bars than the retention window is a defined breach: record the tick, discard the partial, reseed the phase at vtt mod F, keep the price.
Trade Execution And Trade Persistence
Trade execution is downstream of signals, not outputs.
The runtime flow is:
- The route-local
StrategyEventCoordinatoris the sole collector of eachStrategyExecutorsignal stream. It briefly enters the process-wideTradingLeadershipfence and captures the active trading-run UUID, if any. - It immediately enqueues the signal with that immutable provenance for
DataRecorder, publishes the raw signal to the processor/DataBridge stream, and—only when a run was captured—places aTradeExecutionRequeston the route's bounded trade mailbox. TradeExecutorconsumes those requests using the strategy-specific trade settings selected fromordersRepository.tradeSettingUpdates. It enters the fence again before calling the broker and requires the request's run UUID to equal the active run.- On a failed renewal or ownership change, the adapter clears the fence before failing its coroutine, so the former leader process terminates and Kubernetes restarts it. A request captured under the former run can be recorded, but cannot submit an order.
TradeExecutorcalls the broker and emitstradeEvents.DataRecorder.tradeConsumerpersists those trade events throughOrdersRepository.saveOrder(...).
Important behavioral contract:
- outputs and signals can still exist even when a trade is not executed
- missing trade settings or broker failures should not suppress outputs or signals
- those situations only affect trade execution and trade-event emission
The resilience tests in backend-app cover those cases explicitly.
Recorder Path
DataRecorder.feedConsumer is the raw market-data persistence path. It is the only feed consumer that opts into recorded-late ticks (recordsLateFeedData): emitted ticks persist with their real derived fields; recorded-late ticks (late tuples, out-of-session prints) persist with synthetic volume = 0 / duration_since_last = 0 for dual-replica union coverage — the derived columns are non-authoritative and recomputed by every canonical reader.
StrategyEventCoordinator passes strategy outputs and signals to DataRecorder with the trading run captured at route admission. DataRecorder enqueues them on the same single ILP/HTTP writer. They persist complete JSON payloads for replay/audit and store queryable IDs, strategy/instrument IDs, event timestamps, and provenance (writer_id, execution role, nullable trading-run ID). The output table also stores occurrence_time; the signal table stores signal_id and strategy_output_id.
It:
- flattens the feed stream with
flatMapMerge - converts it to a
ReceiveChannel - passes the channel to
Recorder.recordMarketData()
Ingester then writes QuestDB rows with:
instrument_idduration_since_lastltpvolumets
The recorder also appends frame_selected audit records via the same ILP writer with an immediate flush — written at strategy initialization, every session switch, and leadership promotion, each carrying writerId, executionRole, tradingRunId, the exact frame as a Decimal-literal string, derivation-rule and calendar versions, and the per-session totals that produced it. The promotion trigger is fold-driven: when a promoted leader folds its next in-session tick under a new trading run, a LEADERSHIP_PROMOTION record audits the frame it adopted (selected earlier, possibly as standby). A promotion observed on a tick that itself switched sessions writes only the session-switch record, and a pod that was already leader at initialization writes no adoption record — its initialization records already carry the LEADER role.
Trade persistence is separate from QuestDB ingestion. Trade events are stored through OrdersRepository in the main application datastore.
Server Publishing Through DataBridgeLauncher
DataBridgeLauncher.launch() starts one coroutine per event family and forwards app state to backend-server.
The main forwarded streams are:
- strategy signal events
- strategy output events
- market feed events
- trade events
- open positions
- readiness
- error snapshots
- recovery events
Three details matter operationally:
DataBridgeLauncherdoes not reuse the shared feed flow fromStartDataFlowTask; it opens freshliveFeed.feed(...)collectors for market-feed forwarding- readiness published to the server is the diagnostic health set, not the pod-readiness set
DataBridgeLauncherdoes not keep a top-level eager kRPC client;launch()starts one lazy background supervisor job that repeatedly creates a fresh session with Arrow retry, marks readiness from the active session, and cancels cleanly onApplicationStopPreparing. All bridge traffic goes toApplicationConfig.INTERNAL_EVENTS_HOST/PORT/PATH.AuthStateCoordinatorruns alongsideDataBridgeLauncherat app-lifetime scope, started byStartAuthStateTask. It collectsrecoveryFlow().recoverable()for auth prompt publication, watchestokenStateUpdates()for obsolete-prompt cleanup, and resumes datasources when the OAuth token recovers.ProvisionedTokenStateCoordinatoris started by the sameStartAuthStateTask. It resumes the market-feed datasource viaResumableDataSource.resumeIfStopped()whenUPSTOX_ANALYTICStransitions toValidafter a Secret rotation (the analytics token powers the market feed and historical candle data).
Current Guarantees And Non-Guarantees
Guaranteed:
- feed batches are processed sequentially
- EMA remains live and intra-bucket
volume_breakoutremains close-gated- reversal-capable strategies emit exit before opposite entry
- open-position removals retain executions until managed cleanup completes
- a slow consumer never blocks the router (non-suspending
trySenddelivery) - route degradation with open positions triggers processor fail-fast
UpstoxDataSourcedetects day boundaries via vtt-decrease or time-delta heuristic, preventing negative volume deltas at session open and preserving opening auction volumeDEGRADEDdatasources do not flip pod readiness red —HealthStatus.DEGRADED.allowsReadinessreturnstrue, so a feed with a terminal token failure reports degraded health without blocking the readiness endpoint
Not guaranteed:
- deterministic ordering across instruments inside the same batch
- reuse of the same live-feed collector between local processing and bridge publishing
- correct day-boundary detection for non-equity Upstox segments (NSE_COM, MCX_FO) whose inter-session gaps may be shorter than the 12-hour threshold (QAPP-113)
Token Auth And Recovery Flow
When the token broker transitions a broker's state to AwaitingAuth, the recovery pipeline activates:
OAuthTokenBrokerService.recoveryFlow(AuthBroker.UPSTOX)emitsRecoverableErrorException, collected byAuthStateCoordinatorviaRecoveryManager.recoverable()RecoveryManagerpublishes the error toerrorFlow, surfaced byDataBridgetobackend-server- The UI displays the auth URL and accepts the auth code
- The code flows back through
RecoveryRouter→DataBridge.recoveryEvents()→RecoveryManager.recover()→TokenRecoveryAction.recover()→OAuthTokenBrokerService.submitAuthCode() - The token broker exchanges the code for an
AccessToken, transitions state toValid AuthStateCoordinatordetects theValidtransition and callsResumableDataSource.resumeIfStopped();awaitValidToken()also resumes inUpstoxApiDecorator, and the market data feed reconnects
RecoveryManager coalesces by recoveryUrl — concurrent AwaitingAuth emissions for the same broker share one recoveryToken UUID and one published error. The recover() method removes only the resolved error and preserves all other active recoverable errors across all recovery keys. Out-of-band transitions (token becomes Valid without explicit recovery) are handled by clearLogicalRecovery(), which removes the stale prompt, resumes suspended .recoverable() collectors, and cleans up the logical recovery entry.
Terminal Token Failure
Permanent OAuth errors (e.g., UDAPI100069 — wrong client ID/secret) throw NonRecoverableTokenException from UpstoxTokenExchangeStrategy.exchange(). The TokenManager surfaces this as OAuthTokenState.Failed, and awaitValidToken() throws TokenUnavailableException. UpstoxDataSource catches this in its outer .catch handler, sets a terminalFailureMessage (reporting DEGRADED health), and does not rethrow — this contains the failure and prevents infinite retry loops. The terminalFailureMessage is cleared only after the datasource successfully receives and maps upstream data following recovery.
Blackout Period
UpstoxTokenExchangeStrategy enforces a blackout period (3:30 AM–8:00 AM IST) via isTokenRequestAllowed(). During this window, submitAuthCode() throws TokenBlackoutException. Tokens expire at 3:30 AM IST daily; the expiry timer in TokenManager proactively transitions Valid → AwaitingAuth without waiting for a 401.
Provisioned Token Flow
Provisioned tokens — UPSTOX_SANDBOX for sandbox HFT order placement and UPSTOX_ANALYTICS for the market data feed and historical candle data — follow a different lifecycle than OAuth tokens:
- Tokens are mounted from a Kubernetes Secret as JSON files at
/var/run/secrets/token-broker/provisioned ProvisionedTokenManagerloads and validates the files at startup; if the file is missing, expired, or unparseable, the manager entersFailedand starts a 30-second poll loop- On 401,
UpstoxApiDecorator's sandbox client callsProvisionedTokenBrokerService.invalidateToken(UPSTOX_SANDBOX)which re-reads the file; if the fingerprint is unchanged, it transitions toFailedand the poller watches for Secret rotation - The analytics client follows the same pattern: on 401 it calls
invalidateToken(UPSTOX_ANALYTICS). While the handle isFailed,awaitValidToken()throwsTokenUnavailableException, surfacing as terminalDEGRADEDinUpstoxDataSource;ProvisionedTokenStateCoordinatorresumes the feed when the handle returns toValidafter the Secret is rotated - When the operator rotates the Secret, the kubelet refreshes the mounted files and the poller recovers on its next cycle — no pod restart is required
ProvisionedTokenManageruses a mutex-owned state machine:becomeValidUnderLockcancels any running poller and old expiry timer, publishesValid, and starts a new expiry timer;becomeFailedUnderLockcancels the expiry timer, publishesFailed, and ensures exactly one poller is running- The expiry timer reacquires the lock and checks token identity before transitioning to
Failed— stale timers from old tokens are silently ignored - Transient I/O errors (file temporarily unreadable) clear
lastFingerprinttonull, so the poller will detect when the same bytes reappear after the I/O issue resolves
Feed And Repository Separation
FeedRepository is storage/KV-only — it persists FeedIdentifier rows and subscribed-instrument KV state, with no dependency on backend-datasources. The live datasource runtime is owned directly by its consumers: NormalizedMarketFeed builds raw-feed streams through DataSourceFactory.feedStream(...), and SetSubscribedInstrumentsTask pushes the subscribed-instrument set through DataSourceFactory.setSubscribedInstruments(...). AddNewFeedsTask passes the dataSources list from backend-datasources to FeedRepository.addMissingFeeds() for diff-based feed creation.
backend-server and backend-repository do not depend on backend-datasources — the live feed runtime is scoped to backend-app only, consistent with the app/server separation invariant.
Public Instruments
Broker.publicInstruments(action) is a top-level extension in backend-broker that fetches instrument catalogs without authentication. UpstoxPublicInstrumentCatalog uses a plain HttpClient (no auth plugins) to fetch from the assets.upstox.com CDN. This is used by backend-sync's LiveInstrumentFeedRefresher instead of DataSourceFactory, since backend-sync does not have OAuthTokenBrokerService connections.
Code Map
If you need to trace the machinery in code, start here:
backend-app: startup tasks, feed fan-out, processor wiring, data bridge launcher, token broker connection,NormalizedMarketFeed(live datasource stream collection throughDataSourceFactory),AuthStateCoordinator(OAuth auth recovery → datasource resume),ProvisionedTokenStateCoordinator(analytics-token recovery → market-feed resume),ResilientTokenBrokerService+ResilientProvisionedTokenBrokerService(kRPC reconnect + shared state flows)backend-app(Processor.kt):RoutingSnapshot,StrategyRoute(withrouteInstanceId),RouteCommand(TopologyDirty/TargetsDirty/Degrade/TeardownStrategy),RouteState(Active/Degraded/Stopping),RouteJobs,CoreJobs(private types), feed routing, route command loop, reconciliation, degradation, stopping lifecycle, and fail-fast logicbackend-token-broker:TokenManager(token state machine, expiry timer, blackout, write-through persistence),OAuthTokenBrokerServiceImpl(kRPC service),ProvisionedTokenManager(mutex-owned state machine with@Volatile lastFingerprint, file-backed polling recovery, expiry timer with identity check),ProvisionedTokenBrokerServiceImpl(kRPC service),UpstoxTokenExchangeStrategy(OAuth token exchange, error classification),AuthBroker(type-safe key for auth-requiring brokers),ProvisionedTokenHandle(type-safe key for provisioned-token handles)backend-broker:BrokerApiResolver(compile-time-safe Broker→BrokerApi mapping, requires bothOAuthTokenBrokerServiceandProvisionedTokenBrokerService),UpstoxApiDecorator(threeHttpClient.withConfiginstances: OAuth for live trading,UPSTOX_ANALYTICSfor market feed + historical,UPSTOX_SANDBOXfor sandbox HFT;UpstoxErrorParserfor shared 401 classification),UpstoxPublicInstrumentCatalog(auth-free instrument fetch),Broker.publicInstruments()extensionbackend-processor:StrategyExecutor(withstart(), per-route bridge jobs,InstrumentRoute),ProcessorMailbox(internal)backend-strategy:StrategyExecution(withstart()),EmaStrategyExecution,VolumeBreakoutStrategyExecutionbackend-trade-executor:TradeExecutor(receivesBrokerApiResolver)backend-datasources:UpstoxDataSource(tick-to-MarketData conversion, vtt delta, day-boundary detection, DEGRADED health on terminal token failure,ResumableDataSource),BitFlyerDataSource,DataSourceFactory(receivesBrokerApiResolver)backend-repository:FeedRepository(storage/KV-only, no datasource dependency)backend-recovery:RecoveryManager(coalescing by recoveryUrl,clearLogicalRecovery()for out-of-band transitions, targetedrecover()that preserves unrelated errors)backend-recorder:DataRecorder,Recorder,Ingester
Related Documents
- Architecture Overview
- Backend Conventions — BrokerApiResolver wiring, FeedRepository split, RecoveryManager coalescing
- Testing Guide
- GitHub Issue Workflow