Operations

This document covers how to check application health, diagnose failures, and inspect build and deployment status using the available MCP tools. It is the primary reference for any agent session that involves debugging, monitoring, or status checking.

MCP Tool Priority

Always prefer MCP tools over shell fallbacks. Use kubectl, gh, or the TeamCity CLI only when an MCP does not cover the operation you need.

Concern	Tool	Fallback
Market data queries (QuestDB)	QuestDB REST API + agent skill	—
Application state queries (Gel)	Gel MCP	`gel` CLI via `pods_exec` on stg
Kubernetes pods, logs, events	Kubernetes MCP	`kubectl` with the relevant kubeconfig
Metrics and log queries	Grafana MCP	—
Alert and dashboard status	Grafana MCP	—
Build status and logs	TeamCity MCP	TeamCity CLI (`teamcity`)
Trigger a build	TeamCity MCP	TeamCity CLI
Issues, PRs, repo state	GitHub MCP	`gh` CLI

Environment Map

Two environments are in use. Always confirm which environment you are targeting before reading logs or querying metrics.

Environment	Kubernetes	Grafana	QuestDB API
Production	local Kubernetes MCP via `$KUBECONFIG_PROD`	`https://grafana.timemanx.com`	`https://recorder.timemanx.com`
Staging	local Kubernetes MCP via `$KUBECONFIG_DEV`	`https://grafanas.timemanx.com`	`https://recorders.timemanx.com`

TeamCity (teamcity.timemanx.com/app/mcp) and GitHub (https://api.githubcopilot.com/mcp/) are shared across both environments.

MCP access and permissions

The Kubernetes, Grafana, and Gel MCP servers are not deployed by Terraform and have no public hosts. They are operator-managed and run locally, configured in the local MCP client against each environment (e.g. a Kubernetes MCP pointed at the environment's kubeconfig, a Grafana MCP against the environment's Grafana UI, a Gel MCP against the environment's Gel instance). Access control is therefore whatever the local configuration and the target's own authentication enforce — there is no Cloudflare Access layer in front of these MCPs anymore.

Treat production as read-only by convention: prefer observation-only operations against prod, and route any mutation (restarting a pod, scaling a deployment, exec into a container, applying a resource change) through staging or through kubectl with $KUBECONFIG_PROD. If your local Kubernetes MCP config disables the write tools for prod, write attempts will be rejected server-side; either way, do not attempt prod writes.

QuestDB recorder endpoint (recorder.timemanx.com / recorders.timemanx.com) — published through the Cloudflare Tunnel as a CNAME only; the shared edge module does not create a Cloudflare Access application for it, so it is reachable without a service token. Treat the /exec query surface as unauthenticated at the edge in both environments.

Namespaces: Application workloads run in the namespace configured per environment in gradle.prod.properties (k8s-namespace) and gradle.stg.properties. Use namespaces_list on the Kubernetes MCP to confirm if unknown.

Kubeconfig access

When interacting with the clusters directly via kubectl (or other CLI tools), you should use the environment variables pointing to your respective kubeconfig files: - KUBECONFIG_DEV: Path to the kubeconfig for the staging/development cluster. (If not set, the default ~/.kube/config is used). - KUBECONFIG_PROD: Path to the kubeconfig for the production cluster.

Ensure these variables are exported in your environment before running operations scripts.

Local `backend-app` leadership

./gradlew :backend-app:run uses the DEV cluster's Kubernetes API for trading-leadership election. It requires KUBECONFIG_DEV; the run task passes it to the Kubernetes client as KUBECONFIG, uses the configured DEV namespace, and assigns a unique local POD_NAME for the Lease holder identity.

The local application competes for the same backend-app-trading Lease as DEV pods. It remains standby while another holder has the Lease and can submit trades only after it becomes leader.

Local Development

`backend-sync`

Use this workflow when reproducing instrument-sync issues against the local Gel and local sync Postgres containers instead of the staging or production stacks.

Run only one backend-sync JVM at a time; multiple local instances will race for the same ports and produce overlapping sync records.
Local Gel must be reachable on 127.0.0.1:5656.
Local sync Postgres must be reachable on 127.0.0.1:5432.
backend-sync expects the target Feed rows to already exist in Gel.

Bootstrap note:

backend-sync does not create Feed rows itself. Feed registration is currently owned by backend-app startup via AddNewFeedsTask / FeedRepository.addNewFeeds().
If you wipe Gel and run backend-sync by itself before feed bootstrap has happened, the scheduler can record Feed not found for data source: ... errors for any datasource whose Feed row is still missing.

Reset and rerun:

Stop every running backend-sync process.
Reset the local sync Postgres database, and wipe Gel instrument state too if you want a full instrument re-sync.
Start the IDE backend-sync run configuration.

Observe progress from three places:

the backend-sync application log for Started, Completed, Failed, and per-row decode warnings
the local sync Postgres syncrecords table for persisted scheduler state
Gel Feed / Instrument counts to confirm data actually landed in the main datastore

Example local entry points:

podman ps --format '{{.ID}} {{.Image}} {{.Names}}'
podman exec <sync-postgres-container> sh -lc 'psql -U "$POSTGRES_USER" -d "$POSTGRES_DB"'
podman exec <gel-container> sh -lc 'GEL_PASSWORD="$GEL_SERVER_PASSWORD" gel -H 127.0.0.1 -P 5656 -u "$GEL_SERVER_USER" --tls-security insecure'

Useful local queries:

select name, type, status::text
from syncrecords
order by coalesce(
  (status->>'completion_time')::timestamptz,
  (status->>'start_time')::timestamptz,
  (status->>'scheduled_time')::timestamptz
) desc
limit 12;

select Feed {
  name,
  instrument_count := count(.<feed[is Instrument])
}
order by .name;

Current Upstox behavior:

UpstoxApiDecorator currently logs and skips malformed source rows one-by-one during decode.
A sync can therefore finish with SyncStatus.Success even when some source instruments were dropped; today those skipped-row warnings live only in application logs and are not retained in syncrecords.status.

Sync Postgres monitoring

backend-sync runs postgres_exporter as the backend-sync-db-metrics sidecar. Alloy discovers that container directly and scrapes it every 15 seconds with job="backend-sync-postgres"; the pod's existing prometheus.io/* annotations continue to scrape the application endpoint separately.

Start with these PromQL queries in Grafana:

pg_up{job="backend-sync-postgres"}

pg_exporter_last_scrape_error{job="backend-sync-postgres"}

pg_database_size_bytes{job="backend-sync-postgres"}

pg_stat_database_numbackends{job="backend-sync-postgres"}

The exporter connects using the least-privilege sync_metrics Postgres role. Terraform owns its password in the backend-sync-metrics Secret. See Sync Postgres metrics setup before applying terraform/kubernetes. Terraform must run before the rendered backend-sync manifest on first deployment; a later password change triggers a backend-sync rollout, whose bootstrap init container updates the database role before the exporter starts.

Prometheus evaluates BackendSyncPostgresExporterUnhealthy when Alloy cannot scrape the exporter, pg_up != 1, or pg_exporter_last_scrape_error != 0 for five minutes. It is a warning-only rule; Alertmanager is currently disabled, so the rule is observable in Prometheus but does not deliver notifications.

Instrument sync performance

The instrument sync path writes directly to Instrument (no staging table). Key parameters:

Parameter	Value	Where
`INSTRUMENT_BATCH_SIZE`	2,000	`DataStoreImpl` companion
`INSTRUMENT_BATCH_CONCURRENCY`	6	`DataStoreImpl` companion
`FEED_INSTRUMENT_PAGE_SIZE`	5,000	`DataStoreImpl` companion
`FEED_INSTRUMENT_READ_CONCURRENCY`	6	`DataStoreImpl` companion

GelClientPool instance, gelClient, has a default pool size of 50 connections. Insert batches and read pages borrow connections from the shared pool and return them after use.

feedInstruments(feedId) uses a concurrent read pattern: a count query determines total pages, then all pages are fetched in parallel with flatMapMerge(concurrency=6) and reassembled in offset order. The syncTime query runs concurrently with the page fetches.

Expected sync times for a full Upstox catalog (~136k instruments):

Machine	Initial sync	Incremental (10% churn)
M4	< 60s	< 30s
A1	2–3 min	< 60s

If sync times exceed these targets, check:

Gel connection pool exhaustion (default pool size is 50; INSTRUMENT_BATCH_CONCURRENCY and FEED_INSTRUMENT_READ_CONCURRENCY should not exceed available connections)
Network latency between backend-sync and the Gel instance
Whether the .feed index is present (gel describe type Instrument should show index on (.feed))

QuestDB (Market Data)

QuestDB is accessed via its REST API, not an MCP. The QuestDB agent skill embeds QuestDB-specific SQL knowledge into the agent — correct syntax for SAMPLE BY, LATEST ON, ASOF JOIN, window functions, and common PostgreSQL→QuestDB mistakes to avoid.

Endpoints: - Production: https://recorder.timemanx.com - Staging: https://recorders.timemanx.com

Querying

All queries go to the /exec endpoint. Use curl -G --data-urlencode for anything non-trivial:

# Latest row per instrument
curl -G "https://recorder.timemanx.com/exec" \
  --data-urlencode "query=SELECT * FROM market_data LATEST ON ts PARTITION BY instrument_id"

# Average per-minute volume for one instrument over the last hour
curl -G "https://recorder.timemanx.com/exec" \
  --data-urlencode "query=SELECT avg(avg_volume) AS interval_avg FROM (SELECT avg(volume) AS avg_volume FROM market_data WHERE instrument_id = cast('<uuid>' as uuid) AND ts > dateadd('h', -1, now()) SAMPLE BY 1m)"

Key QuestDB SQL patterns for this project

-- Latest row per instrument
SELECT * FROM market_data
LATEST ON ts PARTITION BY instrument_id;

-- Ingestion rate over the last hour (rows per minute)
SELECT ts, count()
FROM market_data
WHERE ts > dateadd('h', -1, now())
SAMPLE BY 1m;

-- Recent rows for one instrument
SELECT ts, duration_since_last, ltp, volume
FROM market_data
WHERE instrument_id = cast('<uuid>' as uuid)
  AND ts > dateadd('h', -1, now())
ORDER BY ts DESC
LIMIT 20;

-- Latest rows needed to satisfy a target volume
SELECT ts, duration_since_last, ltp, volume
FROM (
  SELECT *,
    SUM(volume) OVER (ORDER BY ts DESC ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW) AS volume_sum
  FROM market_data
  WHERE instrument_id = cast('<uuid>' as uuid)
)
WHERE volume_sum - volume <= 90.0
ORDER BY ts ASC;

-- Average volume across time buckets for one instrument
SELECT avg(avg_volume) AS interval_avg
FROM (
  SELECT avg(volume) AS avg_volume
  FROM market_data
  WHERE instrument_id = cast('<uuid>' as uuid)
    AND ts > dateadd('h', -1, now())
  SAMPLE BY 1m
);

-- Instruments whose latest row is older than five minutes
SELECT instrument_id, ts AS last_seen
FROM (
  SELECT instrument_id, ts
  FROM market_data
  WHERE ts > dateadd('h', -1, now())
  LATEST ON ts PARTITION BY instrument_id
)
WHERE ts < dateadd('m', -5, now())
ORDER BY ts ASC;

The actual market_data DDL is CREATE TABLE IF NOT EXISTS market_data (instrument_id UUID, duration_since_last LONG, ltp DOUBLE, volume DOUBLE, seq LONG, ts TIMESTAMP) TIMESTAMP(ts) PARTITION BY HOUR WAL DEDUP UPSERT KEYS(ts, instrument_id, seq). In practice: instrument_id is stored as a UUID and written from the recorder as a dashed UUID string, duration_since_last is stored as milliseconds in a LONG, ltp and volume are DOUBLE, seq is a LONG broker-provided monotonic discriminator (Upstox cumulative vtt; BitFlyer tick_id) that disambiguates same-millisecond trades, and ts is the designated timestamp. Query timestamps in UTC. Duplicate writes for the same (ts, instrument_id, seq) keep the latest values. There is currently no symbol/name column in QuestDB for market data.

Table inspection

curl "https://recorder.timemanx.com/exec?query=SHOW+TABLES"
curl "https://recorder.timemanx.com/exec?query=SHOW+COLUMNS+FROM+market_data"

NATS (Raw Market Data Store)

NATS JetStream stores the raw protobuf bytes captured from Upstox (pre-decode), retained for 7 days (MaxAge=7d), queryable by time of storage for later cross-checks against processed MarketData. It is cluster-internal (no ingress, no public host) and has no first-party UI — monitoring is JSON HTTP on the monitoring port plus kubectl. There is no NATS MCP, so reach it via kubectl. NATS is deployed by the backend-raw-store Gradle module (StatefulSet + Service + PV/PVC + ConfigMap); the nats-accounts Secret (accounts config) is Terraform-managed.

NATS runs with an accounts config: the non-secret directives (jetstream {}, monitor_port, system_account: SYS, no_auth_user: anon) live in the backend-raw-store-config ConfigMap (mounted at /etc/nats/nats.conf), which includes the accounts {} block from a Terraform-managed nats-accounts Secret (mounted at /etc/nats-secrets/accounts.conf, quant-app namespace). The SYS system account has a surveyor user (bcrypt-hashed password) for nats-surveyor, scoped via a permissions block to publish only $SYS.REQ.SERVER.PING, $SYS.REQ.SERVER.PING.JSZ, and $SYS.REQ.ACCOUNT.PING.STATZ (the server-ping, JetStream-statz, and account-statz request subjects surveyor polls) and subscribe only _INBOX.> (its reply inboxes) — so a compromised surveyor pod cannot invoke other system-account operations; the APP account has JetStream enabled and an anonymous anon user (no_auth_user: anon) so backend-app connects without credentials. No credential material is embedded in any Git-tracked manifest — the surveyor password is supplied as a single sensitive Terraform variable (nats_surveyor_password); its bcrypt hash is derived from it in Terraform by the bcrypt_hash.nats_surveyor resource (viktorradnai/bcrypt provider) so the cleartext and hash can never diverge. nats-surveyor receives its password via the NATS_SURVEYOR_PASSWORD env var, sourced from a nats-surveyor-credentials Secret (monitoring namespace); the NATS_SURVEYOR_USER env var comes from the surveyor_user Terraform variable (the username is not secret — a public account name in the NATS accounts config). A NetworkPolicy (nats-ingress, JKube-owned by backend-raw-store via backend-raw-store/src/main/jkube/backend-raw-store-np.yml) restricts NATS ingress to backend-app and nats-surveyor only — it protects the raw-store pod selector and ports, and the cross-namespace surveyor allowance is part of that service contract.

First-deploy ordering: the nats-accounts Secret is Terraform-owned (terraform/kubernetes/app.tf) but the NATS StatefulSet is JKube-rendered by backend-raw-store and mounts that Secret by name (backend-raw-store/build.gradle.kts). Apply terraform/kubernetes — which creates nats-accounts and nats-surveyor-credentials — before applying the backend-raw-store manifests for the first time. Until the nats-accounts Secret exists the NATS pod will fail to start (unresolvable Secret volume mount), leaving NATS unavailable; backend-app's StartRawStoreTask will then fail-fast and the app pod will not become ready. This mirrors the backend-token-broker provisioned-token Secret ordering. The nats-ingress NetworkPolicy is JKube-owned (it is created by the backend-raw-store manifests alongside the StatefulSet, not by Terraform).

ConfigMap changes: the backend-raw-store-config ConfigMap (nats.conf: jetstream {}, monitor_port, system_account, no_auth_user, the include path) is JKube-owned, not Terraform-managed, so Terraform cannot observe its content and the nats_rollout_trigger does not fire on a ConfigMap edit. NATS does not hot-reload its config (the kubelet refreshes the mounted file, but NATS only reads it at startup), so a ConfigMap change requires a NATS pod restart. To make a ConfigMap change roll NATS automatically, the StatefulSet pod template carries a checksum/nats-config annotation set to the SHA-256 of the ConfigMap file (computed at build time in backend-raw-store/build.gradle.kts); editing nats.conf changes the annotation → the next Gradle k8sApply updates spec.template.metadata.annotations → Kubernetes rolls the StatefulSet → NATS starts with the new ConfigMap. (The ConfigMap file is also a k8sResource task input, so the manifest re-renders on edit.) No manual kubectl rollout restart is needed for a ConfigMap change. In practice the ConfigMap content is near-static operational config and rarely changes post-first-deploy; the dynamic credential/permissions content lives in the Terraform-owned nats-accounts Secret, which is covered by the rollout trigger (see "Credential rotation").

Credential rotation

Rotating the surveyor password means updating the single nats_surveyor_password Terraform variable (cleartext) — its bcrypt hash is derived from it by the bcrypt_hash.nats_surveyor resource, so the cleartext and hash can never diverge — then terraform apply. A rotation triggers a rollout of both workloads:

NATS — NATS does not hot-reload accounts.conf (it reads auth/accounts at startup). The StatefulSet is JKube-owned, so Terraform cannot annotate its pod template; instead null_resource.nats_rollout_trigger (terraform/kubernetes/app.tf) runs kubectl rollout restart statefulset/backend-raw-store whenever the rendered accounts.conf changes, then waits on kubectl rollout status (180s). The trigger keys on sha256 of the full rendered accounts.conf (the nats-accounts Secret's data) — not just the password hash — so any content change (a password rotation, a permissions edit, or adding/removing users) rolls NATS. It depends_on the nats-accounts Secret so Terraform has written the new accounts.conf before NATS restarts — otherwise NATS could start with the stale content and never reload it. kubectl + the kubeconfig (var.kube_config_path) must be available on the terraform runner — the same kubeconfig the kubernetes provider uses.
nats-surveyor — env vars (NATS_SURVEYOR_PASSWORD via secret_key_ref) are immutable post-start, and a Secret data change does not alter the Deployment spec (the ref is by name). So the surveyor pod template carries a checksum/nats-surveyor-credentials annotation set to sha256(bcrypt_hash.nats_surveyor.id) — derived from the bcrypt hash (itself derived from the single nats_surveyor_password variable), not the cleartext password, so the annotation does not expose a hash of the cleartext; a rotation changes the annotation → Terraform updates the Deployment → Kubernetes rolls it.
Ordering — backend-app connects to NATS anonymously (no_auth_user: anon), so a surveyor-password rotation does not affect the live trading path — only surveyor metrics scraping is briefly interrupted. Both workloads roll on the same terraform apply (the null_resource and the surveyor Deployment update are independent; no depends_on ordering is enforced, which also keeps the surveyor module out of CI's targeted-apply graph). There is an inherent brief surveyor auth-failure window during rotation regardless of order: while one side has the new password and the other still has the old, surveyor cannot auth. Eliminating that window entirely would require dual-password overlap (add the new password alongside the old, roll both, then remove the old) — not warranted for a personal platform.

On first deploy (terraform-before-JKube ordering), the nats-accounts Secret is created but the NATS StatefulSet does not yet exist. The trigger captures kubectl get statefulset's stderr and matches the NotFound marker to detect exactly that case and exit 0 — every other kubectl failure (expired kubeconfig, RBAC denial, DNS failure, unreachable API server, restart/timeout failure) propagates as exit 1, failing the local-exec provisioner and the terraform apply, so a broken rotation is never silently recorded as complete and NATS is never left running the stale hash.

Logical JetStream usage (what NATS has stored)

The monitoring server (port 8222, from monitor_port: 8222 in the ConfigMap) serves JSON endpoints. Port-forward it first — dev/stg use $KUBECONFIG_DEV; for prod use kubectl with $KUBECONFIG_PROD:

kubectl --kubeconfig "$KUBECONFIG_DEV" -n quant-app port-forward pod/<nats-pod> 8222:8222

# Total bytes on disk + the configured max_file cap
curl -s localhost:8222/varz | jq '.jetstream'
# Per-stream bytes/messages + the retained time range
curl -s 'localhost:8222/jsz?streams=true' | jq '{bytes, messages, streams: [.streams[] | {name, bytes: .state.bytes, messages: .state.messages, first: .state.first_ts, last: .state.last_ts}]}'
# Health (also the readiness/liveness probe)
curl -s localhost:8222/healthz

/varz → jetstream.stats.storage (total bytes on disk) + max_file (the 45 GiB JetStream disk cap).
/jsz?streams=true → per-stream state.bytes / state.messages + first_ts/last_ts.

Physical PVC space (consumed vs available on disk)

NATS does not report filesystem free space — only its own accounting. For the actual disk:

# Used / available / Use% on the mounted volume
kubectl --kubeconfig "$KUBECONFIG_PROD" -n quant-app exec <nats-pod> -- df -h /data
# Capacity + phase (Bound) of the claim
kubectl --kubeconfig "$KUBECONFIG_PROD" -n quant-app get pvc backend-raw-store

The PVC is 50Gi, local-path, ReadWriteOnce (backend-raw-store/src/main/jkube/backend-raw-store-{pv,pvc}.yml); resize after measuring actual raw-frame volume.

Continuous metrics (Prometheus / Grafana)

NATS's monitoring server is JSON-only (no /metrics). Continuous Prometheus/Grafana metrics come from nats-surveyor (Terraform-managed in terraform/kubernetes/monitoring.tf, resource nats_surveyor): it scrapes NATS /jsz + /varz and exposes /metrics on :7777, which Alloy discovers via the pod's prometheus.io/* annotations and remote-writes to Prometheus. PVC free space in Grafana comes from kubelet kubelet_volume_stats_*.

Optional guardrail

JetStream is capped at 45 GiB, leaving 5 GiB of headroom beneath the 50 GiB PVC allocation. It therefore returns "disk resource exhausted" before it fills the node filesystem through the local-path volume.

Raw-vs-QuestDB Reconciliation

Reconciliation is an on-demand cross-check between the raw broker bytes captured in the NATS raw store and the processed MarketData rows persisted to QuestDB. It replays raw frames over a window, decodes each to UpstoxMarketDataFeed, applies the same emission gate the live path uses (so ticks the live path deliberately dropped don't show as false "missing in store"), and diffs the passthrough fields ltt (QuestDB ts) and ltp against the market_data table. The join key is instrumentId + (ltt, seq) — the store dedup key — not storage time, so same-millisecond trades (which share ts but differ on seq) are matched individually rather than collapsing via last-write-wins.

The comparison logic lives in backend-reconciliation (a pure library) and is wired in backend-app, where both inputs are already in scope: RawDataRetriever (NATS replay) and HistoricalDataProvider (QuestDB read-back). backend-server only proxies the requests over the existing AppDataBridge kRPC — it owns no reconciliation logic.

Only the passthrough fields are compared. The derived fields volume (per-tick delta from cumulative vtt) and duration_since_last are intentionally not reconciled — reproducing them would require replaying the full stateful UpstoxStream transform. Reconciliation is also Upstox-only: BitFlyer streams JSON and is not captured as raw bytes. Raw frames whose instrument name cannot be resolved via the live InstrumentMap snapshot are skipped (expected ticks can't be built for them), but whole-instrument capture loss is still detected: the iteration set is unioned with every instrument id that has stored rows in the window (HistoricalDataProvider.distinctInstrumentIds), so an instrument recorded to the store but entirely absent from the raw replay surfaces all its stored rows as MISSING_IN_RAW — there is no subscription-snapshot time-variance blind spot.

`POST /reconcile` — summary report

A synchronous, retried call returning aggregate counts plus a bounded sample (≤50) of discrepancies. Reach the backend-server REST surface:

# Production
curl -sS -X POST https://app.timemanx.com/reconcile \
  -H 'Content-Type: application/json' \
  -d '{"startTime":"2026-07-31T03:45:00Z","endTime":"2026-07-31T04:00:00Z","broker":"upstox"}' | jq

# Staging (note apps host)
curl -sS -X POST https://apps.timemanx.com/reconcile \
  -H 'Content-Type: application/json' \
  -d '{"startTime":"2026-07-31T03:45:00Z","endTime":"2026-07-31T04:00:00Z","broker":"upstox"}' | jq

Request body (ReconciliationRequest):

Field	Type	Notes
`startTime`	ISO-8601 instant	Bounds the raw NATS replay by storage (publish) time
`endTime`	ISO-8601 instant	Must be `>= startTime`
`broker`	string	Required (e.g. `upstox`)

Response (ReconciliationReport):

Field	Meaning
`totalExpected`	Ticks the raw replay expected in QuestDB (after the emission gate)
`totalStored`	QuestDB rows read back for the window (padded ±5s)
`matched`	Present in both with equal `ltp` (within `1e-9` epsilon)
`missingInStore`	Expected from raw but absent in the recorded store
`missingInRaw`	Present in QuestDB but absent from the raw replay (or filtered out by the gate)
`ltpMismatches`	Present in both but `ltp` differs beyond epsilon
`sampleDiffs`	First ≤50 discrepancies (see diff shape below)

`GET /reconcile/diffs` — SSE diff stream

Streams every discrepancy as it is computed. Query parameters carry the window (no JSON body):

curl -sS -N "https://app.timemanx.com/reconcile/diffs?startTime=2026-07-31T03:45:00Z&endTime=2026-07-31T04:00:00Z&broker=upstox"

startTime, endTime, and broker are all required (ISO-8601 instants for the times; e.g. upstox for broker). Bad query parameters are reported as an SSE error event before the stream closes. Each emitted event is a ReconciliationDiff:

Field	Meaning
`instrumentId`	Internal instrument id
`ltt`	Exchange last-trade-time (the QuestDB `ts` join key)
`kind`	`MISSING_IN_STORE` / `MISSING_IN_RAW` / `LTP_MISMATCH`
`rawLtp`	Raw price (present for `MISSING_IN_STORE` and `LTP_MISMATCH`)
`storedLtp`	QuestDB price (present for `MISSING_IN_RAW` and `LTP_MISMATCH`)

Retry caveat: the summary POST /reconcile is a suspend call fully inside the AppDataBridge retry wrapper, so transient backend-app disconnects are retried. The GET /reconcile/diffs SSE stream is a cold flow — kRPC streaming happens during collection, outside the retry wrapper, so a mid-stream disconnect aborts the stream rather than retrying. Acceptable for an on-demand debugging tool; re-run the request to resume.

Gel MCP (Application State)

Gel owns all application state: strategies, feeds, instruments, orders, and positions. The Gel MCP exposes native EdgeQL queries — not the limited PostgreSQL compatibility layer. It runs locally (operator-managed) and is configured against the target environment's Gel instance.

Tools

try_query — run an EdgeQL query inside a transaction that is always rolled back. Use this for all exploratory queries, especially on prod. It cannot modify data.

execute_query — run an EdgeQL query that commits. Use on stg for writes when needed. Use on prod only when truly necessary — prefer try_query on prod by habit since both are safe for reads.

list_rules / fetch_rule — access Gel's schema rules and conventions. Useful when writing schema migrations or understanding type constraints.

list_examples / fetch_example — access EdgeQL code examples for advanced patterns.

Key EdgeQL patterns for this project

-- List registered strategies with their instruments
SELECT Strategy {
  id,
  name,
  display_value,
  instruments: {
    id,
    name,
    display_value,
    type,
    exchange,
    feed: { id, name }
  }
}
ORDER BY .name;

-- List instruments for a feed and show linked strategies
SELECT Instrument {
  id,
  name,
  display_value,
  type,
  exchange,
  feed: { id, name, display_value },
  strategies: { id, name, display_value }
}
FILTER .feed.name = 'feed-name'
ORDER BY .display_value;

-- Recent orders
SELECT Order {
  id,
  time,
  instrument: { id, name, display_value },
  transaction_type,
  price,
  volume,
  broker,
  exchange
}
  ORDER BY .time DESC
  LIMIT 20;

-- Open positions (no exit recorded yet)
SELECT Position {
  id,
  strategy: { id, name, display_value },
  trade_type,
  entry_signal_id,
  entry_order: {
    id,
    time,
    instrument: { id, name, display_value },
    transaction_type,
    price,
    volume
  }
}
FILTER not exists .exit_order
ORDER BY .entry_order.time DESC;

Strategy supports select, insert, and update, but not delete. Feed and Instrument support select, insert, and delete, but deny updates. Order and InstrumentsUpdateRecord are append-only for writes. Position allows inserts plus a single exit update: exit_signal_id and exit_order must be set together and cannot be changed once present. TradeSetting is unique on (strategy, instrument).

Always use try_query when inspecting state on prod — it is functionally identical for reads and eliminates any risk of accidental mutation.

Kubernetes MCP Provides direct access to pods, logs, events, and all Kubernetes resources.

Treat prod as read-only. Any tool that mutates cluster state (pods_delete, pods_exec, pods_run, resources_create_or_update, resources_delete, resources_scale writes) must be run against the stg cluster or via kubectl with the prod kubeconfig. If your local Kubernetes MCP config disables the write tools for prod, write attempts are rejected server-side — never attempt a write operation against prod.

Key tools for health and diagnosis

pods_list — list all pods across namespaces, or filter by namespace, label, or field selector. Use fieldSelector=status.phase=Running to isolate unhealthy pods, or omit to see all.

pods_log — fetch pod logs by name and namespace. Always try previous=true first when a pod has restarted — it gives the log from the crashed container. Use tail to limit output.

pods_get — inspect a pod's full spec and status. Useful for checking container states, restart counts, and readiness/liveness probe failures.

events_list — list recent cluster events, optionally scoped to a namespace. The first tool to reach for when something is broken but you don't know where to look. Warnings and errors surface here before they appear in logs.

resources_get — get any Kubernetes resource by apiVersion, kind, and name. Use for StatefulSets, Deployments, Services, ConfigMaps, and anything else outside the pod-specific tools.

resources_scale — check or update replica count for a StatefulSet or Deployment.

pods_exec — run a command inside a running container. Use sparingly; prefer logs and describe first.

nodes_top and pods_top — resource usage from the Metrics Server. Useful when investigating OOMKill or CPU throttling.

Application module → resource mapping

Module	Kind	Typical name pattern
`backend-app`	StatefulSet	`backend-app`
`backend-server`	StatefulSet	`backend-server`
`backend-recorder` (QuestDB)	StatefulSet	`backend-recorder`
`backend-database` (Gel)	StatefulSet	`backend-database`
`backend-sync`	StatefulSet	`backend-sync`
`backend-web-app` (Caddy)	Deployment	`backend-web-app`

Grafana MCP

Runs locally (operator-managed) and is configured against the target environment's Grafana UI (https://grafana.timemanx.com for prod, https://grafanas.timemanx.com for stg — see environment map above). Covers Prometheus metrics and Loki logs, dashboard queries, and alert rule status.

Key tools for health and diagnosis

list_datasources — discover available datasource UIDs and types before querying. Do this once at the start of a session if you don't know the datasource UID.

query_prometheus — execute PromQL against a Prometheus datasource. Supports instant and range queries. Useful for ingestion lag, processor throughput, pod restart counts, JVM heap, etc.

query_loki_logs — execute LogQL against a Loki datasource. Use for structured log search across pods. Prefer this over pods_log when you need to search across multiple pods or query by label.

alerting_manage_rules — list alert rules and their current state (firing, normal, error). Start here when you want an overview of what is currently broken.

search_dashboards — find dashboards by title. Use to locate the right dashboard before drilling into panel queries.

get_dashboard_summary — get a compact overview of a dashboard (panels, variables, metadata) without loading the full JSON. Use before get_dashboard_by_uid.

get_dashboard_panel_queries — extract the PromQL or LogQL queries from every panel in a dashboard. Useful for understanding what a dashboard measures so you can run the same queries directly.

generate_deeplink — generate a direct URL to a Grafana dashboard or panel. Use when you want to point a human to the right place.

Useful PromQL starting points

These assume a Prometheus datasource. Replace <namespace> with the target namespace.

# Pod restart count in the last hour
increase(kube_pod_container_status_restarts_total{namespace="<namespace>"}[1h])

# Container OOMKill events
kube_pod_container_status_last_terminated_reason{namespace="<namespace>", reason="OOMKilled"}

# JVM heap usage for backend-app
jvm_memory_used_bytes{namespace="<namespace>", pod=~"backend-app.*", area="heap"}

# CPU usage by pod
rate(container_cpu_usage_seconds_total{namespace="<namespace>"}[5m])

Useful LogQL starting points

# All logs from backend-app in the last 15 minutes
{namespace="<namespace>", app="backend-app"} | limit 100

# Error logs only
{namespace="<namespace>"} |= "ERROR" | limit 100

# Logs from a specific pod
{namespace="<namespace>", pod="backend-app-0"} | limit 100

TeamCity MCP

Endpoint: teamcity.timemanx.com/app/mcp. Exposes three tools.

teamcity_build_log — retrieve the full build log for a specific build ID. Supports pagination and filtering to warnings/errors only. Use when a build has failed and you need to know why.

teamcity_rest_get — send GET requests to the TeamCity REST API. Use for: - List projects: /app/rest/projects - List build configurations: /app/rest/buildTypes?locator=project:<projectId> - Find the last build for a config: /app/rest/builds?locator=buildType:<btId>,count:1 - Find the last failed build: /app/rest/builds?locator=buildType:<btId>,status:FAILURE,count:1 - Check a running build: /app/rest/builds?locator=buildType:<btId>,running:true - List investigations: /app/rest/investigations - List muted problems: /app/rest/mutes

teamcity_rest_post — trigger a build by posting to /app/rest/buildQueue. All agent-triggered builds are marked personal=true. Use to re-run a failed build or trigger a selective deploy.

Fallback: When an operation is not covered by these three tools (e.g. managing agents, editing parameters, bulk operations), use the TeamCity CLI:

teamcity run list --project <projectId>
teamcity run log <buildId>
teamcity api /app/rest/...   # raw REST access

GitHub MCP

Endpoint: https://api.githubcopilot.com/mcp/. Repo: timemanx/quant-app.

Use for: listing and creating issues, inspecting PRs, checking workflow run status, searching code. For operations not covered by the MCP, fall back to gh CLI or scripts/github_issues.py for issue management.

Diagnostic Workflows

These are the most common situations that require an agent to check system state.

Is the system healthy?

alerting_manage_rules (Grafana MCP, prod) — check for any firing alert rules.
events_list (Kubernetes MCP, prod) — scan for recent Warning events in the application namespace.
pods_list (Kubernetes MCP, prod) — check that all application pods are Running and not restarting.

If any of these show a problem, proceed to the relevant workflow below.

A pod is crashlooping or restarting

pods_get — check containerStatuses for the restart count and last termination reason.
pods_log with previous=true — get the log from the crashed container. This is the most direct path to the failure cause.
events_list scoped to the application namespace — check for OOMKill, probe failures, or image pull errors.
If the pod will not start at all, resources_get the owning StatefulSet or Deployment and check its conditions.

A build failed

teamcity_rest_get to /app/rest/builds?locator=buildType:<btId>,status:FAILURE,count:1 — find the failed build ID.
teamcity_build_log with that build ID, filtering to warnings and errors first — identify the failure.
If the failure is a test, note the test name and check the relevant module's test suite in docs/testing.md.
If the failure is a deploy step, check the Kubernetes MCP for pod state in the target environment.

Market data ingestion is lagging or stopped

Query QuestDB directly for ingestion rate — use SAMPLE BY 1m on market_data to see rows-per-minute and spot where the feed dropped.
Check for stale instruments — query LATEST ON ts PARTITION BY instrument_id and compare last_seen against now().
query_prometheus (Grafana MCP) — check recorder-level metrics if available.
pods_log for backend-recorder-0 — look for ILP sender errors, QuestDB connection failures, or backpressure.
query_loki_logs — search for ERROR in backend-app logs filtered to StartDataFlowTask or RecoveryManager.
Check the Known Gaps in AGENTS.md — remaining pgwire paths are a known ingestion risk.

DataBridge is disconnected

query_prometheus — check DataBridgeLauncher readiness metrics or bridge connection health.
pods_log for backend-app-0 — look for connection errors in DataBridgeLauncher.
pods_log for backend-server-0 — look for DataBridgeReadinessReporter state changes.
events_list — check for pod restarts on either side that would explain a reconnect.

The bridge reconnects automatically with Arrow retry; a transient disconnect is not always an incident. Check whether it has recovered before escalating.

A strategy is not producing signals

try_query (Gel MCP, prod) — confirm the strategy is registered: SELECT Strategy { id, name, display_value } FILTER .name = 'strategy-name'.
try_query (Gel MCP) — confirm the strategy still has the expected instruments and feeds attached: SELECT Strategy { name, instruments: { id, name, feed: { name }, exchange, type } } FILTER .name = 'strategy-name'.
pods_log for backend-app-0 — check Processor, StrategyExecutor, or StartDataFlowTask for errors.
query_loki_logs — search for the strategy name in backend-app logs.
Check whether M2 (Strategy Execution in Production) is complete in docs/ROADMAP.md.

Check the status of a recent deployment

teamcity_rest_get to /app/rest/builds?locator=buildType:<deployBtId>,count:1 — confirm the deploy build succeeded.
pods_list (Kubernetes MCP, target environment) — verify pods are Running and ready.
resources_get for the relevant StatefulSet — check observedGeneration matches the expected revision.

Fallback Tools

When an MCP does not cover an operation, use these in order of preference:

# Kubernetes — use the appropriate kubeconfig
kubectl --kubeconfig $KUBECONFIG_PROD get pods -n <namespace>
kubectl --kubeconfig $KUBECONFIG_PROD logs backend-app-0 --previous
kubectl --kubeconfig ${KUBECONFIG_DEV:-~/.kube/config} get events -n <namespace>

# GitHub
gh issue list --repo timemanx/quant-app
gh run list --repo timemanx/quant-app
python3 scripts/github_issues.py list --state open

# Grafana CLI (gcx)
gcx --context grafana <command>    # for prod
gcx --context grafanad <command>   # for stg

# TeamCity CLI
teamcity run list --project QuantApp
teamcity run log <buildId>
teamcity api /app/rest/investigations

architecture-overview.md — module boundaries and runtime topology
live-market-pipeline.md — feed, processor, and DataBridge internals
backend-conventions.md — DI patterns and module manifest details
ROADMAP.md — current milestone status

Operations

MCP Tool Priority

Environment Map

MCP access and permissions

Kubeconfig access

Local backend-app leadership

Local Development

backend-sync

Sync Postgres monitoring

Instrument sync performance

QuestDB (Market Data)

Querying

Key QuestDB SQL patterns for this project

Table inspection

NATS (Raw Market Data Store)

Credential rotation

Logical JetStream usage (what NATS has stored)

Physical PVC space (consumed vs available on disk)

Continuous metrics (Prometheus / Grafana)

Optional guardrail

Raw-vs-QuestDB Reconciliation

POST /reconcile — summary report

GET /reconcile/diffs — SSE diff stream

Gel MCP (Application State)

Tools

Key EdgeQL patterns for this project

Kubernetes MCP Provides direct access to pods, logs, events, and all Kubernetes resources.

Key tools for health and diagnosis

Application module → resource mapping

Grafana MCP

Key tools for health and diagnosis

Useful PromQL starting points

Useful LogQL starting points

TeamCity MCP

GitHub MCP

Diagnostic Workflows

Is the system healthy?

A pod is crashlooping or restarting

A build failed

Market data ingestion is lagging or stopped

DataBridge is disconnected

A strategy is not producing signals

Check the status of a recent deployment

Fallback Tools

Related Documents

Local `backend-app` leadership

`backend-sync`

`POST /reconcile` — summary report

`GET /reconcile/diffs` — SSE diff stream