Skip to content

Infrastructure

This document covers the Terraform-managed edge and Cloudflare infrastructure for quant-app.

For how Kubernetes resources are generated from Gradle, see build-system.md. For CI/CD pipeline details, see ci-cd.md.

Overview

Infrastructure is split across three Terraform stacks:

  • terraform/edge/oci — OCI and Cloudflare-managed production edge: tunnel, DNS, Zero Trust access policies
  • terraform/edge/local — dev edge stack for the local cluster: equivalent Cloudflare tunnel deployment
  • terraform/kubernetes — in-cluster monitoring and application workloads that sit behind the edge; reads the Grafana MCP public host from the selected edge state

Terraform Variable Loading

Each Terraform configuration directory contains environment files that load variable values into the shell before running any terraform command:

  • .env — production variable values
  • .env.stg — staging variable values

Source the appropriate file before running Terraform:

# Production
cd terraform/edge/oci
source .env && terraform plan

# Staging
cd terraform/edge/local
source .env.stg && terraform plan

This applies to all three stacks (terraform/edge/oci, terraform/edge/local, terraform/kubernetes). Never run terraform plan, terraform apply, or terraform destroy in any of these directories without sourcing the correct env file first — commands will fail or use incorrect variable values.

Terraform Edge (terraform/edge)

Production Edge (terraform/edge/oci)

  • Reads application hosts from gradle.prod.properties.
  • Fails fast if any app host does not belong to the configured Cloudflare zone.
  • Grafana and Grafana MCP remain Terraform-owned via grafana_subdomain / grafana_mcp_subdomain; they are not sourced from Gradle properties.

Local Dev Edge (terraform/edge/local)

  • Reads application hosts from gradle.stg.properties.
  • Deploys cloudflared-local into the local cluster.
  • Uses dev as the Cloudflare tunnel/app/policy environment label plus the Terraform-owned Grafana hosts.
  • Fails fast if any app host does not belong to the configured Cloudflare zone.

Shared Modules

Shared edge logic lives in child modules. Prefer updating these rather than the stack-local files when changing cross-cutting behavior:

  • terraform/modules/app_hosts — hostname parsing and validation
  • terraform/modules/cloudflare_edge — tunnel publishing, Grafana MCP IP allowlist
  • terraform/modules/versions — shared runtime version pins

Configure the Grafana MCP IP allowlist through allowed_ips in the shared module. Override hostname labels only if the public Grafana or Grafana MCP subdomains change.

Cloudflare API Token Requirements

The API token used by both edge stacks must include: Zone:Read, DNS:Edit, Tunnel:Edit, and Access: Apps and Policies:Edit.

Terraform Kubernetes (terraform/kubernetes)

  • Owns in-cluster monitoring and application workloads behind the edge.
  • Resolves the Grafana MCP ingress host from the edge state configured by edge_state_path.
  • Default: terraform/edge/oci/terraform.tfstate. Point at terraform/edge/local/terraform.tfstate when applying against the local cluster.
  • Important: To prevent overwriting the OCI production state, use Terraform workspaces. The production state resides in the default workspace. When applying to the local cluster, you must switch to or create a separate workspace (e.g., terraform workspace new stg or terraform workspace select stg) before running source .env.stg && terraform apply.

Provisioned Token Secret

terraform/kubernetes owns the token-broker-provisioned-tokens Kubernetes Secret, which mounts provisioned token JSON files into the backend-token-broker pod.

  • Secret data keys: upstox-sandbox.json, upstox-analytics.json
  • File format: {"token": "....", "expiry": "2027-07-15T00:00:00Z"}
  • Variables: upstox_sandbox_provisioned_token_json, upstox_analytics_provisioned_token_json (both sensitive, required — no defaults)
  • Terraform validation enforces that both token and expiry are non-empty JSON strings and that expiry is a valid RFC 3339 timestamp — catching malformed values at terraform plan/apply time rather than deferring failures to the broker
  • The token JSON values are stored in Terraform state — this is acceptable only if Terraform state storage is already treated as sensitive infrastructure data
  • Apply sequencing: terraform/kubernetes must be applied before deploying backend-token-broker for the first time, so the Deployment can reference the Secret
  • Rotation: After updating the Terraform inputs and reapplying, the kubelet refreshes the mounted files and the token broker recovers on its next poll cycle — no pod restart is required

Deployment Environments

Environment Cluster Edge stack Properties source Env file
Production OCI k3s terraform/edge/oci gradle.prod.properties .env
Staging local k3d terraform/edge/local gradle.stg.properties .env.stg