Skip to content

Infrastructure

This document covers the Terraform-managed edge and Cloudflare infrastructure for quant-app.

For how Kubernetes resources are generated from Gradle, see build-system.md. For CI/CD pipeline details, see ci-cd.md.

Overview

Infrastructure is split across four Terraform stacks:

  • terraform/edge/oci — OCI and Cloudflare-managed production edge: tunnel, DNS, Zero Trust access policies
  • terraform/edge/contabo — Contabo VPS and Cloudflare-managed staging edge: VPS, firewall, tunnel, DNS, Zero Trust access policies (mirrors the OCI stack)
  • terraform/edge/local — dev edge stack for the local cluster: equivalent Cloudflare tunnel deployment
  • terraform/kubernetes — in-cluster monitoring and application workloads that sit behind the edge

Terraform Variable Loading

Each Terraform stack loads variable values into the shell before running any terraform command. The file to source depends on the stack and target environment:

  • terraform/edge/oci (production) and terraform/edge/contabo (staging) each use a .env file created in the stack directory (see each stack's README).
  • terraform/edge/local (dev) is driven by inline export commands rather than a sourced env file — see its README.
  • terraform/kubernetes uses .env for the default (production) workspace and .env.stg for the stg (local dev) workspace.

Source the appropriate file before running Terraform:

# Production
cd terraform/edge/oci
source .env && terraform plan

# Staging
cd terraform/edge/contabo
source .env && terraform plan

This applies to all four stacks (terraform/edge/oci, terraform/edge/contabo, terraform/edge/local, terraform/kubernetes). Never run terraform plan, terraform apply, or terraform destroy in any of these directories without sourcing the correct env file (or exporting the required variables for the local stack) first — commands will fail or use incorrect variable values.

Terraform Edge (terraform/edge)

Production Edge (terraform/edge/oci)

  • Reads application hosts from gradle.prod.properties.
  • Fails fast if any app host does not belong to the configured Cloudflare zone.
  • Grafana remains Terraform-owned via grafana_subdomain; it is not sourced from Gradle properties.

Staging Edge (terraform/edge/contabo)

  • Provisions a Contabo VPS bootstrapped with K3s + cloudflared via cloud-init, mirroring the OCI production stack.
  • K3s version pinning: K3s is pinned to a single release via the shared terraform/modules/versions module (k3s output), consumed by both the OCI (production) and Contabo (staging) edge stacks and passed to the installer as INSTALL_K3S_VERSION. The k8s integration tests run against the same release via the conventions.kubernetes.integration-test convention plugin, which generates the K3S_IMAGE build config field into the k8sIntegrationTest source set from the k3s-test version catalog entry. The local dev cluster (k3d) is operator-managed and not pinned by Terraform.
  • A Contabo firewall (default-deny allowlist) permits only SSH (from ssh_allowed_ips) and ICMP; all application ingress flows through the Cloudflare Tunnel.
  • First-boot firewall window (accepted exception): the Contabo API/provider cannot attach a firewall before the instance boots — contabo_firewall polls for status=="running" before assigning, so the instance briefly sits on Contabo's default-open inbound policy on first boot. This is mitigated by a persistent host-level nftables firewall (the qapp_hostfw table, K3s/Flannel-aware: it allows pod traffic on cni0/flannel.1 in both input and forward while dropping unsolicited public-IP→pod/NodePort forwarding) loaded by a dedicated qapp-hostfw.service systemd unit, plus key-only SSH. The unit manages only its own table and is never applied via a global flush, so it does not clobber K3s's own iptables/nft rules; it is ordered Before=network-pre.target k3s.service so it is active before networking/K3s on reboots (on first boot it is enabled from cloud-init runcmd, the earliest point after the nftables package is installed). Steady state is therefore stricter than OCI production (which has no host firewall): Contabo network firewall + persistent host firewall + key-only SSH. Assumptions that must hold or the ruleset breaks K3s: single-node cluster (no cross-node VXLAN/8472 allow — add it before multi-node), default Flannel (cni0/flannel.1), ClusterIP-only ingress (cloudflared→traefik.kube-system.svc.cluster.local), and remote kubectl to 6443 is not reachable (use SSH port-forwarding — parity with OCI). The distro nftables.service is disabled and masked so it can never run a global flush that would erase qapp_hostfw on reboot; qapp-hostfw.service is the single authoritative nftables unit. The bootstrap runs under set -e and asserts the unit is active and the table is loaded before installing K3s, so a firewall activation failure fails closed rather than proceeding unprotected. The input chain explicitly allows DHCPv4/v6 (UDP 67/68 and 546/547) so the pre-network firewall cannot drop lease acquisition/renewal on a DHCP-based image; the K3s installer is downloaded and validated before execution (a curl | sh pipeline masks curl failures) and pinned via INSTALL_K3S_VERSION; SSH hardening (sshd -t + reload/restart) fails closed under set -e. The residual first-boot window before runcmd is an inherent Contabo platform limitation; see ISSUES.md (QAPP-114).
  • Reads application hosts from gradle.stg.properties.
  • Fails fast if any app host does not belong to the configured Cloudflare zone.
  • Grafana remains Terraform-owned via grafana_subdomain.
  • The Contabo product_id (plan, e.g. "VPS 4") is a required variable with no default — look it up at the Contabo product list before applying.

Local Dev Edge (terraform/edge/local)

  • Reads application hosts from gradle.dev.properties (the dev build environment).
  • Deploys cloudflared-local into the local cluster.
  • Uses dev as the Cloudflare tunnel/app/policy environment label plus the Terraform-owned Grafana host.
  • Dev app hosts are localhost and are NOT published to Cloudflare (the dev cluster is accessed locally). Only the Terraform-owned d-suffixed Grafana subdomain (grafanad) is published through the dev tunnel.
  • Real (non-localhost) app hosts still fail fast if they do not belong to the configured Cloudflare zone; the shared app_hosts module exempts only localhost/loopback hosts from publishing and zone validation.
  • Owns the d-suffixed host namespace, distinct from staging's s-suffix (apps, dbs, …) and production's unsuffixed hosts. The earlier conflict with the staging stack (QAPP-113) is resolved.

Shared Modules

Shared edge logic lives in child modules. Prefer updating these rather than the stack-local files when changing cross-cutting behavior:

  • terraform/modules/app_hosts — hostname parsing and validation
  • terraform/modules/cloudflare_edge — tunnel publishing, Grafana UI IP allowlist
  • terraform/modules/versions — shared runtime version pins

Configure the Grafana UI IP allowlist through allowed_ips in the shared module. Override the hostname label only if the public Grafana subdomain changes.

Terraform and Provider Version Pinning

required_version and required_providers { version = ... } accept only literal strings — they cannot reference variables, locals, or tfvars — so there is no native single file for them. Each root stack pins only the providers it uses, and scripts/check_terraform_versions.py (run in CI right after terraform fmt) enforces consistency: the same provider source must be pinned to the same version in every root, required_version must be identical across roots, roots must pin every provider they declare, and shared modules stay source-only (the root decides the version, per HashiCorp module guidance). Runtime versions (K3s, cloudflared) are the exception — they are values, not constraints, and live in terraform/modules/versions.

GitHub Actions installs the exact Terraform version pinned in the workflow-level TERRAFORM_VERSION env (.github/workflows/build.yml) rather than the latest release. The CI version-consistency step passes that pin to scripts/check_terraform_versions.py --ci-version, which fails if the pin stops satisfying the roots' required_version — so bumping required_version without updating TERRAFORM_VERSION (or vice versa) breaks CI loudly instead of drifting silently.

Cloudflare API Token Requirements

The API token used by all edge stacks must include: Zone:Read, DNS:Edit, Tunnel:Edit, and Access: Apps and Policies:Edit.

Terraform Kubernetes (terraform/kubernetes)

  • Owns in-cluster monitoring and application workloads behind the edge.
  • Important: To prevent overwriting the OCI production state, use Terraform workspaces. The production state resides in the default workspace. When applying to the local cluster, you must switch to or create a separate workspace (e.g., terraform workspace new stg or terraform workspace select stg) before running source .env.stg && terraform apply.

Token Broker Secrets

terraform/kubernetes owns the Kubernetes Secrets consumed by backend-token-broker. Apply Terraform before deploying the broker for the first time.

OAuth credentials

upstox-oauth-credentials supplies UPSTOX_CLIENT_ID and UPSTOX_CLIENT_SECRET to the pod as Secret-backed environment variables. Provide their environment-specific values through the sensitive, required upstox_client_id and upstox_client_secret Terraform inputs. The generated Deployment contains only the Secret name and keys; the secret values reside in Terraform state.

Provisioned tokens

token-broker-provisioned-tokens mounts provisioned token JSON files into the pod.

  • Secret data keys: upstox-sandbox.json, upstox-analytics.json
  • File format: {"token": "....", "expiry": "2027-07-15T00:00:00Z"}
  • Variables: upstox_sandbox_provisioned_token_json, upstox_analytics_provisioned_token_json (both sensitive, required — no defaults)
  • Terraform validation enforces that both token and expiry are non-empty JSON strings and that expiry is a valid RFC 3339 timestamp — catching malformed values at terraform plan/apply time rather than deferring failures to the broker
  • The token JSON values are stored in Terraform state — this is acceptable only if Terraform state storage is already treated as sensitive infrastructure data
  • Rotation: After updating the Terraform inputs and reapplying, the kubelet refreshes the mounted files and the token broker recovers on its next poll cycle — no pod restart is required

Sync Postgres metrics

sync_metrics_password is a required, sensitive Terraform input for the sync_metrics role used by the backend-sync Postgres exporter. Set a unique value in the target Kubernetes environment file before sourcing it and running Terraform:

export TF_VAR_sync_metrics_password="<unique-secret>"

Do not commit the value, Gradle properties, or rendered manifests containing it. Terraform creates the backend-sync-metrics Secret; apply terraform/kubernetes before deploying the backend-sync manifest for the first time. Updating the value and reapplying Terraform rolls backend-sync, whose bootstrap init container updates the database role before the exporter starts.

Deployment Environments

Environment Cluster Edge stack Properties source Env file
Production OCI k3s terraform/edge/oci gradle.prod.properties .env
Staging Contabo k3s terraform/edge/contabo gradle.stg.properties .env
Dev local k3d terraform/edge/local gradle.dev.properties .env.stg