replica-omnisciente/firmware/telemetry/INGEST_API.md

Telemetry Ingest API — Bare-Metal Backend Spec

Version: 1.0 Target: Any HTTP-capable 32-bit CPU with MMU/MPU (ARM Cortex-A, RISC-V, x86) Runtime: Rust (Axum) or C++ (CivetWeb) — NO Python on bare metal.

Purpose

Single HTTP endpoint that receives telemetry from ALL surfaces:

Surface	Language	How it reaches this endpoint
Bare-metal agent (ESP32, STM32, RISC-V)	C++/Rust/Zig/Go	HttpTransport::transmit() POSTs JSON
aurelio-vscode extension	TypeScript	HttpTransport (fetch) POSTs JSON
aurelio-theia backend	TypeScript	HttpTransport (axios/fetch) POSTs JSON
Python data-sharing CLI	Python	http transport POSTs JSON
Web portal (aurelio-web)	TypeScript	Same HttpTransport from @aurelio/shared

All surfaces produce the SAME JSON envelope (WIRE_FORMAT.md, schema_version 1).

Endpoints

POST /api/telemetry/ingest

Receives a telemetry payload. Validates schema. Stores for aggregation.

Request headers:

Content-Type: application/json
X-Aurelio-Source: <source-identifier>     (e.g. "vscode", "theia", "esp32-s3-01", "cli")
X-Aurelio-Transport: http
Authorization: Bearer <token>              (optional, per-client)

Request body — the wire format envelope (see WIRE_FORMAT.md):

{
  "schema_version": 1,
  "collected_at": "2026-07-30T17:00:00Z",
  "device_id": "esp32-s3-01",
  "platform": "esp-idf v5.5.4",
  "consent": {
    "categories": ["tool_calls", "environment"],
    "retention_days": 90,
    "redact_secrets": true
  },
  "environment": { "os": "FreeRTOS", "chip": "ESP32-S3" },
  "tool_calls": [...],
  "_summary": { "environment": 1, "tool_calls": 3 }
}

Response:

• 202 Accepted — payload valid, queued for storage
• 400 Bad Request — invalid schema (missing required fields, wrong version)
• 401 Unauthorized — bad/missing token (when auth is required)
• 413 Payload Too Large — body exceeds size limit (default 64KB)

Validation rules:

1. schema_version MUST be 1
2. collected_at MUST be a valid ISO 8601 timestamp
3. device_id MUST be a non-empty string
4. consent MUST be present with a categories array
5. Every top-level category key MUST be in consent.categories (no unconsented data)
6. Body size MUST NOT exceed 64KB (bare-metal friendly)

GET /api/telemetry/health

Health check. Returns {"status":"ok","uptime_s":<seconds>}.

GET /api/telemetry/stats

Aggregate stats (for dashboards). Returns counts per source, per category, per day.

Storage

On bare metal, payloads are stored in a ring buffer on flash/NVRAM. On server-class targets (Linux), payloads go to SQLite or append-only JSONL.

Storage layout (JSONL, one line per payload):

/data/telemetry/2026-07-30.jsonl
/data/telemetry/2026-07-31.jsonl

Retention: consent.retention_days (default 90). Older files are deleted by a background task (on server) or on boot (bare metal).

Implementation Targets

Rust (Axum) — primary

Single binary, ~2MB static. Targets: ARM Cortex-A (Linux), RISC-V (Linux), x86_64 (Linux). Can also run as a service on Proxmox CTs.

[dependencies]
axum = "0.8"
tokio = { version = "1", features = ["full"] }
serde = { version = "1", features = ["derive"] }
serde_json = "1"

C++ (CivetWeb) — minimal footprint

Single binary, ~500KB. Targets: ARM Cortex-A (Linux), RISC-V (Linux), x86 (Linux). For constrained environments where Rust toolchain is unavailable.

Go (TinyGo) — microcontroller

For ESP32/STM32 with networking (WiFi/Ethernet). Runs the ingest endpoint directly on the device — peer-to-peer telemetry without a server.

Deployment

Target	How
Proxmox CT	systemd service, binary at /usr/local/bin/aurelio-telemetry-ingest
Docker	docker run -p 8080:8080 aurelio/telemetry-ingest
Bare metal (Yocto)	bitbake aurelio-telemetry-ingest — systemd unit
ESP32 (TinyGo)	tinygo flash -target=esp32-coreboard-v2 ./cmd/ingest
STM32 (Rust)	probe-rs run --chip STM32F407VGTx target/thumbv7em-none-eabihf/release/ingest

Relationship to existing infra

This endpoint REPLACES the Proxmox-hardcoded sync.py --push for telemetry. The old POST /api/brain/push endpoint (Gabinete daemon) remains for brain session sync — this new endpoint is for the consent-gated telemetry stream.

Before:  device → sync.py --push → ssh root@192.168.0.38 → pct push → CT 208
After:   device → POST /api/telemetry/ingest → any backend (bare metal, CT, Docker)