39fb44fe0e
- infrastructure/proxmox/: CT provisioning configs - infrastructure/fabric/gitops/: GitOps layer - fleet.yaml: GPU inventory (Dell GTX 1050 vfio-pci passthrough) - firmware/: ESP32 firmware tree (67 files, 6.9MB) - realms/savearth/: Savearth team realm with heteronyms - lab-gateway: ESP client + manager - CI: Forgejo + GitHub Actions workflows Co-authored-by: Álvaro de Campos <campos@portugalfuturista.org>
133 lines
4.5 KiB
Markdown
133 lines
4.5 KiB
Markdown
# Telemetry Ingest API — Bare-Metal Backend Spec
|
|
|
|
Version: 1.0
|
|
Target: Any HTTP-capable 32-bit CPU with MMU/MPU (ARM Cortex-A, RISC-V, x86)
|
|
Runtime: Rust (Axum) or C++ (CivetWeb) — NO Python on bare metal.
|
|
|
|
## Purpose
|
|
|
|
Single HTTP endpoint that receives telemetry from ALL surfaces:
|
|
|
|
| Surface | Language | How it reaches this endpoint |
|
|
|---------|----------|------------------------------|
|
|
| Bare-metal agent (ESP32, STM32, RISC-V) | C++/Rust/Zig/Go | `HttpTransport::transmit()` POSTs JSON |
|
|
| aurelio-vscode extension | TypeScript | `HttpTransport` (fetch) POSTs JSON |
|
|
| aurelio-theia backend | TypeScript | `HttpTransport` (axios/fetch) POSTs JSON |
|
|
| Python data-sharing CLI | Python | `http` transport POSTs JSON |
|
|
| Web portal (aurelio-web) | TypeScript | Same `HttpTransport` from `@aurelio/shared` |
|
|
|
|
All surfaces produce the SAME JSON envelope (WIRE_FORMAT.md, schema_version 1).
|
|
|
|
## Endpoints
|
|
|
|
### `POST /api/telemetry/ingest`
|
|
|
|
Receives a telemetry payload. Validates schema. Stores for aggregation.
|
|
|
|
**Request headers:**
|
|
```
|
|
Content-Type: application/json
|
|
X-Aurelio-Source: <source-identifier> (e.g. "vscode", "theia", "esp32-s3-01", "cli")
|
|
X-Aurelio-Transport: http
|
|
Authorization: Bearer <token> (optional, per-client)
|
|
```
|
|
|
|
**Request body** — the wire format envelope (see WIRE_FORMAT.md):
|
|
```json
|
|
{
|
|
"schema_version": 1,
|
|
"collected_at": "2026-07-30T17:00:00Z",
|
|
"device_id": "esp32-s3-01",
|
|
"platform": "esp-idf v5.5.4",
|
|
"consent": {
|
|
"categories": ["tool_calls", "environment"],
|
|
"retention_days": 90,
|
|
"redact_secrets": true
|
|
},
|
|
"environment": { "os": "FreeRTOS", "chip": "ESP32-S3" },
|
|
"tool_calls": [...],
|
|
"_summary": { "environment": 1, "tool_calls": 3 }
|
|
}
|
|
```
|
|
|
|
**Response:**
|
|
- `202 Accepted` — payload valid, queued for storage
|
|
- `400 Bad Request` — invalid schema (missing required fields, wrong version)
|
|
- `401 Unauthorized` — bad/missing token (when auth is required)
|
|
- `413 Payload Too Large` — body exceeds size limit (default 64KB)
|
|
|
|
**Validation rules:**
|
|
1. `schema_version` MUST be `1`
|
|
2. `collected_at` MUST be a valid ISO 8601 timestamp
|
|
3. `device_id` MUST be a non-empty string
|
|
4. `consent` MUST be present with a `categories` array
|
|
5. Every top-level category key MUST be in `consent.categories` (no unconsented data)
|
|
6. Body size MUST NOT exceed 64KB (bare-metal friendly)
|
|
|
|
### `GET /api/telemetry/health`
|
|
|
|
Health check. Returns `{"status":"ok","uptime_s":<seconds>}`.
|
|
|
|
### `GET /api/telemetry/stats`
|
|
|
|
Aggregate stats (for dashboards). Returns counts per source, per category, per day.
|
|
|
|
## Storage
|
|
|
|
On bare metal, payloads are stored in a ring buffer on flash/NVRAM. On
|
|
server-class targets (Linux), payloads go to SQLite or append-only JSONL.
|
|
|
|
Storage layout (JSONL, one line per payload):
|
|
```
|
|
/data/telemetry/2026-07-30.jsonl
|
|
/data/telemetry/2026-07-31.jsonl
|
|
```
|
|
|
|
Retention: `consent.retention_days` (default 90). Older files are deleted
|
|
by a background task (on server) or on boot (bare metal).
|
|
|
|
## Implementation Targets
|
|
|
|
### Rust (Axum) — primary
|
|
|
|
Single binary, ~2MB static. Targets: ARM Cortex-A (Linux), RISC-V (Linux),
|
|
x86_64 (Linux). Can also run as a service on Proxmox CTs.
|
|
|
|
```toml
|
|
[dependencies]
|
|
axum = "0.8"
|
|
tokio = { version = "1", features = ["full"] }
|
|
serde = { version = "1", features = ["derive"] }
|
|
serde_json = "1"
|
|
```
|
|
|
|
### C++ (CivetWeb) — minimal footprint
|
|
|
|
Single binary, ~500KB. Targets: ARM Cortex-A (Linux), RISC-V (Linux),
|
|
x86 (Linux). For constrained environments where Rust toolchain is unavailable.
|
|
|
|
### Go (TinyGo) — microcontroller
|
|
|
|
For ESP32/STM32 with networking (WiFi/Ethernet). Runs the ingest endpoint
|
|
directly on the device — peer-to-peer telemetry without a server.
|
|
|
|
## Deployment
|
|
|
|
| Target | How |
|
|
|--------|-----|
|
|
| Proxmox CT | systemd service, binary at /usr/local/bin/aurelio-telemetry-ingest |
|
|
| Docker | `docker run -p 8080:8080 aurelio/telemetry-ingest` |
|
|
| Bare metal (Yocto) | `bitbake aurelio-telemetry-ingest` — systemd unit |
|
|
| ESP32 (TinyGo) | `tinygo flash -target=esp32-coreboard-v2 ./cmd/ingest` |
|
|
| STM32 (Rust) | `probe-rs run --chip STM32F407VGTx target/thumbv7em-none-eabihf/release/ingest` |
|
|
|
|
## Relationship to existing infra
|
|
|
|
This endpoint REPLACES the Proxmox-hardcoded `sync.py --push` for telemetry.
|
|
The old `POST /api/brain/push` endpoint (Gabinete daemon) remains for brain
|
|
session sync — this new endpoint is for the consent-gated telemetry stream.
|
|
|
|
```
|
|
Before: device → sync.py --push → ssh root@192.168.0.38 → pct push → CT 208
|
|
After: device → POST /api/telemetry/ingest → any backend (bare metal, CT, Docker)
|
|
```
|