fabiorafaelcoutada 223cdc2fb9 docs(compose): aurelio-theia migration plans + proxmox new-server migration plan/spec

2026-07-12 20:53:08 +01:00

13 KiB

Raw Blame History

Proxmox New Server Migration — Implementation Plan

For agentic workers: REQUIRED SUB-SKILL: Use compose:subagent (recommended) or compose:execute to implement this plan task-by-task. Steps use checkbox (- [ ]) syntax for tracking.

Goal: Add a new mid-range server to the Proxmox cluster and migrate CTs 208, 212, 213, 216 from ASUS to relieve the 55.5 GB overcommit on a 16 GB host.

Architecture: Join the new server as a 3rd node in the portugalfuturi cluster. Migrate CTs via vzdump backup + restore. Update Cloudflare tunnel routes, MCP registry, sync scripts, and proxy backends to new IPs.

Tech Stack: Proxmox VE, Corosync, Cloudflare Tunnel, Python sync scripts, Node.js proxy server

Global Constraints

ASUS host: 192.168.0.38 (current primary, must remain operational during migration)
Gigabyte host: 192.168.0.104 (existing cluster node)
New server IP: TBD (must be on 192.168.0.x subnet, outside DHCP range)
All Cloudflare routes use hostnames — only origin IPs change
Brain sync (sync.py) uses pct push/pct pull via SSH to ASUS — must verify this still works after migration
CT 208 should be downsized from 8 GB to 2 GB (only uses 72 MB)

Task 1: Pre-migration snapshot and backup

Covers: [S4]

Files:

Modify: None (operational commands only)

Verification:

Run: ssh root@192.168.0.38 "pct list" — all CTs listed
Run: ssh root@192.168.0.38 "vzdump 208 --storage local --compress zstd --dumpdir /tmp/vzdump/" — backup created
Step 1: Verify current CT state on ASUS

ssh root@192.168.0.38 "pct list"

Expected: CTs 207, 208, 209, 212, 213, 214, 215, 216, 220 listed as running.

Step 2: Create vzdump backups of CTs to migrate

ssh root@192.168.0.38 "mkdir -p /tmp/vzdump && vzdump 208 --storage local --compress zstd --dumpdir /tmp/vzdump/"
ssh root@192.168.0.38 "vzdump 212 --storage local --compress zstd --dumpdir /tmp/vzdump/"
ssh root@192.168.0.38 "vzdump 213 --storage local --compress zstd --dumpdir /tmp/vzdump/"
ssh root@192.168.0.38 "vzdump 216 --storage local --compress zstd --dumpdir /tmp/vzdump/"

Expected: 4 .tar.zst files in /tmp/vzdump/ on ASUS.

Step 3: Verify backups exist

ssh root@192.168.0.38 "ls -la /tmp/vzdump/"

Expected: 4 backup files, each non-empty.

Step 4: Record current service state for rollback reference

ssh root@192.168.0.38 "for ct in 208 212 213 216; do echo \"CT \$ct:\"; pct exec \$ct -- systemctl list-units --type=service --state=running --no-pager 2>/dev/null | head -10; echo; done"

Expected: List of running services per CT. Save this output for comparison after migration.

Task 2: Install and join new server to cluster

Covers: [S3]

Files:

Create: None (Proxmox installer + cluster join)

Verification:

Run: corosync-cfgtool -s on any node — shows 3 members
Run: pct list on new node — operational
Step 1: Install Proxmox VE on new server

Follow standard Proxmox VE installer. Assign static IP (e.g., 192.168.0.40 — verify no conflict first).

# On ASUS, check for IP conflicts
ssh root@192.168.0.38 "arp-scan --localnet | grep 192.168.0.40"

Expected: No response (IP is free).

Step 2: Add new node to cluster

On the new node, join the existing cluster:

# On ASUS (existing node), get join info
ssh root@192.168.0.38 "pveam key --master 1"

On the new node:

# Replace <JOIN-info> with output from step above
pvecm add 192.168.0.38 --key <JOIN-info>

Step 3: Verify cluster health

# On any node
corosync-cfgtool -s
pvecm status
pvecm nodes

Expected: 3 nodes listed (ASUS, Gigabyte, new), quorum achieved.

Step 4: Verify new node is ready for CT restore

ssh root@<NEW_IP> "pct list && pveam available | head -5"

Expected: Empty CT list, storage available.

Task 3: Migrate CTs to new node

Covers: [S4]

Files:

None (operational commands)

Verification:

Run: pct list on new node — shows CTs 208, 212, 213, 216
Run: Service health checks on each CT
Step 1: Transfer backups to new node

NEW_IP="192.168.0.40"  # Replace with actual IP
scp /tmp/vzdump/vzdump-lxc-208-*.tar.zst root@${NEW_IP}:/tmp/
scp /tmp/vzdump/vzdump-lxc-212-*.tar.zst root@${NEW_IP}:/tmp/
scp /tmp/vzdump/vzdump-lxc-213-*.tar.zst root@${NEW_IP}:/tmp/
scp /tmp/vzdump/vzdump-lxc-216-*.tar.zst root@${NEW_IP}:/tmp/

Step 2: Restore CT 208 (Gabinete) with downsized RAM

ssh root@${NEW_IP} "pct restore 208 /tmp/vzdump-lxc-208-*.tar.zst --memory 2048 --hostname pf-gabinete-dos-heteronimos"

Expected: CT 208 created with 2 GB RAM.

Step 3: Restore CT 212 (Antigravity)

ssh root@${NEW_IP} "pct restore 212 /tmp/vzdump-lxc-212-*.tar.zst --memory 8192 --hostname pf-antigravity"

Step 4: Restore CT 213 (Immich)

ssh root@${NEW_IP} "pct restore 213 /tmp/vzdump-lxc-213-*.tar.zst --memory 4096 --hostname pf-immich"

Step 5: Restore CT 216 (Management) with downsized RAM

ssh root@${NEW_IP} "pct restore 216 /tmp/vzdump-lxc-216-*.tar.zst --memory 4096 --hostname pf-management"

Step 6: Start all restored CTs

ssh root@${NEW_IP} "pct start 208 && pct start 212 && pct start 213 && pct start 216"

Step 7: Verify services on each CT

# CT 208 - Gabinete health
ssh root@${NEW_IP} "pct exec 208 -- curl -s http://localhost:18000/api/health"

# CT 216 - Authentik
ssh root@${NEW_IP} "pct exec 216 -- systemctl status authentik --no-pager | head -5"

# CT 212 - Antigravity
ssh root@${NEW_IP} "pct exec 212 -- curl -s http://localhost:8900/ 2>/dev/null | head -1 || echo 'CDP Bridge check needed'"

# CT 213 - Immich
ssh root@${NEW_IP} "pct exec 213 -- curl -s http://localhost:2283/api/server/about 2>/dev/null | head -1 || echo 'Immich check needed'"

Expected: CT 208 returns health OK, CT 216 shows authentik active, others respond or are verifiable.

Step 8: Stop migrated CTs on ASUS

ssh root@192.168.0.38 "pct stop 208 && pct stop 212 && pct stop 213 && pct stop 216"

Step 9: Verify ASUS resource improvement

ssh root@192.168.0.38 "free -h && uptime"

Expected: RAM usage drops significantly, load decreases.

Task 4: Update Cloudflare tunnel routes

Covers: [S5]

Files:

Modify: nervura-electrica/infrastructure/proxmox/config.yml
Modify: /etc/cloudflared/config.yml on ASUS (deployed copy)

Verification:

Run: curl -s https://mcp.portugalfuturista.org/api/health — returns OK
Run: curl -s https://auth.portugalfuturista.org — returns Authentik page
Step 1: Get new IP assignments for migrated CTs

ssh root@${NEW_IP} "pct exec 208 -- hostname -I"
ssh root@${NEW_IP} "pct exec 212 -- hostname -I"
ssh root@${NEW_IP} "pct exec 213 -- hostname -I"
ssh root@${NEW_IP} "pct exec 216 -- hostname -I"

Record the IPs. Update the variable below:

NEW_CT208_IP=<from step>
NEW_CT212_IP=<from step>
NEW_CT213_IP=<from step>
NEW_CT216_IP=<from step>

Step 2: Update config.yml origin IPs

Edit nervura-electrica/infrastructure/proxmox/config.yml:

# CT 208 → new node
- hostname: mcp.portugalfuturista.org
  service: http://${NEW_CT208_IP}:18000

# CT 212 → new node
- hostname: antigravity.portugalfuturista.org
  service: http://${NEW_CT212_IP}:8900
- hostname: antigravity-ws.portugalfuturista.org
  service: http://${NEW_CT212_IP}:8901

# CT 213 → new node
- hostname: photos.portugalfuturista.org
  service: http://${NEW_CT213_IP}:2283

# CT 216 → new node
- hostname: auth.portugalfuturista.org
  service: http://${NEW_CT216_IP}:9000
- hostname: vault.portugalfuturista.org
  service: http://${NEW_CT216_IP}:8088
- hostname: scrobbler.portugalfuturista.org
  service: http://${NEW_CT216_IP}:9078
- hostname: maloja.portugalfuturista.org
  service: http://${NEW_CT216_IP}:42010

Step 3: Deploy updated config to Cloudflare tunnel

scp nervura-electrica/infrastructure/proxmox/config.yml root@192.168.0.38:/etc/cloudflared/config.yml
ssh root@192.168.0.38 "systemctl restart cloudflared"

Step 4: Verify tunnel is healthy

ssh root@192.168.0.38 "systemctl status cloudflared --no-pager | head -5"

Expected: Active (running).

Step 5: Test critical routes

curl -s -o /dev/null -w "%{http_code}" https://mcp.portugalfuturista.org/api/health
curl -s -o /dev/null -w "%{http_code}" https://auth.portugalfuturista.org
curl -s -o /dev/null -w "%{http_code}" https://vault.portugalfuturista.org
curl -s -o /dev/null -w "%{http_code}" https://photos.portugalfuturista.org

Expected: All return 200 or 302 (redirect to login).

Task 5: Update MCP registry and sync scripts

Covers: [S5]

Files:

Modify: nervura-electrica/infrastructure/proxmox/services/gabinete/mcp_registry.json
Modify: replica-omnisciente/.aurelio/sync.py (if CT 208 target changes)
Modify: nervura-electrica/infrastructure/proxmox/services/gabinete/deploy_gabinete_208.sh

Verification:

Run: curl http://<NEW_CT208_IP>:18000/api/services — returns spoke list
Run: python3 .aurelio/sync.py --push — completes successfully
Step 1: Update mcp_registry.json spoke URLs

Edit nervura-electrica/infrastructure/proxmox/services/gabinete/mcp_registry.json:

Replace all 192.168.0.125 references with ${NEW_CT208_IP} (the actual IP from Task 4).

Also update the savearth-mcp URL if CT 212 moved (it was on 192.168.0.212).

Step 2: Update deploy_gabinete_208.sh

The deploy script uses pct exec 208 on the Proxmox host. After migration, CT 208 lives on the new node. The script needs to target the new node:

Edit nervura-electrica/infrastructure/proxmox/services/gabinete/deploy_gabinete_208.sh:

# Change PROXMOX_HOST to new node IP
PROXMOX_HOST="${NEW_NODE_IP}"  # was 192.168.0.38

Step 3: Update sync.py if needed

Check if sync.py hardcodes CT 208 target. If it uses pct push 208 on ASUS, it needs to either:

SSH to the new node instead, or
Use the cluster-aware approach (Proxmox cluster allows pct commands on any node for any CT)

grep -n "208\|pct push\|pct exec" replica-omnisciente/.aurelio/sync.py

If pct push 208 is hardcoded to ASUS, update the SSH target to the new node.

Step 4: Update aurelio-web server.cjs proxy backends

Edit nervura-electrica/infrastructure/proxmox/aurelio-web/server.cjs:

// Update GABINETE_URL if CT 208 IP changed
const GABINETE_URL = process.env.GABINETE_URL || 'http://${NEW_CT208_IP}:18000';

Step 5: Deploy updated aurelio-web

cd nervura-electrica/infrastructure/proxmox
./deploy-aurelio-web.sh

Step 6: Test brain sync

cd replica-omnisciente
python3 .aurelio/sync.py --push

Expected: Sync completes without errors.

Task 6: Post-migration cleanup and documentation

Covers: [S6, S7]

Files:

Modify: nervura-electrica/AGENTS.md (update CT table, IPs, resource allocation)

Verification:

Run: ssh root@192.168.0.38 "free -h" — RAM usage < 4 GB
Run: All Cloudflare endpoints return expected responses
Step 1: Clean up /tmp on ASUS

ssh root@192.168.0.38 "rm -rf /tmp/vzdump/ /tmp/local_brain.tar.gz"

Step 2: Remove old CT configs from ASUS (optional, for cleanliness)

# Only if CT configs are cluttering; the CTs are stopped, not deleted
ssh root@192.168.0.38 "pct destroy 208 --purge"  # Only after verifying migration success

Note: Do NOT run this until Task 4 and Task 5 verification passes completely.

Step 3: Update AGENTS.md with new topology

Edit nervura-electrica/AGENTS.md:

Update CT table: CTs 208, 212, 213, 216 now on new node
Update resource allocation: ASUS now ~4 GB used
Add new server to physical infrastructure table
Update Cloudflare tunnel documentation
Update deployment commands (deploy scripts target new node)
Step 4: Final verification sweep

# Cluster health
corosync-cfgtool -s
pvecm nodes

# ASUS resources
ssh root@192.168.0.38 "free -h && uptime && df -h /"

# All endpoints
for host in mcp auth vault photos antigravity; do
  code=$(curl -s -o /dev/null -w "%{http_code}" https://${host}.portugalfuturista.org)
  echo "${host}: ${code}"
done

# Brain sync
cd replica-omnisciente && python3 .aurelio/sync.py --push

Expected: Cluster healthy, ASUS light, all endpoints responsive, sync works.

Step 5: Commit documentation updates

cd nervura-electrica
git add AGENTS.md infrastructure/proxmox/config.yml infrastructure/proxmox/services/gabinete/
git commit -m "ops: migrate CTs 208,212,213,216 to new cluster node

- CT 208 (Gabinete): downsized 8GB → 2GB, moved to new node
- CT 216 (Management): downsized 8GB → 4GB, removed duplicate *arr apps
- Updated Cloudflare tunnel routes to new IPs
- Updated MCP registry spoke URLs
- Updated deploy scripts for new node target"

13 KiB Raw Blame History

Proxmox New Server Migration — Implementation Plan

Global Constraints

Task 1: Pre-migration snapshot and backup

Task 2: Install and join new server to cluster

Task 3: Migrate CTs to new node

Task 4: Update Cloudflare tunnel routes

Task 5: Update MCP registry and sync scripts

Task 6: Post-migration cleanup and documentation

13 KiB

Raw Blame History