replica-omnisciente/.aurelio/mirrors/README.md

Mirror Pipeline — Upstream Replication into Self-Hosted Mirrors

Constant, one-way replication of Savearth's upstream sources into Portugal Futurista's self-hosted mirrors. The agent fleet operates on the mirrors; the originals stay authoritative.

  UPSTREAM (source of truth)              MIRROR (agents work here)
  ──────────────────────────              ─────────────────────────
  GitHub (SavearthTech/*)  ──────►  Forgejo   code.portugalfuturista.org
  Jira Cloud               ──────►  Plane     plane.portugalfuturista.org
  Confluence Cloud         ──────►  Outline   docs.portugalfuturista.org
                                         │
                                sync outcomes
                                         ▼
                          trajectory-rewards/mirror-sync.jsonl
                                         │
                                         ▼
                        GRPO/DPO RL pipeline (rl-agent-trajectory-training)

Architecture

Single source of truth, matching the repo's established provider/connector pattern:

File	Role
.aurelio/mirrors/sync-mirrors.yaml	Edit this — declares sources, targets, auth refs
.aurelio/mirrors/dist/mirrors.catalog.json	Generated — catalog for downstream surfaces
.aurelio/mirrors/state/last-sync.json	Generated — last sync summary (for drift detection)
scripts/sync-mirrors.py	The engine: three sync modes
.forgejo/workflows/mirror-sync.yml	CI: runs every 15 min
.forgejo/workflows/ci.yml	CI: --check guard on every push

Usage

1. Provision tokens

All tokens are API tokens/PATs. Fetch from Vaultwarden, export as env vars:

export FORGEJO_MIRROR_TOKEN=...    # Forgejo admin (manage_repo, write:org)
export GITHUB_MIRROR_TOKEN=...     # GitHub PAT classic (repo, read:org)
export PLANE_API_TOKEN=...         # Plane personal API token
export OUTLINE_API_TOKEN=...       # Outline API token (read+write)
export JIRA_API_TOKEN=...          # Atlassian API token
export JIRA_USER_EMAIL=...         # Atlassian account email
export CONFLUENCE_API_TOKEN=...    # Same Atlassian account
export CONFLUENCE_USER_EMAIL=...

See .env.example for generation instructions per service.

2. Configure upstream endpoints

Edit .aurelio/mirrors/sync-mirrors.yaml:

targets:
  plane:
    sources:
      - id: savearth-jira
        endpoint: https://savearth.atlassian.net   # ← set this
        projects: [SAVE, IOT, HW]                   # ← or null for all

The Jira and Confluence endpoint fields are null by default — the pipeline skips them until you set the real URL.

3. Run the sync

# One-time / manual:
python3 scripts/sync-mirrors.py --sync              # all targets
python3 scripts/sync-mirrors.py --sync forgejo      # one target
python3 scripts/sync-mirrors.py --sync --dry-run    # preview

# Regenerate catalog after editing the YAML:
python3 scripts/sync-mirrors.py --write-in-place

# CI guard:
python3 scripts/sync-mirrors.py --check             # exit 1 if dist stale

4. Constant sync (CI)

.forgejo/workflows/mirror-sync.yml runs every 15 minutes. Tokens must be set as Forgejo repository secrets:

Settings → Secrets → Actions → New Secret:
  FORGEJO_MIRROR_TOKEN, GITHUB_MIRROR_TOKEN, PLANE_API_TOKEN,
  OUTLINE_API_TOKEN, JIRA_API_TOKEN, JIRA_USER_EMAIL,
  CONFLUENCE_API_TOKEN, CONFLUENCE_USER_EMAIL

The workflow commits last-sync.json + reward signals back to the repo.

What each mirror does

GitHub → Forgejo (code)

• Lists all repos in the SavearthTech org (+ personal matching repos).
• Creates a Forgejo migration with mirror: true — Forgejo polls upstream on its own schedule (native_interval: 8h) after the initial clone.
• Skips forks and archived repos by default.
• Preserves private status, description, topics→labels.

Jira → Plane (issues)

• Lists Jira projects, creates matching Plane projects (by key/name).
• Imports issues with field mapping (summary→name, description→description_html, priority, status, assignee, labels).
• Converts Atlassian Document Format (ADF) to HTML for Plane.
• Idempotent: skips issues that already exist by name match.

Confluence → Outline (docs)

• Lists Confluence spaces, creates an Outline collection ("Savearth Mirror").
• Imports pages with [SPACE_KEY] title naming.
• Converts Confluence storage format (XHTML) to Markdown.
• Handles headings, code blocks, lists, links, bold/italic.
• Deduplicates by title within the collection.

RL feedback loop

Each sync run emits reward signals to .aurelio/brain/trajectory-rewards/mirror-sync.jsonl:

Signal	Reward	When
sync_hit	+0.1	Target synced cleanly (no errors)
sync_miss	-0.2	Target had sync errors (drift/staleness)
upstream_overwrite	-0.5	Agent's mirror change was overwritten by upstream sync

These feed into the GRPO/DPO pipeline (see rl-agent-trajectory-training skill). The reward shaping is configured in sync-mirrors.yaml under rl_feedback.signals.

Direction: read-only mirror

Upstream is the source of truth. Agents work on the Forgejo/Plane/Outline copies. Changes do NOT propagate back to GitHub/Jira/Confluence. This is by design — the pipeline is structured so bidirectional write-back can be added per-target later without rewriting the engine.

Troubleshooting

Symptom	Fix
missing env FORGEJO_MIRROR_TOKEN	Export the token (see step 1)
Jira endpoint is null	Set endpoint: in sync-mirrors.yaml
0 repos visible from GitHub	Token lacks read:org or repos are private — check PAT scopes
Plane 401/403	Regenerate API token in Plane settings
Outline OAuth error	Outline needs ENABLE_LOCAL_AUTH=true or Authentik OIDC (see fleet/identity.md)