replica-omnisciente/sessions/2026-07-10_replan-mycelium-universalisos/PLAN.md
2026-07-12 20:53:09 +01:00

23 KiB
Raw Blame History

Replanned Sprint — Mycelium & Universalisos (evidence-based)

Created: 2026-07-10 (re-analysis) Supersedes: the antigravity multi-agent plan of the same date Analyst: Kimi (root agent, replica-omnisciente) Repos (siblings of this brain, not the in-tree mycelium-rd/):

  • /home/fabiorafaelcoutada/portugalfuturista/mycelium
  • /home/fabiorafaelcoutada/portugalfuturista/universalisos

0. TL;DR — the old plan is built on stale premises

The antigravity plan frames both projects as early-construction efforts with a few compile errors. That is not reality. I built, tested, booted and nm-inspected both trees today. The corrected picture:

Project Old-plan assumption Verified reality (2026-07-10)
Mycelium "3 parser compile errors block cargo check; pipeline (lowering/emitters/CLI) not yet built" cargo check --workspace passes. The whole pipeline (parser → IR lowering → C/asm/binary/CRC emitters → drop-in CLI) already exists and compiles, with 11 tests, golden VMIT fixtures and a round-trip harness. The only failing gate is one test (offline_test_passes): the header emitter generates malformed C.
Universalisos ARMv7 "fix _start warning; shrink vm_page_tables[16][4096] to fix 20 MB BSS" _start warning is real but the cause is inverted (entry symbol is reset_handler, not _start); the 20 MB BSS is real but does not come from vm_page_tables (256 KB) — it comes from g_guest_manager (16 MiB) + ramdisk_memory (2 MiB).
Universalisos AArch64 "builds, healthy (17 KB text / 79 KB BSS); just remove the wfi isolation loop to reach MP0 A/B" Build numbers are correct, but qemu-system-aarch64 produces zero UART output — the kernel does not even print its banner. MP0 is gated by boot/UART bring-up, upstream of the scheduler handoff.

Net: less "build from scratch", more "fix specific, root-caused defects and finish in-progress work".


1. Method & evidence

Every claim below was reproduced, not inferred:

# Mycelium
cargo check --workspace                 # exit 0 (2 warnings)
cargo test  --workspace                 # 1 failure: offline_test_passes
# manual replay of the round-trip steps → captured real gcc diagnostics

# Universalisos (from kernel/)
make ARCH=armv7   PLATFORM=qemu-arm-virt      # exit 0, 2 ld warnings
make ARCH=aarch64 PLATFORM=qemu-aarch64-virt  # exit 0, 1 ld warning
arm-none-eabi-nm --print-size --size-sort …   # located the 20 MB BSS owner
timeout 6s qemu-system-aarch64 … -kernel …     # NO output

Tooling present: cargo/rustc, arm-none-eabi-gcc, aarch64-linux-gnu-{gcc,g++,ld,size,nm}, qemu-system-arm, qemu-system-aarch64. (Note: AArch64 uses the linux-gnu toolchain in -ffreestanding -nostdlib mode — see arch/aarch64/arch.mk:10; there is no bare-metal aarch64-none-elf- on this box.)


2. Verified reality vs. the antigravity claims

Mycelium

  • WS-M1 (fix 3 parser compile errors)cargo check already passes. The line numbers cited (L507512, L711, L103 Attribute) do not correspond to current compile errors. The file was edited since (WIP on xs:extension); the only parser-side diagnostic is a dead-field warning (extension_attributes, parser.rs:111).
  • The pipeline is implemented:
    • mycelium-model/src/lower.rs (543 lines) — XSD→IR lowering exists.
    • mycelium-codegen/src/{header,binary,crc,test_inc}.rs — emitters exist.
    • mycelium-cli/src/main.rs — CLI is a superset of the old WS-M4 flag list (--enum/--struct, --enum-asm, --bin, --crc, --ifndef/--comment/--include, --wordsize/--endian/--align-tail, --catalog, --xml, --out, plus --test).
  • Real defect: the round-trip test (crates/mycelium-codegen/tests/runner_roundtrip.rs) drives the CLI then gcc-compiles a harness — gcc rejects the generated header. Root cause is in header.rs struct-body emission (see §4 M0).

Universalisos

  • AArch64 build is healthy: text 16912 / data 0 / bss 79440. Only an RWX LOAD-segment warning (/usr/bin/aarch64-linux-gnu-ld: aviso: … segmento LOAD com permissões RWX).
  • ARMv7 builds but with two ld warnings and 20,540,000 bytes of BSS:
    arm-none-eabi-ld: warning: cannot find entry symbol _start; defaulting to 40000000
    arm-none-eabi-ld: warning: … has a LOAD segment with RWX permissions
       text    data      bss     dec      hex
     172288   179280  20540000  20891568  13ec7b0
    
  • 🔧 The _start warning is misdiagnosed in the old plan. armv7/linker.ld:8 already declares ENTRY(_start), but armv7/boot.S defines the entry as reset_handler (boot.S:5455) — there is no _start symbol. Fix is ENTRY(reset_handler) (or add a _start alias), not "add ENTRY(_start)".
  • 🔧 The 20 MB BSS is mis-aimed in the old plan. nm --size-sort shows the owners:
    16777292  b  g_guest_manager        ← guest.cpp:15  (image_buffer[16 MiB])
     2097152  b  ramdisk_memory         ← block.cpp:820 ([MAX_BLOCK_DEVICES][RAMDISK_MAX_BYTES])
      262144  b  g_uos_vm_pgdir_pool
      262144  b  vm_page_tables         ← only 256 KB, NOT the problem
      206976  b  e1000_states
      173176  d  g_gic_state
      ...
    
    guest.h:42 sets MAX_GUEST_IMAGE_SIZE = 16*1024*1024, embedded as uint8_t image_buffer[…] inside guest_manager_t (guest.h:230). Shrinking vm_page_tables (old U1.2) saves ~240 KB of a 20 MB problem — pointless.
  • AArch64 MP0 is further away than stated. A 6 s qemu-system-aarch64 run printed nothing — no banner, no CurrentEL, nothing. So the "remove the wfi loop and observe interleaved A/B" task (old U2.1) is blocked by an earlier bring-up gap: the kernel does not reach (or does not drive) the PL011. (The wfi isolation loop at kernel_aarch64.cpp:126132 is confirmed present and the real eret path at L135138 is unreachable — that part of the old plan is accurate, just not the next blocker.)

3. Open questions — answered with evidence

  • Q1 (AArch64 IPC strategy). Premature. AArch64 does not boot to a banner yet; IPC is two milestones out. Defer the decision. When reached, prefer the RISC-V pattern (standalone, arch-local) over refactoring src/core/ipc/ — AGENTS.md is explicit that AArch64/RISC-V are self-contained and ARMv7 owns core/; pulling core IPC into AArch64 risks cross-arch header coupling for no near-term payoff.
  • Q2 (Mycelium XML data parsing). Already implemented. The --bin and --test paths read an XML instance (--xml= or positional) — main.rs:112153 — and the round-trip test exercises configtest.xml. The gap is not parsing; it is binary cross-reference / IDREF tables (documented in CONFIGCONV_CONTRACT.md#open-gaps). Treat Q2 as closed.
  • Q3 (QEMU availability). Present: qemu-system-aarch64 and qemu-system-arm are in /usr/bin. No install needed.

4. Project 1 — Mycelium (re-scoped)

Goal unchanged: drop-in pikeos-configconv. Status: ~90% done; one correctness bug blocks the round-trip gate, plus documented open gaps and in-progress xs:extension work.

M0 — Fix the C-header emitter (the keystone) [highest priority]

File: crates/mycelium-codegen/src/header.rs (struct-body loop, L148160).

Current (broken) emission drops the field name for every child and mis-places array bounds:

// header.rs:148-160 (today)
for c in &rs.children {
    if c.app_info.annotation.ignore { continue; }
    if c.max > 1 && c.max < u32::MAX {
        out.push_str(&format!("    {}[{}];\n", c.c_type, c.max));   // no name; T[N];  is ill-formed
    } else {
        out.push_str(&format!("    {};\n", c.c_type));                // no name at all
    }
}

Captured gcc diagnostics prove this exactly:

configtest.h:160: error: expected identifier … before '['   →  struct teebag_t[10];
configtest.h:173: error: expected specifier-qualifier-list   →  vehicle_t;
configtest.h:202: error: … before '['                          →  char[32] str;
configtest.h:197: error: unknown type name 'uw_t'              →  uw_t uw;
configtest.h:200/201: unknown type name 'stdIPv4_t'/'stdMAC_t'

Three concrete fixes:

  1. Emit the child field name: T name; and T name[N]; (use the child's resolved name; the ResolvedChild/field carries it — confirm field name in lower.rs).
  2. Inline strings: the IR stores c_type = "char[32]", so the attribute path (header.rs:146, {c_type} {fname}) yields char[32] str. Split into base + bound and emit char str[32]; (either carry {base, size} in the IR or parse the trailing [N] in the emitter). Prefer carrying structured size from lowering.
  3. Missing typedefs (uw_t, stdIPv4_t, stdMAC_t, vehicle_t): named simple types are used as field c_type but never declared. In the first pass (today only enums are emitted), also emit typedef <base> <Name>_t; for non-enum, non-cx:external simple types that lower to a named alias; honour cx:external by assuming the typedef is provided by the --included header and emitting nothing.

⚠️ The unit test emits_configtest_header (header.rs:194) codifies the bug: assert!(h.text.contains("char[32]"));. Update this assertion to char … [32] (name before bound) when fixing the emitter, otherwise the test will lock in the regression.

Acceptance

  • cargo test -p mycelium-codegen --test runner_roundtripoffline_test_passes prints pass configtest. and exits 0.
  • Generated configtest.h compiles standalone: gcc -fsyntax-only -I host-shim configtest.h.
  • cargo test --workspace is green end-to-end (all 11 + round-trip).

M1 — Warning hygiene [trivial, same PR as M0]

  • mycelium-codegen: pub mod test_inc; (lib.rs:18) is compiled as a normal module, so its use … SimpleStorage; (test_inc.rs:15) is "unused" outside tests. Gate it #[cfg(test)] mod test_inc; (and move it to a tests/ integration file if it is only used by tests), or mark the import #[allow(unused_imports)] if the type is part of the public test surface intentionally.
  • mycelium-xsd: extension_attributes (parser.rs:111) is parsed but unread. Resolve by M2 (consume it) — do not just #[allow(dead_code)] it.

Acceptance: cargo check --workspace0 warnings.

M2 — Finish xs:extension / complexContent / simpleContent [in progress]

The uncommitted WIP (267 insertions across parser.rs, model.rs, lower.rs, extension_fixtures.rs) already parses xs:extension into ContentModel and stashes extension_attributes. Finish the vertical slice per AGENTS.md "Adding a new XSD construct":

  1. Lowering: merge base-type attributes/children into the derived struct (flatten) or emit a nested base field — pick one and document it in docs/ANNOTATION_VOCAB.md.
  2. Consume extension_attributes (clears the M1 warning legitimately).
  3. Emitters: verify header.rs/binary.rs handle the flattened fields (M0's field-name fix is a prerequisite — do M0 first).
  4. Tests: crates/mycelium-xsd/tests/extension_fixtures.rs (parse) + crates/mycelium-model/tests/lower_fixtures.rs (lowering) + a codegen assertion.

Acceptance: xs:extension fixtures lower + emit; VMIT-derived types that were "missed" (per the contract) are recovered; cargo test green.

M3 — Documented open gaps (backlog, prioritise by VMIT need)

From docs/CONFIGCONV_CONTRACT.md#open-gaps:

  • Binary IDREF / cross-reference tables (round-trip today only covers attribute values).
  • --catalog resolution from pikeos-make-cat output (flag is accepted but ignored; imports resolve from the input file's directory).
  • dumpbin-compatible binary header (today's magic is MYC1, 16 B; PikeOS' header is narrower). Needed only if dumpbin-xml consumers must read mycelium blobs verbatim.
  • Golden-VMIT structural conformance (already the policy per AGENTS.md — "match the contract, not the bytes"); keep conformance/golden/vmit/ as the reference.

Acceptance (per item): a fixture + test + a one-line note closing the gap in the contract.


5. Project 2 — Universalisos (re-scoped)

Reality: ARMv7 is the working demo target (boots, runs scheduler/IPC/HM). AArch64 builds but does not produce UART output today; RISC-V gets past OpenSBI but UART is unverified. The old plan over-rotated on AArch64 MP1 (Stage-2/vCPU) while MP0 bring-up is incomplete.

U1 — ARMv7 quality (real defects, corrected targets) [independent, start now]

U1.1 Entry-symbol warningarmv7/linker.ld:8 says ENTRY(_start) but the symbol is reset_handler (armv7/boot.S:54). Fix one of:

/* linker.ld */  ENTRY(reset_handler)            /* preferred: matches boot.S */

or add a _start alias in boot.S. Do not "add ENTRY(_start)" — it is already there.

U1.2 RWX LOAD segment — affects both armv7/linker.ld and aarch64/linker.ld (the old plan only flagged AArch64). Add explicit PHDRS and assign sections, e.g.:

PHDRS { text PT_LOAD FLAGS(5);   /* R-X */
        data PT_LOAD FLAGS(6); } /* RW- */
SECTIONS {
  .text   : { … } :text
  .rodata : { … } :text
  .data   : { … } :data
  .bss    : { … } :data
}

U1.3 20 MB BSS → < 1 MB — the real owners are not vm_page_tables:

  • guest_manager_t g_guest_manager (guest.cpp:15) embeds uint8_t image_buffer[16 MiB] (guest.h:230, MAX_GUEST_IMAGE_SIZE = 16*1024*1024, guest.h:42). For the QEMU demo this is absurd. Drop MAX_GUEST_IMAGE_SIZE to a demo value (e.g. 12 MiB) and/or reduce MAX_GUESTS (guest.h:30, 16). Keep a compile-time guard so real targets can raise it.
  • ramdisk_memory[MAX_BLOCK_DEVICES][RAMDISK_MAX_BYTES] (block.cpp:820, 2 MiB): reduce RAMDISK_MAX_BYTES or device count for the qemu-arm-virt platform.
  • Note: .bss is NOBITS (does not bloat the on-disk ELF), but it is reserved RAM at runtime under the flat identity map; the project's stated ceiling is < 1 MB.

Acceptance

  • make ARCH=armv7 PLATFORM=qemu-arm-virt0 linker warnings.
  • make ARCH=aarch64 PLATFORM=qemu-aarch64-virt0 linker warnings.
  • arm-none-eabi-size …/armv7/…/universalisos.elfbss < 1 048 576.
  • ARMv7 still boots in QEMU and completes the demo sequence (regression check).

U2 — AArch64 boot / UART bring-up [NEW prerequisite, was missing from the old plan]

A timed QEMU run produced no output. Before any scheduler work, establish the banner:

  1. Confirm _startkernel_main is reached (boot.S) — e.g. write a known pattern to the PL011 DR (0x0900_0000 on qemu-aarch64-virt) before MMU/GIC/timer init.
  2. Verify uart_pl011.cpp base address and board_qemu_virt.h match the machine (-M virt,gic-version=3 -cpu cortex-a53 -m 512M).
  3. Bisect the init order in kernel_aarch64.cpp: UART → LPAE S1 (mmu_lpae) → GICv3 → timer. An early fault (vector at VBAR) or an MMU-enable trap would silently hang; print before each step to localise.

Acceptance: timeout 5s qemu-system-aarch64 … -kernel … prints the === UniversalisOS AArch64 @ EL2 === banner and CurrentEL=… EL2.

U3 — AArch64 MP0 scheduler handoff (old U2.1) [after U2]

kernel_aarch64.cpp:126132 is the "PURE TIMER ISOLATION TEST" wfi loop; the real path (L135138) is unreachable. Per the file's own design comment (L116121) the correct sequence is mask → arm → eret (SPSR unmasks IRQs atomically). Remove the daifclr #2 at L124 and the wfi loop:

__asm__ volatile("msr daifset, #2" ::: "memory");   // mask IRQ
timer_set_oneshot(tick_us);                          // arm; pending, masked
sched_task_t* first = sched_peek_ready_head();       // eret into A (unmasks IRQ via SPSR)
sched_set_current(first);
first->state = TASK_RUNNING;
aarch64_preempt_run(&first->ctx);                    // noreturn

Acceptance: sustained interleaved A/B output; timer ISR at 100 Hz (10 ms); clean Ctrl-A x exit.

U4 — AArch64 MP1 (Stage-2 MMU + vCPU + VGIC) [defer until U2U3 pass]

Unchanged in spirit from the old WS-U3 (s2mmu.{cpp,h}, vcpu.{cpp,h}, VGIC in gicv3.cpp), but do not start until MP0 boots and preempts. RISC-V references remain arch/riscv/{vm.cpp,page_table.cpp}. Re-open Q1 (IPC) only when a second partition must talk.


6. Dependencies & parallel tracks

Mycelium track (independent of Universalisos)
  M0 (header emitter) ─┬─> M1 (warnings)   [same PR]
                       └─> M2 (xs:extension) ─> M3 (backlog gaps)

Universalisos track
  U1 (ARMv7 quality)         [independent; start now]
  U2 (AArch64 boot/UART) ──> U3 (MP0 handoff) ──> U4 (MP1 Stage-2/vCPU) [defer]

Two agents can work in parallel without collision:

  • Track A (Rust/Mycelium): M0 → M1 → M2 → M3. Single repo, fast cargo feedback.
  • Track B (C/Universalisos): U1 now (ARMv7, the verified target); in parallel begin U2 diagnosis on AArch64 (read-only at first — UART base, boot flow). U3/U4 are sequential after.

The old "Claude Code vs Codex" split is incidental; the real axis is Rust codegen vs. bare-metal kernel. Staff by toolchain familiarity, not by agent brand.


7. Risks & decisions needed

  1. BSS-shrink policy (U1.3): picking MAX_GUEST_IMAGE_SIZE is a product decision (demo vs. real target). Recommend a per-platform cap (qemu-arm-virt small, hardware larger) rather than a global cut. Needs a one-line decision before editing guest.h.
  2. AArch64 UART root cause (U2): unconfirmed — could be wrong PL011 base, MMU-enable fault, exception vector, or -kernel/EL-entry assumption. Diagnose before promising MP0 dates.
  3. Header-emitter fix scope (M0): the IR currently stores c_type = "char[32]". Cleaning this may ripple into binary.rs/test_inc.rs layout math. Re-run the whole suite, not just the header test, after changing the representation.
  4. Don't touch universalisos from the mycelium lane (AGENTS.md lane rule). Keep edits in the owning repo; the mycelium AGENTS.md forbids editing kernel sources.

Appendix A — verification cheat-sheet

# Mycelium
cd …/mycelium
cargo check --workspace                                   # 0 warnings (after M1)
cargo test  --workspace                                   # all green (after M0)
cargo test  -p mycelium-codegen --test runner_roundtrip   # the keystone gate
./target/debug/mycelium --enum --struct --ifndef=__T --comment=t --include='<x.h>' \
    conformance/offline-test/configtest.xsd --out=/tmp/c.h && gcc -fsyntax-only \
    -I conformance/host-shim /tmp/c.h                     # header must parse

# Universalisos
cd …/universalisos/kernel
make ARCH=armv7   PLATFORM=qemu-arm-virt                  # 0 ld warnings (after U1)
make ARCH=aarch64 PLATFORM=qemu-aarch64-virt              # 0 ld warnings (after U1.2)
arm-none-eabi-size   build/armv7/qemu-arm-virt/universalisos.elf        # bss < 1 MiB
timeout 5s qemu-system-aarch64 -M virt,gic-version=3 -cpu cortex-a53 -m 512M \
    -nographic -kernel build/aarch64/qemu-aarch64-virt/universalisos.elf  # banner (U2) → A/B (U3)

Appendix B — files that matter (with verified anchors)

Mycelium

  • crates/mycelium-codegen/src/header.rs:148-160 — child-field emission (M0 root cause)
  • crates/mycelium-codegen/src/header.rs:146 — attribute path, char[32] str ordering
  • crates/mycelium-codegen/src/header.rs:217 — test that codifies the bug ("char[32]")
  • crates/mycelium-codegen/src/lib.rs:18pub mod test_inc; (M1 warning)
  • crates/mycelium-codegen/src/test_inc.rs:15 — unused SimpleStorage
  • crates/mycelium-xsd/src/parser.rs:111 — unread extension_attributes (M2)
  • docs/CONFIGCONV_CONTRACT.md#open-gaps — authoritative backlog

Universalisos

  • kernel/src/arch/armv7/linker.ld:8ENTRY(_start) vs reset_handler (U1.1)
  • kernel/src/arch/armv7/boot.S:54reset_handler: (the real entry)
  • kernel/src/arch/aarch64/linker.ld — single region, no PHDRS (U1.2)
  • kernel/src/core/guest.cpp:15 + guest.h:30/42/230g_guest_manager 16 MiB (U1.3)
  • kernel/src/platform/drivers/block.cpp:820ramdisk_memory 2 MiB (U1.3)
  • kernel/src/arch/aarch64/kernel_aarch64.cpp:124-138daifclr + wfi loop vs eret (U3)
  • kernel/src/arch/aarch64/{boot.S,uart_pl011.cpp} + inc/board_qemu_virt.h — U2 bring-up

Progress log (2026-07-10) — Track B executed, AArch64 MP0 gate PASSED

Outcome: ARMv7 quality fixed, AArch64 boots at EL2, and IRQ-driven preemptive scheduling works (sustained, balanced interleaved A/B, 0 traps). Tree left clean: no debug probes, both arches link with 0 warnings.

What shipped (all verified by build + QEMU boot)

  • U1.1 armv7/linker.ld: ENTRY(exception_vectors) (the real global at 0x40000000) — entry warning gone.
  • U1.2 armv7/linker.ld + aarch64/linker.ld: RX/RW PHDRS — RWX LOAD-segment warning gone on both arches.
  • U1.3 BSS 20.5 MB → 3.6 MB: MAX_GUEST_IMAGE_SIZE 16→1 MiB (guest.h), RAMDISK_MAX_BYTES 256→128 KiB (block.cpp). Real owners were g_guest_manager (16 MiB) + ramdisk_memory (2 MiB), not vm_page_tables. Full ARMv7 demo still passes.
  • U2 AArch64 silent-hang root cause: QEMU hands -kernel EL1 without virtualization=on; the EL2 hypervisor trapped on its first EL2 register write. Fixed by (a) requiring virtualization=on (documented in universalisos/AGENTS.md), (b) hardening boot.S EL entry (cmp #2/#3; EL0/EL1 → clean wfe halt; EL3 → drop to EL2).
  • U3 MP0 preempt gate — four compounding defects fixed:
    1. Timer source was CNTV (INTID 27), never delivered at EL2 → switched to CNTHP (INTID 26) in timer.cpp (matching timer.h/board_qemu_virt.h); handler matches 26.
    2. HCR_EL2 was never programmed, so no physical IRQ routed to EL2 → boot.S now sets HCR_EL2 = FMO|IMO|AMO (0x38).
    3. Forced ICC_CTLR_EL1.EOImode = 0 in gicv3.cpp (reliable single-EOI deactivation).
    4. Re-arm via absolute CNTHP_CVAL (timer left enabled).
    5. Preemptive context-switch bug: el2_irq_entry never wrote the interrupted task's SP into the frame (TF_SP_EL0 held stale stack data), so a resumed task got a bogus SP and halted after ~23 switches. Fixed by recording sp + TF_SIZE into [sp, #TF_SP_EL0] in exceptions.S.
    • Proof: timer fires at 100 Hz indefinitely (bisection confirmed); with reschedule enabled, A/B counts balance (~50/50) with interleaved output.

Files changed (universalisos)

kernel/src/arch/armv7/linker.ld, kernel/src/arch/aarch64/linker.ld, kernel/src/core/guest.h, kernel/src/platform/drivers/block.cpp, kernel/src/arch/aarch64/boot.S, kernel/src/arch/aarch64/kernel_aarch64.cpp, kernel/src/arch/aarch64/timer.cpp, kernel/src/arch/aarch64/el2_trap.cpp, kernel/src/arch/aarch64/exceptions.S, kernel/src/arch/aarch64/gicv3.cpp, AGENTS.md.

Remaining (north star: PikeOS/Bao parity)

  • Synthesize a gap matrix from the in-repo analyses (UNIVERSALISOS_VS_PIKEOS_5.0.md, UNIVERSALISOS_PIKEOS_ANALYSIS.md, HYPERVISOR.md, RISCV_PIKEOS_PARITY_PLAN.md) into an ordered backlog.
  • Next major virtualization track: MP1 — Stage-2 MMU (VTTBR_EL2), vCPU context, VGIC.