universalisos/.hermes/plans/2025-07-10_221500-aarch64-el2-gap-closure.md

Task 1: Split kernel_main into two paths — guest always on MP1, scheduler always on MP0

Objective: kernel_main must start mp0_scheduler_run() on MP0 regardless of PSCI result, and the guest must only run on MP1 (not on MP0 fallback). This ensures the preemption IRQ path actually runs.

Files:

• Modify: kernel/src/arch/aarch64/kernel_aarch64.cpp:193-211 (the PSCI CPU_ON + fallback block)
• Modify: kernel/src/arch/aarch64/kernel_aarch64.cpp:88-130 (mp0_scheduler_run, task creation, final comment)

Step 1: Read the current code section

# Already read — lines 193-206 are the PSCI block, mp0_scheduler_run is lines 113-130.

Step 2: Rewrite the PSCI / scheduler dispatch block

Current code (lines 193–211):

    /* 10. Run the host scheduler on MP0. */
    mp0_scheduler_run();

    /* unreachable */
    for (;;) { __asm__ volatile("wfe"); }

But the PSCI failure block (lines 200–206) currently bails to mp1_run_guest():

    if (ret != PSCI_RET_SUCCESS) {
        uart_puts("MP0: PSCI CPU_ON failed, ret=");
        ...
        uart_puts("\nMP0: falling back to local guest run\n");
        mp1_run_guest();
        for (;;) { __asm__ volatile("wfe"); }
    }

Replace lines 193–213 (the "10. Run the host scheduler on MP0." block + unreachable loop) with:

    /* 10. MP0 scheduler: always run on MP0. The guest stays on MP1 only;
     *    if PSCI CPU_ON failed (e.g. single PE), just run the scheduler
     *    without a guest — preemptive A/B interleaved output proves the
     *    IRQ→sched path regardless. */
    uart_puts("MP0: starting host scheduler (MP0)");
    if (ret == PSCI_RET_SUCCESS) {
        uart_puts(" + guest on MP1\n");
    } else {
        uart_puts(" (guest MP1 offline — PSCI failed, ret=");
        universalisos::uart::print_hex(ret);
        uart_puts(")\n");
    }
    mp0_scheduler_run();

    /* unreachable */
    for (;;) { __asm__ volatile("wfe"); }

And remove the old mp1_run_guest() fallback (lines 202-206 death loop). Keep lines 193-200 (the PSCI CPU_ON attempt and its failure logging), just replace the fallback.

Step 3: Verify the new flow reads correctly

After the change, lines ~193-213 should read:

    /* 10. MP0 scheduler: always run on MP0. ... */
    uart_puts("MP0: starting host scheduler (MP0)");
    if (ret == PSCI_RET_SUCCESS) {
        uart_puts(" + guest on MP1\n");
    } else {
        uart_puts(" (guest MP1 offline — PSCI failed, ret=");
        universalisos::uart::print_hex(ret);
        uart_puts(")\n");
    }
    mp0_scheduler_run();

    /* unreachable */
    for (;;) { __asm__ volatile("wfe"); }

Step 4: Build (ARMv7 must still compile — AArch64 is the target)

cd kernel
make ARCH=armv7 PLATFORM=qemu-arm-virt
# Expected: clean build (unchanged — ARMv7 has its own kernel.cpp)
make ARCH=aarch64 PLATFORM=qemu-aarch64-virt
# Expected: clean build, no warnings

Step 5: Boot AArch64 and verify scheduler banner appears

cd kernel
qemu-system-aarch64 -M virt,gic-version=3,virtualization=on -cpu cortex-a53 -m 512M \
  -nographic -kernel build/aarch64/qemu-aarch64-virt/universalisos.elf
# Expected output: after GIC/timer/scheduler init + PSCI log,
#   "MP0: starting host scheduler (MP0) (guest MP1 offline — PSCI failed, ret=...)"
#   then "[A-start][B-start]" then interleaved A/B

Step 6: Verify A/B interleaving works

With single PE (-smp 1 or no -smp), the scheduler should preempt task_a and task_b, producing interleaved A/B output. The CNTHP timer (PPI 26) fires every 10ms via vector 0x280 → el2_irq_entry → el2_irq_handler → sched_tick → g_need_reschedule → asm preempt-switch.

timeout 10 qemu-system-aarch64 -M virt,gic-version=3,virtualization=on \
  -cpu cortex-a53 -m 512M -nographic \
  -kernel build/aarch64/qemu-aarch64-virt/universalisos.elf 2>&1 || true
# Expected: interleaved A/B output (not just "A" forever or "B" forever)

Step 7: Commit

git add kernel/src/arch/aarch64/kernel_aarch64.cpp
git commit -m "fix(aarch64): always run MP0 scheduler instead of guest fallback"

---

Task 2: Enable SMP (-smp 4) so PSCI CPU_ON succeeds and guest runs on MP1

Objective: QEMU -M virt with -smp 4 wakes all 4 PEs; PSCI CPU_ON target 0x1 succeeds because CPU1 is a real PE. The guest boots on MP1 while MP0 schedules.

Files:

• Modify: kernel/src/arch/aarch64/kernel_aarch64.cpp:197 (PSCI target: keep 0x1ULL)
• Document: AGENTS.md (update verified boot command)

Step 1: Add -smp 4 to the AArch64 verified boot command

Current AGENTS.md verified boot command for AArch64:

qemu-system-aarch64 -M virt,gic-version=3,virtualization=on -cpu cortex-a53 -m 512M \
  -nographic -kernel build/aarch64/qemu-aarch64-virt/universalisos.elf

Replace with:

qemu-system-aarch64 -M virt,gic-version=3,virtualization=on -cpu cortex-a53 -m 512M \
  -smp 4 -nographic -kernel build/aarch64/qemu-aarch64-virt/universalisos.elf

Step 2: Boot with -smp 4 and verify PSCI CPU_ON succeeds

cd kernel
timeout 20 qemu-system-aarch64 -M virt,gic-version=3,virtualization=on \
  -cpu cortex-a53 -m 512M -smp 4 -nographic \
  -kernel build/aarch64/qemu-aarch64-virt/universalisos.elf 2>&1 || true

Expected: PSCI CPU_ON returns SUCCESS (0). Output shows:

• "MP0: waking CPU1 via PSCI CPU_ON"
• "MP1: online, enabling Stage-2"
• Guest hypercalls (HVC from EL1, x0=...)
• "MP0: starting host scheduler (MP0) + guest on MP1"
• Interleaved A/B from MP0 scheduler

Step 3: Update AGENTS.md

Modify AGENTS.md line ~84 (the Verified Boot Command for AArch64) to include -smp 4.

Step 4: Commit

git add AGENTS.md
git commit -m "docs(aarch64): add -smp 4 to verified boot command for SMP guest"

---

Task 3: Smoke-test verification matrix — single PE and SMP

Objective: Prove both paths work: (a) single-PE scheduler-only preemption, (b) SMP scheduler + guest on separate PE.

Files:

• None (verification only)

Step 1: Single-PE boot (scheduler-only, no guest)

cd kernel
timeout 15 qemu-system-aarch64 -M virt,gic-version=3,virtualization=on \
  -cpu cortex-a53 -m 512M -nographic \
  -kernel build/aarch64/qemu-aarch64-virt/universalisos.elf 2>&1 || true

Expected: No guest. Scheduler banner "MP0: starting host scheduler (MP0) (guest MP1 offline...)". Then [A-start][B-start] + interleaved A/B. No "[vmexit]" lines. No trap dumps.

Step 2: SMP boot (scheduler on MP0 + guest on MP1)

cd kernel
timeout 20 qemu-system-aarch64 -M virt,gic-version=3,virtualization=on \
  -cpu cortex-a53 -m 512M -smp 4 -nographic \
  -kernel build/aarch64/qemu-aarch64-virt/universalisos.elf 2>&1 || true

Expected: Guest HVC output interleaved with scheduler A/B. Both streams visible. No trap dumps. Guest parks cleanly ("[vmexit] guest payload parked").

Step 3: Rebuild and boot ARMv7 (ensure no cross-contamination)

cd kernel
make ARCH=armv7 PLATFORM=qemu-arm-virt
timeout 15 qemu-system-arm -M virt -cpu cortex-a15 -m 512M \
  -nographic -kernel build/armv7/qemu-arm-virt/universalisos.elf 2>&1 || true

Expected: Full ARMv7 demo pipeline unchanged (IPC, HM, preemption, etc.).

---

Risks and Tradeoffs

1. PSCI CPU_ON might still fail with -smp 4. QEMU's virt machine PSCI implementation is known to be picky. If 0x1ULL target still fails, try 0x0ULL (CPU0) as a fallback test — CPU0 is always awake, so PSCI CPU_ON to itself is a no-op that returns success. This would prove the PSCI conduit works at the cost of guest-on-MP0. If that's the case, emit a MP1: PSCI CPU_ON target 0x1 INVALID_PARAM — try qemu-system-aarch64 -M virt-2.12 or virt,gic-version=3 with different -smp diagnostic.

2. Interleaved A/B might not appear if the timer IRQ is masked or the vector table is misaligned. The current vector table is 2KB-aligned (.align 11), VBAR_EL2 is set, HCR_EL2.IMO=1, and CNTHP_CTL_EL2 has ENABLE=1 IMASK=0. If the timer fires but the IRQ handler dumps a trap, check that SP_EL0 is initialized (boot.S:76 already does msr sp_el0, x0 → lines 76-77).

3. Guest on MP1 may race with MP0 scheduler output on UART. Both PEs write to PL011 without locking. Acceptable for this bring-up milestone; the output will be interleaved but readable.

4. ARMv7 build must not break. The kernel_aarch64.cpp file is arch-specific — ARMv7 uses kernel/src/core/kernel.cpp, not the AArch64 one. No cross-contamination, but verify anyway.

---

Open Questions

• Should MP0 print scheduler tick counts or just A/B letters? Current ARMv7 preempt demo prints only A/B. Keep parity — no tick logging.
• Should the PSCI CPU_ON failure log remain visible, or should we suppress it on SMP success? Keep visible — useful for debugging, harmless noise.

---

Summary

Three tasks, all in kernel/src/arch/aarch64/kernel_aarch64.cpp + AGENTS.md.

1. Fix kernel_main dispatch: always run mp0_scheduler_run(), never fallback guest on MP0.
2. Enable -smp 4 so PSCI CPU_ON wakes a real MP1 PE.
3. Verify single-PE (preemption-only) and SMP (scheduler + guest) both work, ARMv7 unchanged.

After these three tasks, AArch64 EL2 has parity with ARMv7 on:

• Timer-driven preemptive scheduling with interleaved task output
• Guest execution on a separate PE via PSCI
• Full exception vector table exercised (sync + IRQ + guest sync + guest IRQ vectors all hit)