universalisos/docs/esp32-boot-chain-comparison.md

ESP32 Boot Chain: ESP-IDF vs UniversalisOS (MCUboot→U-Boot→UOS)

1. Complete Boot Sequence Comparison

1.1 ESP-IDF Flow

ROM (1st stage, mask ROM @ 0x40000000)
 │  - Reads ESP image header (magic 0xE9) from flash @ 0x1000
 │  - Parses segment headers, loads segments per Memory Types:
 │    · BYTE_ACCESSIBLE/DRAM → memcpy to DRAM
 │    · CACHE_APP → flash-MMU maps flash pages to virtual address
 │    · IRAM → memcpy to IRAM
 │  - Sets up SPI flash, initial cache/MMU config
 │  - Jumps to entry point (bootloader)
 │
 └─ 2nd Stage Bootloader (IDF bootloader @ flash 0x1000)
     │  bootloader_init():
     │    - bootloader_init_mem()  → clear BSS, check stack
     │    - bootloader_clock_configure()
     │    - bootloader_console_init()
     │    - bootloader_reset_mmu():
     │        Cache_Read_Disable(0)
     │        Cache_Flush(0)
     │        mmu_init(0)              ← HARD reset of MMU controller
     │        DPORT unmask DROM0
     │    - bootloader_flash_update_id()
     │    - bootloader_init_spi_flash()  → SPI flash init
     │    - bootloader_enable_random()
     │
     │  bootloader_utility_load_image():
     │    - Parse app partition, read ESP image header
     │    - For each segment: memcpy IRAM/DRAM segments to RAM
     │    - Record DROM/IROM segment addrs for MMU mapping
     │    - load_image() → unpack_load_app()
     │
     │  set_cache_and_start_app():
     │    Cache_Read_Disable(0)          ← disable cache
     │    Cache_Flush(0)                 ← flush D-cache to SRAM
     │    mmu_hal_unmap_all()            ← SOFT reset: clear page table only
     │    cache_flash_mmu_set(0, DROM)   ← map DROM flash pages
     │    cache_flash_mmu_set(1, DROM)   ← map DROM for APP CPU too
     │    cache_flash_mmu_set(0, IROM)   ← map IROM flash pages
     │    cache_flash_mmu_set(1, IROM)   ← map IROM for APP CPU too
     │    cache_ll_l1_get_bus() / enable ← enable cache bus masks
     │    Cache_Read_Enable(0)           ← re-enable cache
     │    isync
     │    bootloader_atexit()            ← clean up (WDT, uart flush)
     │    (*entry)()                     ← jump to app IROM entry
     │
     └─ Application (FreeRTOS app)
         - Runs from IROM (flash-cached @ 0x400D0000+)
         - data/bss in DRAM

1.2 UniversalisOS Flow (our chain)

ROM (1st stage, mask ROM)
 │  - Reads ESP image header from flash @ 0x1000
 │  - Loads MCUboot segments (CACHE_APP + DRAM + IRAM)
 │  - Jumps to MCUboot entry
 │
 └─ MCUboot (2nd stage bootloader @ flash 0x1000)
     │  bootloader_init():  [same ESP-IDF HAL, no major diffs]
     │    - bootloader_reset_mmu() (via bootloader_init → init_ext_mem)
     │
     │  main() → do_boot():
     │    - Validate image, check swap type
     │    - start_cpu0_image():
     │        esp_app_image_load():
     │          - Parse MCUboot's esp_image_load_header_t (magic 0xACE637D3)
     │          - Load DRAM via memcpy (→ 0x3FFAF000+)
     │          - **IROM path**: cache_flash_mmu_set to map flash→IROM
     │          - **IRAM path**: memcpy (BROKEN on ESP32 — D-bus can't write I-bus)
     │        jump to entry_addr
     │
     └─ U-Boot (loaded by MCUboot @ IROM 0x400D0000+)
         │  _start:
         │    - Print 'X', set VECBASE, unpack reloc, init UART
         │    - board_init_f → board_init_r → U-Boot prompt
         │
         └─ UniversalisOS (loaded by U-Boot)

1.3 Critical Difference: IRAM vs IROM

Aspect	ESP-IDF	UniversalisOS
Bootloader code location	IRAM (0x40078000) + CACHE_APP segment	IRAM (0x40078000 loader) + IRAM (0x40090000 main)
App code location	IROM (flash-cached, 0x400D0000+)	Was IRAM (SRAM, 0x40080000) — BROKEN
App code loading	flash-MMU maps flash pages to virtual IROM	Was memcpy to SRAM — ESP32 D-bus can't write I-bus
Handoff sequence	Disable cache → unmap_all → map DROM → map IROM → bus enable → Enable cache → isync → jump	Same pattern (current code)
MMU reset	mmu_hal_unmap_all() (soft, entries only)	Was mmu_init(0) (hard, corrupts controller) — fixed

2. Memory Region Comparison

2.1 ESP32 Physical Memory Map

Bus     Address              Size    Content
────    ────────             ────    ───────
        0x4000_0000-0x4005_FFFF  384KB  ROM (mask ROM, read-only)
I-bus   0x4007_0000-0x4007_7FFF   32KB  SRAM0 — ROM cache (reserved)
I-bus   0x4007_8000-0x4007_A800   10KB  MCUboot LOADER (CACHE_APP segment)
I-bus   0x4007_A800-0x4008_FFFF   86KB  GAP — free
I-bus   0x4008_0000-0x4008_003F   —     UserExceptionVector (Xtensa hw)
I-bus   0x4008_0100-0x4008_01xx   —     KernelExceptionVector
I-bus   0x4008_03C0-0x4008_xxxx   —     DoubleExceptionVector
I-bus   0x4009_0000-0x4009_E800   58KB  MCUboot IRAM MAIN (RECLAIMABLE by app)
I-bus   0x400A_0000-0x400B_FFFF  128KB  SRAM1 (free, dual-bus)
I-bus   0x400C_0000-0x40FF_FFFF    4MB  IROM — flash-cached execution
D-bus   0x3F40_0000-0x3F80_0000    4MB  DROM — flash-cached data
D-bus   0x3FFA_E000-0x3FFC_FFFF  200KB  SRAM2 — DRAM (data/bss/stack)
D-bus   0x3FFE_0000-0x3FFF_FFFF  128KB  SRAM1 — DRAM (dual-bus mirror)

2.2 IDF App Memory Layout

Code:  0x400D_0000+  (IROM, flash-cached via MMU)
RO:    0x3F40_0000+  (DROM, flash-cached via MMU)
Data:  0x3FFB_0000+  (DRAM, SRAM2/SRAM1)
BSS:   0x3FFx_xxxx   (DRAM, after data)
Stack: 0x3FFF_xxxx   (DRAM, top of SRAM1)

2.3 U-Boot Memory Layout (new IROM design)

Code:   0x400D_0000+  (IROM, flash-cached — same as IDF app)
RO:     0x3FFB_0000+  (DRAM, same as IDF)
Data:   0x3FFB_xxxx   (DRAM, after RO)
BSS:    0x3FFB_xxxx   (DRAM, after data)
Stack:  0x4000_0000-   (DRAM, top of address space)

2.4 The IRAM Problem (why memcpy never worked)

ESP32 has split bus architecture:

• I-bus (instruction bus): fetches code from 0x40000000+
• D-bus (data bus): reads/writes data from 0x3FFxxxxx+

A memcpy((void*)0x40080000, ...) uses D-bus to write to I-bus address. The D-bus does NOT route to SRAM0 at that address. The write goes through the flash cache controller, which requires MMU entries for the target address. If no MMU entry exists, the write either goes nowhere or faults.

The only safe way to execute code on ESP32 is from IROM (flash-cached). Direct IRAM loading via memcpy is architecturally impossible from software.

3. Cache / MMU Handling Comparison

3.1 ROM Initial State

After ROM boots the 2nd stage:

• VECBASE = 0x4000_0000 (ROM exception vectors)
• DROM0 bus: unmasked (ROM clears mask)
• IROM0: mapped by ROM to the bootloader's CACHE_APP segments
• D-cache: enabled, contains bootloader code cache lines
• I-cache: enabled, contains bootloader IROM cache lines
• MMU table: populated with bootloader's flash page mappings

3.2 IDF Bootloader Init

bootloader_reset_mmu() (bootloader_esp32.c:43-69):

Cache_Read_Disable(0);  // stop cache reads
Cache_Read_Disable(1);  // APP CPU too
Cache_Flush(0);         // write back D-cache to SRAM
Cache_Flush(1);
mmu_init(0);            // HARD RESET: clear ALL MMU entries + controller state
mmu_init(1);            // APP CPU too
DPORT unmask DROM0;      // restore DROM0 bus so flash reads work

After this, the bootloader must re-initialize SPI flash and re-read its own header from flash. ALL previous MMU mappings are destroyed.

3.3 IDF App Load

set_cache_and_start_app() (bootloader_utility.c:1036-1158):

Cache_Read_Disable(0);  // stop cache reads
Cache_Flush(0);         // write back D-cache
mmu_hal_unmap_all();     // SOFT RESET: clear page table, keep controller state
cache_flash_mmu_set(0, DROM); // map app's data from flash
cache_flash_mmu_set(1, DROM); // for APP CPU
cache_flash_mmu_set(0, IROM); // map app's code from flash
cache_flash_mmu_set(1, IROM); // for APP CPU
cache_ll_l1_enable_bus();    // enable bus masks
Cache_Read_Enable(0);   // re-enable cache
isync;                   // flush prefetch pipeline
bootloader_atexit();     // clean up WDT, flush UART
(*entry)();              // JUMP

Key: uses mmu_hal_unmap_all() NOT mmu_init(). The soft reset only clears the page table entries, leaving the MMU controller configured. mmu_init() hard-resets the controller which requires full re-init.

3.4 Our MCUboot Flow

Cache_Read_Disable(0);  // ✓
Cache_Flush(0);         // ✓
// (no mmu_hal_unmap_all — we don't have this HAL function)
cache_flash_mmu_set(0, IROM); // ✓ rc=0
Cache_Read_Enable(0);   // ✓
isync;                   // ✓
(*entry)();              // JUMP

Missing vs IDF:

1. No mmu_hal_unmap_all() — but we don't need it since we're adding entries on top of MCUboot's existing mappings
2. No DROM mapping for U-Boot — U-Boot's RO data is in DRAM, not DROM
3. No bus mask enable — MCUboot's existing bus masks should still be active
4. No APP CPU mapping — ESP32 single-core during boot

4. Image Formats

4.1 ESP Image Format (ROM loader)

[ESP image header 24B]    magic=0xE9, segment_count, entry_addr
[Segment header 8B] × N   load_addr, data_len
[Segment data]             raw bytes
[Checksum 1B]
[SHA256 32B] (optional)

Segment types determined by esptool from ELF segment VMA:

• 0x3FFxxxxx → BYTE_ACCESSIBLE, DRAM
• 0x4007xxxx → CACHE_APP (flash-mapped, no memcpy)
• 0x4008xxxx → IRAM (memcpy to SRAM)
• 0x400Dxxxx → CACHE_APP (flash-mapped, no memcpy)

4.2 MCUboot Image Format (our chain)

[MCUboot header 32B]      magic=0x96f3b83d, image size, version
[Load header 96B]         magic=0xACE637D3, entry_addr, iram_*, irom_*, dram_*
[IRAM data]               code segment (legacy, memcpy)
[DRAM data]               data segment
[MCUboot TLV]             hash, signatures

Our wrapper uses:

• irom_map_addr, irom_flash_offset, irom_size → flash-MMU mapped
• dram_dest_addr, dram_flash_offset, dram_size → memcpy to DRAM

5. Feature Comparison Matrix

Feature	ESP-IDF Bootloader	MCUboot (our chain)	U-Boot (our chain)
ROM loading	ESP image (0xE9)	ESP image (0xE9)	N/A (loaded by MCUboot)
Code execution	IROM (flash-cached)	IRAM (SRAM)	IROM (flash-cached)
Data storage	DRAM (memcpy)	DRAM (memcpy)	DRAM (memcpy)
Flash MMU	mmu_hal_unmap_all + cache_flash_mmu_set × 4	cache_flash_mmu_set × 1	N/A (runs from MCUboot's MMU)
Cache handling	Disable→Flush→Unmap→Map→Enable→isync	Disable→Flush→Map→Enable→isync	Cache already enabled by MCUboot
VECBASE	App sets it (IRAM or IROM)	Set to 0x40080000 (old IRAM)	Sets VECBASE to IROM 0x400D0000
Exception vectors	App provides, at VECBASE offsets	None (ROM vectors)	Built-in, at IROM offsets
Secure boot	Yes (RSA/ECDSA)	Yes (MCUboot signature validation)	N/A (trusts MCUboot)
OTA updates	Yes (factory/OTA partitions)	Yes (primary/secondary slots)	N/A (loaded by MCUboot)
Flash encryption	Yes (AES-XTS)	Yes (MCUboot supports)	N/A
Image validation	SHA256 checksum	SHA256 via imgtool TLV	N/A
Multi-image	No (single app)	Yes (multiple slots)	N/A
Stack location	DRAM (top of SRAM1)	DRAM (MCUboot's loader segment)	DRAM (0x40000000 - 16)
Boot time	~50ms	~100ms (MCUboot validation)	~200ms (U-Boot init)

6. Root Cause Summary

The fundamental issue was: ESP32 D-bus cannot write to I-bus addresses.

Our original approach loaded U-Boot code to IRAM (0x40080000+) via memcpy, which uses D-bus stores. ESP32's bus architecture routes D-bus writes to the flash cache controller, not directly to SRAM0. The MMU must have valid entries for the target address to route writes to physical SRAM.

The fix: Run U-Boot from IROM (flash-cached at 0x400D0000+). This matches how ESP-IDF runs applications. The flash MMU maps flash pages to IROM virtual addresses. The CPU fetches instructions via I-cache → MMU → flash SPI controller. No memcpy needed — the code executes directly from flash through the cache.

This required:

1. U-Boot linker script migrated from IRAM (0x40080000) to IROM (0x400D0000)
2. MCUboot esp_loader.c updated to map IROM via cache_flash_mmu_set
3. Wrapper tool updated to emit IROM-segment load headers
4. Flash offset aligned to 64KB (ESP32 MMU page size)
5. VECBASE moved from IRAM to IROM address space