universalisos/docs/esp-qemu-timer-memmodel-audit.md

ESP-QEMU CCOUNT/CCOMPARE Timer & ESP32 Memory Model Audit

Source: /home/fabiorafaelcoutada/portugalfuturista/hypervisor_ref/esp-qemu/ Date: 2026-07-15 Context: Debugging timer wheel hang — tasks wake once then system freezes. Timer fires repeatedly but tasks don't wake from second delay.

---

1. How QEMU Increments CCOUNT — Virtual Clock, NOT Instruction Count

Answer: CCOUNT is derived from QEMU_CLOCK_VIRTUAL (nanosecond wall-clock of the VM), not from an instruction counter.

Mechanism

CCOUNT is lazily computed on every read/write. It is NOT incremented per-instruction. The value is derived from the elapsed virtual-clock nanoseconds converted to CPU clock ticks:

target/xtensa/op_helper.c:39-47 — HELPER(update_ccount):

void HELPER(update_ccount)(CPUXtensaState *env)
{
    XtensaCPU *cpu = env_archcpu(env);
    uint64_t now = qemu_clock_get_ns(QEMU_CLOCK_VIRTUAL);   // <-- virtual clock

    env->ccount_time = now;
    env->sregs[CCOUNT] = env->ccount_base +
        (uint32_t)clock_ns_to_ticks(cpu->clock, now - env->time_base);
}

• ccount_base is an offset adjusted when the guest writes CCOUNT (wsr_ccount, line 54).
• time_base is set once at IRQ init to the virtual clock value at that time (pic_cpu.c:107).
• clock_freq_khz = 40000 for ESP32 → 240 MHz... wait, 40000 kHz = 40 MHz? No — core-esp32.c:34: .clock_freq_khz = 40000 → 40 MHz base clock. But the ESP32 actual CPU frequency can be changed at runtime via esp32_clk_update (esp32.c:224-246) which calls clock_update_hz().

When CCOUNT is Updated

CCOUNT is updated (re-synced from virtual clock) on these operations:

• rsr CCOUNT — translate.c:2068-2076: calls gen_helper_update_ccount before reading.
• wsr CCOUNT — translate.c:2449-2455: calls gen_helper_wsr_ccount.
• xsr CCOUNT — translate.c:2640-2653: calls gen_helper_update_ccount then wsr_ccount.
• wsr CCOMPAREn — indirectly via update_ccompare → update_ccount (op_helper.c:67).
• waiti — not directly, but check_interrupts reads INTSET/INTENABLE.

Key Implication: Between reads, CCOUNT in env->sregs[CCOUNT] is STALE. It only reflects the true virtual-clock-derived value when update_ccount runs. QEMU TCG does not increment it per-instruction.

CCOUNT ↔ CCOMPARE Timer

The CCOMPARE timer is a QEMU QEMUTimer on the virtual clock:

hw/xtensa/pic_cpu.c:106-113 — xtensa_irq_init:

if (xtensa_option_enabled(env->config, XTENSA_OPTION_TIMER_INTERRUPT)) {
    env->time_base = qemu_clock_get_ns(QEMU_CLOCK_VIRTUAL);
    env->ccount_base = env->sregs[CCOUNT];
    for (i = 0; i < env->config->nccompare; ++i) {
        env->ccompare[i].env = env;
        env->ccompare[i].timer = timer_new_ns(QEMU_CLOCK_VIRTUAL,
                xtensa_ccompare_cb, env->ccompare + i);
    }
}

The timer is armed in update_ccompare (op_helper.c:60-72):

void HELPER(update_ccompare)(CPUXtensaState *env, uint32_t i)
{
    XtensaCPU *cpu = env_archcpu(env);
    uint64_t dcc;

    qatomic_and(&env->sregs[INTSET], ~(1u << env->config->timerint[i]));  // clear timer IRQ
    HELPER(update_ccount)(env);
    dcc = (uint64_t)(env->sregs[CCOMPARE + i] - env->sregs[CCOUNT] - 1) + 1;
    timer_mod(env->ccompare[i].timer,
              env->ccount_time + clock_ticks_to_ns(cpu->clock, dcc));
    env->yield_needed = 1;
}

• dcc = delta in CCOUNT ticks between CCOMPARE and current CCOUNT.
• timer_mod schedules the QEMU timer at ccount_time + dcc_in_ns on the virtual clock.

---

2. CCOMPARE0 Firing: Does QEMU Auto-Clear INTSET?

Answer: QEMU does NOT auto-clear the timer interrupt bit in INTSET when CCOMPARE fires. The guest MUST clear it.

What Happens When the Timer Fires

hw/xtensa/pic_cpu.c:85-92 — xtensa_ccompare_cb:

static void xtensa_ccompare_cb(void *opaque)
{
    XtensaCcompareTimer *ccompare = opaque;
    CPUXtensaState *env = ccompare->env;
    unsigned i = ccompare - env->ccompare;

    qemu_set_irq(env->irq_inputs[env->config->timerint[i]], 1);
}

This calls qemu_set_irq(..., 1) which routes to xtensa_set_irq:

hw/xtensa/pic_cpu.c:66-83 — xtensa_set_irq:

static void xtensa_set_irq(void *opaque, int irq, int active)
{
    CPUXtensaState *env = opaque;
    ...
    uint32_t irq_bit = 1 << irq;

    if (active) {
        qatomic_or(&env->sregs[INTSET], irq_bit);       // SET the bit
    } else if (env->config->interrupt[irq].inttype == INTTYPE_LEVEL) {
        qatomic_and(&env->sregs[INTSET], ~irq_bit);     // only LEVEL type auto-clears on deassert
    }
    check_interrupts(env);
}

Critical: Timer Interrupts Are INTTYPE_TIMER, NOT INTTYPE_LEVEL

For ESP32, the timer interrupts map as follows:

core-esp32/core-isa.h:417, 426-427, 455-457:

Interrupt 6  (CCOMPARE0) → INTTYPE_TIMER  (XTHAL_INTTYPE_TIMER)
Interrupt 15 (CCOMPARE1) → INTTYPE_TIMER
Interrupt 16 (CCOMPARE2) → INTTYPE_TIMER

Since inttype == INTTYPE_TIMER (not INTTYPE_LEVEL), the else if branch in xtensa_set_irq never fires for timer interrupts. The bit stays set in INTSET once asserted.

How the Guest Clears It

The guest must clear the timer interrupt via wsr intclear or wsr ccompare (which also clears):

target/xtensa/exc_helper.c:128-143 — HELPER(intset) / HELPER(intclear):

void HELPER(intclear)(CPUXtensaState *env, uint32_t v)
{
    intclear(env, v & (env->config->inttype_mask[INTTYPE_SOFTWARE] |
                       env->config->inttype_mask[INTTYPE_EDGE]));
}

⚠️ CRITICAL BUG-CANDIDATE: intclear only allows clearing INTTYPE_SOFTWARE and INTTYPE_EDGE masked bits. INTTYPE_TIMER bits are NOT in either mask. The only way to clear a timer interrupt is via wsr CCOMPAREn (which calls update_ccompare, clearing the specific timer bit at op_helper.c:65-66).

core-esp32/core-isa.h:446-449 — masks:

INTTYPE_MASK_SOFTWARE  = 0x20000080   (ints 7, 29)
INTTYPE_MASK_EXTERN_EDGE = 0x50400400 (ints 10, 22, 28, 30)
INTTYPE_MASK_TIMER     = 0x00018040   (ints 6, 15, 16) ← NOT clearable via intclear!

Summary of Clearing Paths for Timer IRQ (int 6/15/16)

Operation	Clears timer bit?	Reference
wsr CCOMPAREn (same index)	✅ YES	op_helper.c:65-66
wsr intclear	❌ NO (INTTYPE_TIMER not in mask)	exc_helper.c:139-142
xtensa_set_irq(irq, 0)	❌ NO (not INTTYPE_LEVEL)	pic_cpu.c:77-78
QEMU auto-clear on delivery	❌ NO	no such code path exists

⚠️ This is the most likely cause of the timer wheel hang. If the guest ISR does not re-write CCOMPARE0 (or writes a new CCOMPARE value), the interrupt bit stays set in INTSET. With PS.EXCM set during ISR, and the interrupt still pending, the system enters a tight re-exception loop or deadlocks.

---

3. CCOMPARE Re-arming from ISR Context — Known Issues

The Re-arm Sequence

For the timer to fire again, the guest MUST:

1. Read current CCOUNT (or know it implicitly).
2. Write the new compare value to wsr CCOMPARE0.
3. This triggers update_ccompare which: (a) clears the timer IRQ bit, (b) re-arms the QEMU timer.

Potential Issues

Issue A: INTSET bit never cleared if CCOMPARE not re-written

As detailed in section 2, the timer interrupt bit (int 6) stays in INTSET until wsr CCOMPARE0 runs. If the guest only does wsr intclear (common pattern in FreeRTOS/RTOS ISRs), the bit remains set → check_interrupts (pic_cpu.c:35-64) sees the interrupt is still pending → immediately re-enters ISR → livelock or hang.

Issue B: yield_needed flag

update_ccompare sets env->yield_needed = 1 (op_helper.c:71). This triggers a HELPER(exception)(env, EXCP_YIELD) at the next TCG exit. If the yield mechanism is not properly handled in the hypervisor/OS context, this can cause the CPU to stall.

Issue C: Timer fires while in EXCM state

When the timer fires and the CPU is already in EXCM (exception mode, e.g., inside another ISR or exception handler), handle_interrupt (exc_helper.c:161-203) checks:

if (level > xtensa_get_cintlevel(env) && ...)

For CCOMPARE0 at level 1 (core-isa.h:379: XCHAL_INT6_LEVEL = 1), if cintlevel >= 1 (which it is when PS.EXCM is set, since excm_level = 3), the interrupt will NOT be delivered immediately. It remains pending. This is correct behavior but means:

• If the ISR runs at level ≥ 1 (which it always does with EXCM), a timer that fires DURING the ISR is deferred.
• When the ISR returns via rfi, EXCM is cleared, cintlevel drops, and the pending timer interrupt fires — but only if INTSET still has the bit set AND INTENABLE has it enabled.

Issue D: Timer interrupt at level 1 only

CCOMPARE0 → int 6 → level 1 (core-isa.h:379). XCHAL_EXCM_LEVEL = 3 (core-isa.h:351). This means PS.EXCM masks level 1-3. A level-1 timer interrupt CANNOT preempt any exception handler. The OS must use rfi to lower cintlevel to 0 for the timer ISR to run.

For the timer wheel hang: If the first delay works (timer fires, ISR runs, task wakes), but the second delay fails, the most likely scenario is:

1. First ISR clears the timer condition by writing CCOMPARE0 (re-arming).
2. Second fire happens, but either: (a) INTENABLE was cleared and not re-enabled, (b) the CCOMPARE write didn't happen because the ISR returned before re-arming, or (c) the update_ccompare timer_mod computed dcc from a stale CCOUNT (if update_ccount wasn't called recently, the virtual clock may have advanced significantly).

---

4. ESP32 DRAM Memory Map

Primary Memory Regions

hw/xtensa/esp32.c:61-76 — esp32_memmap[]:

Region	Base	Size	Type	Notes
DROM	0x3FF90000	0x10000 (64 KB)	ROM	Alias of IROM+0x60000, per-CPU
IROM	0x40000000	0x70000 (448 KB)	ROM	Per-CPU, flash cache
DRAM	0x3FFAE000	0x52000 (336 KB)	RAM	Main SRAM, shared
IRAM	0x40080000	0x40000 (256 KB)	RAM	Instruction RAM, shared
ICACHE0	0x40070000	0x8000 (32 KB)	RAM	Shared
ICACHE1	0x40078000	0x8000 (32 KB)	RAM	Shared
RTCSLOW	0x50000000	0x2000 (8 KB)	RAM	RTC slow memory, shared
RTCFAST_I	0x400C0000	0x2000 (8 KB)	RAM	PRO CPU only
RTCFAST_D	0x3FF80000	0x2000 (8 KB)	RAM	Alias of RTCFAST_I, PRO CPU only
FRAMEBUF	0x20000000	variable	RAM	Virtual framebuffer

DRAM Details

• Base: 0x3FFAE000
• Size: 0x52000 = 336 KB
• End: 0x3FFAE000 + 0x52000 = 0x40000000
• Type: memory_region_init_ram — plain RAM, shared between both CPUs (sys_mem)
• Allocation: esp32.c:300-302:

  memory_region_init_ram(dram, NULL, "esp32.dram", 0x52000, &error_fatal);
  memory_region_add_subregion(sys_mem, 0x3FFAE000, dram);
  

Memory Aliases

1. DROM is an alias of IROM: esp32.c:296:

   memory_region_init_alias(drom, NULL, name, irom, 0x60000, 0x10000);
   

   DROM at 0x3FF90000 maps to IROM offset 0x60000 → physical 0x40060000.

2. RTCFAST_D is an alias of RTCFAST_I: esp32.c:326:

   memory_region_init_alias(rtcfast_d, NULL, "esp32.rtcfast_d", rtcfast_i, 0, 0x2000);
   

3. APB register mirror: Peripheral devices registered at both DPORT base and APB base (esp32.c:248-257):

   // DPORT base and APB base mirror
   memory_region_add_subregion_overlap(dest, dport_base_addr, mr, 0);
   memory_region_add_subregion_overlap(dest, dport_base_addr - DR_REG_DPORT_APB_BASE + APB_REG_BASE, mr_apb, 0);
   

4. PSRAM (optional): If s->dport.has_psram, a cache region dram1 is overlaid at dram1->base (esp32.c:365-372). This extends DRAM into the 0x3F800000 range via the flash cache controller.

CPU-Specific vs Shared Memory

• Per-CPU (via s->cpu_specific_mem[i]): IROM, DROM, RTCFAST_I, RTCFAST_D, cache regions (drom0, iram0, dram1).
• Shared (via sys_mem): DRAM, IRAM, ICACHE0, ICACHE1, RTCSLOW, all peripherals.

---

5. Nested Exception / Double Exception Handling

PS Register Fields

target/xtensa/cpu.h:658-665 — xtensa_get_cintlevel:

static inline int xtensa_get_cintlevel(const CPUXtensaState *env)
{
    int level = (env->sregs[PS] & PS_INTLEVEL) >> PS_INTLEVEL_SHIFT;
    if ((env->sregs[PS] & PS_EXCM) && env->config->excm_level > level) {
        level = env->config->excm_level;
    }
    return level;
}

For ESP32: excm_level = 3 (core-isa.h:351). When PS.EXCM is set, cintlevel is at least 3, masking all level 1-3 interrupts.

Exception Entry

target/xtensa/exc_helper.c:48-69 — HELPER(exception_cause):

void HELPER(exception_cause)(CPUXtensaState *env, uint32_t pc, uint32_t cause)
{
    uint32_t vector;

    env->pc = pc;
    if (env->sregs[PS] & PS_EXCM) {
        // ALREADY in exception mode → DOUBLE EXCEPTION
        if (env->config->ndepc) {
            env->sregs[DEPC] = pc;      // save PC in DEPC
        } else {
            env->sregs[EPC1] = pc;
        }
        vector = EXC_DOUBLE;            // → double exception vector
    } else {
        env->sregs[EPC1] = pc;          // normal: save in EPC1
        vector = (env->sregs[PS] & PS_UM) ? EXC_USER : EXC_KERNEL;
    }

    env->sregs[EXCCAUSE] = cause;
    env->sregs[PS] |= PS_EXCM;          // set EXCM

    HELPER(exception)(env, vector);
}

Interrupt-Driven Double Exception

target/xtensa/exc_helper.c:161-203 — handle_interrupt:

static void handle_interrupt(CPUXtensaState *env)
{
    int level = env->pending_irq_level;

    if ((level > xtensa_get_cintlevel(env) && ...) || level == env->config->nmi_level) {
        if (level > 1) {
            // High-priority interrupt
            env->sregs[EPC1 + level - 1] = env->pc;    // save PC
            env->sregs[EPS2 + level - 2] = env->sregs[PS]; // save PS
            env->sregs[PS] = (env->sregs[PS] & ~PS_INTLEVEL) | level | PS_EXCM;
            env->pc = relocated_vector(env, env->config->interrupt_vector[level]);
        } else {
            // Level-1 interrupt → converted to exception
            env->sregs[EXCCAUSE] = LEVEL1_INTERRUPT_CAUSE;

            if (env->sregs[PS] & PS_EXCM) {
                // Already in EXCM → DOUBLE EXCEPTION
                env->sregs[DEPC] = env->pc;   // (if ndepc)
                cs->exception_index = EXC_DOUBLE;
            } else {
                env->sregs[EPC1] = env->pc;
                cs->exception_index = (env->sregs[PS] & PS_UM) ? EXC_USER : EXC_KERNEL;
            }
            env->sregs[PS] |= PS_EXCM;
        }
    }
}

Double Exception Behavior Summary

1. Normal exception (PS.EXCM=0): PC→EPC1, vector=EXC_USER/EXC_KERNEL, PS.EXCM set.
2. Exception while EXCM=1: PC→DEPC (if ndepc configured), vector=EXC_DOUBLE. For ESP32, ndepc is set (DEPC exists).
3. Level-1 interrupt while EXCM=1: Treated as double exception (EXCCAUSE=LEVEL1_INTERRUPT_CAUSE, EXC_DOUBLE vector). This is the path a timer interrupt takes if it fires while already handling another level-1 exception.
4. High-priority interrupt (level > 1): Uses separate EPCn/EPSn registers. Can nest. PS.EXCM is set, INTLEVEL raised to the interrupt level.

Double Exception Vector

target/xtensa/exc_helper.c:224-250 — xtensa_cpu_do_interrupt:

case EXC_DOUBLE:
    if (env->config->exception_vector[cs->exception_index]) {
        vector = env->config->exception_vector[cs->exception_index];
        env->pc = relocated_vector(env, vector);
    }

The double exception vector is fetched from exception_vector[EXC_DOUBLE], which comes from EXCEPTION_VECTORS in overlay_tool.h, derived from XCHAL_DOUBLEEXC_VECTOR_* in core-isa.h.

---

Summary: Root Cause Hypotheses for Timer Wheel Hang

Most Likely: Timer INTSET bit not cleared between fires

Mechanism: CCOMPARE0 fires → int 6 set in INTSET → ISR runs → ISR does NOT re-write CCOMPARE0 (or writes intclear which doesn't clear INTTYPE_TIMER) → int 6 stays in INTSET → check_interrupts sees pending interrupt → on rfi lowering cintlevel, immediately re-enters ISR → infinite loop / hang.

Evidence: pic_cpu.c:77-78 — only INTTYPE_LEVEL interrupts auto-clear on deassert. Timer is INTTYPE_TIMER. exc_helper.c:139-142 — intclear only clears INTTYPE_SOFTWARE | INTTYPE_EDGE.

Fix: The guest ISR MUST write wsr CCOMPARE0 with the next compare value, which calls update_ccompare and clears the bit (op_helper.c:65-66). Writing wsr INTCLR is insufficient.

Secondary: yield_needed causing unexpected TCG exit

update_ccompare sets yield_needed = 1 (op_helper.c:71). If the hypervisor doesn't handle EXCP_YIELD properly, the CPU may stall after the first timer re-arm.

Tertiary: Virtual clock drift vs instruction count

Since CCOUNT is tied to virtual clock (not instruction count), the timer fires based on wall-clock VM time, not execution progress. In a heavily loaded QEMU or under TCG single-step, the virtual clock and instruction count can diverge significantly, causing timers to fire at unexpected points relative to code execution.