Allow SGIs to be generated from non-SMP kernels.
Signed-off-by: Kent McLeod <kent@kry10.com>
Signed-off-by: Gerwin Klein <gerwin.klein@proofcraft.systems>
Use atomic operations to get and set the IPI pending flag to
ensure that if an IPI is observed, the IPI arguments are also
observed.
Using atomics makes this requirement explicit to the compiler.
Without this change there is a race between setting up an IPI
and another core checking for ipi == 1. This can happen before
IPI_MEM_BARRIER has been executed.
Found by Yanyan Shen.
Signed-off-by: Indan Zupancic <indan@nul.nu>
Align the kernel lock to the exclusive reservation granule size.
Without this nearby writes can delay atomic operations implemented
with looping exclusive load/store instructions for an undefined time.
Cleanup IPI code.
Signed-off-by: Indan Zupancic <Indan.Zupancic@mep-info.com>
Remove cpu argument from clh_lock_acquire/release, it is never
valid to call them with something else than getCurrentCPUIndex().
Use clh_is_self_in_queue() instead of poking into lock internals
in ipiStallCoreCallback().
Add some comments from the paper to make it easier to match the
implementation with the description in the paper it's based on.
Don't use confusingly different naming, but follow naming from
the paper: Rename 'node' to 'myreq' and 'next' to 'watch', etc.
Signed-off-by: Indan Zupancic <Indan.Zupancic@mep-info.com>
This reverts commit 3d2ae69f9c:
"ARM/SMP: Re-implement atomic exchange taking sel4 IPI into account"
Also removed riscv's try_arch_atomic_exchange_rlx().
This was added because CAS can take very long to finish on ARM if
the exclusive reservation granule (ERG) is large for the platform,
as any writes on other cores within up to 2Kb could make the CAS
fail.
The correct fix is to add padding around the global lock equal to ERG
to make CAS fast, which would make the extra IPI check redundant.
Signed-off-by: Indan Zupancic <Indan.Zupancic@mep-info.com>
Strengthen the clh_lock_acquire to use release on the atomic_exchange
that makes the node public. Otherwise (on ARM & RISCV), the store to
the node value which sets its state to CLHState_Pending can become
visible some time after the node is visible.
In that window of time, the next thread which attempts to acquire the
lock will still see the old state (CLHState_Granted) and enters the
critical section, leading to a mutual exclusion violation.
Signed-off-by: jonas <s9joober@gmail.com>
The implementation of try_arch_atomic_exchange does not correctly pass
RELEASE memory ordering (or stronger) to the exchange operation.
To acknowledge this, try_arch_atomic_exchange is replaced by a relaxed
try_arch_atomic_exchange_rlx which does not apply any memory ordering.
Instead, the memory ordering is now added manually by
sel4_atomic_exchange. This provides better latency for interrupts as no
barriers are evoked inside the loop which performs the relaxed exchange
and checks for interrupts.
Furthermore, the new manual application of barriers ensures the memory
ordering passed to sel4_atomic_exchange.
Signed-off-by: jonas <s9joober@gmail.com>
All the kernel header files now use pargma once rather than the ifndef,
as the pre-processed C files do not change while header files
are protected with pargma once. This will also solve any naming issues
caused by ifndef.
This commit also converts our own copyright headers to directly use
SPDX, but leaves all other copyright header intact, only adding the
SPDX ident. As far as possible this commit also merges multiple
Data61 copyright statements/headers into one for consistency.
Added support for injecting remote IPI calls towards given VCPU's
on SMP configured systems. This introudcing a new type of
IpiRemoteCall and handlers for updating the vgic state on
incoming/outgoing IPI's.
Co-authored-by: Yanyan Shen <yanyan.shen@data61.csiro.au>
Explicitly create a struct definition for irq_t on SMP Arm
configurations. This makes it a lot harder to mistakenly use the wrong
irq encoding when moving an irq between a cnode index and hardware irq
number / core. A couple areas where this was being handled incorrectly
was fixed as part of the refactor. When performing an ipi for masking
PPI interrupts, the idx encoding is used as it fits into a single word.
This provides a more robust and efficient implementation that only forces memory reloads
at explicit known points, instead of the over approximation of every time they are
accessed that is forced by volatile.
- Restart TCB from inside the lock if it is waiting for anything other than IRQ
- Only replace the TCB with idle thread if it is in ThreadState_RunningVM state
Also, this makes the design generic to be shared with arm.
Currently building ARM/SMP is broken. This commit:
1- Makes it possible to build ARM/SMP with stubs. Run-time SMP for ARM
DOES NOT WORK.
2- Can be a reference for future SMP targets to follow in order to
layout/add the minimal required files and functions needed to support SMP.
3- Builds for Sabre only. In order to support other platforms, ipi
interrupt ID should be defined in machine.h
This commit is a re-arrangement of SMP directory structure to make it
easier for other architectures/platforms (in general) and ARM (in
particular) to add SMP support.
* new include/smp directory to act as a centralised container of
"shared" architecture-independent SMP headers.
This makes it clearer what's needed for other architecture/platform to
support SMP.
* Each platform can define its own unique ipi.[h|c] that make sense,
since ipi implementation is SoC/platform dependent.
