The Arch_hasVCPU function instead of cheaking for whether a thread has a VCPU was
specifically checking for whether a thread on aarch32 had a VCPU, as that was the
condition needed to be passed into sanitiseRegister. This made the implementation
on x86 extremely confusing as Arch_hasVCPU was implemented to return false due to
there being no need for sanitiseRegister on x86 to know whether there was a vcpu.
This commit changes hasVCPU variables to be an abstract archInfo boolean that can
be used arbitrarily by an architecture. Arch_hasVCPU function was also changed to
become Arch_getSanitiseRegisterInfo whose result only passed to sanitiseRegister.
This version of compile_assert is not expanded during verification builds and
can be used in cases where you would like an assertion that contains syntax
that is not supported by the C parser
In the original slow path, armv_contextSwitch_HWASID shoots first.
Verification can't reorder machine operations on hardware state we don't
model. This puts the fastpath into the same order as the slow path.
invokeVCPUReadReg should not be setting message registers for the return message unless
the user performed a call. In doing so we must refactor the call to readVCPUReg to
outside the introduced `if` condition since, as it performs machine operations, it
should always happen
The fields in the multiboot_info structure have precise sizes and should not use pointers
types as these sizes change between ia32 and x86_64 builds. This commit changes such
fields to be the correct fixed size uint32_t
When performing non standalone builds a separate autoconf.h file is generated and defines
the configuration of the kernel. Currently the directory with that file as well as the
kernels include/plat/${PLAT} directory are both listed as include directories to the
compiler. Due to the directory with the generated autoconf.h being earlier in the -I
list the generated autoconf.h will be used in preference to the standalone autoconf.h.
Relying on the order of -I directives is too unstable and so this commit moves the
autoconf.h files to a dedicated configs directory, which is only added as a -I
directive if a standalone build is being performed.
Provides a configuration option for enabling HDCR.TD* traps, or saving and loading debug
state on VCPU switches. Currently verification only plans to support the trap setting.
As this option complicates all of the #ifdef's related to debug registers even further,
abstractions for enabling/disabling each individual piece of the debug code for different
configuration options are also implemented.
Part of these refactored #ifdef guards was to remove the guards completely from libsel4
around the definitions of the number of breakpoints and watchpoints.
We were previously using UNPREDICTABLE behaviour, because we were setting
HDCR.TDE without setting the other debug-related bits in HDCR.
Also, don't duplicate code: just call setHDCRTrapDebugExceptionState
within initHDCR().
A more efficient way of setting up threads' initial context is implemented
by this patch: essentially instead of reading the debug coprocessor everytime
we create a thread, we read the coprocessor once at boot and then assign
those pre-read values when initializing threads.
Previous implementation read a 64-bit value from two registers in one go
and implicitly cast down to paddr_t.
C Parser cannot handle an asm register read into anything other than a
word.
Workaround: perform two 32-bit reads instead, returning one of them.
struct vcpu { struct tcb* tcb; ...
struct arch_tcb { struct vcpu* vcpu; ...
and
struct tcb { struct arch_tcb tcbArch; ...
These conspire to generate a type error on verification side due to
assumptions about non-colliding names.
- Restart TCB from inside the lock if it is waiting for anything other than IRQ
- Only replace the TCB with idle thread if it is in ThreadState_RunningVM state
Also, this makes the design generic to be shared with arm.
This removes some duplicated constants and moves some constants into debug.h such that
they are visible without the user debugging API enabled. These constants are useful
for configuring other debugging/performance monitoring features.
This patch disables the debug register state save and restore on ARM-hyp,
such that Guest VMs can now interfere with one another and trigger
one another for certain limited scenarios. This will be undone later on.
This patch enables the hype-mode kernel to intercept non-secure PL0
debug exceptions. Since it uses the HDCR register to do this (and not the SDCR),
it consequentially has no effect on any secure-monitor that may be executing
alongside the hypmode kernel.
For native seL4-hyp user threads, when the CPU sees a need to raise a debug
exception that was taken from PL0 (user), it will naturally try to raise that
exception to PL1 (supervisor).
But since seL4 is in PL2 and not PL1, seL4 has not set up an exception vector
table (VBAR) for PL1. So when the CPU attempts to vector into the PL1 VBAR,
it just gets a data abort, which it then delivers to PL2.
This patch causes the PL2 seL4 hypervisor to directly catch the debug exception
straight from PL0 and process the debug exception directly, for native threads.
Guest VM VCPUs are unaffected by this change. Their debug exceptions are delivered
to the Guest VM's PL1 VBAR as per usual.
As things are now, the Guest VMs can modify the debug registers at will
from non-secure PL1, and the kernel does nothing to ensure that guest VM
debug coprocessor registers are preserved.
This is a preliminary patch that simply hooks into vcpu_save and saves the
CPU's debug coprocessor registers when saving VCPU state.
For restoring the debug registers on switching to a VCPU, we just re-use
restore_user_debug_context, which is already called in restore_user_context.
(Restore_user_context is called by c_handle_vcpu_fault()). Specifically,
we modify the used_breakpoints_bf so that restore_user_debug_context()
will always pop all the debug context.
This patch only covers the breakpoint and watchpoint registers,
and it doesn't cover the entire debug coprocessor, which is another
conversation.
* commit '5253dca8b1e6a5cda3b6c7adce55f398ee3274fe':
SELFOUR-746: ARM/SMP - Reserve the first two SGI IDs for seL4 IPI
SELFOUR-746: ARM/GIC - Provide IPI support
SELFOUR-746: ARM/GIC - bank active_irq for each core
Similar to initIRQController(), cpu_initLocalIRQController() has to
be implemented for each platform. For those that are gonna support
SMP, it should be implemented proprely (if required), otherwise
it can be stubbed for single core implementations.
On 64-bit platforms physical address could be >2^32 offset from a virtual address. This
changes offsets to match the word size of the target architecture
In SMP context, init_cpu is meant to be called for each CPU, where
initialising private timer, user access and benchmarks (basically CCNT)
are per-CPU operations and involve per-CPU hardware
resources/interrupts.
When the user has no PD in order to schedule them we must still set something as
the active PD, just so that they can then fault. Previously when hypervisor was
enabled `0` was at as the PD. This worked as platforms typically do not have
anything valid, and certainly nothing resembling a page directory, at physical
address 0. This assumption is not very clean and unknown physical addresses
should not be accessed, so this change adds an explicit PD that is empty that
can be switched to.
Currently building ARM/SMP is broken. This commit:
1- Makes it possible to build ARM/SMP with stubs. Run-time SMP for ARM
DOES NOT WORK.
2- Can be a reference for future SMP targets to follow in order to
layout/add the minimal required files and functions needed to support SMP.
3- Builds for Sabre only. In order to support other platforms, ipi
interrupt ID should be defined in machine.h
In handleSyscall the current thread may be preempted to handle a pending interrupt.
With kernel mode interrupts in x86 this handling was delayed until we were about
to switch back to user mode. This change unifies the handling with ARM, where the
interrupt is handled prior to calling the thread scheduler.
Change AP boot code to use a memory region that is lower down, although also
smaller. As long as the boot code can fit in this smaller region booting is
more reliable as depending on the machine, bios, firmware and bootloader
varying amounts of the low memory will be available.
This commit changes the stack so that it is declared a single time in stack.c,
with the correct attributes and alignment, instead of being declared in stack.h
and relying on the linker to collapse the multiple declarations.
Previously if the `ksCurThread` of the destination core is not the thread that is
receiving the notification in `sendSignal` then the notification would be delayed
until the VMM for that VM next did a `seL4_VMEnter`. This adds a `tcb_t*` parameter
to `VMCheckBoundNotification` instead of having it implicitly talk about `ksCurThread`.
A TCB parameter is safe to send here as the `doRemoteVMCheckBoundNotification`
happens synchronously and so the TCB is guaranteed to still exist.
Provides a mechanism where if af GP fault is potentially expected it can be caught
and the calling code returned to. The GP happening can be detected by setting
a specific return to handler and additionally by the return to handler being
cleared if a GP happened.
This commit allows x86 to completely handle a pending interrupt without switching
out to user mode. To handle an interrupt on x86 the APIC *must* generate an exception,
prior to you being able to acknowledge it. Previously we only allow exceptions (i.e.
interrupts) to be generated outside of kernel mode when we are in user mode.
This change allows us to 'poll' for an interrupt and transition the APIC whilst in kernel
mode by enabling and taking interrupts at carefully defined points. A pending interrupt
will be stored by the exception handler, allowing us to then handle the interrupt and
acknowledge the hardware APIC. Handling is done by waiting until after we have 'left' the
kernel and are about to switch to user mode and then 'entering' the kernel again by jumping
to the interrupt entry point.
Handling interrupts entirely in kernel mode provides two advantages
* It will allow, in the future, the ability to handle kernel interrupts in situations
where we need to handle the interrupt before actually performing the hardware switch
back to user mode. This case happens where the user thread is using vt-x and so
pending interrupts do not generate an interrupt exception, but rather cause an exception
to be generated telling the system that there is a pending interrupt
* Where there are multiple pending interrupts it is more efficient to avoid additional
switches in and out of the user thread
Whilst this change does not enable pre-emption points to handle the interrupt before
returning out of `handleSyscall` it should be easily implementable with what is provided.
Adds space in the vcpu_t struct for the registers from the different operating modes
and saves/restores these registers as part of saving and restoring the VCPU
Previously any thread could be set to run in any mode (except HYP mode).
Whilst this causes no security issues for the kernel, different execution
modes in ARM have different banked registers. These registers are not
currently saved and restored allowing for threads to manipulate registers
that will be seen (or potentially are being actively used) by other threads.
Saving and restoring these banked registers for all threads is a performance
cost (even if only done for the actual mode the thread runs in) for no real
benefit, as there is no clear reason to run a thread in other modes if you
do not have a vcpu. Therefore this commit restricts being in modes other than
user to threads that have a vcpu. When multiple VCPUs are properly supported
the switching of VCPUs will then save/restore these banked registers.
Prior to this commit virtual addresses in the bitfield were considered
to be 48-bits, with sign extension to 64-bits. This meant that we were
actually sign extended bit number 47. Whilst this gives correct cannonical
addresses it prevents a portion of the address space from being used.
This commit changes the bitfield to use 49-bit virtual addresses, so that
after the bit 48 is used for sign extension there is still 48-bits of
usable address.
The ARM_MPCORE configuration was added to capture which systems have the additional
ThreadID registers. This was an unneeded indirection as any arm version from v7 and
beyond has these registers. For the purposes of seL4 this means any arm version
except v6. This change scrubs ARM_MPCORE and replaces it with negated usages of
ARCH_ARM_V6. Additionally the IPC buffer config defaults were updated as the
ARM_MPCORE config was explicitly selecting the optimal one.
This commit makes seL4 hold the stack address on TPIDRPRW register (PL1 only
Thread ID Register, VMSA, see B4.1.150 section of ARMv7-A and ARMv7-R
edition reference manual).
Originally, the kernel stack was (re-)mapped to a fixed kernel virtual address and
on traps sp is loaded with this virtual address. Reason is to shave up some
cycles/instructions on (fast) traps path instead of having two instructions
(loading global variable pointer and then memory-load stack address from
it).
With ARM pipelining getting smart/complex (from ARMv7 onwards), there is
no noticeable difference in performance between the two implementations.
Furthermore, this change makes it easy for SMP to load different stacks
for each core and do CPU ID arithmetic efficiently. It also avoids the
issues involving (re-)mapping stacks for different cores if we
followed the original design.
Note: the main side-effect to this change is that the kernel stack for
ARM is no longer mapped with execute-never attribute.
* commit 'be77963e5bb4c5c28ad3ebd8746c292e7b5876f6':
Define CONFIG_MAX_NUM_NODES and CONFIG_KERNEL_STACK_BITS in plat/autoconf.h
Unify kernel stack definition/declaration and share it between architecures/modes
Defines the vtd structures to be their full 52-bit size when in 64-bit mode by
making the hardware.bf per mode in the pc99 platform.
Updates some variables in iospace.c that were hard defined to be 32-bit
Rather than defining a kernel stack in random places for each
architecure for each mode (32/64) and for single/SMP modes, make the
stack definition shareable between all of the above. This is also useful
for the future ARM SMP work.
This commit is a re-arrangement of SMP directory structure to make it
easier for other architectures/platforms (in general) and ARM (in
particular) to add SMP support.
* new include/smp directory to act as a centralised container of
"shared" architecture-independent SMP headers.
This makes it clearer what's needed for other architecture/platform to
support SMP.
* Each platform can define its own unique ipi.[h|c] that make sense,
since ipi implementation is SoC/platform dependent.
Circular dependency prevents extern definition of benchmark_log_utilisation_enabled from
getting included in include/arch/arm/arch/benchmark_overflowHandler.h.
arch-level headers shouldn't depend on arch-independent headers to avoid
such circular dependency issues (in the future).
This looks like we're just swapping the positions of capEPBadge and capEPPtr,
but it turns out that the bitwise op being performed on capEPPtr to set the
high bits were part of the data dependency critical path, so this actually
does improve the speed by moving the bitwise op to capEPBadge (albeit it's
now an AND instead of an OR)
I initially set the field size to 32 bits, but it turns out that causes gcc
to emit an instruction (mov r32, r32) that causes the instruction decoder
to switch to the legacy decode path for the rest of the fast path for some
reason.
Previous code to return to user level performed
popf
sysexit
The popf was just before the sysexit as there is a one instruction
delay on the trap flag taking effect and ensured we did not attempt
to single step the kernel. Unfortunately there is not a one instruction
delay on enabling the interrupt flag, and as a result an interrupt
can be taken prior to executing the sysexit instruction. It is
possible to exploit this to escalate a user level thread such that
it is running with CPL0
This commit changes the restore paths to perform
sti
sysexit
Which will correctly delay interrupts until the completion of sysexit.
As the popf is now being done earlier to prevent single stepping the
kernel we return via an iret, instead of sysexit, for threads that
have single stepping enabled. To achieve this we
* When loading debug state if we enable the Trap flag we also manipulate
the register state such that the iret return path will be picked
* As fastpath_restore does not have an iret return path we forbid
the fastpath from switching to threads that have single stepping
enabled
* Adds object and cap definitions for VT-x structures (VCPU and EPT).
* Extends the asid_map implementation to support ASIDs in the EPT
* Adds size definitions for VCPU and EPT objects
* commit '6cd485204f0b3ca09a42770c545f0724189e7b78':
x86: Add Skylake-related info
x86: Add support for more architectures in reading the TSC frequency
This actually leads to better code. Copies of the halt loop inlined
in various places will instead be single instructions 'bl halt'. It's
also important for the translation validation to avoid having
pointless loops everywhere, especially inside the bodies of other
loops.
The vast majority of the DONT_TRANSLATE markers in the kernel are used
to hide __asm__ statements and builtin functions
(e.g. __builtin_unreachable ()) from the C-to-Isabelle parser.
The parser now supports underscore identifiers and many __asm__ statements,
and the builtin functions are prototyped, meaning the vast majority of the
DONT_TRANSLATE markers can be dropped. The remaining markers cover functions
that must be treated specially.
Add compatible prototypes for compiler builtins
__builtin_unreachable, __builtin_ctzl, __builtin_clzl,
and __builtin_popcountl.
The compiler ignores these, but they are necessary for the Isabelle
C parser to handle them. This is needed to drop DONT_TRANSLATE markers
from various functions which call these builtins.
Previously we threw away any modifications the kernel may have made
to the EFER when a VM exit happens. In x86-64 there are modifications
to the EFER that must be preserved
This unifies the interface provided by ia32 and x86-64. This interface
is neccessary as it is not always sufficient to talk about the current
PD (or vspace root), as CR3 can contain more than just the root.
x86-64 has better ways (`swapgs`) of managing per core kernel stacks.
This commit hides the `kernelSP` member of a `vcpu_t` under x86-64
so as not to cause confusion and accidental attempted usage
Previously the GP registers for a VCPU were defined in the 32-bit
arch registerset. This does not actually make sense as the mode
for the VCPU should be decoupled (and well defined) regardless of
the execution mode of the kernel. This commit provides an explicit
definition and register order for VCPU GP registers.
Uninitialized data structures in BOOT and PHYS code currently get
placed in sections that are allocated in the file of the final image.
Whilst these sections will get reclaimed during kernel boot, so no
runtime memory is being wasted, it results in kernel images that
are much larger to load and transport than necesary.
This change adds explicit BSS regions for both BOOT and PHYS code
and moves all appropriate data structures into them
seL4 is always in the top of memory, so the high bits of pointers are always 1.
The autogenerated unpacking code doesn't know that, however, so will try to
conditionally sign extend (in 64-bit mode), which wastes cycles in the fast
path. Instead, we can do the unpacking ourselves and explicitly set the high
bits.
This commit moves the write to FS and GS base, allowing for a much
more efficient write to GS base under x86-64 SMP. When writing
GS base was in Arch_switchToThread it was neccessary to write to
an MSR such that when swapgs was performed on kernel exit the new
value of GS base would be retrieved. Unfortunately writing to an
MSR is very expensive and we would much prefer to use the writegsbase
instructions instead. By moving this code to restore user context
we are able to call swapgs earlier and then use the normal
writegsbase instruction
When using, SSE (and other variants), mxcsr would assume similar role as
control word in i387 FPU. When initializing the FPU, it should have valid value.
In the common case this function expands to a single check, with a blank
body. Whilst forcively inling will cause code bloat, will be code bloat
in the uncommon code path, which we do not care about
config_default was intended to either evaluated to the passed configuration
value, or the a default value if the config didn't exist. For integer values
this does not actually work, and the default value always gets returned.
This commit reimplements the desired functionality as config_ternary, which
takes 3 arguments, a config to switch on and a desired true and false expansion
* commit 'ed95f84a438aea6365762a180cc493113e9282e0':
SELFOUR-413: changes for verification
SELFOUR-567: use seL4_CapRights_t from libsel4
SELFOUR-413: refactor libsel4 fault API
Split fault types into arch/generic
This change
* changes seL4_CapRights from the kernel to be seL4_CapRights_t in
libsel4
* deprecates the duplicated seL4_CapRights in libsel4, which is
now the bitfield generated type seL4_CapRights_t.
* fixes all usages in kernel and libsel4
Impact: for verification, this will require the type to change name
from cap_rights to seL4_CapRights_t.
This is a breaking libsel4 API change, although most code uses
seL4_AllRights or similar constants, which will not break
at a source level as these constants have been updated.
This is a *breaking API change*
This commit:
* makes seL4_Fault_tag_t common between the kernel and libsel4
* deprecates the existing functions from sel4/messages.h includes
* introduces a new fault API in sel4/faults.h and
* sel4/sel4_arch/faults.h
* deprecates seL4_GetTag(), as the function did not work without
the user calling seL4_SetTag() first (seL4_MessageInfo is passed
in registers and not set in the IPC buffer)
* removes previously deprecated functions (deprecated prior to 3.0.0)
* updates the seL4 manual to reflect the changes
Prior to this commit faults were separate
per architecture. This commit extracts the common
fault types and introduces arch specific faults,
reducing code duplication across architectures.
Adds a heuristic to switch to a NULL fpu state if we think the FPU
is not presently in use. A NULL fpu state is more efficient as
we do not have to enable/disable the FPU when switching threads
The compiler fails to realize that inlining these functions is
a performance benefit due to fact that after inlining their
bodies can be optimized with other inlined functions.
The use of 'volatile' on the asm blocks used by getCurrentCPUIndex
resulted in an inability for the compiler to elide repeated invocations
of getCurrentCPUIndex when inlining. The volatile is not needed
as we already want to claim that these functions are CONST and so
the asm block can be move/reordered/deleted as the compiler wishes
When sending a signal to a notification object that has a bound thread that is
in the 'runningVM' start we, ordinarily, want to deliver it by switching back
to the native execution of that thread and delivering the message. If the thread
is in the 'runningVM' state on a different core, then we must not modify its
thread state. Previously we would modify the thread state (changing it to
Running) and then attemptSwitchTo. Switching would fail, as its affinity would
be for a different core, resulting in us enqueing the thread into the
scheduling queue, despite the thread also presently running.
This commit both adds a check to prevent us manipulating a thread that
is actively running on a different core, as well as a remote operation
to tell that core that if it is running a VM it should recheck its
endpoint for a message.
benchmark.h requires a definition of KS_LOG_PPTR, but its previous
placement in machine.h resulted in a circular include. This commit
factors out KS_LOG_PPTR and related definitions to a separate header,
creates the corresponding header for x86, and prevents circular
includes in the x86 builds
Adds support for per-core kernel stacks through the use of
thread local storage and swapgs. In addition to the main
kernel stack the IRQ stack also needs to be made per core
Defines the TLB bitmap to be located, virtually, just below the kernel
window. To get an initialized bitmap in each new address space we
change copyGlobalMappings to copy from USER_TOP (which includes the
initialized TLB bitmap in the global address space) instead of just
the kernel window base
This commit adds support for using the ThreadID registers of the ARM MPCore
platforms for storing the address of the IPC buffer instead of the globals
frame. The choice of using the user readable/writeable ThreadID register
is chosen, even though it means the user cannot use it for its own
purposes, as it leaves room in the future for doing TLS support in the
user read only register, where compilers expect it.
This commit adds support for saving/restoring an additional register
that exists on ARM MPCore platforms. As this register is user wrieable,
as well as readable, it must saved and restored by the kernel to
prevent gross information channels.
Although this function is called from the fastpath inside of an
`unlikely` macro and the function itself gets inlined, the knowledge
that this conjunction is unlikely is somehow lost. Explicitly putting
a `likely` here fixes it
Removes the recycle operation and adds an operation
to cancel any badged sends on and endpoint. Calling
Revoke + CancelBadgedSend is equivalent to Recycle
on a badged endpoint
This removes usages of CONST Arch_activateIdleThread. According to
gcc 'a function that has pointer arguments and examines the data
pointed to must not be declared const'. Whilst these functions do
not presently examine their tcb_t pointer argument, it seems
pointless to pass something that can never be safely looked at.
Further a function that returns void and is CONST is utterly meaningless
This commit does the following:
* invert the bit field scheduler for better cache performance for high priority threads
* peeks into the bitfield scheduler to allow fastpath to be leveraged when IPC occurs from lo --> hi priority threads if correct
Previously generic functions for manipulating page table and page
directory entries were placed in mode/structures.h. These are moved
to a mode/vspace.h instead so that they can use existing functions
defined in arch/vspace.h.
The x86_make_pde_mapping function is dropped in this move as it is
not used. Instead it is replaced with a function for creating an
empty mapping for whatever is the vspace root.
1. Missing loop counter when accessing to PD in 'tlb_bitmap_get'.
2. Missing 'TLBBITMAP_PD_INDEX' in 'TLBBITMAP_PD_MAKE_INDEX'. The index generated by 'TLBBITMAP_PD_MAKE_INDEX' should be valid in PD while the original one used to generate just 0, 1, etc. The correct index is '0 + TLBBITMAP_PD_INDEX', '1 + TLBBITMAP_PD_INDEX' and so on.
When checking if a thread is using the FPU or not we now check on that
threads core, under the assumption that if a thread migrates its FPU
state is explicitly (and not lazily) migrated.
