XNU Build Consolidation

Introduction and motivation

XNU is supported on approximately 20 different targets. Whilst in some cases the differences between two given targets are small (e.g. when they both support the same ISA), XNU has traditionally required to have separate builds in cases where the topology of the targets differ (for example, when they feature different core/cluster counts or cache sizes). Similarly, SoC-specific fix-ups are usually conditionally compiled based on the target.

Given the time it takes to compile all three different variants (release, debug and development) for each supported SoC, usually several times a day for various teams across Apple, the goal of this project was to reduce the number of existing builds, as well as to set up a simple framework that makes it easier to share builds across different SoCs moving forward.

Although this effort could be extended to KEXTs, and hence lead to shared KernelCaches across devices, the scope of this document only includes XNU. In cases where KEXTs also differ across targets, or perhaps the required KEXTs are completely different in the first place, the kernel still needs to be linked appropriately with different sets of KEXTs and hence KernelCaches cannot be shared.

Changes required in XNU

The kernel itself is relatively SoC-agnostic, although strongly architecture-dependent; this is because most of the SoC-specific aspects of the KernelCache are abstracted by the KEXTs. Things that pertain to the kernel include:

• Number of cores/clusters in the system, their physical IDs and type.
• Addresses of PIO registers that are to be accessed from the XNU side.
• L1/L2 cache geometry parameters (e.g. size, number of set/ways).
• Just like other components, the kernel has its share of responsibility when it comes to setting up HID registers and applying fix-ups at various points during boot or elsewhere at runtime.
• Certain kernel-visible architectural features are optional, which means that two same-generation SoCs may still differ in their feature set.

All of these problems can be solved through a mix of relying more heavily on device tree information and performing runtime checks. The latter is possible because both the ARM architecture and the Apple's extensions provide r/o registers that can be checked at runtime to discover supported features as well as various CPU-specific parameters.

Obtaining cache geometry parameters at runtime

Although not often, the kernel may still require deriving, one way or another, parameters like cache sizes and number of set/ways. XNU needs most of this information to perform set/way clean/invalidate operations. Prior to this work, these values were hardcoded for each supported target in proc_reg.h, and used in caches_asm.s. The ARM architecture provides the CCSIDR_EL1 register, which can be used in conjunction with CSSELR_EL1 to select the target cache and obtain geometry information.

Performing CPU/Revision-specific checks at runtime

CPU and revision checks may be required at various places, although the focus here has been the application of tunables at boot time.

Tunables are often applied:

• On a specific core type of a specific SoC.
• On a subset of all of the CPU revisions.
• On all P-cores or all E-cores.

This has led in the past to a number of nested, conditionally-compiled blocks of code that are not easy to understand or manage as new tunables are added or SoCs/revisions are deprecated.

The changes applied as part of this work focus mainly on:

1. Decoupling the tunable-application code from start.s.
2. Splitting the tunable-application code across different files, one per supported architecture (e.g. tunables_h7.h, or tunables_h11.h).
3. Providing "templates" for the most commonly-used combinations of tunables.
4. Providing a family of assembly macros that can be used to conditionally execute code on a specific core type, CPU ID, revision(s), or a combination of these.

All of the macros live in the 64-bit version of proc_reg.h, and are SoC-agnostic; they simply check the MIDR_EL1 register against a CPU revision that is passed as a parameter to the macro, where applicable. Similarly, where a block of code is to be executed on a core type, rather than a specific core ID, a couple of the provided macros can check this against MPIDR_EL1.

Checking for feature compatibility at runtime

Some architectural features are optional, which means that, when disabled at compile-time, this may cause two same-generation SoCs to diverge.

Rather than disabling features, and assuming this does not pose security risks or performance regressions, the preferred approach is to compile them in, but perform runtime checks to enable/disable them, possibly in early boot. The way these checks are performed varies from feature to feature (for example, VHE is an ARM feature, and the ARM ARM specifies how it can be discovered). For Apple-specific features, these are all advertised through the AIDR_EL1 register. One of the changes is the addition of a function, ml_feature_supported(), that may be used to check for the presence of a feature at runtime.

Deriving core/cluster counts from device tree

One of the aspects that until now has been hardcoded in XNU is the system topology: number of cores/clusters and their physical IDs. This effort piggybacks on other recent XNU changes which aimed to consolidate topology-related information into XNU, by parsing it from the device tree and exporting it to KEXTs through well-defined APIs.

Changes applied as part of the XNU consolidation project include:

• Extending the ml_* API to extract cluster information from the topology parser. New APIs include the following:
  • ml_get_max_cluster_number()
  • ml_get_cluster_count()
  • ml_get_first_cpu_id()
• Removing hardcoded core counts (CPU_COUNT) and cluster counts (ARM_CLUSTER_COUNT) from XNU, and replacing them with ml_* calls.
• Similarly, deriving CPU physical IDs from the topology parser.

Allocating memory that is core size/cluster size/cache size aligned

In some cases, certain statically-allocated arrays/structures need to be cache line-aligned, or have one element per core or cluster. Whilst this information is not known precisely at compile time anymore, the following macros have been added to provide a reasonably close upper bound:

• MAX_CPUS
• MAX_CPU_CLUSTERS
• MAX_L2_CLINE

These macros are defined in board_config.h, and should be set to the same value for a group of targets sharing a single build. Note that these no longer reflect actual counts and sizes, and the real values need to be queried at runtime through the ml_ API.

The L1 cache line size is still hardcoded, and defined as MMU_CLINE. Since this value is always the same and very often checked at various places across XNU and elsewhere, it made sense to keep it as a compile time macro rather than relying on runtime checks.

Restrictions on conditional compilation

Currently, a family of per-SoC macros are defined at build time to enable XNU to conditionally compile code for different targets. These are named ARM[64]_BOARD_CONFIG_[TARGET_NAME], and have historically been used in different places across the kernel; for example, when applying tunables, various fixes, or enabling disabling features. In order not to create divergences in the future across same-generation SoCs, but also to keep the codebase consistent, the recommendation is to avoid the use of these macros whenever possible.

Instead, XNU itself defines yet another family of macros that are defined for all targets of a particular generation. These are named after the P-CORE introduced by each (for example, APPLEMONSOON, or APPLEVORTEX), and are preferred over the SoC-specific ones. Where a generation macro is not enough to provide correctness (which happens, for example, when the code block at hand should not be executed on a given SoC of the same family), appropriate runtime checks can be performed inside the conditionally-compiled code block. machine_read_midr() and get_arm_cpu_version() may be used for this purpose.