third_party/dawn/third_party/vulkan-loader/src/docs/LoaderInterfaceArchitecture.md
Vulkan is a layered architecture, made up of the following elements:
The general concepts in this document are applicable to the loaders available for Windows, Linux, Android, and macOS systems.
While this document is primarily targeted at developers of Vulkan applications, drivers and layers, the information contained in it could be useful to anyone wanting a better understanding of the Vulkan runtime.
The application sits at the top and interfaces directly with the Vulkan loader. At the bottom of the stack sits the drivers. A driver can control one or more physical devices capable of rendering Vulkan, implement a conversion from Vulkan into a native graphics API (like MoltenVk, or implement a fully software path that can be executed on a CPU to simulate a Vulkan device (like SwiftShader or LavaPipe). Remember, Vulkan-capable hardware may be graphics-based, compute-based, or both. Between the application and the drivers, the loader can inject any number of optional layers that provide special functionality. The loader is critical to managing the proper dispatching of Vulkan functions to the appropriate set of layers and drivers. The Vulkan object model allows the loader to insert layers into a call-chain so that the layers can process Vulkan functions prior to the driver being called.
This document is intended to provide an overview of the necessary interfaces between each of these.
The loader was designed with the following goals in mind:
Layers are optional components that augment the Vulkan development environment. They can intercept, evaluate, and modify existing Vulkan functions on their way from the application down to the drivers and back up. Layers are implemented as libraries that can be enabled in different ways and are loaded during CreateInstance. Each layer can choose to hook, or intercept, Vulkan functions which in turn can be ignored, inspected, or augmented. Any function a layer does not hook is simply skipped for that layer and the control flow will simply continue on to the next supporting layer or driver. Because of this, a layer can choose whether to intercept all known Vulkan functions or only a subset it is interested in.
Some examples of features that layers may expose include:
Because layers are optional and dynamically loaded, they can be enabled and disabled as desired. For example, while developing and debugging an application, enabling certain layers can assist in making sure it properly uses the Vulkan API. But when releasing the application, those layers are unnecessary and thus won't be enabled, increasing the speed of the application.
The library that implements Vulkan, either through supporting a physical hardware device directly, converting Vulkan commands into native graphics commands, or simulating Vulkan through software, is considered "a driver". The most common type of driver is still the Installable Client Driver (or ICD). The loader is responsible for discovering available Vulkan drivers on the system. Given a list of available drivers, the loader can enumerate all the available physical devices and provide this information for an application.
Vulkan allows multiple ICDs each supporting one or more devices.
Each of these devices is represented by a Vulkan VkPhysicalDevice object.
The loader is responsible for discovering available Vulkan ICDs via the standard
driver search on the system.
VkConfig is a tool LunarG has developed to assist with modifying the Vulkan environment on the local system. It can be used to find layers, enable them, change layer settings, and other useful features. VkConfig can be found by either installing the Vulkan SDK or by building the source out of the LunarG VulkanTools GitHub Repo.
VkConfig generates three outputs, two of which work with the Vulkan loader and layers. These outputs are:
These files are found in different locations based on your platform:
<table style="width:100%"> <tr> <th>Platform</th> <th>Output</th> <th>Location</th> </tr> <tr> <th rowspan="3">Linux</th> <td>Vulkan Override Layer</td> <td>$USER/.local/share/vulkan/implicit_layer.d/VkLayer_override.json</td> </tr> <tr> <td>Vulkan Layer Settings</td> <td>$USER/.local/share/vulkan/settings.d/vk_layer_settings.txt</td> </tr> <tr> <td>VkConfig Configuration Settings</td> <td>$USER/.local/share/vulkan/settings.d/vk_layer_settings.txt</td> </tr> <tr> <th rowspan="3">Windows</th> <td>Vulkan Override Layer</td> <td>%HOME%\AppData\Local\LunarG\vkconfig\override\VkLayerOverride.json</td> </tr> <tr> <td>Vulkan Layer Settings</td> <td>(registry) HKEY_CURRENT_USER\Software\Khronos\Vulkan\LoaderSettings</td> </tr> <tr> <td>VkConfig Configuration Settings</td> <td>(registry) HKEY_CURRENT_USER\Software\LunarG\vkconfig </td> </tr> </table>The Override Meta-Layer is an important part of how VkConfig works. This layer, when found by the loader, forces the loading of the desired layers that were enabled inside of VkConfig as well as disables those layers that were intentionally disabled (including implicit layers).
The Vulkan Layer Settings file can be used to specify certain behaviors and actions each enabled layer is expected to perform. These settings can also be controlled by VkConfig, or they can be manually enabled. For details on what settings can be used, refer to the individual layers.
In the future, VkConfig may have additional interactions with the Vulkan loader.
More details on VkConfig can be found in its GitHub documentation.
Vulkan has a few concepts that provide a fundamental basis for its organization. These concepts should be understood by any one attempting to use Vulkan or develop any of its components.
An important concept to understand, which is brought up repeatedly throughout this document, is how the Vulkan API is organized. Many objects, functions, extensions, and other behavior in Vulkan can be separated into two groups:
A "Vulkan instance" (VkInstance) is a high-level construct used to provide
Vulkan system-level information and functionality.
A few Vulkan objects associated directly with an instance are:
VkInstanceVkPhysicalDeviceVkPhysicalDeviceGroupAn "instance function" is any Vulkan function where the first parameter is an instance object or no object at all.
Some Vulkan instance functions are:
vkEnumerateInstanceExtensionPropertiesvkEnumeratePhysicalDevicesvkCreateInstancevkDestroyInstanceAn application can link directly to all core instance functions through the
Vulkan loader's headers.
Alternatively, an application can query function pointers using
vkGetInstanceProcAddr.
vkGetInstanceProcAddr can be used to query any instance or device entry-points
in addition to all core entry-points.
If vkGetInstanceProcAddr is called using a VkInstance, then any function
pointer returned is specific to that VkInstance and any additional objects
that are created from it.
Extensions to Vulkan are similarly associated based on what type of functions they provide. Because of this, extensions are broken up into instance or device extensions where most, if not all of the functions, in the extension are of the corresponding type. For example, an "instance extension" is composed primarily of "instance functions" which primarily take instance objects. These will be discussed in more detail later.
A Vulkan device (VkDevice), on the other-hand, is a logical identifier used
to associate functions with a particular Vulkan physical device
(VkPhysicalDevice) through a particular driver on a user's system.
A few of the Vulkan constructs associated directly with a device include:
VkDeviceVkQueueVkCommandBufferA "device function" is any Vulkan function which takes any device object as its first parameter or a child object of the device. The vast majority of Vulkan functions are device functions. Some Vulkan device functions are:
vkQueueSubmitvkBeginCommandBuffervkCreateEventVulkan devices functions may be queried using either vkGetInstanceProcAddr or
vkGetDeviceProcAddr.
If an application chooses to use vkGetInstanceProcAddr, each call will have
additional function calls built into the call chain, which will reduce
performance slightly.
If, instead, the application uses vkGetDeviceProcAddr, the call chain will be
more optimized to the specific device, but the returned function pointers will
only work for the device used when querying them.
Unlike vkGetInstanceProcAddr, vkGetDeviceProcAddr can only be used on
Vulkan device functions.
The best solution is to query instance extension functions using
vkGetInstanceProcAddr, and to query device extension functions using
vkGetDeviceProcAddr.
See
Best Application Performance Setup
section in the
LoaderApplicationInterface.md document for more
information on this.
As with instance extensions, a device extension is a set of Vulkan device functions extending the Vulkan language. More information about device extensions can be found later in this document.
Vulkan uses an object model to control the scope of a particular action or operation. The object to be acted on is always the first parameter of a Vulkan call and is a dispatchable object (see Vulkan specification section 3.3 Object Model). Under the covers, the dispatchable object handle is a pointer to a structure, which in turn, contains a pointer to a dispatch table maintained by the loader. This dispatch table contains pointers to the Vulkan functions appropriate to that object.
There are two types of dispatch tables the loader maintains:
vkCreateInstancevkCreateDeviceAt that time the application and the system can each specify optional layers to
be included.
The loader will initialize the specified layers to create a call chain for each
Vulkan function and each entry of the dispatch table will point to the first
element of that chain.
Thus, the loader builds an instance call chain for each VkInstance that is
created and a device call chain for each VkDevice that is created.
When an application calls a Vulkan function, this typically will first hit a trampoline function in the loader. These trampoline functions are small, simple functions that jump to the appropriate dispatch table entry for the object they are given. Additionally, for functions in the instance call chain, the loader has an additional function, called a terminator, which is called after all enabled layers to marshall the appropriate information to all available drivers.
For example, the diagram below represents what happens in the call chain for
vkCreateInstance.
After initializing the chain, the loader calls into the first layer's
vkCreateInstance, which will call the next layer's vkCreateInstance
before finally terminating in the loader again where it will call
every driver's vkCreateInstance.
This allows every enabled layer in the chain to set up what it needs based on
the VkInstanceCreateInfo structure from the application.
This also highlights some of the complexity the loader must manage when using
instance call chains.
As shown here, the loader's terminator must aggregate information to and from
multiple drivers when they are present.
This implies that the loader has to be aware of any instance-level extensions
which work on a VkInstance to aggregate them correctly.
Device call chains are created in vkCreateDevice and are generally simpler
because they deal with only a single device.
This allows for the specific driver exposing this device to always be the
terminator of the chain.
To ensure that the system is safe from exploitation, Vulkan applications which are run with elevated privileges are restricted from certain operations, such as reading environment variables from unsecure locations or searching for files in user controlled paths. This is done to ensure that an application running with elevated privileges does not run using components that were not installed in the proper approved locations.
The loader uses platform-specific mechanisms (such as secure_getenv and its
equivalents) for querying sensitive environment variables to avoid accidentally
using untrusted results.
These behaviors also result in ignoring certain environment variables, such as:
VK_DRIVER_FILES / VK_ICD_FILENAMESVK_ADD_DRIVER_FILESVK_LAYER_PATHVK_ADD_LAYER_PATHVK_IMPLICIT_LAYER_PATHVK_ADD_IMPLICIT_LAYER_PATHXDG_CONFIG_HOME (Linux/Mac-specific)XDG_DATA_HOME (Linux/Mac-specific)For more information on the affected search paths, refer to Layer Discovery and Driver Discovery.
The Application interface to the Vulkan loader is now detailed in the LoaderApplicationInterface.md document found in the same directory as this file.
The Layer interface to the Vulkan loader is detailed in the LoaderLayerInterface.md document found in the same directory as this file.
The Driver interface to the Vulkan loader is detailed in the LoaderDriverInterface.md document found in the same directory as this file.
If your application is crashing or behaving weirdly, the loader provides several mechanisms for you to debug the issues. These are detailed in the LoaderDebugging.md document found in the same directory as this file.
Loader policies with regards to the loader interaction with drivers and layers are now documented in the appropriate sections. The intention of these sections is to clearly define expected behavior of the loader with regards to its interactions with those components. This could be especially useful in cases where a new or specialized loader may be required that conforms to the behavior of the existing loader. Because of this, the primary focus of those sections is on expected behaviors for all relevant components to create a consistent experience across platforms. In the long-run, this could also be used as validation requirements for any existing Vulkan loaders.
To review the particular policy sections, please refer to one or both of the sections listed below:
The filter environment variables provided in certain areas have some common restrictions and behaviors that should be listed.
The filter variables will be compared against the appropriate strings for either drivers or layers. The appropriate string for layers is the layer name provided in the layer's manifest file. Since drivers don’t have a name like layers, this substring is used to compare against the driver manifest's filename.
All of the filter environment variables accept comma-delimited input. Therefore, you can chain multiple strings together and it will use the strings to individually enable or disable the appropriate item in the current list of available items.
To provide enough flexibility to limit name searches to only those wanted by the developer, the loader uses a limited glob format for strings. Acceptable globs are:
"string*""*string""*string*""string"
VK_LAYER_KHRONOS_validation will match the layer name
VK_LAYER_KHRONOS_validation, but not a layer named
VK_LAYER_KHRONOS_validation2 (not that there is such a layer).This is especially useful because it is difficult sometimes to determine the
full name of a driver manifest file or even some commonly used layers
such as VK_LAYER_KHRONOS_validation.
All of the filter environment variables assume the strings inside of the glob are not case-sensitive. Therefore, “Bob”, “bob”, and “BOB” all amount to the same thing.
The values from the disable environment variable will be considered before the enable or select environment variable. Because of this, it is possible to disable a layer/driver using the disable environment variable, only to have it be re-enabled by the enable/select environment variable. This is useful if you disable all layers/drivers with the intent of only enabling a smaller subset of specific layers/drivers for issue triaging.
When multiple VK_LOADER_<DEVICE|VENDOR|DRIVER>_ID_FILTERs are provided, they
apply cummulatively. Only devices matching all of the filters will be presented
to the application.
The following are all the Debug Environment Variables available for use with the Loader. These are referenced throughout the text, but collected here for ease of discovery.
</small></td>
<td><small>
If a global path to the JSON file is not used, issues may be encountered.
<a href="#elevated-privilege-caveats">
Ignored when running Vulkan application with elevated privileges.
</a>
</small></td>
<td><small>
export
VK_ADD_DRIVER_FILES=
<folder_a>/intel.json:<folder_b>/amd.json
set
VK_ADD_DRIVER_FILES=
<folder_a>\nvidia.json;<folder_b>\mesa.json
</small></td>
VK_ADD_LAYER_PATH=
<path_a>:<path_b>
set
VK_ADD_LAYER_PATH=
<path_a>;<path_b></small>
</td>
VK_ADD_IMPLICIT_LAYER_PATH=
<path_a>:<path_b>
set
VK_ADD_IMPLICIT_LAYER_PATH=
<path_a>;<path_b></small>
</td>
This has replaced the older deprecated environment variable
<i>VK_ICD_FILENAMES</i>, however the older environment variable will
continue to work.
</small></td>
<td><small>
This functionality is only available with Loaders built with version
1.3.207 of the Vulkan headers and later.
It is recommended to use absolute paths to JSON files.
Relative paths may have issues due to how the loader transforms relative library
paths into absolute ones.
<a href="#elevated-privilege-caveats">
Ignored when running Vulkan application with elevated privileges.
</a>
</small></td>
<td><small>
export
VK_DRIVER_FILES=
<folder_a>/intel.json:<folder_b>/amd.json
set
VK_DRIVER_FILES=
<folder_a>\nvidia.json;<folder_b>\mesa.json
</small>
</td>
VK_LAYER_PATH=
<path_a>:<path_b>
set
VK_LAYER_PATH=
<path_a>;<path_b>
</small></td>
VK_IMPLICIT_LAYER_PATH=
<path_a>:<path_b>
set
VK_IMPLICIT_LAYER_PATH=
<path_a>;<path_b>
</small></td>
* error (only errors)
* warn (only warnings)
* info (only info)
* debug (only debug)
* layer (layer-specific output)
* driver (driver-specific output)
* all (report out all messages)
To enable multiple options (outside of "all") like info, warning and
error messages, set the value to "error,warn,info".
</small></td>
<td><small>
None
</small></td>
<td><small>
export
VK_LOADER_DEBUG=all
set
VK_LOADER_DEBUG=warn
</small></td>
The value should be "<hex vendor id>:<hex device id>".
<b>NOTE:</b> This DOES NOT REMOVE devices from the list, only reorders them.
</small></td>
<td><small>
<b>Linux Only</b>
</small></td>
<td><small>
set VK_LOADER_DEVICE_SELECT=0x10de:0x1f91
</small></td>
</small></td>
<td><small>
<b>Linux Only</b>
</small></td>
<td><small>
set VK_LOADER_DISABLE_SELECT=1
</small></td>
</small></td>
<td><small>
<b>Use Wisely!</b> This may cause the loader or application to crash.
</small></td>
<td><small>
export
VK_LOADER_DISABLE_INST_EXT_FILTER=1
set
VK_LOADER_DISABLE_INST_EXT_FILTER=1
</small></td>
Since drivers don’t have a name like layers, this glob is used to
compare against the manifest filename.
Known driver manifests being those files that are already found by the
loader taking into account default search paths and other environment
variables (like <i>VK_ICD_FILENAMES</i> or <i>VK_ADD_DRIVER_FILES</i>).
</small></td>
<td><small>
This functionality is only available with Loaders built with version
1.3.234 of the Vulkan headers and later.
If no drivers are found with a manifest filename that matches any of the
provided globs, then no driver is enabled and it <b>may</b> result
in Vulkan applications failing to run properly.
</small></td>
<td><small>
export
VK_LOADER_DRIVERS_SELECT=nvidia*
set
VK_LOADER_DRIVERS_SELECT=nvidia*
The above would select only the Nvidia driver if it was present on the
system and already visible to the loader.
</small></td>
Since drivers don’t have a name like layers, this glob is used to
compare against the manifest filename.
Known driver manifests being those files that are already found by the
loader taking into account default search paths and other environment
variables (like <i>VK_ICD_FILENAMES</i> or <i>VK_ADD_DRIVER_FILES</i>).
</small></td>
<td><small>
This functionality is only available with Loaders built with version
1.3.234 of the Vulkan headers and later.
If all available drivers are disabled using this environment variable,
then no drivers will be found by the loader and <b>will</b> result
in Vulkan applications failing to run properly.
This is also checked before other driver environment variables (such as
<i>VK_LOADER_DRIVERS_SELECT</i>) so that a user may easily disable all
drivers and then selectively re-enable individual drivers using the
enable environment variable.
</small></td>
<td><small>
export
VK_LOADER_DRIVERS_DISABLE=*amd*,*intel*
set
VK_LOADER_DRIVERS_DISABLE=*amd*,*intel*
The above would disable both Intel and AMD drivers if both were present
on the system and already visible to the loader.
</small></td>
Known layers are those which are found by the loader taking into account
default search paths and other environment variables
(like <i>VK_LAYER_PATH</i>).
</i>
</small></td>
<td><small>
This functionality is only available with Loaders built with version
1.3.234 of the Vulkan headers and later.
</small></td>
<td><small>
export
VK_LOADER_LAYERS_ENABLE=*validation,*recon*
set
VK_LOADER_LAYERS_ENABLE=*validation,*recon*
The above would enable the Khronos validation layer and the
GfxReconstruct layer, if both were present on the system and already
visible to the loader.
</small></td>
Known layers are those which are found by the loader taking into account
default search paths and other environment variables
(like <i>VK_LAYER_PATH</i>).
</small></td>
<td><small>
This functionality is only available with Loaders built with version
1.3.234 of the Vulkan headers and later.
Disabling a layer that an application intentionally enables as an
explicit layer <b>may</b> cause the application to not function
properly.
This is also checked before other layer environment variables (such as
<i>VK_LOADER_LAYERS_ENABLE</i>) so that a user may easily disable all
layers and then selectively re-enable individual layers using the
enable environment variable.
</small></td>
<td><small>
export
VK_LOADER_LAYERS_DISABLE=*MESA*,~implicit~
set
VK_LOADER_LAYERS_DISABLE=*MESA*,~implicit~
The above would disable any Mesa layer and all other implicit layers
that would normally be enabled on the system.
</small></td>
Known layers are those which are found by the loader taking into account
default search paths and other environment variables
(like <i>VK_LAYER_PATH</i>).
</small></td>
<td><small>
This functionality is only available with Loaders built with version
1.3.262 of the Vulkan headers and later.
This will not cause layers to be enabled if the normal mechanism to
enable them
</small></td>
<td><small>
export
VK_LOADER_LAYERS_ALLOW=*validation*,*recon*
set
VK_LOADER_LAYERS_ALLOW=*validation*,*recon*
The above would allow any layer whose name is validation or recon to be
enabled regardless of the value of <i>VK_LOADER_LAYERS_DISABLE</i>.
</small></td>
</small></td>
<td><small>
export
VK_LOADER_DISABLE_DYNAMIC_LIBRARY_UNLOADING=1
set
VK_LOADER_DISABLE_DYNAMIC_LIBRARY_UNLOADING=1
</small></td>
VK_LOADER_DEVICE_ID_FILTER=0x7460:0x747e,29827
set
VK_LOADER_DEVICE_ID_FILTER=0x7460:0x747e,29827
</small></td>
VK_LOADER_VENDOR_ID_FILTER=65541
set
VK_LOADER_VENDOR_ID_FILTER=65541
</small></td>
VK_LOADER_DRIVER_ID_FILTER=1-3:13
set
VK_LOADER_DRIVER_ID_FILTER=1-3:13
</small></td>
These environment variables are still active and supported, however support may be removed in a future loader release.
<table style="width:100%"> <tr> <th>Environment Variable</th> <th>Behavior</th> <th>Replaced By</th> <th>Restrictions</th> <th>Example Format</th> </tr> <tr> <td><small><i>VK_ICD_FILENAMES</i></small></td> <td><small> Force the loader to use the specific driver JSON files. The value contains a list of delimited full path listings to driver JSON Manifest files. <b>NOTE:</b> If a global path to the JSON file is not used, issues
may be encountered.
</small></td>
<td><small>
This has been replaced by <i>VK_DRIVER_FILES</i>.
</small></td>
<td><small>
<a href="#elevated-privilege-caveats">
Ignored when running Vulkan application with elevated privileges.
</a>
</small></td>
<td><small>
export
VK_ICD_FILENAMES=
<folder_a>/intel.json:<folder_b>/amd.json
set
VK_ICD_FILENAMES=
<folder_a>\nvidia.json;<folder_b>\mesa.json
</small></td>
VK_INSTANCE_LAYERS=
<layer_a>;<layer_b>
set
VK_INSTANCE_LAYERS=
<layer_a>;<layer_b>
</small></td>
For example, <b>vkCreateDevice()</b>.
</td>
See the
<a href="#dispatch-tables-and-call-chains">Dispatch Tables and Call
Chains</a> section for more information.
</td>
Some Vulkan device functions are:
<b>vkQueueSubmit</b>,
<b>vkBeginCommandBuffer</b>,
<b>vkCreateEvent</b>.
See the <a href="#instance-versus-device">Instance Versus Device</a>
section for more information.
</td>
On <i>Windows/Linux/macOS</i>, the discovery process typically focuses
on searching for Manifest files.
On <i>Android</i>, the process focuses on searching for library files.
</td>
See <a href="#dispatch-tables-and-call-chains">Dispatch Tables and Call
Chains</a> for more information.
</td>
See <a href="#drivers">Drivers</a> section for more information.
</td>
Always first query if an extension exists, and enable it during
<b>vkCreateInstance</b> (if it is an instance extension) or during
<b>vkCreateDevice</b> (if it is a device extension) before attempting
to use it.
Extensions will always have an author prefix or suffix modifier to every
structure, enumeration entry, command entry-point, or define that is
associated with it.
For example, `KHR` is the prefix for Khronos authored extensions and
will also be found on structures, enumeration entries, and commands
associated with those extensions.
</td>
As with the extension the function is defined as part of, it will have a
suffix modifier indicating the author of the extension.
Some example extension suffixes include:
<b>KHR</b> - For Khronos authored extensions,
<b>EXT</b> - For multi-company authored extensions,
<b>AMD</b> - For AMD authored extensions,
<b>ARM</b> - For ARM authored extensions,
<b>NV</b> - For Nvidia authored extensions.
</td>
These are the most common type of Vulkan drivers.
See <a href="#installable-client-drivers">Installable Client Drivers</a>
section for more information.
</td>
A typical examples for a Graphics IHV include (but not limited to):
AMD, ARM, Imagination, Intel, Nvidia, Qualcomm
</td>
See the <a href="#dispatch-tables-and-call-chains">Dispatch Tables and
Call Chains</a> section for more information.
</td>
Some Vulkan instance functions are:
<b>vkEnumerateInstanceExtensionProperties</b>,
<b>vkEnumeratePhysicalDevices</b>,
<b>vkCreateInstance</b>,
<b>vkDestroyInstance</b>.
See the <a href="#instance-versus-device">Instance Versus Device</a>
section for more information.
</td>
See the <a href="#layers">Layers</a> section for more information.
</td>
See <a href="#the-loader">The Loader</a> section for more information.
</td>
See <a href="#dispatch-tables-and-call-chains">Dispatch Tables and Call
Chains</a> for more information.
</td>
See <a href="#dispatch-tables-and-call-chains">Dispatch Tables and Call
Chains</a> for more information.
</td>
See
<a href="LoaderApplicationInterface.md#wsi-extensions">WSI Extensions</a>
for more information.
</td>
</td>
</td>
These functions are:
`vkGetInstanceProcAddr`, `vkGetDeviceProcAddr`,
`vk_icdGetInstanceProcAddr`, `vk_icdGetPhysicalDeviceProcAddr`, and
`vk_layerGetPhysicalDeviceProcAddr`.
</td>