src/plugins/intel_npu/docs/npu-properties-howto.md
What is a property from NPU Plugin's POV and Step-by-step guide on how to add one
Practical manual for NPU plugin properties
| Name | Description | Example |
|---|---|---|
| Property | a plugin interface which can be set or read. Can map to options or metrics` | ov::log::level |
| Option | a configuration entry in our internal configuration.</br>Consists of an OptionBase template descriptor + a template OptionValue value. | LOG_LEVEL |
| Config | Our internal database of configuration keys and their values | _globalConfig |
| Metric | A property which does not map to any configuration key and configuration entry in our internal config.</br>Usually a static value directly read from driver or OS. | ov::device::pci_info |
| Compiler | Npu compiler as viewed from the plugin’s perspective.</br>Can be Compiler-In-Driver or Compiler-In-Plugin | CID |
| “Anonymous” property</br>OR</br>compiler-private property | A setting from application level which the plugin has no knowledge of</br>(it is not registered, plugin is not aware of its datatype)</br> but which the compiler reports as supported via is_supported() API. | N/A |
As it can be observed in the above class hierarchy diagram, a Property is a public interface to an internal information, the top layer of abstraction.
A property can connect internally to an Option or to a Metric.
The main difference between Option and Metric is that while Options are entries in our internal database (Config) which can be modified at any time, Metrics are static read-only pieces of information
which do not exist in the internal database. Metrics can not be set or changed externally, their values are not stored in the plugin and they most often just a direct system or driver call.
Example of Metrics in NPU Plugin are: driver version, compiler version, device architecture, pci information, gops, uuid, luid, etc.
To summarize:
A property can map internally to one of the 2 types:
Implements the option descriptor. This class contains all the details of a config option: name, datatype, default value, parser, public/private, mutability, compiler version (for legacy support), etc. This serves as the key in our configuration map.
Class definition in npu_plugin/al/include/config/config.hpp:
struct OptionBase {
using ValueType = T;
// `ActualOpt` must implement the following method:
// static std::string_view key()
static constexpr std::string_view getTypeName() {
if constexpr (TypePrinter<T>::hasName()) {
return TypePrinter<T>::name();
}
static_assert(TypePrinter<T>::hasName(),
"Options type is not a standard type, please add `getTypeName()` to your option");
}
// Overload this to provide environment variable support.
static std::string_view envVar() {
return "";
}
// Overload this to provide deprecated keys names.
static std::vector<std::string_view> deprecatedKeys() {
return {};
}
// Overload this to provide default value if it wasn't specified by user.
// If it is std::nullopt - exception will be thrown in case of missing option access.
static std::optional<T> defaultValue() {
return std::nullopt;
}
// Overload this to provide more specific parser.
static ValueType parse(std::string_view val) {
return OptionParser<ValueType>::parse(val);
}
// Overload this to provide more specific validation
static void validateValue(const ValueType&) {}
// Overload this to provide more specific implementation.
static OptionMode mode() {
return OptionMode::Both;
}
// Overload this for private options.
static bool isPublic() {
return false;
}
// Overload this for read-only options (metrics)
static ov::PropertyMutability mutability() {
return ov::PropertyMutability::RW;
}
/// Overload this for options conditioned by compiler version
static uint32_t compilerSupportVersion() {
return ONEAPI_MAKE_VERSION(0, 0);
}
static std::string toString(const ValueType& val) {
return OptionPrinter<ValueType>::toString(val);
}
};
is storage for the registered options This is the base map which stores the available optionBase descriptors.
This layer implements the option database manipulation functions: add/remove/has/reset
Note: This layer is static, intialized once in plugin constructor with all the options the Plugin has knowledge of.
is the highlevel configuration “database” which implements the mapping between OptionBase and templatized OptionValue.
Maps and stores the user-defined values for each entry in OptionsDesc layer.
Implements the top level configuration manipulation functions:
get/update/has/getString/toString/fromString and handles typecasts, typeverification, parsing and conversions.
Note: This layer is static, initialized once in plugin constructor with the provided (also static) OptionDesc.
is a derivative class of Config, used only by NPU Plugin, which implements additional filtering layers atop of the base config,
such as enabling/disabling keys based on their availability/support on the current system configuration.
Note: This layer dynamically changes based on system configuration and compiler_type
is the top level class and serves as the NPU Plugin’s interface to OpenVino and the application layer.
It’s main purpose is to implement get_property and set_property interfaces and the callback functions of each property.
Note: This layer dynamically changes based on system configuration and compiler_type
The following steps how to add a new simple property which maps to a compiler configuration option.
*simple in this context means that it has no special callback function required for it, just set/get
First step is to define the new property's name, datatype and string-name in the public header in
openvino/src/inference/include/openvino/runtime/intel_npu/properties.hpp
Example:
static constexpr ov::Property<ExampleType,ov::PropertyMutability::RW> example_property{"NPU_EXAMPLE_PROPERTY"};
Notes:
Second step is to define the optionDesc class for this property in
openvino/src/plugins/intel_npu/al/config/options.hpp
Example:
//
// EXAMPLE_PROPERTY
//
struct EXAMPLE_PROPERTY final : OptionBase<EXAMPLE_PROPERTY, ov::intel_npu::ExampleType> {
static std::string_view key() {
return ov::intel_npu::example_property.name();
}
static constexpr std::string_view getTypeName() {
return "ov::intel_npu::ExampleType";
}
static ov::intel_npu::ExampleType defaultValue() {
return ov::intel_npu::ExampleType::VAL3;
}
static uint32_t compilerSupportVersion() {
return ONEAPI_MAKE_VERSION(5, 5);
}
static bool isPublic() {
return true;
}
static OptionMode mode() {
return OptionMode::Both;
}
static ov::PropertyMutability mutability() {
return ov::PropertyMutability::RW;
}
static std::string_view envVar() {
return "IE_NPU_EXAMPLE_PROPERTY";
}
static ov::intel_npu::ExampleType parse(std::string_view val) {
if (val == "VAL1") {
return ov::intel_npu::ExampleType::VAL1;
} else if (val == "VAL2") {
return ov::intel_npu::ExampleType::VAL2;
} else if (val == "VAL3") {
return ov::intel_npu::ExampleType::VAL3;
}
OPENVINO_THROW("Value '", val, "'is not a valid EXAMPLE_PROPERTY option");
}
static std::string toString(const ov::intel_npu::ExampleType& val) {
std::stringstream strStream;
strStream << val;
return strStream.str();
}
};
Notes:
(!!) None of the member functions are mandatory to be defined.
If any is missing, the default function will be used for its call, as defined in the OptionsBase class
(see class OptionBase in npu_plugin/al/include/config/config.hpp or Class Hierarchy section above)
Third step is to register the new option in the plugin:
plugin.cpp > function init_options():
REGISTER_OPTION(EXAMPLE_PROPERTY);
Notes:
at this point, the npu plugin will take care of registering and managing the option in the internal configuration.
It ensures that it is enabled or disabled based in the current system/environment/application configuration.
Fourth step is to create and register the Property (which is basicly the interface to this configuration option) for both Plugin and CompiledModel (if needed)
npu_plugin/plugin/src/properties.cpp > function registerPluginProperties()
TRY_REGISTER_SIMPLE_PROPERTY(ov::intel_npu::example_property, EXAMPLE_PROPERTY);
Explanation:
this macro will register (if all conditions are met) a property with the name ov::intel_npu::example_property.name()
which maps to our internal configuration named EXAMPLE_PROPERTY
and has a simple callback function of config.get<EXAMPLE_PROPERTY>()
npu_plugin/plugin/src/properties.cpp > function registerCompiledModelProperties()
TRY_REGISTER_COMPILEDMODEL_PROPERTY_IFSET(ov::intel_npu::example_property, EXAMPLE_PROPERTY);
Explanation:
this macro will register (if all conditions are met) a property with the name ov::intel_npu::example_property.name()
which maps to our internal configuration named EXAMPLE_PROPERTY
and has a simple callback function of config.get<EXAMPLE_PROPERTY>()
(!!) ONLY if it has been previously explicitly set at compile time
In order for the property to be property exposed in python API, add python wrapper for the new property in pyOpenvino
openvino/src/bindings/python/src/pyopenvino/core/properties/properties.cpp:
In section // sumbodule npu
wrap_property_RW(m_intel_npu, ov::intel_npu::example_property, "example_property");
Document the new property in the appropaite sections (+ additional information, if required) in:
openvino/docs/articles_en/openvino-workflow/running-inference/inference-devices-and-modes/npu-device.rst
openvino/src/plugins/intel_npu/README.md
As described in the first paragraph, Metrics do not have an entry in our internal Config, they are static immutable data which just needs to be returned at get_property calls.
This means we do not need to define and Option nor register an Option for it.
First step is to define the new property's name, datatype and string-name in the public header in
openvino/src/inference/include/openvino/runtime/intel_npu/properties.hpp
Example:
static constexpr ov::Property<ExampleType,ov::PropertyMutability::RO> example_property{"NPU_EXAMPLE_PROPERTY"};
Notes:
You need to register the new property and define a callback function for it, in properties.cpp.
For plugin: npu_plugin/plugin/src/properties.cpp > function registerPluginProperties()
For compiled-model: npu_plugin/plugin/src/properties.cpp > function registerCompiledModelProperties()
REGISTER_SIMPLE_METRIC(ov::intel_npu::example_property, true, _metrics->GetDriverVersion());
Explanation
this macro will register a property with the name ov::intel_npu::example_property (NPU_EXAMPLE_PROPERTY), which will be public and included in supported_properties (second parameter)
which will execute _metrics->GetDriverVersion() function as it's get_property callback
In order for the property to be property exposed in python API, add python wrapper for the new property in pyOpenvino
openvino/src/bindings/python/src/pyopenvino/core/properties/properties.cpp:
In section // sumbodule npu
wrap_property_RO(m_intel_npu, ov::intel_npu::example_property, "example_property");
Document the new property in the appropaite sections (+ additional information, if required) in:
openvino/docs/articles_en/openvino-workflow/running-inference/inference-devices-and-modes/npu-device.rst
openvino/src/plugins/intel_npu/README.md
By internal convention, what needs to be included in compiled-model properties gets decided based on the following statements:
To facilitate registering properties in compiled-model ONLY if they have been explicitly set prior to model compilation, properties.cpp offers a helper macro:
This macro will register the provided property (OPT_NAME) to the provided option (OPT_TYPE) only if a value for it exists in the config. (default values do not live in config, but directly read from OptionsDesc)
Example:
TRY_REGISTER_COMPILEDMODEL_PROPERTY_IFSET(ov::intel_npu::compilation_mode, COMPILATION_MODE);
If the new property requires a custom callback function, only Step 4. changes.
Instead of using TRY_REGISTER_SIMPLE_PROPERTY macro, you can choose from the following helper macros:
This can be used when callback is standard, but visibility (public/private) is custom.
Instead of using automaticly the value from optionsBase, one can define a custom function to determine
whether the property will be public (included in supported_properties) or private and provide as PROP_VISIBILITY parameter.
This macro can be used whenever a custom callback function is required for this property,
provided as a lambda function as PROP_RETFUNC parameter.
(Standard callback function just returns the value of the config)
This macro bypasses all automatic descriptor fetching, availability checks, and compatibility verifications
and gives you the possibility to register a completely custom property with custom visibility and custom callback function.
As oppposed to option-backed properties, for metric-backed ones we only have 1 extra helper macro,
apart from REGISTER_SIMPLE_METRIC (which lets you define name, visibility and return value as a single function call):
This macro offers the possibility to define a full custom callback function, as a lambda function propided as parameter PROP_RETFUNC,
instead of just a single value function call as REGISTER_SIMPLE_METRIC.
Example:
REGISTER_CUSTOM_METRIC(ov::device::full_name,
!_metrics->GetAvailableDevicesNames().empty(),
[&](const Config& config) {
const auto specifiedDeviceName = get_specified_device_name(config);
return _metrics->GetFullDeviceName(specifiedDeviceName);
});
There is a possibility to get options filtered out right at the options registration phase too, if desired by a specific cornercase.
If the option is not registered, it will be virtually inexistent in the whole stack at runtime, so there will be no added overhead by all the availability checks and verifications the properties manager does.
To achieve this, we can condition the registration of the option in:
plugin.cpp > (function) void Plugin::init_options()
Example:
if (_backend) {
if (_backend->isCommandQueueExtSupported()) {
REGISTER_OPTION(TURBO);
REGISTER_OPTION(WORKLOAD_TYPE);
}
}
The above code will register TURBO and WORKLOAD_TYPE options only if command queue extension is supported by the backend (and of course, first if a backend exists)
Having skipped the option registration (in case command queue extension is not supported), the option will not exist in the stack, will not be checked for compiler support, will not be enabled/disabled dinamically and can not be enabled dynamically anymore at runtime.
Removing a public property can be done by removing everything added in section “How do add a new public property” step-by-step.