src/plugins/intel_npu/docs/npu-properties-howto.md
What is a property from NPU Plugin's POV and Step-by-step guide on how to add one
Practical manual for NPU plugin properties
| Name | Description | Example |
|---|---|---|
| Property | a plugin interface which can be set or read. Can map to options or metrics` | ov::log::level |
| Option | a configuration entry in our internal configuration.</br>Consists of an OptionBase template descriptor + a template OptionValue value. | LOG_LEVEL |
| Config | Our internal database of configuration keys and their values | _globalConfig |
| Metric | A property which does not map to any configuration key and configuration entry in our internal config.</br>Usually a static value directly read from driver or OS. | ov::device::pci_info |
| Compiler | Npu compiler as viewed from the plugin's perspective.</br>Can be Compiler-In-Driver or Compiler-In-Plugin | CID |
| "Anonymous" property</br>OR</br>compiler-private property | A setting from application level which the plugin has no knowledge of</br>(it is not registered, plugin is not aware of its datatype)</br> but which the compiler reports as supported via is_supported() API. | N/A |
As it can be observed in the above class hierarchy diagram, a Property is a public interface to an internal information, the top layer of abstraction.
A property can connect internally to an Option or to a Metric.
The main difference between Option and Metric is that while Options are entries in our internal database (Config) which can be modified at any time, Metrics are static read-only pieces of information
which do not exist in the internal database. Metrics can not be set or changed externally, their values are not stored in the plugin and they most often just a direct system or driver call.
Example of Metrics in NPU Plugin are: driver version, compiler version, device architecture, pci information, gops, uuid, luid, etc.
To summarize:
A property can map internally to one of the 2 types:
Implements the option descriptor. This class contains all the details of a config option: name, datatype, default value, parser, public/private, mutability, compiler version (for legacy support), etc. This serves as the key in our configuration map.
Class definition in npu_plugin/al/include/config/config.hpp:
struct OptionBase {
using ValueType = T;
// `ActualOpt` must implement the following method:
// static std::string_view key()
static constexpr std::string_view getTypeName() {
if constexpr (TypePrinter<T>::hasName()) {
return TypePrinter<T>::name();
}
static_assert(TypePrinter<T>::hasName(),
"Options type is not a standard type, please add `getTypeName()` to your option");
}
// Overload this to provide environment variable support.
static std::string_view envVar() {
return "";
}
// Overload this to provide deprecated keys names.
static std::vector<std::string_view> deprecatedKeys() {
return {};
}
// Overload this to provide default value if it wasn't specified by user.
// If it is std::nullopt - exception will be thrown in case of missing option access.
static std::optional<T> defaultValue() {
return std::nullopt;
}
// Overload this to provide more specific parser.
static ValueType parse(std::string_view val) {
return OptionParser<ValueType>::parse(val);
}
// Overload this to provide more specific validation
static void validateValue(const ValueType&) {}
// Overload this to provide more specific implementation.
static OptionMode mode() {
return OptionMode::Both;
}
// Overload this for private options.
static bool isPublic() {
return false;
}
// Overload this for read-only options (metrics)
static ov::PropertyMutability mutability() {
return ov::PropertyMutability::RW;
}
/// Overload this for options conditioned by compiler version
static uint32_t compilerSupportVersion() {
return ONEAPI_MAKE_VERSION(0, 0);
}
static std::string toString(const ValueType& val) {
return OptionPrinter<ValueType>::toString(val);
}
};
is storage for the registered options This is the base map which stores the available optionBase descriptors.
This layer implements the option database manipulation functions: add/remove/has/reset
Note: This layer is static, intialized once in plugin constructor with all the options the Plugin has knowledge of.
is the highlevel configuration "database" which implements the mapping between OptionBase and templatized OptionValue.
Maps and stores the user-defined values for each entry in OptionsDesc layer.
Implements the top level configuration manipulation functions:
get/update/has/getString/toString/fromString and handles typecasts, typeverification, parsing and conversions.
Note: This layer is static, initialized once in plugin constructor with the provided (also static) OptionDesc.
is a derivative class of Config, used only by NPU Plugin, which implements additional filtering layers atop of the base config,
such as enabling/disabling keys based on their availability/support on the current system configuration.
Note: This layer dynamically changes based on system configuration and compiler_type
is the top level class and serves as the NPU Plugin's interface to OpenVino and the application layer.
It's main purpose is to implement get_property and set_property interfaces and the callback functions of each property.
Note: This layer dynamically changes based on system configuration and compiler_type
The following steps how to add a new simple property which maps to a compiler configuration option.
*simple in this context means that it has no special callback function required for it, just set/get
First step is to define the new property's name, datatype and string-name in the public header in
openvino/src/inference/include/openvino/runtime/intel_npu/properties.hpp
Example:
static constexpr ov::Property<ExampleType,ov::PropertyMutability::RW> example_property{"NPU_EXAMPLE_PROPERTY"};
Notes:
Second step is to define the optionDesc class for this property in
openvino/src/plugins/intel_npu/al/config/options.hpp
Example:
//
// EXAMPLE_PROPERTY
//
struct EXAMPLE_PROPERTY final : OptionBase<EXAMPLE_PROPERTY, ov::intel_npu::ExampleType> {
static std::string_view key() {
return ov::intel_npu::example_property.name();
}
static constexpr std::string_view getTypeName() {
return "ov::intel_npu::ExampleType";
}
static ov::intel_npu::ExampleType defaultValue() {
return ov::intel_npu::ExampleType::VAL3;
}
static uint32_t compilerSupportVersion() {
return ONEAPI_MAKE_VERSION(5, 5);
}
static bool isPublic() {
return true;
}
static OptionMode mode() {
return OptionMode::Both;
}
static ov::PropertyMutability mutability() {
return ov::PropertyMutability::RW;
}
static std::string_view envVar() {
return "IE_NPU_EXAMPLE_PROPERTY";
}
static ov::intel_npu::ExampleType parse(std::string_view val) {
if (val == "VAL1") {
return ov::intel_npu::ExampleType::VAL1;
} else if (val == "VAL2") {
return ov::intel_npu::ExampleType::VAL2;
} else if (val == "VAL3") {
return ov::intel_npu::ExampleType::VAL3;
}
OPENVINO_THROW("Value '", val, "'is not a valid EXAMPLE_PROPERTY option");
}
static std::string toString(const ov::intel_npu::ExampleType& val) {
std::stringstream strStream;
strStream << val;
return strStream.str();
}
};
Notes:
(!!) None of the member functions are mandatory to be defined.
If any is missing, the default function will be used for its call, as defined in the OptionsBase class
(see class OptionBase in npu_plugin/al/include/config/config.hpp or Class Hierarchy section above)
Third step is to register the new option in the plugin:
plugin.cpp > function init_options():
REGISTER_OPTION(EXAMPLE_PROPERTY);
Notes:
at this point, the npu plugin will take care of registering and managing the option in the internal configuration.
It ensures that it is enabled or disabled based in the current system/environment/application configuration.
Fourth step is to create and register the Property (which is basicly the interface to this configuration option) for both Plugin and CompiledModel (if needed)
npu_plugin/plugin/src/plugin_property_manager.cpp > function PluginPropertyManager::registerProperties()
try_register_property_based_on_config<EXAMPLE_PROPERTY>(_config, _properties, ov::intel_npu::example_property.name());
Explanation:
this helper function will register (if all conditions are met) a property with the name ov::intel_npu::example_property.name()
which maps to our internal configuration named EXAMPLE_PROPERTY
and has a simple callback function of config.get<EXAMPLE_PROPERTY>()
npu_plugin/plugin/src/compiled_model_property_manager.cpp > function CompiledModelPropertyManager::registerProperties()
try_register_property_based_on_config_if_set_as_read_only<EXAMPLE_PROPERTY>(_config, _properties, ov::intel_npu::example_property.name());
Explanation: this helper function registers the compiled-model property with the name ov::intel_npu::example_property.name() which maps to our internal configuration named EXAMPLE_PROPERTY and has a simple callback function of config.get<EXAMPLE_PROPERTY>() (!!) ONLY if it has been previously explicitly set at compile time.
In order for the property to be property exposed in python API, add python wrapper for the new property in pyOpenvino
openvino/src/bindings/python/src/pyopenvino/core/properties/properties.cpp:
In section // sumbodule npu
wrap_property_RW(m_intel_npu, ov::intel_npu::example_property, "example_property");
Document the new property in the appropaite sections (+ additional information, if required) in:
openvino/docs/articles_en/openvino-workflow/running-inference/inference-devices-and-modes/npu-device.rst
openvino/src/plugins/intel_npu/README.md
As described in the first paragraph, Metrics do not have an entry in our internal Config, they are static immutable data which just needs to be returned at get_property calls.
This means we do not need to define and Option nor register an Option for it.
First step is to define the new property's name, datatype and string-name in the public header in
openvino/src/inference/include/openvino/runtime/intel_npu/properties.hpp
Example:
static constexpr ov::Property<ExampleType,ov::PropertyMutability::RO> example_property{"NPU_EXAMPLE_PROPERTY"};
Notes:
You need to register the new property and define a callback function in the owner-specific property manager. For plugin: npu_plugin/plugin/src/plugin_property_manager.cpp > function PluginPropertyManager::registerProperties() For compiled-model: npu_plugin/plugin/src/compiled_model_property_manager.cpp > function CompiledModelPropertyManager::registerProperties()
register_property_with_custom_function(_properties, ov::intel_npu::example_property.name(), true, _metrics->GetDriverVersion());
Explanation
this helper function will register a property with the name ov::intel_npu::example_property (NPU_EXAMPLE_PROPERTY), which will be public and included in supported_properties (second parameter)
which will execute _metrics->GetDriverVersion() as its get_property callback.
Note: the first argument is the property name string (property.name()), not the property object itself.
In order for the property to be property exposed in python API, add python wrapper for the new property in pyOpenvino
openvino/src/bindings/python/src/pyopenvino/core/properties/properties.cpp:
In section // sumbodule npu
wrap_property_RO(m_intel_npu, ov::intel_npu::example_property, "example_property");
Document the new property in the appropaite sections (+ additional information, if required) in:
openvino/docs/articles_en/openvino-workflow/running-inference/inference-devices-and-modes/npu-device.rst
openvino/src/plugins/intel_npu/README.md
By internal convention, what needs to be included in compiled-model properties gets decided based on the following statements:
To facilitate registering properties in compiled-model ONLY if they have been explicitly set prior to model compilation, property_registration.hpp provides a helper function:
This helper function registers the provided property to the provided option type only if a value for it exists in the config. (default values do not live in config, but are directly read from OptionsDesc) Example:
try_register_property_based_on_config_if_set_as_read_only<COMPILATION_MODE>(_config,
_properties,
ov::intel_npu::compilation_mode.name());
If the new property requires a custom callback function, only Step 4. changes. Instead of using try_register_property_based_on_config helper function, you can choose from the following helper functions:
This can be used when callback is standard, but visibility (public/private) is custom. Instead of using automatically the value from the option descriptor, one can pass a runtime bool to determine whether the property will be public (included in supported_properties) or private. Example:
try_register_property_based_on_config_with_visibility<RUN_INFERENCES_SEQUENTIALLY>(
_config,
_properties,
ov::intel_npu::run_inferences_sequentially.name(),
[&] { return _backend && _backend->getInitStructs(); }());
This helper function can be used whenever a custom callback function/implementation is required for this property, provided as a lambda function. The getter receives the Config object and must return an ov::Any value. (Standard callback function just returns the value of the config) Example:
try_register_property_based_on_config_with_custom_function(_config, _properties, ov::intel_npu::stepping.name(),
[&](const Config& config) {
if (!config.has<STEPPING>()) {
return static_cast<int64_t>(_metrics->GetSteppingNumber(specifiedDeviceName));
}
return config.get<STEPPING>();
});
Apart from register_property_with_custom_function, two additional helper functions are available for metric-backed properties:
Conditionally registers a metric property. The property is only added when shouldRegister is true.
Use this when the availability of a metric depends on a runtime condition (e.g. backend capability check).
Example:
try_register_property_with_custom_function(_properties,
ov::device::full_name.name(),
!_metrics->GetAvailableDevicesNames().empty(),
true,
[&](const Config& config) {
const auto specifiedDeviceName = get_specified_device_name(config);
return _metrics->GetFullDeviceName(specifiedDeviceName);
});
Same as try_register_property_with_custom_function, but for properties whose getter also receives an ov::AnyMap of additional
arguments at get_property call time. Use this for properties such as ov::compatibility_check that accept
extra input arguments.
Example:
try_register_property_with_custom_function_and_args(_properties,
ov::compatibility_check.name(),
_compatibilityCheckSupported,
true,
[this](const Config&, const ov::AnyMap& arguments) {
return validateCompatibilityDescriptor(_backend, arguments);
});
There is a possibility to get options filtered out right at the options registration phase too, if desired by a specific cornercase. If the option is not registered, it will be virtually inexistent in the whole stack at runtime, so there will be no added overhead by all the availability checks and verifications the properties manager does. To achieve this, we can condition the registration of the option in: plugin.cpp > (function) void Plugin::init_options() Example:
if (_backend) {
if (_backend->isCommandQueueExtSupported()) {
REGISTER_OPTION(TURBO);
REGISTER_OPTION(WORKLOAD_TYPE);
}
}
The above code will register TURBO and WORKLOAD_TYPE options only if command queue extension is supported by the backend (and of course, first if a backend exists) Having skipped the option registration (in case command queue extension is not supported), the option will not exist in the stack, will not be checked for compiler support, will not be enabled/disabled dinamically and can not be enabled dynamically anymore at runtime.
Removing a public property can be done by removing everything added in section "How do add a new public property" step-by-step.