docs/docs/features/ml-hardware-acceleration.md
This feature allows you to use a GPU to accelerate machine learning tasks, such as Smart Search and Facial Recognition, while reducing CPU load. As this is a new feature, it is still experimental and may not work on all systems.
:::info You do not need to redo any machine learning jobs after enabling hardware acceleration. The acceleration device will be used for any jobs that run after enabling it. :::
/dev/mali0 must be available in the host server
ls /dev to check that it existslibmali.so firmware (possibly with an additional firmware file)
hwaccel.ml.yml file assumes the path to it is /usr/lib/libmali.so, so update accordingly if it is elsewherehwaccel.ml.yml file assumes an additional file /lib/firmware/mali_csffw.bin, so update accordingly if your device's driver does not require this file.env file, see environment variables for ARM NN specific settings
MACHINE_LEARNING_ANN_FP16_TURBO can significantly improve performance at the cost of very slightly lower accuracyHSA_OVERRIDE_GFX_VERSION environmental variable: HSA_OVERRIDE_GFX_VERSION=<a supported version, e.g. 10.3.0>. If this doesn't work, you might need to also set HSA_USE_SVM=0.docker exec -t immich_machine_learning ls -la /dev/dri. If this is not the case execute getent group render and getent group video on the WSL host, then add those groups to hwaccel.ml.yaml
openvino-wsl:
devices:
- /dev/dri:/dev/dri
- /dev/dxg:/dev/dxg
volumes:
- /dev/bus/usb:/dev/bus/usb
- /usr/lib/wsl:/usr/lib/wsl
group_add:
- 44 # Replace this number with the number you found with getent group video
- 992 # Replace this number with the number you found with getent group render
cat /sys/kernel/debug/rknpu/version to check the version.env file, see environment variables for RKNN specific settings
MACHINE_LEARNING_RKNN_THREADS to 2 or 3 can dramatically improve performance for RK3576 and RK3588 compared to the default of 1, at the expense of multiplying the amount of RAM each model uses by that amount.hwaccel.ml.yml file and ensure it's in the same folder as the docker-compose.yml.immich-machine-learning, add one of -[armnn, cuda, rocm, openvino, rknn] to the image section's tag at the end of the line.docker-compose.yml under immich-machine-learning, uncomment the extends section and change cpu to the appropriate backend.immich-machine-learning container with these updated settings.You can confirm the device is being recognized and used by checking its utilization. There are many tools to display this, such as nvtop for NVIDIA or Intel, intel_gpu_top for Intel, and radeontop for AMD.
You can also check the logs of the immich-machine-learning container. When a Smart Search or Face Detection job begins, or when you search with text in Immich, you should either see a log for Available ORT providers containing the relevant provider (e.g. CUDAExecutionProvider in the case of CUDA), or a Loaded ANN model log entry without errors in the case of ARM NN.
Some platforms, including Unraid and Portainer, do not support multiple Compose files as of writing. As an alternative, you can "inline" the relevant contents of the hwaccel.ml.yml file into the immich-machine-learning service directly.
For example, the cuda section in this file is:
deploy:
resources:
reservations:
devices:
- driver: nvidia
count: 1
capabilities:
- gpu
You can add this to the immich-machine-learning service instead of extending from hwaccel.ml.yml:
immich-machine-learning:
container_name: immich_machine_learning
# Note the `-cuda` at the end
image: ghcr.io/immich-app/immich-machine-learning:${IMMICH_VERSION:-release}-cuda
# Note the lack of an `extends` section
deploy:
resources:
reservations:
devices:
- driver: nvidia
count: 1
capabilities:
- gpu
volumes:
- model-cache:/cache
env_file:
- .env
restart: always
Once this is done, you can redeploy the immich-machine-learning container.
If you want to utilize multiple NVIDIA or Intel GPUs, you can set the MACHINE_LEARNING_DEVICE_IDS environmental variable to a comma-separated list of device IDs and set MACHINE_LEARNING_WORKERS to the number of listed devices. You can run a command such as nvidia-smi -L or glxinfo -B to see the currently available devices and their corresponding IDs.
For example, if you have devices 0 and 1, set the values as follows:
MACHINE_LEARNING_DEVICE_IDS=0,1
MACHINE_LEARNING_WORKERS=2
In this example, the machine learning service will spawn two workers, one of which will allocate models to device 0 and the other to device 1. Different requests will be processed by one worker or the other.
This approach can be used to simply specify a particular device as well. For example, setting MACHINE_LEARNING_DEVICE_IDS=1 will ensure device 1 is always used instead of device 0.
Note that you should increase job concurrencies to increase overall utilization and more effectively distribute work across multiple GPUs. Additionally, each GPU must be able to load all models. It is not possible to distribute a single model to multiple GPUs that individually have insufficient VRAM, or to delegate a specific model to one GPU.
MACHINE_LEARNING_ANN_FP16_TURBO is enabled)MACHINE_LEARNING_RKNN_THREADS is at the default of 1, RKNPU will have substantially lower throughput for ML jobs than ARM NN in most cases, but similar latency (such as when searching)MACHINE_LEARNING_RKNN_THREADS is set to 3, it will be somewhat faster than ARM NN at FP32, but somewhat slower than ARM NN if MACHINE_LEARNING_ANN_FP16_TURBO is enabledMACHINE_LEARNING_RKNN_THREADS is at the default of 1, but significantly higher if greater than 1 (which is necessary for it to fully utilize the NPU and hence be comparable in speed to ARM NN)