docs/en/integrations/ambarella.md
Deploying Ultralytics YOLO models on Ambarella SoCs requires model compilation using Ambarella's compilation tools, and models optimized for the CVflow architecture perform better at inference time. This fork of Ultralytics integrates Ambarella's SpongeTorch compression toolkit directly into the train, validate, and export pipeline, enabling developers to generate optimized models for efficient deployment on Ambarella hardware.
This guide covers the current object detection deployment workflow, from compression-aware training through on-device inference (see the Workflow Overview for the full pipeline).
The AmbaPB checkpoint format is an Ambarella-specific extension of the ONNX IR specification that supports CVflow computational primitives and packages the artifacts generated by the SDK tools. It is the host-side artifact used to validate compiled-model accuracy before deployment; the separate Cavalry binary produced for the target device is what runs on the board.
!!! note
This workflow depends on proprietary Ambarella SDK components that are not available on PyPI. To obtain the required SDK packages, register with the [Ambarella Developer Zone](https://www.ambarella.com/developer/) and request access through the Cooper™ Developer Platform.
!!! note "Support"
This integration is maintained by Ambarella. Report issues with the fork, SpongeTorch, or the SDK to [Ambarella support](https://www.ambarella.com/developer/).
Ambarella, headquartered in Santa Clara, California, is a semiconductor company that designs edge AI SoCs. Its processors combine image signal processing, video encoding, and on-chip AI compute, and are used in security, automotive, robotics, industrial, and consumer devices.
CVflow is Ambarella's vision processing architecture. It uses a dedicated vision engine, separate from the CPU and GPU, to run computer vision and neural network workloads. Models trained in frameworks like PyTorch are compiled to CVflow's native format with the Ambarella SDK before they run on the engine.
Current CVflow SoC families and their typical applications:
| SoC family | Typical applications |
|---|---|
| CV72 / CV75 | 4K AI security cameras, smart cameras, industrial vision |
| CV5 / CV52 | Drones, action cameras, robotics, multi-camera systems |
| N1-655 | On-premise generative AI and multi-stream video analytics appliances |
The pipeline has six stages:
amba_config) so SpongeTorch applies unstructured pruning progressively during training. Where post-training quantization (PTQ) accuracy is not acceptable, SpongeTorch also supports quantization-aware training (QAT), but that path is not yet wired into this Ultralytics integration and is planned for a future release.amba_config, preserving the compression structure in the ONNX graph.*.ambapb.ckpt.onnx model through Ultralytics predict/val via the AmbaPB backend to verify accuracy before deployment.SpongeTorch train and export workflow is optional and can be replaced by a plain ONNX export (see Exporting Without SpongeTorch).
Install this Ultralytics fork, then set up the Ambarella CVflow SDK — which includes the compilation tools and the cvflowbackend library — and install the spongetorch wheel distributed alongside it:
!!! tip "Installation"
=== "CLI"
```bash
# Install this Ultralytics fork from source
git clone https://github.com/Ambarella-Inc/ultralytics
cd ultralytics
git checkout amba_v8.4.46
pip install -e .
# Access and set up the Ambarella SDK compilation tools
# After the environment is ready, install the spongetorch library
pip install /path/to/spongetorch-*.whl
```
The AutoBackend locates cvflowbackend through the SDK compilation tools' tv2 command (tv2 -libpath cvflowbackend), so the SDK compilation tools must be installed and on your PATH before running inference or validation with compiled models.
SpongeTorch is driven by a SpongeKit configuration file (protobuf-text format, .prototxt) that defines the pruning passes, including sparsity targets and the compression schedule. Obtain example configurations and the matching schema documentation from your Ambarella SDK release. To maintain consistency between training, validation, and deployment, use the training configuration whenever validation needs to re-prepare the model, and always use the same configuration when exporting a compressed checkpoint.
Two arguments control the SpongeTorch integration across train, val, and export modes:
| Argument | Type | Default | Description |
|---|---|---|---|
amba_config | str | None | Path to the SpongeKit config passed to spongetorch.prepare(). Enables compression-aware training and SpongeTorch-aware export. |
amba_chipset | str | None | Target chipset name passed to spongetorch.set_target_chipset(), e.g. CV72. |
The fork also adds a general export argument:
| Argument | Type | Default | Description |
|---|---|---|---|
export_file | str | None | Custom export output path/name, e.g. '/tmp/model.onnx' or 'model.onnx'. |
Train (or fine-tune) your model with SpongeTorch compression enabled:
!!! example "Usage"
=== "Python"
```python
from ultralytics import YOLO
model = YOLO("yolo26n.pt")
model.train(
data="coco8.yaml",
epochs=100,
amba_config="config.prototxt",
amba_chipset="CV72",
)
```
=== "CLI"
```bash
yolo train model=yolo26n.pt data=coco8.yaml epochs=100 \
amba_config=config.prototxt amba_chipset=CV72
```
When amba_config is set, the trainer wraps the model and optimizer with spongetorch.prepare() at setup. Compression is applied progressively on a step schedule, so the network learns to stay accurate while becoming sparse. The trained checkpoint stores SpongeTorch's sparse state (_orig/_mask tensors), which the export step later requires. The config file is copied into the run directory as amba_config.prototxt for reproducibility.
!!! note "Checkpoint gating"
`best.pt` and `last.pt` are intentionally not saved until the SpongeTorch compression schedule crosses its `end_step` — a half-compressed checkpoint would not be usable. Ensure `epochs` is long enough for the schedule in your config to complete; the log reports when checkpoint saving begins. If training ends before the schedule completes, the final epoch is saved anyway with a warning, but such a checkpoint should not be deployed.
!!! tip "Fine-tune instead of training from scratch"
For best accuracy, first train your model normally (or start from a pretrained checkpoint), then run a shorter compression fine-tune with `amba_config` on the trained weights.
Validate accuracy before compiling, using the same config:
!!! example "Usage"
=== "CLI"
```bash
yolo val model=runs/detect/train/weights/best.pt data=coco8.yaml \
amba_config=config.prototxt amba_chipset=CV72
```
The validator re-applies spongetorch.prepare() when required and disables Conv+BN fusion so the compression structure is preserved. Compare mAP against your uncompressed baseline; if the accuracy drop is too large, adjust the SpongeKit config and retrain.
Export the compressed checkpoint with the same amba_config used in training:
!!! example "Usage"
=== "Python"
```python
from ultralytics import YOLO
model = YOLO("runs/detect/train/weights/best.pt")
model.export(
format="onnx",
amba_config="config.prototxt",
amba_chipset="CV72",
)
```
=== "CLI"
```bash
yolo export model=runs/detect/train/weights/best.pt format=onnx \
amba_config=config.prototxt amba_chipset=CV72
```
The exporter rebuilds the model, re-applies spongetorch.prepare() with your config, reloads the sparse checkpoint weights into the prepared structure, and traces to ONNX with Conv+BN fusion disabled — producing a graph in the exact form the SDK compilation tools expect.
ONNX export embeds the model task, class names, stride, and input size in the ONNX file, while the AmbaPB backend reads this information from a metadata.yaml sidecar next to the compiled model. Unless your SDK compilation tools create this sidecar, extract it from the ONNX model before compilation:
import onnx
from ultralytics.utils import YAML
model = onnx.load("model.onnx")
YAML.save("metadata.yaml", {item.key: item.value for item in model.metadata_props})
Keep metadata.yaml in the same directory as the compiled *.ambapb.ckpt.onnx or *.ambapb.fastckpt.onnx file.
!!! warning
- The checkpoint must include SpongeTorch compression state. Attempting to export an uncompressed checkpoint with `amba_config` set raises: *"Checkpoint has no SpongeTorch pruning state... Use a compressed checkpoint from amba training before export."*
- The configuration must match the configuration used during training. Using a different configuration may prevent the checkpoint weights from being loaded correctly.
Compile the exported ONNX model for your target chipset using the SDK compilation tools, following the SDK's compilation guide. The tools map the graph onto the CVflow AI engine — applying PTQ, scheduling, and memory planning — and produce the AmbaPB checkpoint for host validation.
PTQ applies INT8 quantization using calibration images (prepared as described in the SDK's compilation guide), and the compilation tools balance accuracy against runtime latency: mapping more operations to INT8 lowers latency but can reduce accuracy, while keeping more operations in FP16 preserves accuracy at higher latency. When PTQ cannot reach your accuracy target at the INT8 level required for your latency budget, QAT with SpongeTorch is the intended remedy — it trains the model to tolerate more aggressive INT8 quantization, recovering accuracy at a lower-latency operating point. QAT is not yet available in this integration and is planned for a future release.
!!! note
For Ultralytics to recognize the compiled model, its filename must end with `.ambapb.ckpt.onnx` or `.ambapb.fastckpt.onnx`.
The compiled AmbaPB model loads directly through the Ultralytics API — AutoBackend detects the .ambapb suffix and routes inference through cvflowbackend, executing the model as it will run on the AI engine:
!!! example "Usage"
=== "Python"
```python
from ultralytics import YOLO
model = YOLO("model.ambapb.ckpt.onnx")
# Inference
results = model("https://ultralytics.com/images/bus.jpg")
# Validation
metrics = model.val(data="coco8.yaml")
```
=== "CLI"
```bash
yolo predict model=model.ambapb.ckpt.onnx source='https://ultralytics.com/images/bus.jpg'
yolo val model=model.ambapb.ckpt.onnx data=coco8.yaml
```
This is the final accuracy check before hardware deployment, including all compiler quantization effects. If a metadata.yaml file sits next to the compiled model, the backend reads class names, stride, and task information from it. The backend uses CVflow inference mode acinf by default; set the environment variable ULTRALYTICS_AMBAPB_DEBUG=1 to log input/output details for debugging.
After the AmbaPB checkpoint passes host validation, use the SDK compilation tools to convert it into a Cavalry binary for your target device, following the SDK's compilation guide. The Cavalry binary is the form executed by the SDK runtime library on the board.
Load the Cavalry binary on your Ambarella device using the Ambarella SDK runtime. Preprocessing and postprocessing must match what the detection model was compiled for: letterboxed RGB input in the 0–255 range, and standard YOLO detection decoding on the outputs. Refer to the SDK deployment documentation for runtime APIs.
If you do not need SpongeTorch's training-time pruning, the standard Ultralytics pipeline also produces a model the SDK tools can compile:
!!! example "Usage"
=== "CLI"
```bash
yolo export model=yolo26n.pt format=onnx
```
Compile the resulting ONNX with the SDK compilation tools, which perform post-training quantization themselves. This path trades some runtime performance and quantized accuracy for a simpler workflow with no spongetorch dependency at training time.
Ultralytics YOLO models on Ambarella CVflow SoCs power always-on vision at the edge:
This guide outlined the current workflow to deploy Ultralytics YOLO models on Ambarella CVflow SoCs: compression-aware training with SpongeTorch (amba_config/amba_chipset), ONNX export of the compressed checkpoint, offline compilation to an AmbaPB checkpoint with the SDK tools, host validation through Ultralytics, and conversion to a Cavalry binary for on-device deployment with the Ambarella SDK.
For other edge AI targets, see the related Hailo, Rockchip RKNN, Sony IMX500, Qualcomm QNN, DEEPX, and Axelera guides. For the full list of export formats, visit the Export mode documentation and the integrations page.
model.export()?No. There is no format="ambarella" target. Export to ONNX (optionally with SpongeTorch compression via amba_config), then compile the ONNX model to AmbaPB offline with the Ambarella SDK compilation tools.
Any CVflow-based SoC supported by your SDK compilation tools may be targeted, including the CV72/CV75 families for AI cameras and CV5/CV52 for drones and robotics. The amba_chipset argument configures SpongeTorch's optimization target; select the matching target separately when compiling. Accepted chipset strings and availability depend on the installed SDK release.
SpongeTorch is the PyTorch flavor of Ambarella's SpongeKit model compression library (which also has Caffe and TensorFlow variants), integrated into the Ambarella fork of Ultralytics for training-time unstructured pruning (quantization-aware training is planned for a future release). It is optional: a plain Ultralytics ONNX export can also be compiled with the SDK compilation tools, which perform the quantization themselves, at some cost in runtime performance and quantized accuracy.
They are proprietary and not on PyPI. Register on the Ambarella Developer Zone to request SDK access; the SDK includes the compilation tools (with cvflowbackend), and the separately distributed spongetorch wheel ships alongside it.
Run yolo val model=model.ambapb.ckpt.onnx data=your_data.yaml with the Ambarella fork installed. The AmbaPB backend executes the compiled model as it runs on the CVflow AI engine, so the reported mAP includes all compiler quantization effects.