Back to Cutlass

Ahead-of-Time (AOT) Compilation

media/docs/pythonDSL/guides/ahead_of_time_compilation.rst

4.6.216.4 KB
Original Source

.. _dsl_ahead_of_time_compilation: .. |DSL| replace:: CuTe DSL

Ahead-of-Time (AOT) Compilation

This guide demonstrates how to use |DSL|'s Ahead-of-Time (AOT) compilation features to export compiled kernels for use in production environments.

Overview

|DSL| Ahead-of-Time (hereinafter referred to as AOT) compilation allows you to:

  • Compile once, enable cross-compilation: Write kernels in Python and cross-compile them for multiple GPU architectures, and cross-compile the host object for other CPU architectures (see :ref:dsl_aot_host_cross_compilation).
  • Remove JIT overhead: Eliminate compilation delays in production by pre-compiling kernels.
  • Flexible integration: Easily integrate compiled kernels into both Python and C/C++ codebases using flexible deployment options.

We provide 2 levels of AOT ABI:

  1. Low-Level CuTe ABI: This ABI is expressed using CuTe DSL types and tensors, mirroring the original Python function.
  2. High-Level Apache TVM FFI ABI: For interop with various frameworks (e.g., PyTorch, JAX), and offer high-level stable ABI access.

This guide will focus on the CuTe ABI AOT. For the Apache TVM FFI AOT, please refer to the section "Exporting Compiled Module" in :doc:tvm_ffi_compilation.

CuTe ABI AOT Workflow

Export Interface


The ``export_to_c`` interface is provided by the ``JitCompiledFunction`` class. It accepts the following parameters:

* ``file_path``: The path to the directory where the header and object files will be saved.
* ``file_name``: The base name for the header and object files. The same file name will always overwrite existing files.
* ``function_prefix``: The prefix of the function symbol in the generated object file. This should be a unique identifier to avoid symbol conflicts. Users should ensure the function prefix is unique for each exported function. Defaults to the ``file_name``.

It generates the following files:

* ``{file_path}/{file_name}.h``: A C header file containing API function declarations. This header specifies the runtime function signatures in C, mirroring the original Python function interfaces.
* ``{file_path}/{file_name}.o``: A standard object file containing the compiled kernel code. You can link this object file into either a static or shared library. It includes the host entry function, fatbin data, and helper functions such as ``cuda_init`` and ``cuda_load_to_device``. Additionally, it embeds metadata for runtime loading and version verification.

Example:

.. code-block:: python

   import cutlass.cute as cute
   import cutlass.cute.cuda as cuda

   @cute.kernel
   def print_tensor_kernel(a: cute.Tensor):
       cute.printf("a: {}", a)

   @cute.jit
   def print_tensor(a: cute.Tensor, stream: cuda.CUstream):
       print_tensor_kernel(a).launch(grid=(1, 1, 1), block=(1, 1, 1), stream=stream)

   compiled_func = cute.compile(print_tensor)
   # Export compiled functions to object files and headers
   compiled_func.export_to_c(file_path="./artifacts", file_name="print_tensor_example", function_prefix="print_tensor")

Loading in Python

Load pre-compiled object files or shared libraries into Python for execution.

.. code-block:: python

import cutlass.cute as cute import torch from cutlass.cute import from_dlpack import cutlass.cute.cuda as cuda

Load module from object file

module = cute.runtime.load_module("./artifacts/print_tensor_example.o")

or

module = cute.runtime.load_module("./artifacts/libprint_tensor_example.so")

Prepare data

a = torch.arange(160, dtype=torch.float32, device="cuda").reshape(16, 10) a_cute = from_dlpack(a).mark_layout_dynamic() stream = cuda.CUstream(0)

Call the function (no JIT compilation needed!)

module.print_tensor(a_cute, stream=stream)

This will fail because 'non_existing_api' was not exported:

module.non_existing_api()

C++ Integration with Static Linking


Integrate compiled kernels directly into your C++ executable during the build process. The generated header file supplies the necessary API for loading the module and invoking the function.

Example:

.. code-block:: cpp

   #include "print_tensor_example.h"
   #include <cuda_runtime.h>

   void run_print_tensor() {
       // Prepare tensor, the tensor declaration is in the header file
       print_tensor_Tensor_a_t tensor_a;
       tensor_a.data = nullptr; // GPU memory is set to nullptr.
       // Set dynamic shapes and strides
       tensor_a.dynamic_shapes[0] = 32;
       tensor_a.dynamic_shapes[1] = 16;
       tensor_a.dynamic_strides[0] = 16;
       
       // Create stream
       cudaStream_t stream;
       cudaStreamCreate(&stream);
       
       // Load module before calling the kernel
       print_tensor_Kernel_Module_t module;
       print_tensor_Kernel_Module_Load(&module);
       
       // Call the kernel; the kernel wrapper function is defined in the header file
       cute_dsl_print_tensor_wrapper(&module, &tensor_a, stream);
       
       // Cleanup
       print_tensor_Kernel_Module_Unload(&module);
       cudaStreamDestroy(stream);
   }

The ``print_tensor_example.h`` header file is generated by the ``export_to_c`` interface. It includes:

* The ``print_tensor_Kernel_Module_t`` type: Represents the kernel module.
* The ``print_tensor_Tensor_a_t`` type: A tensor-specific type that defines the ABI for a particular CuTe tensor.
* The ``cute_dsl_print_tensor_wrapper`` function: The user-facing entry point to invoke the kernel.

The compilation of the C++ executable requires the ``libcuda_dialect_runtime.so`` or ``libcuda_dialect_runtime_static.a`` library which is involved in ``<wheel_install_path>/lib``, along with the CUDA driver and runtime libraries, to function properly.


C++ Integration with Dynamic Loading

Dynamically load pre-compiled object files or shared libraries at runtime. By including the CuteDSLRuntime.h header, you can load the module, look up exported functions, and invoke them.

.. code-block:: cpp

#include "CuteDSLRuntime.h" #include <cuda_runtime.h>

void run_print_tensor() { // Load module from shared library CuteDSLRT_Module_t *module = nullptr; CuteDSLRT_Error_t err = CuteDSLRT_Module_Load( &module, "./artifacts/libprint_tensor_example.so" ); // or CuteDSLRT_Error_t err = CuteDSLRT_Module_Load( &module, "./artifacts/print_tensor_example.o" ); check_error(err);

   // Lookup function
   CuteDSLRT_Function_t *func = nullptr;
   err = CuteDSLRT_Module_Get_Function(&func, module, "print_tensor");
   check_error(err);
   
   // Prepare arguments, matching the argument type defined in the header file
   typedef struct {
       void *data;
       int32_t dynamic_shapes[2];
       int64_t dynamic_strides[1];
   } print_tensor_Tensor_a_t;

   print_tensor_Tensor_a_t tensor_a;
   tensor_a.data = nullptr;
   tensor_a.dynamic_shapes[0] = 32;
   tensor_a.dynamic_shapes[1] = 16;
   tensor_a.dynamic_strides[0] = 16;

   // Create stream
   cudaStream_t stream;
   cudaStreamCreate(&stream);

   // Call the function; the runtime function accepts packed arguments, refer to the wrapper in the header file
   int ret;
   void* args[] = {&tensor_a, &stream, &ret};
   err = CuteDSLRT_Function_Run(func, args, 3);
   check_error(err);
   cudaStreamSynchronize(stream);
   
   // Cleanup
   CuteDSLRT_Module_Destroy(module);
   cudaStreamDestroy(stream);

}

The CuteDSLRuntime.h header file can be found in <wheel_install_path>/include. It includes:

  • The CuteDSLRT_Error_t type: Indicates error status.
  • The CuteDSLRT_Module_Load function: Loads the module.
  • The CuteDSLRT_Module_Get_Function function: Gets a function from the loaded module. The runtime API will load the CUDA module for kernel execution.
  • The CuteDSLRT_Function_Run function: Runs the function.
  • The CuteDSLRT_Module_Destroy function: Destroys the module.

The compilation of the C++ executable requires the libcute_dsl_runtime.so library which is involved in <wheel_install_path>/lib, along with the CUDA driver and runtime libraries, to function properly.

.. _dsl_aot_host_cross_compilation:

Host Cross-Compilation for AArch64

By default, the object file produced by export_to_c targets the same machine that runs cute.compile (the build host, typically x86_64). |DSL| can instead emit the host portion of the object for a different CPU architecture, so you can build a kernel on an x86_64 machine and deploy it to an AArch64 Linux machine. Only the host code is affected; the GPU device code is still selected by --gpu-arch as described above.

Build Host vs Target


* **Build host**: the machine where you run Python, ``cute.compile``, and ``export_to_c``.
* **Target**: the machine where the resulting shared library or executable will run.

When the host target is left empty (the default), the ``.o`` is compiled for the build host. When a target is requested, the ``.o`` is an ELF object for that target's triple, and the accompanying ``.h`` uses only fixed-width and opaque types, so the C ABI is identical regardless of any word-size difference between the build host and the target.

Selecting the Host Target

The host target is chosen at cute.compile time through the --host-target option. Two input formats are accepted:

  • A preset tag. linux-aarch64 maps to the triple aarch64-unknown-linux-gnu.
  • A long form for explicit CPU and feature tuning: llvm -mtriple=<triple> [-mcpu=<cpu>] [-mattr=<features>].

.. code-block:: python

import cutlass.cute as cute

Preset tag

compiled = cute.compile( my_function, *args, options="--gpu-arch sm_100a --host-target linux-aarch64")

Long form with explicit CPU and features

compiled = cute.compile( my_function, *args, options="--gpu-arch sm_100a " "--host-target 'llvm -mtriple=aarch64-unknown-linux-gnu " "-mcpu=neoverse-n1 -mattr=+lse'")

Invalid input, such as an unknown preset tag or a malformed long form, is rejected immediately when cute.compile is called, rather than later during export.

Exporting the Cross-Compiled Object


Once the function is compiled with a host target, export it exactly as in the native workflow; no additional arguments are required.

.. code-block:: python

   compiled.export_to_c(file_path="./artifacts", file_name="kernel",
                        function_prefix="kernel")

The resulting ``kernel.o`` is an AArch64 ELF object, and ``kernel.h`` is portable across the build host and the target.

Resolving Runtime Symbols at Link Time

Linking the cross-compiled .o into a shared library or executable on the build host requires the |DSL| runtime library, but the real libcute_dsl_runtime.so only exists on the AArch64 target. To make link-time symbol resolution possible on the build host, the wheel ships a link-time stub: an empty-body libcute_dsl_runtime.so with the same SONAME and exported symbols as the real library, installed under lib/stubs/<triple>/. The linker resolves against the stub on the build host; at runtime on the target the dynamic loader binds against the real library.

Use the aot_config helper to discover the linker flags for a target triple:

.. code-block:: bash

python -m cutlass.cute.export.aot_config --libdir --target aarch64-unknown-linux-gnu python -m cutlass.cute.export.aot_config --ldflags --target aarch64-unknown-linux-gnu python -m cutlass.cute.export.aot_config --libs --target aarch64-unknown-linux-gnu

  • --libdir prints the resolved runtime library directory for the target (the stub subtree when no cross-built runtime is shipped).
  • --ldflags prints the -L<dir> linker search flag.
  • --libs prints the runtime -l flag (-lcute_dsl_runtime). Pass --with-tvm-ffi to also include the TVM FFI library.

If no per-triple or stub subtree exists for the requested target, the helper exits with a non-zero status and prints a message to standard error.

Cross-Linking the Shared Library


Link the object with your AArch64 cross toolchain (for example ``aarch64-linux-gnu-g++``) and a sysroot that provides the CUDA headers and libraries for the target:

.. code-block:: bash

   aarch64-linux-gnu-g++ -shared -o kernel.so kernel.o \
       $(python -m cutlass.cute.export.aot_config --ldflags --target aarch64-unknown-linux-gnu) \
       $(python -m cutlass.cute.export.aot_config --libs    --target aarch64-unknown-linux-gnu)

Linking succeeds against the stub on the build host. Deploy ``kernel.so`` to the AArch64 target, where it binds against the real runtime library.

Limitations
~~~~~~~~~~~

* **AArch64 only.** Only ``aarch64-unknown-linux-gnu`` is currently supported. Other architectures fail with a "target not registered" error during code generation.
* **Not compatible with TVM FFI.** Combining ``--enable-tvm-ffi`` with ``--host-target`` raises an error; drop ``--enable-tvm-ffi`` and use the plain AOT export path described here.
* **CUDA runtime version must match.** The exported object depends on the CUDA runtime; the target's CUDA runtime/toolkit version must match the one |DSL| was built against (see :ref:`dsl_aot_object_compat`).
* **Linking is your responsibility.** You must supply your own cross toolchain and a target sysroot with the CUDA headers and libraries; the stub only resolves the |DSL| runtime symbols.

Supported Argument Types
------------------------

|DSL| supports the following argument types:

* ``cute.Tensor``
* ``cute.Shape`` / ``cute.Coord`` / ``cute.Tile`` / ``cute.IntTuple`` / ``cute.Stride``
* ``cuda.CUstream``
* ``cutlass.Int8`` / ``cutlass.Int16`` / ``cutlass.Int32`` / ``cutlass.Int64`` / ``cutlass.Boolean``
* ``cutlass.Uint8`` / ``cutlass.Uint16`` / ``cutlass.Uint32`` / ``cutlass.Uint64``
* ``cutlass.Float32`` / ``cutlass.TFloat32`` / ``cutlass.Float64`` / ``cutlass.Float16``

Note that:

1. ``cute.Tensor`` is a dynamic tensor type that only contains dynamic shapes and strides in its ABI representation. As a result, different compilations may produce different tensor ABIs. This is why declarations for each tensor type are included in the generated header file.
2. ``strides`` in ``cute.Tensor`` are determined by the ``use_32bit_strides`` compile argument. When ``use_32bit_strides`` is set to ``True``, the strides are 32-bit; when set to ``False``, they are 64-bit.
3. Currently, custom types are not supported for AOT compilation.

.. _dsl_aot_object_compat:

Object File Compatibility Issues
--------------------------------

The object file generated by |DSL| depends on the CUDA runtime library. Therefore, ensure that the version of the CUDA runtime/toolkit library matches the version used by |DSL|. Otherwise, ABI compatibility with the CUDA runtime cannot be guaranteed.

When using C++ static linking integration, compatibility is assured because the header and object files are generated together and guaranteed to match.

For C++ dynamic loading integration and Python loading, the binary file is loaded at runtime. To ensure compatibility, version information is embedded in the metadata of the generated binary file. At runtime, this version information is checked, and if it does not match the expected version, the binary file will be rejected.

Relation to Apache TVM FFI AOT
------------------------------

Apache TVM FFI AOT offers a comparable capability, enabling TVM functions to be compiled into binary files that can be loaded and executed at runtime.
For more information, see the section "Exporting Compiled Module" in :doc:`tvm_ffi_compilation`.

The primary distinction is that, when TVM FFI is enabled, |DSL| generates a dedicated wrapper function on top of the underlying CuTe ABI. This wrapper adheres to the calling conventions defined by TVM FFI.
In contrast, the CuTe ABI entry function is specified directly in the generated header file, which affects how arguments must be provided.

For instance, with the TVM FFI wrapper function, users are able to pass in arguments such as ``torch.Tensor`` directly. However, when calling the CuTe ABI entry function, arguments should be provided as ``cute.Tensor`` types.