Tsavorite Native Device (C++)

We use CMake to build the native device (optionally used on Linux and Windows). To build, create one or more build directories and use CMake to set up build scripts for your target OS. Once CMake has generated the build scripts, it will try to update them, as needed, during ordinary build.

The resulting library is named native_device.dll on Windows and libnative_device.so on Linux. Both expose the same C ABI; selection of the underlying IO backend at runtime is done via the C# IoBackend enum, which Garnet surfaces as --device-io-backend default|libaio|uring.

Note on the prebuilt artifacts. Prebuilt binaries are checked in at libs/storage/Tsavorite/cs/src/core/Device/runtimes/{linux-x64,win-x64}/native/ and copied into bin/ at build time. If you change C++ sources you must rebuild and re-check in the matching prebuilt for the platform you changed, otherwise consumers will not see your changes. See "Updating the shipped prebuilt binaries" below.

Runtime dependencies (end users)

We ship two Linux prebuilt native libraries next to each other under runtimes/linux-x64/native/:

1. libnative_device.so — built with USE_URING=ON, links libaio AND liburing. Exposes both IoBackend.Libaio and IoBackend.Uring to the C# caller. Loaded when liburing2 is present on the host.
2. libnative_device_libaio.so — built with USE_URING=OFF, links libaio only. Exposes IoBackend.Libaio. Loaded automatically as a fallback when liburing2 is not present on the host so the libaio backend (the default) still works without any extra setup.

The C# NativeStorageDevice loader tries libnative_device.so first and silently falls back to libnative_device_libaio.so if the dynamic linker reports liburing.so.2: cannot open shared object file. The libaio backend always works out of the box on any Linux distribution that ships libaio (essentially all of them).

If a caller explicitly selects IoBackend.Uring on a host without liburing installed, construction throws a TsavoriteException with the install command for the user's distribution (apt-get install -y liburing2, dnf install -y liburing, apk add liburing). We never silently downgrade Uring to Libaio.

# libaio (always required — both shipped binaries link it):
#   Debian / Ubuntu 24.04+: libaio1t64
#   Older Debian / Ubuntu:  libaio1
#   Fedora / RHEL / AzureLinux: libaio
#   Alpine: libaio

# liburing (optional — only needed if you want IoBackend.Uring):
#   Debian / Ubuntu: liburing2
#   Fedora / RHEL / AzureLinux: liburing
#   Alpine: liburing

Windows users do not need any of this; the shipped native_device.dll uses the Windows ThreadPool API and has no native runtime dependencies beyond the standard CRT.

Building on Windows

The native device builds with MSVC (Visual Studio 2022, Visual Studio 2026 Insiders, or the standalone Build Tools). You need the Desktop development with C++ workload installed, plus the Spectre-mitigated CRT libraries (see prerequisites below).

1. Confirm CMake and Visual Studio are installed

CMake 3.21 or newer is required for the Visual Studio 2022 generator; 3.31 or newer is recommended for Visual Studio 2026 Insiders. Check with:

cmake --version

To list what Visual Studio installs CMake will discover:

& "${env:ProgramFiles(x86)}\Microsoft Visual Studio\Installer\vswhere.exe" -prerelease -products * -format json |
    Select-String -Pattern '"displayName"|"installationPath"|"installationVersion"'

Pre-release / Insiders editions only show up with the -prerelease flag — omitting it is the most common reason for a missing-VS error.

2. Install the Spectre-mitigated CRT libraries

Our CMakeLists.txt builds with /Qspectre /guard:cf /sdl (source-level Spectre v1 mitigation, Control Flow Guard, and Security Development Lifecycle checks) and the project requires the matching Spectre-mitigated CRT runtime libraries at link time. They are an optional component in the VS Installer:

1. Open the Visual Studio Installer, click Modify on your VS 2022 or VS 2026 Insiders install.
2. Switch to the Individual components tab.
3. Search for Spectre and check the entry that matches your toolset and architecture — e.g. MSVC v143 - VS 2022 C++ x64/x86 Spectre-mitigated libs (Latest) for VS 2022, or the equivalent v144 / v18 entry for VS 2026 Insiders.
4. Click Modify to install (about 200 MB).

Skipping this step will surface as error MSB8040: Spectre-mitigated libraries are required for this project during cmake --build.

3. Pick a generator

If cmake --help does not list your generator, your CMake is too old — upgrade with winget install Kitware.CMake or download a fresh build from https://cmake.org/download/.

4. Configure and build

Create a build directory under Tsavorite\cc and configure from there:

cd libs\storage\Tsavorite\cc
mkdir build
cd build

# Pick the line matching your VS edition:
cmake -G "Visual Studio 18 2026" -A x64 ..    # VS 2026 Insiders (current dev tooling)
cmake -G "Visual Studio 17 2022" -A x64 ..    # VS 2022 (any edition) / Build Tools 2022

cmake --build . --config Release

If configure/link fails to find the toolset or the Spectre-mitigated CRT (e.g. the generator defaults to a newer toolset than the one whose Spectre libs you installed — common with VS 2026 Insiders), select the MSVC toolset explicitly with -T. For example, to use the v143 (VS 2022) toolset:

cmake -G "Visual Studio 18 2026" -A x64 -T v143 ..   # force the v143 toolset

Use the toolset that matches your installed Spectre-mitigated libs (v143 for VS 2022, v144/v145 for VS 2026 Insiders).

That produces build\src\Release\native_device.dll (and a matching native_device.pdb). The CMake configuration also creates a Tsavorite.sln that you can open in Visual Studio for interactive development; both Debug and Release profiles are added for x64.

USE_URING is a no-op on Windows; the Windows DLL only exposes the Default (IOCP) backend.

Generator-agnostic fallback (Ninja)

If you have Build Tools without an IDE, an unsupported / preview VS version, or you just want a faster command-line build, use Ninja. Open the x64 Native Tools Command Prompt for VS <version> (the start-menu shortcut that ships with every VS install) and run:

cd libs\storage\Tsavorite\cc
mkdir build && cd build
cmake -G Ninja -DCMAKE_BUILD_TYPE=Release ..
cmake --build .

Ninja picks up cl.exe from the developer prompt automatically, so it bypasses the generator-string lookup entirely.

Building on Linux

1. Install the required packages

The native device links against aio (libaio) and, when USE_URING=ON (the default), uring (liburing). Plus the usual build toolchain.

On Debian / Ubuntu (including Ubuntu 24.04 with the t64 ABI transition):

sudo apt-get install build-essential cmake libaio-dev liburing-dev

On Fedora / RHEL / Azure Linux:

sudo dnf install gcc-c++ cmake libaio-devel liburing-devel

On Alpine (musl): the prebuilt won't load — Alpine images use the managed LocalStorageDevice instead.

CMake 3.21 or newer is required.

2. Configure and build

CMake on Linux generates scripts for either Debug or Release per build directory — run CMake twice, in two different directories, if you want both.

cd libs/storage/Tsavorite/cc
mkdir -p build/Release && cd build/Release
cmake -DCMAKE_BUILD_TYPE=Release ../..
make -j"$(nproc)"

Or, for a Debug build:

mkdir -p build/Debug && cd build/Debug
cmake -DCMAKE_BUILD_TYPE=Debug ../..
make -j"$(nproc)"

The output is build/<config>/libnative_device.so. The build links libaio (always) and, when USE_URING=ON (the default), also links liburing — the resulting binary exposes the Libaio and (if USE_URING=ON) Uring backends to the C# caller via the IoBackend enum (--device-io-backend libaio|uring). The Linux default is libaio; Uring must be explicitly selected.

The repo ships two prebuilt Linux binaries. To support hosts that may or may not have liburing2 installed, the repo ships both: libnative_device.so (built with USE_URING=ON, links libaio + liburing) and libnative_device_libaio.so (built with USE_URING=OFF, links libaio only). The C# loader tries the uring-enabled binary first and falls back to the libaio-only binary on liburing.so.2: cannot open. If you rebuild the prebuilts, rebuild both — once with -DUSE_URING=ON, once with -DUSE_URING=OFF — and copy each over the corresponding file in runtimes/linux-x64/native/ (see the "Updating the shipped prebuilt binaries" section below for the rebuild script).

Ubuntu 24.04 (t64 ABI) note. Since the t64 transition the system ships libaio.so.1t64 instead of libaio.so.1. CMake's -laio will find /usr/lib/x86_64-linux-gnu/libaio.so which is a symlink to the .1t64.0.2 file, so the build itself succeeds — but the resulting binary records libaio.so.1t64 in its NEEDED list. To restore the portable name, run patchelf --replace-needed libaio.so.1t64 libaio.so.1 libnative_device.so after the build (see the prebuilt update section below).

Disabling io_uring at build time (optional)

If you want a build that does not have a runtime dependency on liburing at all (e.g., for a constrained image where liburing is unavailable), pass -DUSE_URING=OFF to cmake:

cmake -DCMAKE_BUILD_TYPE=Release -DUSE_URING=OFF ../..

The resulting binary links only libaio. Requesting the Uring backend at runtime against this build is rejected at construction with the same clear exception (pointing the user at the install command for liburing2). This is the build flavour that ships as libnative_device_libaio.so.

Updating the shipped prebuilt binaries

The C# project copies the prebuilt native libraries from libs/storage/Tsavorite/cs/src/core/Device/runtimes/<rid>/native/ into the output directory at build time. After rebuilding, copy the new artifact over the corresponding file in that directory and check it in.

Linux (linux-x64):

# from libs/storage/Tsavorite/cc, after a Release build with USE_URING=ON:
cp build/Release/libnative_device.so \
   ../cs/src/core/Device/runtimes/linux-x64/native/libnative_device.so

The shipped Linux binary records libaio.so.1 and liburing.so.2 as its NEEDED entries. On Ubuntu 24.04 and later (post-t64 ABI transition) the linker records libaio.so.1t64 instead; the Dockerfiles in this repo install libaio1t64 and create a libaio.so.1 -> libaio.so.1t64 compat symlink, but we keep the checked-in NEEDED name portable by patching it:

patchelf --replace-needed libaio.so.1t64 libaio.so.1 \
   ../cs/src/core/Device/runtimes/linux-x64/native/libnative_device.so

Windows (win-x64):

A VS generator places the binary under src\Release\ (or src\Debug\); a Ninja build places it under src\ directly. Adjust the source path accordingly:

:: from libs\storage\Tsavorite\cc, after a Release build with the VS generator:
copy /Y build\src\Release\native_device.dll ^
   ..\cs\src\core\Device\runtimes\win-x64\native\
copy /Y build\src\Release\native_device.pdb ^
   ..\cs\src\core\Device\runtimes\win-x64\native\

To verify that the new entrypoints are present in the rebuilt binary:

# Linux
nm -D --defined-only libnative_device.so | grep NativeDevice_

:: Windows
dumpbin /exports native_device.dll | findstr NativeDevice_

You should see NativeDevice_CreateWithBackend and NativeDevice_AvailableBackends in addition to the legacy entrypoints.

Other options

You can try other generators (compilers) supported by CMake. The main CMake build script is the CMakeLists.txt located in the root directory (Tsavorite/cc).