Fix native device loading on Debian 13 / Ubuntu 24.04 (libaio t64 transition)

The t64 ABI transition renamed libaio.so.1 to libaio.so.1t64, breaking
libnative_device.so which has a hard DT_NEEDED of libaio.so.1. Previously
we worked around this with system-wide symlinks in every Dockerfile and CI
workflow.

Fix this properly in the loader itself:

- CMakeLists.txt now sets RPATH=$ORIGIN (via INSTALL_RPATH +
  BUILD_WITH_INSTALL_RPATH + --disable-new-dtags) so libnative_device.so
  searches its own directory for dependencies. This lets a managed-side
  compat symlink next to the native library satisfy the linker without
  any LD_LIBRARY_PATH contortions from the caller.

- libaio_compat.h (new) pins the libaio entry points to the specific
  symbol versions that make libaio's userspace fast paths kick in:
    io_setup    @LIBAIO_0.4
    io_destroy  @LIBAIO_0.4
    io_getevents@LIBAIO_0.4  (userspace ring fast path)
    io_submit   @LIBAIO_0.1
  Older libaio-dev marked LIBAIO_0.4 as the default version so a plain
  link picked these up automatically. On t64 (libaio1t64-dev) the default
  is gone and libaio.h has no .symver redirects for x86_64, so a fresh
  link produces UNVERSIONED references that at runtime resolve to the
  slower LIBAIO_0.1 io_getevents which always syscalls and blocks -
  which caused NativeStorageDevice probe/TryComplete paths to hang.

- NativeStorageDevice.ImportResolver now resolves NativeLibraryPath to
  an absolute path (fixing a latent bug where the relative path
  bypassed .NET's runtimes/ probing) and, on Linux, catches
  DllNotFoundException referencing libaio.so.1, locates
  libaio.so.1t64 in standard multiarch paths, and drops a compat
  symlink next to libnative_device.so. The symlink creation tolerates
  the race where multiple processes start simultaneously and another
  process has already created a usable symlink. If repair still fails,
  the loader throws a descriptive DllNotFoundException explaining the
  t64 transition and offering three remediation options.

- VectorManager.Initialize() and ResumePostRecovery() now early-return
  when IsEnabled is false. Vector Set preview is off by default; there
  is no reason these paths should touch storage when the feature is
  disabled.

With the loader + build fixes in place, remove the now-redundant
workarounds:

- Dockerfile and Dockerfile.ubuntu: drop the ln -sf libaio.so.1 line.
  (Dockerfile.alpine and Dockerfile.azurelinux ship libaio.so.1 natively.
  Dockerfile.chiseled uses a restricted runtime and was not touched.)

- .github/workflows/ci.yml and nightly.yml: drop the ubuntu-latest
  libaio pre-step; the managed ImportResolver now handles repair
  automatically and the test suite actually exercises the repair path.

- validate_docker_images.py: accept either libaio.so.1 or
  libaio.so.1t64, since the former is only materialized lazily (on
  first native device init) for glibc images now.

The bundled libnative_device.so has been rebuilt against the above
sources with '-O3 -g -DNDEBUG' (project Release defaults). Verified
via objdump -T that io_* references are correctly versioned.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Author: badrishc

.github/workflows/ci.yml

@@ -170,10 +170,6 @@ jobs:
     steps:
       - name: Check out code
         uses: actions/checkout@v4
-      - name: Set workaround for libaio on Ubuntu 24.04 (see https://askubuntu.com/questions/1512196/libaio1-on-noble/1512197#1512197)
-        run: |
-          sudo ln -s /usr/lib/x86_64-linux-gnu/libaio.so.1t64 /usr/lib/x86_64-linux-gnu/libaio.so.1
-        if: ${{ matrix.os == 'ubuntu-latest' }}
       - name: Set environment variable for Linux
         run: echo "RunAzureTests=yes" >> $GITHUB_ENV
         if: ${{ matrix.os == 'ubuntu-latest' }}

.github/workflows/nightly.yml

@@ -43,11 +43,6 @@ jobs:
       - name: Check out code
         uses: actions/checkout@v4
 
-      - name: Set workaround for libaio on Ubuntu 24.04 (see https://askubuntu.com/questions/1512196/libaio1-on-noble/1512197#1512197)
-        run: |
-           sudo ln -s /usr/lib/x86_64-linux-gnu/libaio.so.1t64 /usr/lib/x86_64-linux-gnu/libaio.so.1
-        if: ${{ matrix.os == 'ubuntu-latest' }}
-
       - name: Set environment variable for Linux
         run: echo "RunAzureTests=yes" >> $GITHUB_ENV
         if: ${{ matrix.os == 'ubuntu-latest' }}

Dockerfile

@@ -45,7 +45,6 @@ RUN apt-get update \
        liblua5.4-0 \
     && ARCH="$(uname -m)" \
     && case "$ARCH" in x86_64) MULTIARCH="x86_64-linux-gnu";; aarch64) MULTIARCH="aarch64-linux-gnu";; *) MULTIARCH="$ARCH-linux-gnu";; esac \
-    && ln -sf "/usr/lib/${MULTIARCH}/libaio.so.1t64" "/usr/lib/${MULTIARCH}/libaio.so.1" \
     && DN_DIR=$(ls -d /usr/share/dotnet/shared/Microsoft.NETCore.App/* 2>/dev/null | head -n1 || true) \
     && if [ -n "$DN_DIR" ]; then ln -sf "/usr/lib/${MULTIARCH}/liblua5.4.so.0" "$DN_DIR/liblua54.so"; fi \
     && rm -rf /var/lib/apt/lists/*

Dockerfile.ubuntu

@@ -46,7 +46,6 @@ RUN apt-get update \
        liblua5.4-0 \
     && ARCH="$(uname -m)" \
     && case "$ARCH" in x86_64) MULTIARCH="x86_64-linux-gnu";; aarch64) MULTIARCH="aarch64-linux-gnu";; *) MULTIARCH="$ARCH-linux-gnu";; esac \
-    && ln -sf "/usr/lib/${MULTIARCH}/libaio.so.1t64" "/usr/lib/${MULTIARCH}/libaio.so.1" \
     && DN_DIR=$(ls -d /usr/share/dotnet/shared/Microsoft.NETCore.App/* 2>/dev/null | head -n1 || true) \
     && if [ -n "$DN_DIR" ]; then ln -sf "/usr/lib/${MULTIARCH}/liblua5.4.so.0" "$DN_DIR/liblua54.so"; fi \
     && rm -rf /var/lib/apt/lists/*

libs/server/Resp/Vector/VectorManager.cs

@@ -116,6 +116,8 @@ public VectorManager(int dbId, GarnetServerOptions serverOptions, Func<IMessageC
         /// </summary>
         public void Initialize()
         {
+            if (!IsEnabled) return;
+
             using var session = (RespServerSession)getCleanupSession();
             if (session.activeDbId != dbId && !session.TrySwitchActiveDatabaseSession(dbId))
             {
@@ -153,6 +155,8 @@ public void Initialize()
         /// </summary>
         public void ResumePostRecovery()
         {
+            if (!IsEnabled) return;
+
             using var session = (RespServerSession)getCleanupSession();
 
             ref var ctx = ref session.storageSession.vectorBasicContext;

libs/storage/Tsavorite/cc/src/CMakeLists.txt

@@ -26,6 +26,7 @@ set (NATIVE_DEVICE_HEADERS ${NATIVE_DEVICE_HEADERS}
 else()
 set (FNATIVE_DEVICE_HEADERS ${NATIVE_DEVICE_HEADERS}
   device/file_linux.h
+  device/libaio_compat.h
 )
 endif() 
 
@@ -48,4 +49,15 @@ endif()
 add_library(native_device SHARED ${NATIVE_DEVICE_SOURCES} ${NATIVE_DEVICE_HEADERS})
 if (UNIX)
    target_link_libraries(native_device PRIVATE stdc++fs aio)
+   # Search the library's own directory for DT_NEEDED dependencies. This lets the C# loader drop
+   # a compat symlink (e.g., libaio.so.1 -> libaio.so.1t64 for the Debian/Ubuntu t64 ABI
+   # transition) next to libnative_device.so and have the dynamic linker pick it up without
+   # requiring LD_LIBRARY_PATH to be set by the caller. Use RPATH (DT_RPATH) instead of RUNPATH
+   # because RPATH is searched before LD_LIBRARY_PATH and applies transitively to all dependency
+   # lookups, whereas RUNPATH only applies to the direct library and is overridden by
+   # LD_LIBRARY_PATH.
+   set_target_properties(native_device PROPERTIES
+     INSTALL_RPATH "$ORIGIN"
+     BUILD_WITH_INSTALL_RPATH TRUE)
+   target_link_options(native_device PRIVATE "LINKER:--disable-new-dtags")
 endif()

libs/storage/Tsavorite/cc/src/device/file_linux.h

@@ -7,6 +7,7 @@
 #include <cstdint>
 #include <string>
 #include <libaio.h>
+#include "libaio_compat.h"
 #include <sys/types.h>
 #include <sys/stat.h>
 #include <unistd.h>

libs/storage/Tsavorite/cc/src/device/libaio_compat.h

@@ -0,0 +1,64 @@
+// Copyright (c) Microsoft Corporation.
+// Licensed under the MIT license.
+
+// Force specific versioned libaio symbols at link time.
+//
+// Background: libaio.so exports multiple versions of the core AIO entry points:
+//   LIBAIO_0.1 - original syscall-only ABI
+//   LIBAIO_0.4 - adds a userspace ring-buffer fast path for io_getevents
+//   LIBAIO_0.5 / LIBAIO_0.6 - later additions
+//
+// On older distros libaio-dev shipped a library that marked LIBAIO_0.4 as the
+// default version, so linking `-laio` recorded `io_getevents@LIBAIO_0.4` etc.
+// automatically. Starting with the t64 ABI transition (Debian 13 / Ubuntu
+// 24.04+), libaio1t64 no longer advertises a default version for these
+// symbols, and libaio.h has no `.symver` redirects for x86_64, so a fresh
+// link produces UNVERSIONED references. At runtime the dynamic linker then
+// picks the first matching definition - LIBAIO_0.1, whose `io_getevents`
+// always goes through the syscall and blocks until `min_nr` events arrive.
+// That caused Garnet's NativeStorageDevice probe/TryComplete paths to hang
+// indefinitely on an empty context when they expected the LIBAIO_0.4 fast
+// path to return 0 immediately.
+//
+// To keep rebuilds correct regardless of which distro they happen on, we
+// pin the exact versions we want:
+//   io_setup    @LIBAIO_0.4
+//   io_destroy  @LIBAIO_0.4
+//   io_getevents@LIBAIO_0.4
+//   io_submit   @LIBAIO_0.1
+// These match the versions that the original (pre-t64) build produced and
+// that Garnet has shipped against for years.
+//
+// The mechanism: declare aliases via `.symver`, then #define the unadorned
+// names to route to those aliases. This has no runtime cost; the resulting
+// binary simply records versioned UND references.
+//
+// IMPORTANT: Include this header before any use of io_setup / io_destroy /
+// io_getevents / io_submit. (file_linux.h already includes <libaio.h> then
+// this header, so including file_linux.h is sufficient.)
+
+#pragma once
+
+#ifdef __linux__
+
+#include <libaio.h>
+
+__asm__(".symver io_setup_0_4, io_setup@LIBAIO_0.4");
+__asm__(".symver io_destroy_0_4, io_destroy@LIBAIO_0.4");
+__asm__(".symver io_getevents_0_4, io_getevents@LIBAIO_0.4");
+__asm__(".symver io_submit_0_1, io_submit@LIBAIO_0.1");
+
+extern "C" {
+int io_setup_0_4(int maxevents, io_context_t* ctxp);
+int io_destroy_0_4(io_context_t ctx);
+int io_getevents_0_4(io_context_t ctx, long min_nr, long nr,
+                     struct io_event* events, struct timespec* timeout);
+int io_submit_0_1(io_context_t ctx, long nr, struct iocb** iocbs);
+}
+
+#define io_setup    io_setup_0_4
+#define io_destroy  io_destroy_0_4
+#define io_getevents io_getevents_0_4
+#define io_submit   io_submit_0_1
+
+#endif // __linux__

libs/storage/Tsavorite/cs/src/core/Device/NativeStorageDevice.cs

@@ -54,10 +54,179 @@ static NativeStorageDevice()
 
         static IntPtr ImportResolver(string libraryName, Assembly assembly, DllImportSearchPath? searchPath)
         {
-            IntPtr libHandle = IntPtr.Zero;
-            if (libraryName == NativeLibraryName && NativeLibraryPath != null)
-                libHandle = NativeLibrary.Load(NativeLibraryPath);
-            return libHandle;
+            if (libraryName != NativeLibraryName || NativeLibraryPath == null)
+                return IntPtr.Zero;
+
+            var resolvedPath = ResolveNativeLibraryPath(assembly);
+
+            try
+            {
+                return NativeLibrary.Load(resolvedPath);
+            }
+            catch (DllNotFoundException ex) when (RuntimeInformation.IsOSPlatform(OSPlatform.Linux)
+                                                  && ex.Message.Contains("libaio.so.1", StringComparison.Ordinal))
+            {
+                // Debian 13 (trixie) / Ubuntu 24.04 (noble) renamed libaio1 to libaio1t64 as part of the
+                // 64-bit time_t ABI transition. The package now ships libaio.so.1t64 (SONAME "libaio.so.1t64"),
+                // which does NOT satisfy our DT_NEEDED of "libaio.so.1". Try to repair by dropping a
+                // libaio.so.1 -> libaio.so.1t64 symlink next to libnative_device.so; this works because the
+                // native library is built with RPATH=$ORIGIN.
+                if (TryCreateLibaioCompatSymlink(resolvedPath, out var symlinkedPath))
+                {
+                    try
+                    {
+                        return NativeLibrary.Load(resolvedPath);
+                    }
+                    catch (DllNotFoundException)
+                    {
+                        // Fall through to the detailed error below.
+                    }
+                }
+
+                throw new DllNotFoundException(BuildLibaioDiagnostic(symlinkedPath, ex), ex);
+            }
+        }
+
+        /// <summary>
+        /// Resolve NativeLibraryPath (which is a NuGet-style "runtimes/&lt;rid&gt;/native/&lt;lib&gt;" relative
+        /// path) to an absolute filesystem path. We probe (in order) the assembly's own directory, the
+        /// application's base directory, and finally the current working directory. Falls back to the
+        /// raw relative path if none of these exist, so dlopen's error message surfaces as before.
+        /// </summary>
+        static string ResolveNativeLibraryPath(Assembly assembly)
+        {
+            string[] searchRoots =
+            [
+                Path.GetDirectoryName(assembly?.Location),
+                AppContext.BaseDirectory,
+                Directory.GetCurrentDirectory(),
+            ];
+
+            foreach (var root in searchRoots)
+            {
+                if (string.IsNullOrEmpty(root))
+                    continue;
+                var candidate = Path.Combine(root, NativeLibraryPath);
+                if (File.Exists(candidate))
+                    return Path.GetFullPath(candidate);
+            }
+
+            return NativeLibraryPath;
+        }
+
+        /// <summary>
+        /// Candidate paths for libaio.so.1t64 on Debian/Ubuntu multiarch layouts. These match what
+        /// libaio1t64 installs on amd64 and arm64; add more here if additional architectures appear.
+        /// </summary>
+        static readonly string[] LibaioT64CandidatePaths =
+        [
+            "/usr/lib/x86_64-linux-gnu/libaio.so.1t64",
+            "/usr/lib/aarch64-linux-gnu/libaio.so.1t64",
+            "/lib/x86_64-linux-gnu/libaio.so.1t64",
+            "/lib/aarch64-linux-gnu/libaio.so.1t64",
+            "/usr/lib64/libaio.so.1t64",
+            "/usr/lib/libaio.so.1t64",
+        ];
+
+        /// <summary>
+        /// Locate libaio.so.1t64 and create a libaio.so.1 symlink next to libnative_device.so so that
+        /// the dynamic linker (searching RPATH=$ORIGIN) can satisfy the DT_NEEDED entry. Returns true
+        /// when after the call a usable symlink exists at the expected path - whether we created it or
+        /// a concurrently-starting process did. Sets <paramref name="createdSymlink"/> to the link path
+        /// in that case.
+        /// </summary>
+        static bool TryCreateLibaioCompatSymlink(string resolvedNativeLibraryPath, out string createdSymlink)
+        {
+            createdSymlink = null;
+
+            string t64Path = null;
+            foreach (var candidate in LibaioT64CandidatePaths)
+            {
+                if (File.Exists(candidate))
+                {
+                    t64Path = candidate;
+                    break;
+                }
+            }
+            if (t64Path == null)
+                return false;
+
+            string shimPath;
+            try
+            {
+                var nativeDir = Path.GetDirectoryName(Path.GetFullPath(resolvedNativeLibraryPath));
+                if (string.IsNullOrEmpty(nativeDir) || !Directory.Exists(nativeDir))
+                    return false;
+
+                shimPath = Path.Combine(nativeDir, "libaio.so.1");
+            }
+            catch (Exception)
+            {
+                return false;
+            }
+
+            try
+            {
+                File.CreateSymbolicLink(shimPath, t64Path);
+                createdSymlink = shimPath;
+                return true;
+            }
+            catch (IOException)
+            {
+                // Either a concurrently-starting process already created the symlink (common in
+                // container fleets where multiple Garnet instances share an image), or a stale file
+                // of the same name is present. If it's a symlink resolving to libaio.so.1t64, treat
+                // that as success; otherwise fall through to the diagnostic error.
+                if (IsUsableLibaioShim(shimPath))
+                {
+                    createdSymlink = shimPath;
+                    return true;
+                }
+                return false;
+            }
+            catch (Exception)
+            {
+                return false;
+            }
+        }
+
+        /// <summary>
+        /// Returns true if <paramref name="shimPath"/> is an existing symlink that points to a
+        /// libaio.so.1t64 file (possibly via relative or absolute target).
+        /// </summary>
+        static bool IsUsableLibaioShim(string shimPath)
+        {
+            try
+            {
+                var info = new FileInfo(shimPath);
+                if (!info.Exists) return false;
+                var target = info.LinkTarget;
+                if (string.IsNullOrEmpty(target)) return false;
+                // LinkTarget can be a relative path (e.g., just "libaio.so.1t64"); accept either.
+                return target.EndsWith("libaio.so.1t64", StringComparison.Ordinal);
+            }
+            catch
+            {
+                return false;
+            }
+        }
+
+        static string BuildLibaioDiagnostic(string attemptedSymlinkPath, Exception inner)
+        {
+            var arch = RuntimeInformation.ProcessArchitecture == Architecture.Arm64
+                ? "aarch64-linux-gnu" : "x86_64-linux-gnu";
+            var attempted = attemptedSymlinkPath == null
+                ? "Could not find libaio.so.1t64 in standard multiarch paths; auto-repair skipped."
+                : $"Attempted to create '{attemptedSymlinkPath}' -> libaio.so.1t64 but the load still failed.";
+            return
+                $"Failed to load native storage device library '{NativeLibraryPath}' because its dependency 'libaio.so.1' " +
+                "is not resolvable by the dynamic linker. This typically happens on Debian 13 (trixie) or " +
+                "Ubuntu 24.04 (noble) where the libaio1 package was renamed to libaio1t64 (64-bit time_t ABI " +
+                "transition) and only ships 'libaio.so.1t64'. " + attempted + " " +
+                "To fix, either (a) install the legacy-named package if available for your distro, " +
+                $"(b) as root, create the compat symlink: sudo ln -s /usr/lib/{arch}/libaio.so.1t64 /usr/lib/{arch}/libaio.so.1, " +
+                "or (c) switch to the managed device by setting '--device-type Managed' (or removing '--use-native-device-linux'). " +
+                "Original loader error: " + inner.Message;
         }
 
         /// <summary>