Environment
- Workload: NVIDIA Isaac Sim 4.x / 5.x (with OptiX Denoiser enabled)
- vGPU Mechanism: API Interception (
libvgpu.so hooking CUDA driver calls)
Issue Description
When running NVIDIA Isaac Sim inside a vGPU-enabled container, the process crashes with Segmentation fault (core dumped) (exit code 139) shortly after app ready when OptiX denoiser (librtx.optixdenoising.plugin.so) is initialized.
The crash backtrace shows that libvgpu.so encounters a null pointer dereference inside cuMemGetInfo_v2:
Thread backtrace:
000: libc.so.6!__sigaction+0x50
001: libvgpu.so!cuMemGetInfo_v2+0xd2 <-- Crash inside vGPU interception hook
002: libnvoptix.so.1!rtGetSymbolTable+0x14646d
003: librtx.optixdenoising.plugin.so!carbOnPluginPreStartup+0x3d9f
It seems OptiX calls cuMemGetInfo_v2 from its internal rendering threads, which passes a context descriptor that the vGPU hook library fails to parse safely.
Workaround
We mitigated this at the application level using an LD_PRELOAD C library that bypasses libvgpu.so for cuMemGetInfo_v2 and forwards calls directly to host native libcuda.so.1:
#define _GNU_SOURCE
#include <stdio.h>
#include <dlfcn.h>
typedef int (*pfn_cuMemGetInfo_v2)(size_t *free_mem, size_t *total_mem);
int cuMemGetInfo_v2(size_t *free_mem, size_t *total_mem) {
static pfn_cuMemGetInfo_v2 real_fn = NULL;
if (!real_fn) {
void *h = dlopen("libcuda.so.1", RTLD_LAZY | RTLD_LOCAL);
if (!h) h = dlopen("libcuda.so", RTLD_LAZY | RTLD_LOCAL);
if (h) real_fn = (pfn_cuMemGetInfo_v2)dlsym(h, "cuMemGetInfo_v2");
}
if (real_fn) {
int res = real_fn(free_mem, total_mem);
if (res == 0 && free_mem && *free_mem > 0) return 0;
}
// Fallback safe values
if (free_mem) *free_mem = (size_t)20 * 1024 * 1024 * 1024ULL;
if (total_mem) *total_mem = (size_t)24 * 1024 * 1024 * 1024ULL;
return 0;
}
int cuMemGetInfo(size_t *free_mem, size_t *total_mem) {
return cuMemGetInfo_v2(free_mem, total_mem);
}Question
Does HAMi-core currently have context safety guards in cuMemGetInfo_v2 to handle calls originated from OptiX threads? Any insights or upstream fixes would be greatly appreciated!
Environment
libvgpu.sohooking CUDA driver calls)Issue Description
When running NVIDIA Isaac Sim inside a vGPU-enabled container, the process crashes with
Segmentation fault (core dumped)(exit code 139) shortly afterapp readywhen OptiX denoiser (librtx.optixdenoising.plugin.so) is initialized.The crash backtrace shows that
libvgpu.soencounters a null pointer dereference insidecuMemGetInfo_v2:It seems OptiX calls
cuMemGetInfo_v2from its internal rendering threads, which passes a context descriptor that the vGPU hook library fails to parse safely.Workaround
We mitigated this at the application level using an
LD_PRELOADC library that bypasseslibvgpu.soforcuMemGetInfo_v2and forwards calls directly to host nativelibcuda.so.1:Question
Does HAMi-core currently have context safety guards in
cuMemGetInfo_v2to handle calls originated from OptiX threads? Any insights or upstream fixes would be greatly appreciated!