refactor(pathfinder): descriptor-driven library discovery and loading

[cpcloud]

Summary

Library metadata was scattered across parallel hand-maintained tables, loader internals, and module-level constants. This made it easy for policy to drift between search and load paths, hard to review changes to individual libraries, and fragile to extend. This PR consolidates all of that into a single descriptor model and restructures the search/load pipeline around it.

Why

• Drift risk: library behavior was assembled from multiple overlapping sources (supported_nvidia_libs tables, inline constants, per-platform branching). Changing one without the others caused subtle bugs.
• Review friction: understanding what a library needs at load time required reading across several modules. Now it's one catalog entry.
• Canary fragility: CTK-root canary fallback (critical for nvvm on non-standard installs) was wired through globals that the refactor initially dropped. This PR restores it as descriptor-configured behavior.
• Weak test coverage: some catalog tests were tautological (comparing data to itself). Replaced with structural invariants that catch real errors.

Change progression

1. Unify library policy in descriptors — introduced LibDescriptor and an authored catalog as the single metadata source, then derived runtime and legacy compatibility tables from it.
2. Decouple orchestration from platform specifics — refactored discovery/loading into composable search steps with explicit seams (SearchPlatform, PlatformLoader) so OS-specific behavior is localized and the cascade is auditable.
3. Stabilize during migration — threaded descriptors through loader internals, fixed edge cases in search/load wiring, cleaned up dead code.
4. Restore CTK canary fallback — reintroduced canary-based CTK-root discovery in the refactored flow, with anchor policy on the descriptor and probing subprocess-isolated.
5. Harden correctness guardrails — replaced tautological catalog tests with structural invariants; added focused fallback coverage.

Functional outcome

• Public API unchanged (load_nvidia_dynamic_lib).
• Driver libraries remain system-search-only.
• CTK canary fallback preserved, now descriptor-configured.
• Adding or modifying a library is now a single catalog entry change.

Made with Cursor

[copy-pr-bot]

Auto-sync is disabled for ready for review pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

[cpcloud]

Recommend going per-commit on this one.

[cpcloud]

/ok to test

[github-actions]

Doc Preview CI
🚀 View preview at https://nvidia.github.io/cuda-python/pr-preview/pr-1685/
https://nvidia.github.io/cuda-python/pr-preview/pr-1685/cuda-core/
https://nvidia.github.io/cuda-python/pr-preview/pr-1685/cuda-bindings/
https://nvidia.github.io/cuda-python/pr-preview/pr-1685/cuda-pathfinder/
Preview will be ready when the GitHub Pages deployment is complete.

[rwgk]

Library metadata was scattered across parallel hand-maintained tables, loader internals, and module-level constants.

That's not entirely fair: there is an existing very large degree of centralization of config-like information in supported_nvidia_libs.py.

Essentially, this PR is doing the equivalent of a matrix transposition (row-major vs column-major for the matrix case / libname-oriented (new) vs packaging/platform-oriented (old)). That has pros-and-cons either way.

My main high-level concern: SUPPORTED_LINUX_SONAMES_CTK and SUPPORTED_WINDOWS_DLLS_CTK are generated from scratch starting from extracted CTK distributions, with simple toolshed scripts. After this PR, that'll be a lot more difficult. I think it'd be fine if we essentially kept those two dicts as source of truth, and populate into the descriptors. The non-ctk libs are fine purely descriptor-based.

[cpcloud]

My main high-level concern: SUPPORTED_LINUX_SONAMES_CTK and SUPPORTED_WINDOWS_DLLS_CTK are generated from scratch starting from extracted CTK distributions, with simple toolshed scripts.

I'd rather keep a single source of truth. That is one of two main goals of this PR besides cleaning up all the comingled search and platform logic.

We can adapt the toolshed scripts to the new layout.

[cpcloud]

Just to note a few things:

• The old layout is derived and still exported. Downstream consumers should require zero code changes.
• Changing scripts is basically a one time cost.
• Two sources of truth is worse than one, it's a desync surface and defeats the purpose of a big design aspect of this PR.
• Descriptors are easier for the script to target.

I'll just adapt the scripts in this PR.

[rwgk]

I'll just adapt the scripts in this PR.

That works. I assumed it's difficult/messy.

Two sources of truth is worse than one

Totally, I was suggesting one source of truth, just with one level of indirection, so that we can still easily auto-populate the equivalent of SUPPORTED_LINUX_SONAMES_CTK and SUPPORTED_WINDOWS_DLLS_CTK. If you can achieve that without the indirection, even better.

The indirection I had in mind, roughly:

SUPPORTED_LINUX_SONAMES_CTK = ...
SUPPORTED_WINDOWS_DLLS_CTK = ...
...
    DescriptorSpec(
        name="cudart",
        strategy="ctk",
        linux_sonames=SUPPORTED_LINUX_SONAMES_CTK["cudart"],
        windows_dlls=SUPPORTED_WINDOWS_DLLS_CTK["cudart"],
        site_packages_linux=("nvidia/cu13/lib", "nvidia/cuda_runtime/lib"),
        site_packages_windows=("nvidia/cu13/bin/x86_64", "nvidia/cuda_runtime/bin"),
    ),

You could even hide that behind a DescriptorSpecCtk helper function.

BTW (because I'm copy-pasting that code): Could you please replace strategy with e.g. packaged_with or similar? That's really what it's about: packaged with ctk, or driver, or other. Another idea would be directory_layout, or directory_structure.

[cpcloud]

Could you please replace strategy with e.g. packaged_with or similar? That's really what it's about: packaged with ctk, or driver, or other. Another idea would be directory_layout, or directory_structure.

Yep. I like that much better. strategy is a bit too generic.