Skip to content

fix: probe CUDA context on correct device in set_task_pid()#230

Open
maverick123123 wants to merge 1 commit into
Project-HAMi:mainfrom
maverick123123:fix/set-task-pid-probe-device-mismatch
Open

fix: probe CUDA context on correct device in set_task_pid()#230
maverick123123 wants to merge 1 commit into
Project-HAMi:mainfrom
maverick123123:fix/set-task-pid-probe-device-mismatch

Conversation

@maverick123123

Copy link
Copy Markdown
Contributor

Problem

set_task_pid() in src/utils.c polls NVML processes on the first physical device where nvml_to_cuda_map(i) >= 0, but the probe context is always created on hardcoded CUDA device 0 via cuDevicePrimaryCtxRetain(&pctx, 0).

When CUDA_VISIBLE_DEVICES does not start with the pod's device 0 (e.g., CVD=1 or CVD=1,0), the polled NVML device and the probe context device are different physical GPUs. getextrapid() never finds the new PID, causing a spurious "host pid is error!" and subsequent fake Device 0 OOM due to incorrect memory accounting.

Fix

Save the cudaDev found in the NVML enumeration loop and use it for both cuDevicePrimaryCtxRetain and cuDevicePrimaryCtxRelease, ensuring the probe context runs on the same device being polled.

Also guard against the case where no NVML device maps to a CUDA device (probeDev remains -1) by returning early before retaining any context.

Testing

Verified on an 8x RTX 3090 machine with all CVD combinations:

CUDA_VISIBLE_DEVICES Before fix After fix
0 OK OK
1 host pid is error OK
0,1 OK OK
1,0 host pid is error OK

Related

Fixes #225

@hami-robot

hami-robot Bot commented Jul 23, 2026

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: maverick123123
Once this PR has been reviewed and has the lgtm label, please assign archlitchi for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@hami-robot
hami-robot Bot requested review from archlitchi and chaunceyjiang July 23, 2026 08:32
@coderabbitai

coderabbitai Bot commented Jul 23, 2026

Copy link
Copy Markdown

Warning

Review limit reached

@maverick123123, you've reached your PR review limit, so we couldn't start this review.

Next review available in: 10 minutes

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 284582d1-f21b-48e1-80db-d255aecceaca

📥 Commits

Reviewing files that changed from the base of the PR and between 52f33fc and ffcf6d7.

📒 Files selected for processing (1)
  • src/utils.c
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@maverick123123

Copy link
Copy Markdown
Contributor Author

@rootsongjc ready to merge

Comment thread src/utils.c Outdated
Comment thread src/utils.c
set_task_pid() polls NVML processes on the first device where
nvml_to_cuda_map(i) >= 0, but the probe context was always created
on hardcoded CUDA device 0 via cuDevicePrimaryCtxRetain(&pctx, 0).

When CUDA_VISIBLE_DEVICES does not start with the pod's device 0
(e.g., CVD=1 or CVD=1,0), the polled NVML device and the probe
context device are different physical GPUs. getextrapid() never
finds the new PID, causing a spurious 'host pid is error!' and
subsequent Device 0 OOM due to incorrect memory accounting.

Fix: use the cudaDev found in the NVML enumeration loop for both
cuDevicePrimaryCtxRetain and cuDevicePrimaryCtxRelease, ensuring
the probe context runs on the same device being polled.

Fixes Project-HAMi#225

Signed-off-by: maverick123123 <yuming.wu@dynamia.ai>
@maverick123123
maverick123123 force-pushed the fix/set-task-pid-probe-device-mismatch branch from 01c7c41 to ffcf6d7 Compare July 23, 2026 09:05
@hami-robot hami-robot Bot added size/M and removed size/S labels Jul 23, 2026
@mesutoezdil

Copy link
Copy Markdown
Contributor

/lgtm

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

2 participants