Skip to content

fix the error of illegal memory access caused by cub#10

Open
ShunChengWu wants to merge 1 commit into
graphdeco-inria:mainfrom
ShunChengWu:main
Open

fix the error of illegal memory access caused by cub#10
ShunChengWu wants to merge 1 commit into
graphdeco-inria:mainfrom
ShunChengWu:main

Conversation

@ShunChengWu

Copy link
Copy Markdown

Fix an issue caused by cub.
From my own test, the problem happened when you call cub related functions, such as InclusiveSum Here https://github.com/graphdeco-inria/diff-gaussian-rasterization/blob/main/cuda_rasterizer/rasterizer_impl.cu#L281.
The problem is that the copy process does not go all the way. From my test, the copy somehow stopped at around 80K. After I tried to bypass that function by using std::inclusive_scan. However, illegal memory access happened somewhere in the backward pass.
Afterward, I found that a similar issue kind of happened in custom-built Pytorch functions (NVIDIA/thrust#1341) and found the solution here pytorch/pytorch#52663 (comment).

After the test, I used the modified code to train Gaussian Splatting on the Blender Lego dataset, and it worked like a charm.

Related issues:

@hyqlat

This comment was marked as abuse.

@N-Demir

N-Demir commented Feb 28, 2025

Copy link
Copy Markdown

Strangely it didn't work for me either. Potentially it matters what cuda/torch version you're on? I've tried cuda11.8/torch1.13 and cuda12.4/torch2.4. And this is for using the 3dgs_accel branch

update: I had just messed up some code!

@HenryLyu

Copy link
Copy Markdown

Thank you very much! Your solution is awesome

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants