Does llama.cpp ACTUALLY support pipeline parallelism? #20252
Replies: 5 comments 24 replies
-
|
Yes it is supported. You can read more about how it works in #6017. If you configure it correctly, the PP performance scales nearly linear with the number of devices, even for single request. |
Beta Was this translation helpful? Give feedback.
-
|
Thanks for the reply. In llama-batched-bench
This matches the performance I see in llama-server Full log -ub 512 -b 512
Full log -ub 512 -b 1024
|
Beta Was this translation helpful? Give feedback.
-
|
Hm not sure. Try to run the |
Beta Was this translation helpful? Give feedback.
-
|
I ran the test and saw essentially no difference. I should be seeing some improvement here, right?
I tested with a 24B model (on 2 and 3 gpus) also and saw a similar lack of improvement. |
Beta Was this translation helpful? Give feedback.
-
|
I just tried this on 4xA40 GPUs, and I can see good scaling. |
Beta Was this translation helpful? Give feedback.
-
|
Thanks for testing that. Seems related to my hardware. I've been looking into p2p which they supposedly support, might be causing issues if it's not working properly. |
Beta Was this translation helpful? Give feedback.
-
|
Yes, this was a dual socket system. But I have tested on single socket systems too in the past, and it worked fine. I don't have access to P40s, so I can't test. Can you try capturing nsight trace for batch size 8192, pp8192 case, and share here? |
Beta Was this translation helpful? Give feedback.
-
|
@marlin-oss Hi! Did you figure it out? I see the same thing on dual RX6800. There is exactly zero difference between Seeing that there is a behavior to disable pipeline parallelism if you're low on memory, maybe it's disabled by something else as well, but without logging anything? |
Beta Was this translation helpful? Give feedback.
-
|
I haven't heard of GPU-level profilers/debuggers before, but I can probably figure out how to capture a log with some help. But I'm on AMD, so my tools would be different I suppose? |
Beta Was this translation helpful? Give feedback.
-
|
I missed that. I’ll update this thread if I find anything. Are you running in a vm with GPU pass through? I was going to test if that was causing the issue |
Beta Was this translation helpful? Give feedback.
-
|
FWIW, scaling works well for me on 3x AMD v320. Roughly the same scaling ratio as gaugarg-nv posted above, using the ROCm backend (I haven't tested Vulkan)
One example is having any RPC devices active.
This is a good point; I've had trouble getting this working. For AMD GPUs, you can use rocm-bandwidth-test to see which devices can do P2P and the bandwidth between them. Ideally you should see all the devices able to P2P to each other, and able to communicate at roughly the full theoretical maximum PCIe bandwidth expected for your PCIe generation and bus width. |
Beta Was this translation helpful? Give feedback.
-
|
No passthrough, nothing unusual; the most unusual thing I have is Docker. Is there anything special I have to specify to allow P2P transfers in Docker? I've built rocm-bandwidth-test in that container. But I don't know what am I looking for, or how to read the output. But according to what an AI is telling me, it can't find a valid way to do direct P2P communication. I've got a GA-X79S-UP5-WIFI mainboard - that's socket 2011, Intel C606 chipset, PCI-Express 3.0. No Infinity Fabric, no resizeable BAR. So, I guess no direct P2P is possible? But can't data be passed "the normal way", without P2P? |
Beta Was this translation helpful? Give feedback.
-
I haven't tried docker so I can't say, but if you post your
If you haven't already tried, try the Vulkan build. It is a lot more forgiving. I think it implements layer split through Device->RAM->Device copying. For me the performance is a bit less good compared to ROCm, but if it works for you that might be better than hours of debugging. If you're able, I'd suggest trying at least |
Beta Was this translation helpful? Give feedback.
-
|
According to my tests, ROCm is 20% to 2x faster at prompt processing, and Vulkan is 20% to 3x faster at token generation, and its token generation decays very little compared to ROCm as context grows. I use both. Vulkan does not see any improvement from enabling pipeline parallelism either. Now I'm pretty sure that the problem is my old-ass hardware. It makes sense that it would rely on technologies almost everyone has nowadays. (I can't try it on my host system right now because of... Linux reasons 🥲 ) But I've also just realized that pipeline parallelism does not support partial CPU offloading - not layers, not MoE experts. I know we'd be bottlenecked by the CPU anyway, but wouldn't it still improve things quite a bit as long as only a few experts are offloaded?.. |
Beta Was this translation helpful? Give feedback.
-
I would say to not blame this until you've had a chance to test in the host OS. PCIe P2P is not new, I'd expect any self-respecting C606 motherboard to support it.
Not everything that could be implemented, is implemented. Now is your chance to shine 🙂 |
Beta Was this translation helpful? Give feedback.
-
|
I can't run 7.2.3 on the host, but I've tried 6.3.2 (with an older rocm-bandwidth-test, ROCm 6.3.2 host/Docker |
Beta Was this translation helpful? Give feedback.
-
|
Hmm, models take much longer to load now, but prompt processing... is the same as before. I've made sure to enable pipeline parallelism - I have a fix of my own to fail instead of silently disabling it if there was not enough memory to allocate for it. And I've double-checked that I'm running it with |
Beta Was this translation helpful? Give feedback.
-
|
I've tried both ROCm and Vulkan, tried your branch as-is to make sure my patches don't mess it up - nope, it's even a little slower than running it without RPC (and without PP). Looking back at this: ...I guess this is where I got the idea that ReBAR is necessary. But what's that about IOMMU? I have one, but there is no "configuration" in BIOS other than enabling a few things. But none of that explains why doesn't even RPC help. 😄 I was pretty sure that it would work no matter what. Is there maybe some kind of debug logging? Should I get infinite-verbosity logs? |
Beta Was this translation helpful? Give feedback.
-
Yes. It's not a perfect solution
There are a variety of other reasons which cause PP to be disabled. Are you only terminating for that particular one? Do you see in the logs the positive confirmation that it is enabled? Can you try a smaller model and share the whole command lines you are running for the RPC servers and the bench?
With |
Beta Was this translation helpful? Give feedback.
-
I just thought it might be useful information - I didn't expect it to have any impact; we're not bottlenecked by anything, local network has unlimited bandwidth, so why would it be slower? Oh, because instead of DMA from page cache to VRAM, we're now sending data through CPU?
I know that you can get "infinitely verbose logs" with During startup, llama.cpp tries to initialize PP, checking that all conditions are met, then prints So yes, it's definitely enabled, and it successfully allocates the buffer. Other than that, there is nothing related in the verbose logs. Here are my commands: So, I'm running router mode, which lets me try different models, and also I have automated builds for different Docker images, so I can try ROCm/Vulkan or different branches easily. The model parameters file I'm testing them with my own benchmark: https://github.com/dark-penguin/llm-tools (I've just created a new release so you can download the same version as me without rebuilding). It's not very accurate, but it is very simple, and it lets me test different context lengths at different depth, with real text and a real prompt, not random garbage (like llama-bench AFAIK). And it tests directly llama-server, or any other OpenAPI-compatible endpoint, not a different binary with different flags and conditions. What it does is simply send requests with specified length roughly calculated (assuming about 4 characters per token). Each run mostly hits cache, so you don't waste time reuploading, and you can see how fast each request processes.
(Append My normal run (no RPC, -pp off, -b 512 -ub 512): Reference run (RPC, -pp on, -b 512 -ub 512): Test run (RPC, -pp on, -b 2048 -ub 512): Just to double-check under exactly the same conditions - which you won't be able to do without my patch: Looks like RPC makes prompt processing 5-15% faster (how?!), but token generation 25-60% slower, and those effects get more prominent with higher context depth. But all RPC runs are the same - with PP enabled, with PP disabled, and with PP impossible due to -b being equal to -ub . |
Beta Was this translation helpful? Give feedback.
-
|
I see that with PP disabled, allocated compute buffers are smaller than with it enabled, so at least it's "trying" to get enabled. Maybe we could trace PP behavior by inserting debug-level log messages at each step? Something is going wrong later on - something that's not checked, or intentionally quietly ignored (similar to enabling PP in the first place). By the way, how do you stop the RPC server processes? They don't seem to react to any signals. |
Beta Was this translation helpful? Give feedback.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
-
The log says "llama_context: pipeline parallelism enabled". As far as I can tell, with layer split, it's only "batch parallel" or "pipeline sequential". Based on my understanding of the term "pipeline parallel", a model split between N GPUs should be able process N concurrent requests "roughly" N times faster than a single request (minus overhead)
With 2 GPUs and 2 Requests (prompt processing):
While one gpu is idle, it starts processing the next request - like a pipeline. I do not see this behavior. Only 1 GPU is processing at any time.
I've tried every combination of flags I can think of. Is there a build flag?
I appreciate any help.
Beta Was this translation helpful? Give feedback.
All reactions