Debian and Ubuntu packages - Call for testing and feedback

[ckastner]

Hi,

we have been working on shipping llama.cpp, whisper.cpp, ggml and other ggml-org projects directly from Debian's and Ubuntu's official repositories.

We want users to be able to just sudo apt-get install llama.cpp.

Our packages should be in good shape now (as tested by the CI we built for this task), so now we would like to ask you, the community, for feedback:

• Is everything clear and intuitive? If not, what can we do better?
• What are you missing? eg: which ackends (we've started with only with a few); documentation (what?)
• Bugs, issues, etc.

Please check ggml-Debian for a summary of what we have, how to install/use it, and what we are working on.

Note that packages for trixie-backports and noble-backports are currently still being shipped from our own development repository (instructions included in the link above), but these will be made available in the official channels soon. We want to ship ~monthly updates this way.

Also note that newer GPUs might not be supported by HIP/CUDA in the above releases. Try the Vulkan backend in that case.

[aendk]

@ckastner
what exactly do you mean by

Also note that newer GPUs might not be supported by HIP/CUDA in the above releases. Try the Vulkan backend in that case.

?
That only on the *-backports, newer GPUs are not supported? And what does "new" mean in this context?

[peterwwillis]

As a user of RDNA3, but not someone who wants to use unstable, I would very much like it if the packaged solution could work with LTS. Having the vendors ship them sounds great to me, but how does that work with the llama.cpp packages? What would be shipped by whom?

[ckastner]

(Assuming that with "shipping", we're always talking about an APT repository, so that apt-get update works)

The vendor continue to ship the CUDA resp. ROCm stack through their own APT repo.

For llama.cpp, the current strategy would be for us (the Debian maintainers) to ship the packages using our own APT server, following the example from the gist.

An alternative would be for HF/ggml.ai to set up their own APT server (which I could help with), and to ship them from there. Though this is a bit more tricky because of the build step, and distro integration. For example, in Debian we also re-test all reverse dependencies on every upload, to catch regressions, and do other maintenance work.

Another alternative would be for vendors to include our packages in their APT repo, the one where they ship CUDA resp. ROCm through. Though I'm not sure they'd be open to that.

[peterwwillis]

Debian's llama.cpp package would need to link against the upstream GPU vendors' libs, right? If the vendors kept their own apt repo and packages, do you pull their packages from their apt repo to build your llama.cpp pkgs? If so, how does the package work for the user? I imagine they would need to first add the vendor apt pkg sources, and then install the Debian llama.cpp pkgs? I haven't heard of this being a thing before so I'm assuming I misunderstood something.

But if that's what's proposed, it does sound like the it could be the simplest option... A downside to it would be that upstream vendors would, like you mention, need to test the downstream dependencies for regressions before they release, and I personally doubt they would do that.

Is it at all possible for Debian to pull the apt pkg sources from the vendor's apt repo, and use that to build Debian pkgs of both the GPU vendors' code, as well as llama.cpp pkgs? That way if a vendor update breaks, it won't make it to the end user since you're building/testing everything in tandem. Perhaps if this were its own apt pkg repo, sort of not an official branch but still maintained by Debian?

[ckastner]

Debian's llama.cpp package would need to link against the upstream GPU vendors' libs, right? If the vendors kept their own apt repo and packages, do you pull their packages from their apt repo to build your llama.cpp pkgs? If so, how does the package work for the user? I imagine they would need to first add the vendor apt pkg sources, and then install the Debian llama.cpp pkgs? I haven't heard of this being a thing before so I'm assuming I misunderstood something.

No, you got it exactly right. Users would need to enable two APT repositories: the vendor one, and the one where we ship the llama.cpp packages, which won't be the official Debian APT (deb.debian.org) but rather our team's APT repo.

This reason why we can't use the official one to host these vendor-targeting builds is precisely because these packages would need dependencies from outside of the official sources. So they would be uninstallable with "pure" Debian or Ubuntu sources.list.

But if that's what's proposed, it does sound like the it could be the simplest option... A downside to it would be that upstream vendors would, like you mention, need to test the downstream dependencies for regressions before they release, and I personally doubt they would do that.

Luckily we can do that downstream, in our own CI. So we'd discover such regressions pretty fast.

Is it at all possible for Debian to pull the apt pkg sources from the vendor's apt repo, and use that to build Debian pkgs of both the GPU vendors' code, as well as llama.cpp pkgs?

Not(*) within Debian. We do have our own versions of the GPU vendor's code, that's what the llama.cpp packages in the official Debian/Ubuntu repos are built with. But these are release-specific, and thus get old. That's why we don't support RDNA3 in trixie-backports or noble-backports, because the vendor version is so old.

The vendors ship up-to-date versions of their stacks in their own APT repos. If they wanted to, with help from Debian people, they could ship backports in the official Debian repos, but I suspect it's simpler for them to just do their own APT.

(*) "Not" in the sense of practically not. It would be technically possible, but would require vendor commitment beyond what I suspect they are willing to do.

That way if a vendor update breaks, it won't make it to the end user since you're building/testing everything in tandem. Perhaps if this were its own apt pkg repo, sort of not an official branch but still maintained by Debian?

That would be one way I guess.

In any case, we're open to try many things.

The first attempt will be the one you described: we pull the vendor packages to build our llama.cpp packages, and ship them in our own APT repo. And we can iterate on that, based on user feedback.

[Debianreiser]

LLAMA.CPP DEBE SER INCLUIDO EN DEBIAN DE FORMA INMEDIATA!!! y su codigo fuente tambien!!! y herramientas para crear modelos de IA Y MUCHAS COSAS MAS!!! HAY QUE POTENCIAR DEBIAN CON LA IA Y EL QUE RENIEGUE DE ELLO ARDERA EN EL INFIERNO!!!

[ckastner]

Here's an interesting problem that we've been hearing from our users, for which we could really use more feedback:

Which of ggml's compute backends should the llama.cpp package install by default?

Currently, it only installs the CPU backend. So people with GPUs won't benefit from them unless they know to additionally install the HIP, CUDA or Vulkan backends.

Possible strategies

1. ggml backend meta-package

This strategy involves creating a ggml meta-package libggml0-backend-all recommending all real backends, and have llama.cpp depend on libggml0-backend-all | libggml0-backend-<specific>.

With such a setup, by default, apt-get install llama.cpp would install all backends. Users could disable/override this with --no-install-recommends.

• Pros
  • Best hardware coverage
• Cons
  • Download and installation of many useless dependencies on any given system
  • This won't work for CUDA because of its placement in Debian non-free resp. Ubuntu multiverse

2. llama.cpp meta-packages

This strategy adds new packages llama.cpp-cuda, llama.cpp-hip, and so on, where each package depends on the right ggml backend.

• Pros
  • Easy to find and install
• Cons
  • Can get confusing quite fast, if the number of meta-packages grows too large

3. Vulkan backend

This strategy installs the Vulkan backend by default.

• Pros
  • Should provide reasonable results on most systems
• Cons
  • Performance for GPUs probably slightly worse than native
  • Installing the backend alone may be insufficient to get it working. For example, for NVIDIA cards, users would have to manually install nvidia-vulkan-icd.

[ckastner]

Just a quick ping, the packages have been updated:

• ggml: 0.9.10
• llama.cpp: b8611
• llama.vim: 4103c5c

As announced, we plan frequent releases, so in future we'll only ping if something noteworthy happens.

[ggerganov]

ggml: 0.9.10

There was an unfortunate regression in the CUDA backend just before that release (#20998 (comment)). I'm preparing v0.9.11 now.

Edit: it's up: https://github.com/ggml-org/ggml/releases/tag/v0.9.11

[ckastner]

There was an unfortunate regression in the CUDA backend just before that release (#20998 (comment)). I'm preparing v0.9.11 now.

No problem, I updated the ggml packages to 0.9.11.

[Debianreiser]

DEBE SER INCLUIDA USANDO EL CODIGO FUENTE SOLAMENTE Y CADA USUARIO QUE LO COMPILE MANUALMENTE , PERO EL CODIGO FUENTE DE LLAMA.CPP DEBE ESTAR PERMANENTEMENTE EN LOS REPOSITORIOS DE DEBIAN PARA QUE EL USUARIO AVANZADO QUE LO DESEE PUEDA COMPILAR LLAMA DESDE DEBIAN SI TIENE REPOSITORIO OFICIAL PROPIO OFFLINE , ADEMAS ES UNA FORMA DE EXPANDIR LA FAMA Y PRESTIGIO DE LLAMA.CPP

[talhaHavadar]

I somehow recall that we discussed about this but can we have llama.cpp to emit a warning if there is a gpu available but backend is missing? Something like "hey I see you have a gpu but you are only using cpu backend, install 'gpu' backend to utilize gpu"

What do you all think about this?

[mbaudier]

Based on your feedback, we have been working over the last two weeks on the packaging of a ggml CUDA backend built against the Nvidia-provided repositories (cc @aendk). It is available for Debian unstable/sid and stable/trixie in the Debian AI team repository (both built against the 'debian13' Nvidia repo), with detailed installation instructions available here:

https://salsa.debian.org/deeplearning-team/ggml/-/blob/debian/unstable/debian/README.vendor-cuda.md

There are various use cases where using the Nvidia-provided packages can be useful or necessary, which in turn require a dedicated build of the ggml CUDA backend:

• On cloud instances with Nvidia GPUs, only the Nvidia-provided driver supports the lightweight cloud kernel. It is possible to use the regular Debian 'contrib' and 'non-free' packages, but it requires switching to the full-fledged kernel, which is not trivial.
• The Nvidia-provided repositories offer multiple versions of CUDA and of the Nvidia driver, which may be useful in order to support newer GPUs or pin down specific versions. This new ggml CUDA backend package is therefore dependent on a given CUDA version and marked as such (currently libggml0-backend-vendor-cuda-13-2 for CUDA v13.2).

Thanks to the ggml and llama.cpp architectures, all the other packages (currently in the Debian AI team repository, and soon in the trixie-backports repository) can be used as such. All this affects only the CUDA backend.

We have tried to provide it for Ubuntu noble-backports as well, but we ran into issues related to the package dependencies in the Nvidia-provided Ubuntu repositories, that we therefore cannot address on our side. Please do not use the Ubuntu packages, except in order to help us troubleshoot these issues.

These ggml CUDA backend packages for deployments using the Nvidia-provided repositories have been systematically tested with Debian Trixie on AWS instances with Nvidia GPUs (amd64 and arm64). As usual, your feedback is welcome and very useful!

[ckastner]

We have tried to provide it for Ubuntu noble-backports as well, but we ran into issues related to the package dependencies in the Nvidia-provided Ubuntu repositories, that we therefore cannot address on our side. Please do not use the Ubuntu packages, except in order to help us troubleshoot these issues.

To expand on this a bit, we use an unversioned nvidia-compute as a package build-time dependency, and resulting backend this generates a versioned run-time dependency of libnvidia-compute-560, which causes installation problems on systems without a matching 560 driver.

However: I'm assuming ggml doesn't interact with the driver directly, only via the CUDA libraries. These all seem to have the same SOVER across driver versions, thus providing the same ABI. So we probably just need to loosen the runtime dependency to something that supports any libnvidia-compute-xxx.

But rather than tap in the dark, we'd thought we'd ping the thread and see what strategies others are using.

[ckastner]

We have tried to provide it for Ubuntu noble-backports as well, but we ran into issues related to the package dependencies in the Nvidia-provided Ubuntu repositories, that we therefore cannot address on our side

We found a solution to these issues. The libggml0-backend-vendor-cuda-13-2 backend provided in the vendor builds should now work with any installed Nvidia driver version.

We only build targeting CUDA 13.2 for now, but we can also build for other versions.

[ckastner]

Good news: llama.cpp has finally migrated to the Debian testing release -> we can now start shipping trixie-backports from the official Debian archive \o/

And @talhaHavadar is looking to take care of official noble-backports and later resolute-backports.

As indicated, we plan to have frequent releases after that. And installation from the official archives will be much simpler compared to our prototype instructions above. I'll submit a brief PR with these soon.

@ggerganov, the first versions we'd ship would be llama.cpp b8681 and ggml 0.9.11. Are these recent enough? Gemma worked fine on my end. If you'd like newer versions on launch, we'd just need a new ggml release.

For me, the current way makes sense and I cannot say with confidence that any of the 3 new options would generally be better.

I also agree. With no general improvement, I think it's best we launch as-is, and wait for broader user feedback.

[Debianreiser]

TENEIS QUE PONERLO EN LA VERSION ESTABLE LLAMA.CPP COMPILADA PARA TODAS LAS NVIDIA DESDE LA RTX200 HASTA LA RTX5090 AL COMPILAR SELECCIONAR QUE VALGA PARA TODAS LAS ARQUITECTURAS DE NVIDIA Y YA ESTA ASI DE FACIL

[David-Else]

@ckastner Is it still coming to backports? Cheers.

[ckastner]

Yep, we were just discussing the topic this week. We're aiming for something like the last week in June.

We'll start with official ones for Debian. Ubuntu is still debating whether to ship the packages are backports or as stable release updates, but I guess that shouldn't take too long, either.

For now, our development APT repo (see ggml-Debian has packages for trixie-backports, noble-backports, and resolute-backports, if you'd like to give these a try.

[talhaHavadar]

For ubuntu we are still debating how to pull this to apt for LTS but I have a snap of llama-cpp available here https://snapcraft.io/llama-cpp

At worst I believe it will be available in backports in ubuntu as well