[Feature]: Combine GPU and CPU builds into one with a command line option

[MrSimmo]

Description

Morning!

Apologies if this has been raised before (I searched, but probably not well).

Wonder if it would be possible to merge/combine the two builds of Audio Separator (CPU + GPU) into one and then give the user a command line option such as --use_gpu to enable the use of the GPU?

Many Thanks

[beveradb]

If I knew of a way to achieve this, I'd definitely like that too! PRs very much welcome 😄

The reason they're separate currently is because of the way python packaging works and what PyPI supports, it has pretty limited options for handling platform/hardware-dependent compiled binaries.

Crucially, a key dependency (ONNX Runtime) has separate python packages for CPU vs. GPU (see here: https://onnxruntime.ai/docs/install/#python-installs ) and there's no way - that I'm aware of - to have conditional dependencies targeting GPUs with packages served via PyPI. PEP 508 defines some environment markers, but nothing for GPU.

There's some discussion about this from 2018 here:

• https://mail.python.org/archives/list/distutils-sig@python.org/thread/LXLF4YSC4WUZOYRX65DW7CESIX7UUBK5/
• [PEP508] Adding gpu or cuda markers in PEP 508 pypa/interoperability-peps#68
• [PEP508] Adding gpu or cuda specification in PEP 508 python/peps#581

Sadly the conclusion as far as I'm aware is to use optional "extras" (like I've done for audio-separator) e.g. pip install "audio-separator[gpu]" vs pip install "audio-separator[cpu]"

If you have any info which could change this and make the user experience better, please share :)

[MrSimmo]

Thanks, thats a pain! I had a look through, managed to find that the GPU package supports running in CPU only mode if you 'hide' the the gpus from it but then causes a bunch of CUDA related issues from what I can see.

I thought about virtualising the PyPi runtime (like you can with pyenv) but couldn't figure out how to achieve that either :(

Fingers crossed the upstream devs will add support but as its been stale since 2018, it likely wont!

[Politrees]

Hi!
As far as I know, onnxruntime-gpu should support both CUDAExecutionProvider and CPUExecutionProvider to support both GPU and CPU with one library. Only onnxruntime-gpu is not optimized for CPU as it is done in onnxruntime. Have you thought about this? Or is it not the case at all and I did not understand the problem? :)

python-audio-separator/audio_separator/separator/separator.py

Line 297 in 69cb8f9

self.onnx_execution_provider = ["CUDAExecutionProvider"]

⠀⠀⠀↓↓↓

self.onnx_execution_provider = ["CUDAExecutionProvider", "CPUExecutionProvider"]

[beveradb]

The onnxruntime-gpu package only has binaries for Linux and Windows: https://pypi.org/project/onnxruntime-gpu/#files - I'm personally a mac user and run it locally so I would definitely not want audio-separator to have a fixed dependency on onnxruntime-gpu, as then I wouldn't be able to install it on my own system 😅

To be honest this issue feels like a non-issue to me anyway, as the existing setup works and gives people the flexibility to use audio-separator on any platform.

The majority of people using audio-separator likely aren't using the models which use ONNX anyway, all the newest models (e.g. roformer) just use PyTorch directly so regardless of whether onnxruntime or onnxruntime-gpu are installed, if a CUDA capable GPU is available to PyTorch, it should be loaded - there's nothing about the audio-separator code itself which is "built" for a specific platform or hardware requirement.