[BUG]: Error loading the LLava model

[aropb]

Models:
https://huggingface.co/benxh/Qwen2.5-VL-7B-Instruct-GGUF
https://huggingface.co/KBlueLeaf/llama3-llava-next-8b-gguf (from here #897)
https://huggingface.co/second-state/Llava-v1.5-7B-GGUF

Error:
External component has thrown an exception.
System.Runtime.InteropServices.SEHException (0x80004005): External component has thrown an exception.
at LLama.Native.SafeLlavaModelHandle.clip_model_load(String mmProj, Int32 verbosity)
at LLama.Native.SafeLlavaModelHandle.LoadFromFile(String modelPath, Int32 verbosity)
at LLama.LLavaWeights.LoadFromFile(String mmProject)

What could be the problem?
I wanted to use a multimodal model to convert image2text.

Environment & Configuration

• Operating system: Windows
• .NET runtime version: NET 9.0.3
• LLamaSharp version: 0.23.0
• CUDA version (if you are using cuda backend): CPU
• CPU & GPU device: CPU

[martindevans]

Does the model you're using work with the corresponding version of llama.cpp? Maybe also test in the previous version of LLamaSharp, to check for regressions.

[aropb]

I'll try. Can you tell me if llava should work with any multimodal models?

I also noticed that the context is created only through llama weight, why is that?

[aropb]

Does the model you're using work with the corresponding version of llama.cpp? Maybe also test in the previous version of LLamaSharp, to check for regressions.

0.21.0 - the same mistake

[aropb]

Does the model you're using work with the corresponding version of llama.cpp?

CPU

Yes, loaded!

[SignalRT]

I tested the master with the model used in the unit test successfully.

Tested on Windows with CUDA.

[aropb]

and on the CPU?

[aropb]

I tested the master with the model used in the unit test successfully.

Models/llava-v1.6-mistral-7b.Q3_K_XS.gguf ?

[SignalRT]

As I explain I tested in GPU. The unit test are running successfully so it should work. I'm talking about Llava. Qwen-VL is not tested, and should not work.

[aropb]

Qwen-VL?

[aropb]

Does LLava not work on the CPU?

[SignalRT]

Qwen-VL?

Is that LlamaSharp documentation?

[aropb]

mmproj-model-f16.gguf - works!
llava-v1.6-mistral-7b-Q5_K_S.gguf - dont works!
https://huggingface.co/bartowski/Qwen2-VL-7B-Instruct-GGUF - dont works!

[aropb]

Is that LlamaSharp documentation?

Yes
https://github.com/ggml-org/llama.cpp

[aropb]

Should the LLava model be mmproj only?
Is it really impossible to use even Qwen2-VL for image2text?

Maybe I don't understand correctly how to work with VLM.

[SignalRT]

mmproj-model-f16.gguf - works! llava-v1.6-mistral-7b-Q5_K_S.gguf - dont works! https://huggingface.co/bartowski/Qwen2-VL-7B-Instruct-GGUF - dont works!

You need both files. Check https://scisharp.github.io/LLamaSharp/0.23.0/QuickStart/ and the Llava example in the examples project