One-click deployment of mainstream LLMs and multimodal models on SOPHGO TPU
Deploy LLMs & VLMs on SOPHGO BM1684X / BM1688 / CV186X with a single command
English · 简体中文
Quick Start · Supported Models · Compilation Flow · Advanced Features · FAQ · Website
| Date | Updates |
|---|---|
| 🔥 2026.07.16 | Falcon-Perception now supports BM1684X, Python demo, referring segmentation (box + mask) → Details |
| 🔥 2026.07.09 | LocateAnything-3B now supports BM1684X / BM1688, Python demo, visual grounding (box / point) → Details |
| 2026.06.30 | MiniCPM-V-4.6 now supports BM1684X / BM1688, Python demo, image & video support → Details |
| 2026.05.21 | Gemma4 now supports BM1684X / BM1688, Python demo, image / video / audio support → Details |
| 2026.04.15 | Qwen3.5 now supports BM1684X / BM1688, Python & C++ demos, image & video support → Details |
| 2025.10.15 | Qwen3-VL now supports BM1684X / BM1688, Python / C++ demos, image & video support → Details |
| 2025.05.22 | InternVL3 now supports BM1684X / BM1688, image & video support → Details |
| 2025.04.30 | Qwen2.5-VL now supports BM1684X / BM1688, Python / C++ demos → Details |
| 2025.04.29 | Reasoning model Qwen3 now supports BM1684X / BM1688 → Details |
| 2025.03.07 | QwQ-32B and DeepSeek-R1-Distill-Qwen-32B multi-chip demos adapted → Details |
| 2025.02.05 | Adapted DeepSeek-R1-Distill-Qwen series (1.5B / 7B / 14B) → Details |
LLM-TPU is an open-source project officially maintained by SOPHGO, dedicated to deploying mainstream generative AI models (LLM / VLM) on SOPHGO BM1684X / BM1688 / CV186X series TPU chips.
┌──────────────┐ tpu-mlir ┌──────────────┐ tpu-runtime ┌──────────────────┐
│ HuggingFace │ ─────────────► │ bmodel │ ───────────────► │ PCIE / SoC │
│ weights │ llm_convert │ (quantized) │ Python / C++ │ BM1684X / 1688 │
└──────────────┘ └──────────────┘ └──────────────────┘
- 🚀 One-click compilation:
llm_convert.pyexports HuggingFace weights directly to bmodel - 🧩 Rich model coverage: Dozens of models including Qwen / Llama / DeepSeek / InternVL / MiniCPM / Phi / ChatGLM
- 🎯 Multimodal: Inference across text, image, video, and audio
- ⚡ Efficient inference: AWQ/GPTQ quantized models, dynamic compilation, KV Cache, multi-chip parallelism
- 🛠️ Dual-language demos: Popular models ship with both Python and C++ reference implementations
- 📦 Ready to use: Pre-compiled bmodels available for download — no compilation required
Compiling models requires the TPU-MLIR environment (Docker or source build both work). Alternatively, use the pre-compiled bmodels provided in each demo. See
models/for the full model list.
Get an LLM running on your TPU device in just two steps:
git clone https://github.com/sophgo/LLM-TPU.git
cd LLM-TPU
./run.sh --model qwen2.5vl| Model | Command |
|---|---|
| Qwen3-4B | ./run.sh --model qwen3 |
| Qwen2.5-VL-3B | ./run.sh --model qwen2.5vl |
| InternVL3-2B | ./run.sh --model internvl3 |
📘 For detailed steps, see Quick Start.
| Model | Supported Chips | One-click Compile | Notes |
|---|---|---|---|
| Falcon-Perception | BM1684X | — | Python, referring segmentation box + mask |
| LocateAnything-3B | BM1684X / 1688 | — | Python, visual grounding box / point |
| Qwen3.5 | BM1684X / 1688 | ✅ | Python + C++, image / video |
| Qwen3-VL | BM1684X / 1688 | ✅ | Python + C++, image / video |
| Qwen2.5-VL | BM1684X / 1688 | ✅ | Python + C++ |
| Qwen2-VL | BM1684X / 1688 | ✅ | — |
| InternVL3 | BM1684X / 1688 | ✅ | Video supported |
| Gemma4 | BM1684X / 1688 | ✅ | Python, image / video / audio |
| Gemma3 | BM1684X / 1688 | ✅ | — |
| Qwen-VL / InternVL2 / MiniCPM-V-2.6 / Llama3.2-Vision | BM1684X / 1688 | — | Deployed |
| Family | Representative Models | One-click Compile |
|---|---|---|
| Qwen | Qwen1.5 / Qwen2 / Qwen2.5 / Qwen3 / QwQ-32B | ✅ |
| DeepSeek | DeepSeek-R1-Distill-Qwen (1.5B / 7B / 14B / 32B) | ✅ |
| Llama | Llama2 / Llama3 | ✅ |
| MiniCPM | MiniCPM4 | ✅ |
| Phi | Phi-3 / Phi-4 | ✅ |
| ChatGLM | ChatGLM3 / ChatGLM4 | ✅ |
| Others | Baichuan2 · CodeFuse · Falcon · Gemma / Gemma2 · Mistral · WizardCoder · Yi · Yi34B · LWM-Text-Chat · Megrez · MiniCPM3 · DeepSeek-V2 | — |
The models/ directory currently contains the following model implementations:
LLM: Baichuan2 · ChatGLM2 · ChatGLM3 · CodeFuse · DeepSeek-V2 · GLM4 · Llama2 · Llama3 · LWM · Megrez · MiniCPM3 · MiniCPM4 · Mistral · Phi-3 · Qwen · Qwen1_5 · Qwen2 · Qwen2_5 · Qwen3 · RWKV6 · RWKV7 · WizardCoder · Yi · Yi34B
Multimodal (Vision / Video / Audio): DriveMM · Falcon-Perception · Gemma3 · Gemma4 · GLM4V · InternVL2 · InternVL3 · Janus-Pro · Llama3_2-Vision · LocateAnything · MiniCPM-V-2_6 · MiniCPMV4 · MiniCPMV4_6 · Molmo · NVILA · Qwen2_5_Omni · Qwen2_5_VL · Qwen2_Audio · Qwen2_VL · Qwen3_5 · Qwen3_ASR · Qwen3_VL · VILA1_5
See each subdirectory for complete source code and conversion details.
Using Qwen2.5-VL as an example:
Prefer AWQ or GPTQ quantized versions for better accuracy.
git lfs install
git clone git@hf.co:Qwen/Qwen2.5-VL-3B-Instruct-AWQRefer to TPU-MLIR
llm_convert.py \
-m /workspace/Qwen2.5-VL-3B-Instruct-AWQ \
-s 2048 --max_input_length 1024 \
-c bm1684x \
--max_pixels 672,896 \
-o qwen2.5vl_3b| Argument | Short | Required | Description |
|---|---|---|---|
--model_path |
-m |
✅ | Path to model weights |
--seq_length |
-s |
✅ | Maximum sequence length |
--max_input_length |
— | Maximum single-input length; defaults to seq_length |
|
--quantize |
-q |
Quantization type: w4bf16 / w4f16 / bf16 / f16 … |
|
--chip |
-c |
✅ | Target platform: bm1684x / bm1688 / cv186x |
--q_group_size |
-g |
Quantization group size, default 64 |
|
--max_pixels |
— | VLM only, max pixels, e.g. 672,896 or 602112 |
|
--do_sample |
— | Include sampling model in output; off by default | |
--out_dir |
-o |
✅ | Output directory |
💡 Choosing quantization: if the model is already quantized, you do NOT need to specify
quantize; unquantized models require it.For more advanced options, see Advanced Features.
Once finished, the output directory will contain the corresponding bmodel and config directory, ready to load for inference.
| Capability | Description | How to Enable | Examples |
|---|---|---|---|
| Dynamic Compilation | Runs inference based on actual input length, reducing latency for short inputs; also recommended for variable-size multimodal images | --dynamic |
Qwen3 · Qwen2.5-VL · MiniCPM4 · InternVL3 · Qwen3-VL |
| Prefill with KV Cache | Keeps historical context as KV Cache, significantly reducing multi-turn conversation latency | --use_block_with_kv--max_input_length--max_prefill_kv_length |
Qwen3-VL · Qwen2.5-VL · Qwen3 · InternVL3 |
| Multi-chip Parallelism | Parallel inference across multiple TPUs, enabling larger models and higher throughput | --num_device N |
Qwen2.5 / 2-8 chips |
| Random Sampling | Sampling with generation.json configuration (greedy by default) |
--do_sample |
Qwen3 · InternVL3 · MiniCPM4 |
| Multi-task Reuse | Load the same model multiple times for multiple tasks; weights are loaded only once per chip | — | Qwen2.5-VL multiuser |
| Shared Prefill Reuse | Prefill a long prompt only once; subsequent conversations share its KV Cache | --share_prompt--max_prefill_kv_length |
Qwen2.5 · Qwen3 |
| Model Encryption | Encrypt bmodel with a third-party library; call the decryption API at inference time | — | Qwen · Qwen1.5 |
- Prefer AWQ / GPTQ quantized models when converting to bmodel — they incur the least accuracy loss.
- If only floating-point weights are available, first apply W4A16 quantization with AutoAWQ or AutoGPTQ, then compile to bmodel.
Please refer to the LLM-TPU FAQ.
- 📘 TPU-MLIR — Main compiler repository
- 📗 TPU-MLIR Quick Start Guide
- 🎬 TPU-MLIR Paper / Full Project Walkthrough (Bilibili)
- ✍️ ChatGLM2 Pipeline Analysis & TPU-MLIR Deployment (Zhihu)
- 🌐 SOPHGO Official Website
Issues and suggestions are welcome via GitHub Issues, and Pull Requests are appreciated to help grow the ecosystem. If you are interested in SOPHGO chips or business cooperation, feel free to reach out through the SOPHGO website.
This project is open-sourced under the Apache 2.0 license. See third-party-licenses/ for third-party component licenses.
⭐ If this project helps you, please give it a Star! ⭐