支持更多引擎/架构优化

CLAUDE.md

@@ -24,19 +24,24 @@ The pipeline runs in a background thread: **Audio Capture (32ms chunks) -> VAD -
 ```
 main.py (LiveTranslateApp)
   |-- model_manager.py     Centralized model detection, download, cache utils
+  |-- pipeline_controller.py Audio capture + VAD + ASR queue controller, incremental ASR coordination
   |-- audio_capture.py     WASAPI loopback via pyaudiowpatch, auto-reconnects on device change
-  |-- vad_processor.py     Silero VAD / energy-based / disabled modes, progressive silence + backtrack split
+  |-- vad_processor.py     Silero VAD / FireRedVAD / energy-based / disabled modes, progressive silence + backtrack split
+  |-- vad_firered.py      FireRedVAD streaming rolling-frame confidence adapter
   |-- asr_client.py        Main-process ASR worker manager (spawn, Pipe IPC, timeouts)
   |-- asr_worker.py        ASR subprocess entrypoint; loads and owns one backend/model
   |-- asr_engine.py        faster-whisper (Whisper) backend
   |-- asr_sensevoice.py    FunASR SenseVoice backend (better for Japanese)
   |-- asr_funasr_nano.py   FunASR Nano backend
   |-- asr_anime_whisper.py Anime-Whisper backend (litagin/anime-whisper, ja anime/galgame)
+  |-- asr_crispasr.py     CrispASR ggml runtime backend (GGUF/bin single-file models)
+  |-- asr_sherpa_onnx.py  sherpa-onnx OfflineRecognizer/OnlineRecognizer backend (local ONNX model dirs)
+  |-- asr_parakeet_cpp.py parakeet.cpp C API backend (local GGUF model + native runtime)
   |-- translator.py        OpenAI-compatible API client, streaming, JSON schema, context history
   |-- subtitle_overlay.py  PyQt6 transparent overlay (2-row header: controls + model/lang combos)
   |-- subtitle_window.py   Standalone subtitle window for OBS capture (outlined text, animations)
   |-- subtitle_settings.py Subtitle window settings UI (grid layout, text line editor)
-  |-- control_panel.py     Settings UI (7 tabs: VAD/ASR, Translation, Style, Subtitle, Benchmark, Cache, Changelog)
+  |-- control_panel.py     Settings UI (8 tabs: ASR, VAD, Translation, Style, Subtitle, Benchmark, Cache, Changelog)
   |-- dialogs.py           Setup wizard, model download/load dialogs, ModelEditDialog
   |-- benchmark.py         Translation benchmark (BENCH_SENTENCES, run_benchmark())
   |-- log_window.py        Real-time log viewer
@@ -45,12 +50,12 @@ main.py (LiveTranslateApp)
 ### Threading / Process Model
 
 - **Main thread**: Qt event loop (all UI)
-- **Capture thread**: `_capture_loop` in `LiveTranslateApp` reads audio and runs VAD
-- **ASR queue thread**: `_asr_loop` drains VAD segments and calls `ASRClient.transcribe()`
+- **Capture thread**: `_capture_loop` in `PipelineController` reads audio and runs VAD
+- **ASR queue thread**: `_asr_loop` in `PipelineController` drains VAD segments and calls ASR via `ASRService`
 - **ASR worker process**: `asr_worker.py` owns the concrete ASR backend/model and runs inference over `multiprocessing.Pipe`
 - **ASR loading**: `_switch_asr_engine()` stops the current worker, then starts the target worker in a background thread; if target loading fails, it restarts the previous worker from its saved config
 - Cross-thread UI updates use **Qt signals** (e.g., `add_message_signal`, `update_translation_signal`)
-- ASR readiness tracked by `_asr_ready` flag; pipeline drops segments while no ready worker exists
+- ASR readiness tracked by `ASRService`; `PipelineController` drops segments while no ready worker exists
 
 ### Configuration
 
@@ -146,6 +151,7 @@ Continuous speech is processed incrementally to reduce latency (enabled by `incr
 
 ### VAD Behavior
 
+- **FireRedVAD mode**: `vad_firered.py` lazy-imports `fireredvad` and adapts LiveTranslate 32ms float32 chunks into FireRedVAD 25ms/10ms streaming frames. FireRedVAD only supplies speech confidence; `VADProcessor` still owns segmentation, silence handling, backtrack splitting, incremental ASR buffer state, and ASR queue contract.
 - **Progressive silence**: Buffer越长接受越短的停顿切分 (<3s=full, 3-6s=half, 6-10s=quarter of silence_limit)
 - **Adaptive silence**: Tracks recent pause durations, sets threshold to P75 × 1.2, auto-adjusts between 0.3s~2.0s
 - **Backtrack split**: Max duration时回溯smoothed confidence history找最低谷切分，remainder保留到下一段
@@ -165,13 +171,23 @@ Continuous speech is processed incrementally to reduce latency (enabled by `incr
 - FunASR Nano: `asr_funasr_nano.py` does `os.chdir(model_dir)` before `AutoModel()` inside the ASR worker process, so relative paths in config.yaml (e.g. `Qwen3-0.6B`) resolve locally instead of triggering HuggingFace Hub network requests, without changing the GUI process cwd
 - `Translator` defaults to 10s timeout via `make_openai_client()` to prevent API calls from hanging indefinitely
 - Log window is created at startup but hidden; shown via tray menu "Show Log"
-- Audio chunk duration is 32ms (512 samples at 16kHz), matching Silero VAD's native window size for minimal latency
+- Audio chunk duration is 32ms (512 samples at 16kHz), matching Silero VAD's native window size for minimal latency. FireRedVAD keeps this capture cadence and internally rolls 25ms windows with 10ms shifts.
+- FireRedVAD model discovery expects the official `Stream-VAD` directory containing `cmvn.ark` and `model.pth.tar`; the model is optional and not part of first-launch required downloads.
+- FireRedVAD input scaling is explicit: LiveTranslate keeps ASR/VAD buffers as float32 PCM, while `vad_firered.py` clips to [-1, 1] and multiplies by 32768 before calling `detect_frame()`.
 - FunASR `disable_pbar=True` required in all `generate()` calls — tqdm crashes in GUI process when flushing stderr
 - ASR engine lifecycle: the GUI process never instantiates `ASREngine`, `FunASREngine`, or `AnimeWhisperEngine` directly. It owns an `ASRClient`; each worker process owns one concrete backend/model. Engine/model/device changes shut down the current worker first, then start a new worker. On target load failure, the saved previous worker config is used to restore ASR.
+- CrispASR is treated as a ggml C++ runtime hub with GGUF/bin single-file weights. The GUI process never imports `crispasr`; `asr_crispasr.py` imports the Python binding only inside the ASR worker process and adapts results to the common ASR dict.
+- sherpa-onnx is treated as a local ONNX ASR runtime. The GUI process never imports `sherpa_onnx`; `asr_sherpa_onnx.py` imports it lazily inside the ASR worker after any CUDA device mapping is finalized. Offline families use `OfflineRecognizer`; `online_transducer` uses `OnlineRecognizer` as a VAD segment wrapper, not true partial streaming.
+- parakeet.cpp is treated as a worker-only native C API backend. The GUI process never loads the parakeet DLL; `asr_parakeet_cpp.py` uses `ctypes` only inside the ASR worker, loads one local `.gguf` once, and returns the common ASR dict.
+- parakeet.cpp model scanning only accepts known official GGUF filename prefixes or explicit sidecar metadata (`family=parakeet_cpp` / `architecture=parakeet`). CrispASR scanning excludes recognized parakeet GGUF files to avoid `.gguf` list pollution.
+- parakeet.cpp C API returned strings must be released with `parakeet_capi_free_string()`, and loaded contexts must be released with `parakeet_capi_free()`.
+- parakeet.cpp runtime DLL paths are added only inside the worker with `os.add_dll_directory()`. CUDA runtime may require the matching `cudart-parakeet-bin-win-cuda-x64.zip` DLLs beside the parakeet runtime.
+- CrispASR packaging has two parts that must stay aligned: `pyproject.toml` installs the pure-Python binding from the CrispASR release tag via `uv sync`, while `install.ps1` and the portable `bootstrap.ps1` download the matching prebuilt Windows `libcrispasr` DLL runtime from GitHub Releases. Do not replace this with plain `crispasr` from PyPI; that package name may not exist in the configured registry and the binding alone does not include `crispasr.dll`.
+- CrispASR native runtime install prefers `libcrispasr-windows-x86_64-cuda.tar.gz` on NVIDIA systems and falls back to `libcrispasr-windows-x86_64.tar.gz`. DLLs are copied next to the installed `crispasr` package so the worker can load them without a global PATH edit.
 - Whisper (ctranslate2) only accepts `device="cuda"` not `"cuda:0"`; device index passed via `device_index` param. Parsed from combo text like `"cuda:0 (RTX 4090)"` in `_switch_asr_engine`
 - ASR text density filter: segments ≥2s producing ≤3 alnum characters are discarded as noise
 - Settings file uses atomic write (write to `.tmp` then `os.replace`) to prevent corruption on crash
-- `stop()` joins pipeline thread before flushing VAD to prevent concurrent `_process_segment` calls
+- `PipelineController.stop()` joins capture/ASR queue threads before flushing VAD to prevent concurrent segment processing
 - Cancelled ASR download leaves the current worker running; failed target worker load attempts to restore the previous worker config
 - `Translator._build_system_prompt` catches format errors in user prompt templates, falls back to DEFAULT_PROMPT
 - Translation prompt presets: `PROMPT_PRESETS` in `translator.py` (daily/esports/anime), selectable via control panel combo

README.md

@@ -21,13 +21,13 @@ Works with any system audio — videos, livestreams, voice chat. No player modif
 ## Features
 
 - **Real-time pipeline**: System audio → VAD → ASR → LLM translation → overlay
-- **Multiple ASR engines**: faster-whisper, SenseVoice, FunASR Nano, Anime-Whisper
+- **Multiple ASR engines**: faster-whisper, SenseVoice, FunASR Nano, Anime-Whisper, CrispASR, sherpa-onnx, parakeet.cpp, Remote Whisper
 - **Remote ASR**: offload speech recognition to a GPU machine over HTTP — see [REMOTE_ASR.md](REMOTE_ASR.md)
 - **Any OpenAI-compatible API**: DeepSeek, Grok, Qwen, GPT, Ollama, vLLM, etc.
 - **Streaming translation display**: Real-time character-by-character translation output
 - **Per-model settings**: Streaming, structured output (JSON), context history, disable thinking
 - **Microphone mix-in**: Optionally mix microphone input with system audio for ASR
-- **Low-latency VAD**: 32ms chunks + Silero VAD with adaptive silence detection
+- **Low-latency VAD**: 32ms chunks + Silero VAD or optional FireRedVAD with adaptive silence detection
 - **Transparent overlay**: Always-on-top, click-through, draggable, 14 color themes
 - **CUDA acceleration**: GPU-accelerated ASR inference
 - **Auto model management**: Setup wizard, ModelScope / HuggingFace dual sources
@@ -71,23 +71,25 @@ To update, double-click **`update.bat`** — it will pull the latest code and up
 <summary>Manual install</summary>
 
 ```bash
-python -m venv .venv
+uv venv --python 3.12 .venv
 .venv\Scripts\activate
 
 # PyTorch (choose one)
-pip install torch torchaudio --index-url https://download.pytorch.org/whl/cu126  # CUDA
-pip install torch torchaudio --index-url https://download.pytorch.org/whl/cu128  # CUDA (RTX 50xx)
-pip install torch torchaudio --index-url https://download.pytorch.org/whl/cpu    # CPU only
+uv pip install torch torchaudio --index-url https://download.pytorch.org/whl/cu126  # CUDA
+uv pip install torch torchaudio --index-url https://download.pytorch.org/whl/cu128  # CUDA (RTX 50xx)
+uv pip install torch torchaudio --index-url https://download.pytorch.org/whl/cpu    # CPU only
 
 # Dependencies
-pip install -r requirements.txt
-pip install funasr --no-deps
+uv sync --locked --inexact --no-install-package torch --no-install-package torchaudio
+uv pip install funasr --no-deps
+uv pip install "sherpa-onnx>=1.13.3" "sherpa-onnx-bin>=1.13.3"
 
 # Launch
 .venv\Scripts\python.exe main.py
 ```
 
-> FunASR uses `--no-deps` because `editdistance` requires a C++ compiler. `editdistance-s` in `requirements.txt` is a pure-Python drop-in replacement.
+> FunASR uses `--no-deps` because `editdistance` requires a C++ compiler. `editdistance-s` in `pyproject.toml` is a pure-Python drop-in replacement.
+> `uv sync` installs the optional FireRedVAD Python package; its model is downloaded separately only if you choose to use that VAD backend.
 
 </details>
 
@@ -97,6 +99,55 @@ pip install funasr --no-deps
 2. Silero VAD + SenseVoice models download automatically (~1GB)
 3. Main UI appears when ready
 
+## FireRedVAD Models
+
+FireRedVAD is optional and is not downloaded during first launch. To use it, download the official model under `models/FireRedVAD`:
+
+```powershell
+modelscope download --model xukaituo/FireRedVAD --local_dir ./models/FireRedVAD
+# or
+huggingface-cli download FireRedTeam/FireRedVAD --local-dir ./models/FireRedVAD
+```
+
+Make sure these files exist:
+
+```text
+models/FireRedVAD/Stream-VAD/cmvn.ark
+models/FireRedVAD/Stream-VAD/model.pth.tar
+```
+
+Then open Settings → VAD/ASR, choose `FireRedVAD`, click Refresh, and select the local Stream-VAD model. The first integration uses FireRedVAD only as a streaming speech-confidence backend; LiveTranslate still owns segmentation, silence handling, backtrack splitting, incremental ASR, and ASR queueing. FireRed AED is not connected.
+
+## sherpa-onnx Models
+
+LiveTranslate supports sherpa-onnx local ONNX models through Python `OfflineRecognizer` and `OnlineRecognizer` APIs. Online models are currently decoded as a VAD-segment wrapper, not as true partial streaming ASR. `install.ps1` installs the CPU wheel by default. CUDA requires replacing it with a CUDA wheel, for example:
+
+```powershell
+powershell -ExecutionPolicy Bypass -File install.ps1 -SherpaOnnxRuntime cuda12
+```
+
+Download sherpa-onnx ASR model archives from the official sherpa-onnx releases, extract them anywhere under `models/`, then open Settings → VAD/ASR, choose `sherpa-onnx (ONNX)`, click Refresh, and select the local model directory. Online transducer scans accept `encoder.onnx`/`decoder.onnx`/`joiner.onnx` and int8 variants such as `encoder.int8.onnx`/`decoder.int8.onnx`/`joiner.int8.onnx`. PR #3671 Nemotron packages are published with names like `sherpa-onnx-nemotron-3.5-asr-streaming-0.6b-560ms-int8-2026-06-11`; unofficial snapshots must still have ONNX files accepted by the installed sherpa-onnx/ONNX Runtime version. The `onnx-community/nemotron-3.5-asr-streaming-0.6b-onnx-int4` layout is not treated as a sherpa-onnx model in this path.
+
+## parakeet.cpp Models
+
+parakeet.cpp is optional and is not downloaded during first launch. It uses local GGUF models plus a native parakeet.cpp runtime. The first integration uses LiveTranslate's existing VAD segments and ASR worker process; parakeet.cpp streaming EOU is not connected yet.
+
+Optional installer commands:
+
+```powershell
+powershell -ExecutionPolicy Bypass -File install.ps1 -InstallParakeetCpp -ParakeetCppBackend cpu
+powershell -ExecutionPolicy Bypass -File install.ps1 -DownloadParakeetCppModel -ParakeetCppModel tdt_ctc-110m-q4_k
+```
+
+Manual layout:
+
+```text
+models/parakeet.cpp/runtime/v0.3.2/<cpu|cuda|vulkan>/
+models/parakeet.cpp/models/tdt_ctc-110m-q4_k.gguf
+```
+
+Then open Settings → VAD/ASR, choose `parakeet.cpp (GGUF)`, click Refresh for both model and runtime, and select them. CUDA runtime may require the matching `cudart-parakeet-bin-win-cuda-x64.zip` asset next to the runtime DLLs.
+
 ## Translation API
 
 Settings → Translation tab:
@@ -108,22 +159,28 @@ Settings → Translation tab:
 | Model | `deepseek-chat` |
 | Proxy | `none` / `system` / custom URL |
 
+Real API keys are stored in `user_settings.json`, which is git-ignored. Keep `config.yaml` free of real credentials.
+
 ## Architecture
 
 ```
-Audio (WASAPI 32ms) → VAD (Silero) → ASR → LLM Translation → Overlay
+Audio (WASAPI 32ms) → VAD (Silero / FireRedVAD / Energy) → ASR → LLM Translation → Overlay
          ↑ optional mic mix-in
 ```
 
 ```
 main.py                 Entry point & pipeline
 ├── audio_capture.py    WASAPI loopback + mic mix-in
-├── vad_processor.py    Silero VAD
+├── vad_processor.py    VAD state machine (Silero / FireRedVAD / energy / disabled)
+├── vad_firered.py      FireRedVAD streaming frame adapter
 ├── asr_engine.py       faster-whisper backend
 ├── asr_funasr.py       Unified FunASR model selector backend
 ├── asr_sensevoice.py   SenseVoice backend
 ├── asr_funasr_nano.py  FunASR Nano backend
 ├── asr_anime_whisper.py Anime-Whisper backend (ja anime/galgame)
+├── asr_crispasr.py     CrispASR ggml runtime backend
+├── asr_sherpa_onnx.py  sherpa-onnx OfflineRecognizer/OnlineRecognizer backend
+├── asr_parakeet_cpp.py parakeet.cpp C API GGUF backend
 ├── asr_remote.py        Remote Whisper client (→ asr_server.py, see REMOTE_ASR.md)
 ├── translator.py       OpenAI-compatible client (streaming, JSON schema, context)
 ├── model_manager.py    Model download & cache
@@ -138,7 +195,11 @@ main.py                 Entry point & pipeline
 - [faster-whisper](https://github.com/SYSTRAN/faster-whisper) — Whisper inference via CTranslate2
 - [FunASR](https://github.com/modelscope/FunASR) — SenseVoice / Fun-ASR-Nano
 - [Anime-Whisper](https://huggingface.co/litagin/anime-whisper) — Japanese anime/galgame ASR
+- CrispASR — ggml C++ ASR runtime hub with GGUF/bin single-file models, used through its Python binding in the ASR worker
+- [sherpa-onnx](https://github.com/k2-fsa/sherpa-onnx) — ONNX ASR runtime used through `OfflineRecognizer` and segment-wrapped `OnlineRecognizer`
+- [parakeet.cpp](https://github.com/mudler/parakeet.cpp) — NVIDIA NeMo Parakeet GGUF inference through the C API in the ASR worker
 - [Silero VAD](https://github.com/snakers4/silero-vad) — Voice activity detection
+- [FireRedVAD](https://github.com/FireRedTeam/FireRedVAD) — Optional streaming VAD confidence backend
 
 ## Star History