Live Whisper

Real-time streaming speech-to-text server using faster-whisper (CTranslate2) on GPU, served over a WebSocket, with a browser page that streams your microphone to it.

How it works

• server.py — the transcription API. FastAPI app exposing /ws, a WebSocket per connection. Each connection streams raw 16kHz mono PCM16 audio; the server buffers it, periodically re-transcribes the trailing window for live "partial" captions, and commits a "final" line once it detects ~1.2s of silence. Runs on port 8000.
• webserver.py — just serves the static webpage (static/index.html) on its own port, separate from the API. Runs on port 8080.
• static/index.html — captures mic audio in the browser, downsamples to 16kHz PCM16, and streams it over a WebSocket to the API server's port (hardcoded in the page as API_PORT, currently 8000 — change it there if you run the API on a different port). Includes a language slider (Italian/English/Spanish/French/German) that's sent as ?lang= on the WebSocket URL — the server transcribes in that fixed language for the whole session (no auto-detection; it was unreliable on short rolling windows).

Model

Using Na0s/Medical-Whisper-Large-v3 — a large-v3 fine-tune on doctor/patient consultations — converted to CTranslate2 format locally at models/medical-whisper-large-v3-ct2/ (that's what WHISPER_MODEL points to by default). To reproduce or update this conversion:

./venv/bin/python3 -c "
from huggingface_hub import snapshot_download
snapshot_download('Na0s/Medical-Whisper-Large-v3', local_dir='models/medical-whisper-large-v3-hf')
"
# that repo ships slow-tokenizer files only; faster-whisper needs tokenizer.json
./venv/bin/python3 -c "
from transformers import WhisperTokenizerFast
tok = WhisperTokenizerFast.from_pretrained('models/medical-whisper-large-v3-hf')
tok.save_pretrained('models/medical-whisper-large-v3-hf')
"
./venv/bin/ct2-transformers-converter \
  --model models/medical-whisper-large-v3-hf \
  --output_dir models/medical-whisper-large-v3-ct2 \
  --copy_files tokenizer.json preprocessor_config.json \
  --quantization float32 --force

Community medical fine-tune — verify accuracy against your own recordings before relying on it; treat output as requiring clinician review, not autonomous documentation. To fall back to stock Whisper, set WHISPER_MODEL=large-v3 (or small, etc.) — faster-whisper will download it from Hugging Face automatically.

Setup

Detected hardware here: 3x NVIDIA GTX 1080 Ti (Pascal, compute capability 6.1). Pascal GPUs don't support efficient float16/int8_float16 execution in CTranslate2 — tested directly, both error out. compute_type=float32 (full precision, no quantization) is the default: benchmarking showed quantization's speed benefit mostly vanishes at the concurrency levels this hardware actually saturates at anyway, so there's no real reason to give up accuracy for it here. If you later run this on a newer GPU (Turing/Ampere+), float16 will be both faster and supported.

python3 -m venv venv
./venv/bin/pip install -r requirements.txt
cp .env.example .env   # then fill in WHISPER_API_TOKEN — see "Authentication" below

The medical model above needs converting once (see "Model" section). Stock Whisper model weights download automatically from Hugging Face on first run (needs internet the first time; cached under ~/.cache/huggingface after that).

TLS (required for the mic to work over the network)

Browsers only expose getUserMedia (mic access) on https:// or localhost origins. Since the page is reached over the network at a non-localhost address, it must be served over HTTPS or the mic prompt silently never appears. A self-signed cert is already generated at certs/cert.pem / certs/key.pem (covers this machine's LAN/public IP, 127.0.0.1, and localhost). Regenerate it if the IP changes:

MY_IP=$(ip -4 addr show enp179s0f0 | grep -oP 'inet \K[\d.]+')
openssl req -x509 -newkey rsa:2048 -nodes -keyout certs/key.pem -out certs/cert.pem -days 365 \
  -subj "/CN=whisper-live" \
  -addext "subjectAltName=IP:$MY_IP,IP:127.0.0.1,DNS:localhost"

Running

Two separate processes: the transcription API, and the webpage. Both need --ssl-keyfile/--ssl-certfile since both are reached from outside localhost.

# terminal 1 — transcription API (port 8000). Defaults already use all 3 GPUs.
./venv/bin/uvicorn server:app --host 0.0.0.0 --port 8000 \
  --ssl-keyfile certs/key.pem --ssl-certfile certs/cert.pem

# terminal 2 — webpage (port 8080)
./venv/bin/uvicorn webserver:app --host 0.0.0.0 --port 8080 \
  --ssl-keyfile certs/key.pem --ssl-certfile certs/cert.pem

To stop them: pkill -f "uvicorn server:app" and pkill -f "uvicorn webserver:app" (or kill <pid> from ps aux | grep uvicorn).

Then open https://<this-machine-ip>:8080/ (note https) from any browser on the network, click Start, allow mic access, and talk. The page connects out to port 8000 (also https/wss) for the actual transcription WebSocket. Your browser will warn about the self-signed certificate on both origins (8080 for the page, 8000 for the API) — click through "Advanced → Proceed" once for each; this is expected since it's not a CA-signed cert.

--host 0.0.0.0 is what makes each one reachable from other machines, not just localhost. If this box has a firewall (ufw, cloud security group, etc.), open both ports 8000 and 8080 for whichever clients need access.

If you serve the webpage on a different port than 8000, update API_PORT at the top of the <script> in static/index.html to match wherever server.py is actually running.

Running with Docker

Dockerfile + docker-compose.yml package both processes as containers built from the same image (nvidia/cuda:12.4.1-runtime-ubuntu22.04 + Python 3 + requirements.txt). The models/ and certs/ directories are bind-mounted read-only rather than baked into the image — the model weights are multiple GB and the certs contain a private key, neither belongs in an image layer.

One-time host setup: NVIDIA Container Toolkit

The transcription container needs GPU access, which requires Docker's nvidia runtime. This is a one-time install on the host (not inside any container):

# 1. Add NVIDIA's apt repo and signing key
curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg

curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | \
  sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | \
  sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list

# 2. Install the toolkit
sudo apt-get update
sudo apt-get install -y nvidia-container-toolkit

# 3. Register the `nvidia` runtime with Docker and restart the daemon
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker

# 4. Verify — should print all GPUs (e.g. the 3x GTX 1080 Ti on this box)
sudo docker run --rm --gpus all nvidia/cuda:12.4.1-base-ubuntu22.04 nvidia-smi

If you're on a very new Ubuntu release and apt-get update complains the repo's codename is unrecognized, NVIDIA's list is a generic (non-codename-pinned) deb repo, so this is usually transient — retry, or check https://github.com/NVIDIA/nvidia-container-toolkit/issues for current guidance.

You'll also want your user in the docker group so you don't need sudo for every docker command: sudo usermod -aG docker $USER, then log out and back in.

Build and run

The API (api) and webpage (web) are separate Compose services behind profiles, so they can be started independently — a bare docker compose up with no profile starts nothing:

# transcription API only (port 8000, GPU)
sudo docker compose --profile api up -d

# webpage only (port 8080)
sudo docker compose --profile web up -d

# both
sudo docker compose --profile api --profile web up -d

docker compose builds the image automatically on first run; rebuild after changing server.py, webserver.py, static/, or requirements.txt with:

sudo docker compose build

Logs: sudo docker compose logs -f api (or web). Stop: sudo docker compose --profile api --profile web down.

Both containers read certs/key.pem / certs/cert.pem from the host via bind mount, so generate the TLS cert (see "TLS" above) before starting them. The api container reads models/ the same way, so the model must already be converted/downloaded on the host first (see "Model" above) — nothing is downloaded inside the container.

WHISPER_MODEL, WHISPER_DEVICE, WHISPER_COMPUTE, WHISPER_DEVICE_INDEX, and WHISPER_NUM_WORKERS (see table below) are all overridable without editing the compose file — either export them in your shell before docker compose up, or put them in a .env file next to docker-compose.yml:

# .env
WHISPER_NUM_WORKERS=2
WHISPER_API_TOKEN=<your-secret>

WHISPER_API_TOKEN isn't optional under Compose — the api service is deliberately set to fail fast (${WHISPER_API_TOKEN:?...}) rather than start unauthenticated. See "Authentication" below for what this protects and how the webpage uses it.

Config via environment variables

Var	Default	Meaning
WHISPER_MODEL	models/medical-whisper-large-v3-ct2	Local CTranslate2 model path, or a stock size (tiny/small/medium/large-v3) to download from Hugging Face instead.
WHISPER_DEVICE	cuda	cuda or cpu.
WHISPER_COMPUTE	float32	CTranslate2 compute type. int8/float16 both fail outright on these Pascal GPUs — see "Setup".
WHISPER_DEVICE_INDEX	0,1,2	Comma-separated GPU indices to replicate the model across — defaults to all 3 GPUs on this box. CTranslate2 does not spread across GPUs unless told to.
WHISPER_NUM_WORKERS	4	Concurrent CTranslate2 inference workers per GPU in WHISPER_DEVICE_INDEX. At 4/GPU with this large-v3-sized model in float32, measured VRAM use is ~9-9.7GB out of 11GB per 1080 Ti — near the practical ceiling for this hardware. Total concurrent capacity ≈ WHISPER_NUM_WORKERS × (GPUs in WHISPER_DEVICE_INDEX).
WHISPER_API_TOKEN	(required, no default)	Shared-secret token required to open /ws, set in .env — see "Authentication" below.

Concurrency notes

• Each WebSocket connection keeps its own audio buffer — multiple people can connect and transcribe independently and simultaneously.
• Transcription calls run in a thread pool (WHISPER_NUM_WORKERS threads) so the asyncio event loop stays free to keep receiving audio from every connection while GPU inference is in progress.
• num_workers on WhisperModel tells CTranslate2 to actually run that many inference calls concurrently against the one loaded set of weights (rather than serializing them), so raising it is what lets the GPU do several transcriptions "insieme". With this large-v3-sized medical model in float32, 4 workers/GPU already uses ~9-9.7GB/11GB — don't push much further without re-checking for OOM.
• All 3 GPUs are used by default (WHISPER_DEVICE_INDEX=0,1,2) — confirmed via nvidia-smi that this actually allocates memory on each listed device, not just GPU 0. Combined with WHISPER_NUM_WORKERS, total concurrent transcription capacity is num_workers × number of GPUs listed (12 by default).
• Measured throughput on the earlier small/int8 config: saturates around ~11 calls/sec, supporting roughly 16 concurrent active speakers before live captions start lagging. The medical large-v3 model is slower per call, so that ceiling is lower now — re-benchmark if you need a precise number for this model.

Authentication

/ws requires a shared-secret token — each connection burns real GPU time, so it must not be left open to anyone who can reach the port. The token is passed as ?token= on the WebSocket URL (browsers' native WebSocket API can't set custom headers, so a query param is the practical option here) and checked with a constant-time comparison (secrets.compare_digest) before the connection is accepted; an invalid or missing token gets the handshake refused with close code 1008.

The token lives in a .env file in the project root (gitignored — never commit it), loaded automatically by both server.py (via python-dotenv) and docker compose (which reads .env natively). server.py refuses to start if WHISPER_API_TOKEN is missing entirely, so it can never come up silently unauthenticated.

cp .env.example .env
python3 -c "import secrets; print(secrets.token_urlsafe(32))"   # paste into .env

A real .env already exists in this checkout with a generated token — rotate it any time by generating a new value and pasting it in; both server.py and any running Compose containers need a restart to pick up the change.

The webpage (static/index.html) has an "API token" field that stores the value in that browser's localStorage and appends it to the /ws URL — paste the token there once per browser.

This only guards the WebSocket endpoint. The static webpage itself (webserver.py, port 8080) still has no auth — it serves no sensitive data on its own, but the token travels through it in the browser's localStorage/URL, so still keep it behind HTTPS (see "TLS" above) so the token isn't sent in the clear. For anything beyond a trusted LAN, also put both ports behind a reverse proxy (nginx/Caddy) with its own access control as defense in depth.