LayerCam

Real-time 4K background removal & multi-layer compositing studio for Windows — captures a USB/UVC camera, isolates the human foreground with offline AI matting, composites it over a stacked layer list (video/image backgrounds, PNG/text overlays), plays synchronized audio, and records the result to .mp4 with hardware-accelerated encoders. Fully offline — no network access at install, build, or run time.

Implements the Project Requirement Specification in requirement.docx.

Two implementations share the project (and the same config/settings.json):

• Native C++ (primary) — built with Visual Studio 2019 (MSVC v142, CMake generator "Visual Studio 16 2019", x64). OpenCV + ONNX Runtime (DirectML) + Dear ImGui/DirectX 11 + bundled ffmpeg. Maximum 4K throughput.
• Python reference — PyQt5 + onnxruntime-directml; the original rapid prototype, kept runnable for development (py-* tasks).

Quick start

build_deploy.bat         REM SDK extract + VS2019 build + self test + deploy apps\LayerCam\LayerCam.exe
run.bat                  REM one-click launch (native build if present)
build_deploy.bat run     REM build (if needed) and launch the native app
build_deploy.bat py-run  REM Python reference implementation

The single command build_deploy.ps1 handles dependency linking, environment initialization, source compilation, verification, and output assembly packaging (spec §3.2). All libraries are housed inside offline-sdk\ — only Visual Studio 2019 itself lives outside the tree.

Features ↔ spec map

Spec	Native C++ (primary)	Python reference
§2.1 USB/UVC capture up to 4K@30	src/capture/camera.cpp — MSMF→DShow→default fallback, MJPG FourCC for 4K, capture thread, center-crop aspect helper	src/capture/camera.py
§2.2 Offline AI segmentation	src/inference/segmenter.cpp — Robust Video Matting, ONNX Runtime C++ API, DirectML→CPU fallback	src/inference/segmenter.py
§2.2 Edge refinement	src/inference/refine.cpp — de-halo alpha shift + bilateral + feather (live sliders)	src/inference/refine.py
§2.3 Multi-layer engine	src/compositor/layer_engine.cpp — background (color/image/video), overlays (PNG/text), top foreground matte	src/compositor/layers.py
§2.4 Audio + A/V sync	src/compositor/audio_out.cpp — waveOut playback; audio sample clock (waveOutGetPosition) is the master timebase for video PTS; external MP3/WAV override	src/compositor/audio.py
§2.5 Recording	src/compositor/recorder.cpp — raw frames piped to bundled ffmpeg; NVENC→QuickSync→AMF→x264/x265 auto-probe; H.264/H.265 .mp4	src/compositor/recorder.py
§3.1 Strict offline	zero network calls; all libs in offline-sdk\	same
§3.2 Single-command build	build_deploy.ps1 — CMake + Visual Studio 16 2019 x64	py-* tasks
§5 UI/UX	src/ui/*.cpp — Dear ImGui + DirectX 11 dark studio theme, drag-and-drop layer reorder, live preview, 1080p↔4K dropdown	src/ui/*.py (PyQt5)

Project layout (spec §4)

layerCam/
├── vs/                  # ── PRIMARY: native C++ project (Visual Studio 2019) ──
│   ├── CMakeLists.txt   #   generator "Visual Studio 16 2019", x64
│   └── src/
│       ├── main.cpp     #   entry point (GUI + --selftest)
│       ├── capture/     #   camera controls and frame ingest
│       ├── inference/   #   RVM segmenter (ONNX/DirectML) + edge refinement
│       ├── compositor/  #   layers, audio clock, ffmpeg recorder, engine
│       └── ui/          #   Dear ImGui + DirectX 11 dark UI
├── python/              # ── Python reference implementation (PyQt5) ──
│   ├── src/             #   same module layout as vs/src
│   ├── tests/smoke_test.py
│   ├── LayerCam.spec    #   PyInstaller config
│   ├── requirements.txt
│   └── run.bat
├── apps/                # Final built binaries (build_deploy output)
│   └── LayerCam/        #   native app (LayerCam-py/ for the Python bundle)
├── build/               # Temporary build targets (cmake, venv, PyInstaller)
├── config/              # settings.json + encoder (hardware) profiles
├── offline-sdk/         # Zero-internet dependency store
│   ├── models/          #   rvm_mobilenetv3.onnx
│   ├── ffmpeg/          #   ffmpeg.exe (decode, encode, mux)
│   ├── opencv/          #   OpenCV 4.10 prebuilt (x64/vc16 = VS2019)
│   ├── onnxruntime/     #   ONNX Runtime DirectML (headers + libs)
│   ├── directml/        #   DirectML.dll redistributable
│   ├── imgui/           #   Dear ImGui sources
│   ├── third_party/     #   nlohmann/json.hpp
│   ├── wheels/          #   all pip wheels (Python reference)
│   ├── python/          #   offline Python 3.10 installer
│   ├── python-portable/ #   zero-install Python runtime
│   └── _downloads/      #   original SDK archives (offline re-extract)
├── ref/                 # validation & literature notes
├── build_deploy.ps1     # unified single-command automation pipeline
└── run.bat              # one-click launcher (native build)

Populating offline-sdk/ (fresh clone)

offline-sdk/ (~2.3 GB of third-party binaries) is not committed — several files exceed GitHub's 100 MB limit. A fresh clone must place these back before building. Download the archives into offline-sdk/_downloads/, then run build_deploy.bat sdk to extract the C++ SDKs into the layout the build expects:

Path	Source
offline-sdk/_downloads/opencv-4.10.0-windows.exe	github.com/opencv/opencv/releases (4.10.0)
offline-sdk/_downloads/onnxruntime-directml-1.18.1.nupkg	nuget.org — Microsoft.ML.OnnxRuntime.DirectML 1.18.1
offline-sdk/_downloads/directml-1.15.0.nupkg	nuget.org — Microsoft.AI.DirectML 1.15.0
offline-sdk/_downloads/imgui-1.90.9.zip	github.com/ocornut/imgui (v1.90.9)
offline-sdk/_downloads/json.hpp	github.com/nlohmann/json (v3.11.3)
offline-sdk/models/rvm_mobilenetv3.onnx	github.com/PeterL1n/RobustVideoMatting (ONNX export)
offline-sdk/ffmpeg/ffmpeg.exe	any recent FFmpeg Windows build
offline-sdk/wheels/ + offline-sdk/python-portable/	Python-reference deps (see python/requirements.txt)

build_deploy.bat sdk extracts OpenCV, ONNX Runtime, DirectML, ImGui and json from _downloads/ into offline-sdk/{opencv,onnxruntime,directml,imgui,third_party}.

Requirements

• Windows 10/11. Python 3.10 — but nothing to install: the bundled offline-sdk\python-portable\ runtime is used automatically.
• Any DX12 GPU gives real-time matting via DirectML (no CUDA toolkit needed). CPU-only works but is ~3× slower; set LAYERCAM_DEVICE=cpu to force it.

Per-layer colour (HSV)

Background and camera (foreground) layers have live Hue / Saturation / Brightness sliders in the Layers panel. Hue shifts in degrees (−180…180), saturation and brightness scale 0…2× (1.0 = unchanged). The grade is applied per frame only when a layer is non-default, so it costs nothing otherwise.

Full-screen

F11 toggles full-screen; Esc exits. In full-screen the toolbar and the right-hand Layers/Output panel are hidden — the composited preview fills the whole screen. Leaving full-screen restores them.

Supported background video formats

.mp4 .mkv .mov .avi .webm .mpg .mpeg .wmv .m4v .flv .ts .vob. Each file is opened with FFmpeg first and falls back to the platform's native decoder (Media Foundation) for odd MPG/AVI variants.

Performance notes

Native C++ compositor throughput (this machine, movie background + overlay, measured by LayerCam.exe --selftest): 1080p ≈ 30 fps (with a movie audio track and recording active) and 4K ≈ 30 fps. The Python reference is slower at high resolution (pure-NumPy compositing) — use the native build for 4K.

RVM matting cost scales with the Quality (downsample ratio) knob, not the canvas size — 4K output with 0.25 matting runs at interactive rates on a mid-range GPU. 0.5 gives the crispest hair edges at a frame-rate cost. 4K capture requires a camera that delivers MJPG 3840×2160 (most 4K UVC cams).

A/V clock: the master timebase is a wall clock aligned to audio start, not the audio device's sample counter — decoupling the render loop from the audio callback keeps a movie background + recording from stalling under load.

Recording

Click ● Record. The first recording probes encoders (NVENC → QuickSync → AMF → software) with a tiny test encode and keeps the best one; the status bar shows which is in use. Audio heard during recording (background-video track or external override) is muxed into the final .mp4. Encoder argument chains are editable in config/encoder_profiles.json.

Troubleshooting

• No camera image — another app may hold the camera; check Windows Settings ▸ Privacy ▸ Camera. The app auto-tries Media Foundation, DirectShow and default backends and shows the reason on-canvas.
• onnxruntime DLL initialization failed — usually antivirus scanning freshly-extracted DLLs, or a OneDrive-synced folder. Keep the project on a local path; the app retries and degrades to the raw feed instead of crashing.
• 4K only reaches ~15 fps — lower matting Quality to 0.25/0.125, enable GPU inference, and prefer NVENC/QSV for recording.