Frame-accurate video decode via mediabunny — fixes video/pose mismatch

[ericleonardis]

Frame-accurate video decode via mediabunny — fixes video/pose mismatch (#115)

Closes #115.

Summary

When stepping through frames, the pose skeleton could sit on the wrong video frame
(most visible on fast-moving nodes like the tail), making predictions look misplaced
"reprojection-like". This PR fixes it by decoding video frames through sleap-io.js's
frame-accurate MediaBunnyVideoBackend (WebCodecs) instead of relying on HTML5
<video>.currentTime seeking, which is not frame-accurate.

Confirmed fixed on the original (un-re-encoded) HEVC session on real hardware:
video and pose now match while stepping.

Root cause

The pose data was never the problem — it imports byte-for-byte correctly and exports
perfectly (verified across all three SLP backends and confirmed by round-tripping through
the SLEAP GUI). The bug is display-only:

OnDemandVideoDecoder extracts every frame by seeking the HTML5 <video> element
(currentTime = frameIdx / fps). That seek is not frame-accurate — for a long-GOP
video it can return a frame a whole GOP behind the one requested (currentTime reads
back exactly right, but the displayed/captured frame is stale). The correct skeleton
overlay then draws on top of a stale video image, so the animal has moved and the tail
misses.

(HTML5 was the only live extraction path — the class's WebCodecs branch was disabled from
the start, and its decodeRange was never actually exercised.)

The fix

sleap-io.js already ships MediaBunnyVideoBackend, which decodes exact frames via
WebCodecs (sink.getSample(frameTimestamp)) and handles keyframe seeking correctly.
mediabunny was stubbed out in LUCID; this PR vendors the real library and routes the
per-frame path through the backend.

• Vendor mediabunny — lib/mediabunny/mediabunny.min.mjs (npm mediabunny@1.30.0
  browser ESM, matching sleap-io.js's ^1.30.0). Importmap mediabunny repointed from
  the stub to it in index.html + tests/test-runner.html; MediaBunnyVideoBackend
  exposed on window.SleapIO.
• loading/video.js — OnDemandVideoDecoder gains a frame-accurate path:
  init() builds MediaBunnyVideoBackend.fromBlob (_initMediabunny) and adopts its
  authoritative frame count / fps; getFrame() decodes exact frames through it with a
  transparent HTML5 fallback on any miss/error; playback still uses the HTML5 element;
  close() frees the backend.
• VideoSample-leak patch — lib/sleap-io/chunk-M65RB7KH.js: sleap-io's backend called
  sample.toVideoFrame() but never closed the underlying VideoSample
  ("A VideoSample was garbage collected without first being closed"), which can exhaust
  the WebCodecs frame pool over a long session. Both sites now sample.close(), marked
  // LUCID local patch (#115). ⚠️ Re-apply on any sleap-io re-vendor (grep the
  marker); should be reported upstream.

Enabling it (currently opt-in)

Off by default (zero change to current behavior). Enable per-browser:

localStorage.LUCID_VIDEO_BACKEND = 'mediabunny'   // DevTools console, then reload + reload session

Confirm it engaged — the console shows [video] … [mediabunny frame-accurate], or:

window.__lucid.state.views.forEach(v =>
  console.log(v.name, v.decoder && v.decoder._mbBackend ? 'MEDIABUNNY ✅' : 'html5'));

Disable with delete localStorage.LUCID_VIDEO_BACKEND + reload.

Why opt-in, not default (yet): this couldn't be validated in headless CI — headless
Chromium's software decoder is itself frame-inaccurate, so every WebCodecs decoder
(mediabunny, a hand-rolled one, and even a raw <video>) returns the same wrong frame
there. It's verified only on real hardware so far, and it's only been exercised on one
dataset. Flipping it to default-on (with an ='html5' escape hatch) is a reasonable
follow-up once it's seen more videos.

Testing

• tests/test-mediabunny-backend.js (new, 6 cases) — guards the integration wiring:
  flag detection (_mediabunnyEnabled), getFrame routing through the backend, HTML5
  fallback on null/throw, HTML5-direct when no backend, and close() releasing it. These
  deliberately don't assert decode frame-accuracy (not headless-testable — see above).
• Full browser suite: 1064/1064 passing with the real mediabunny loaded + the leak
  patch (was 1058 + 6 new). No page errors from the importmap swap.
• Real-hardware: confirmed the video/pose mismatch is gone on the original HEVC session.

Not needed: re-encoding

We also tried re-encoding the HEVC to short-GOP H.264, which helped the HTML5 path a
little but did not fully fix it — the mediabunny backend fixes it on the original
videos, so no re-encode is required.

Files changed

File	Change
lib/mediabunny/mediabunny.min.mjs, PROVENANCE.txt	vendored mediabunny 1.30.0 (new)
lib/sleap-io/chunk-M65RB7KH.js	VideoSample-leak patch (2 sites)
index.html, tests/test-runner.html	importmap → real mediabunny; export backend
loading/video.js	opt-in mediabunny backend in OnDemandVideoDecoder
tests/test-mediabunny-backend.js	new tests
MODULES.md, CLAUDE.md	docs (incl. the local-patch note)

The old lib/sleap-io/mediabunny-stub.js is now unused (left in place; harmless).

Base branch

This branch (eric/prediction-mismatch) is stacked on the cross-view-tracker PR (#131).
Target this PR at that branch, or rebase onto main after #131 lands, so the diff is only
the files above.

[github-actions]

Preview Deployment

Preview has been removed.

[ericleonardis]

Hacky quick fix: snap-to-frame on pause

Playback now runs on the native <video> path by default (smooth), with the three
heavy per-frame updates — info panel, timeline redraw(), and the 3D viewport scene
rebuild — throttled to ~10 Hz during playback so the overlay can keep up. That gets us
most of the way, but native playback can still settle a hair off-frame when you pause
(the residual "tracking leads the video" lag).

Rather than chase perfect real-time sync on the native path, this PR adds a deliberate
band-aid on the pause action:

On user-pause, stop playback and then do a frame-accurate mediabunny seek one frame
forward (seekToFrame(currentFrame + 1)), so we always land exactly on-frame with
the pose overlay.

This is literally automating the thing we already knew fixed it — pressing "next frame"
after pausing snaps the video and tracking back into alignment, because a step goes
through the frame-accurate mediabunny decode path. Now it happens for you on pause.

Scope (intentionally narrow):

• Only the explicit pause controls call it — the play button and spacebar
  (VideoController.pausePlayback()).
• Internal stops — scrubbing, session teardown, source switch, natural end-of-video —
  still call stopPlayback() directly and get no forward nudge.

Known trade-off: pausing advances the frame by one (that's the "forward" in the fix).
It's the simplest thing that guarantees an on-frame landing; if it feels like an
overshoot we can re-seek the same frame instead (one-line change).

This is explicitly a quick fix, not the real solution — see the follow-up issue on the
residual playback lag for the proper path (composite the <video> element directly under
a transparent overlay, or hardware-accelerated decode).

Tests: pausePlayback covered in tests/test-video-controller.js (steps one frame
forward on pause, clamps at the last frame, no-ops when not playing).