x86: H.265/HEVC video fails to play — Qt6 VAAPI hardware decode errors under cage (FormatError, 0 frames)

[vpetersson]

Summary

On x86, an H.265/HEVC video (RTSP stream tested; likely any HEVC source) renders black on the display. The C++ viewer dispatches it, but QMediaPlayer immediately emits code=2 (FormatError) "Input/output error" and 0 frames are decoded/rendered. H.264 plays fine on the same x86 board, and H.265 hardware-decodes fine on the Pi 5 (Hantro G2), so this is x86-specific.

This was found while testing streaming support (PR #3068 follow-up / PR #3071). It is not caused by the streaming classification changes — it reproduces with any HEVC source on x86.

Environment

• x86 (Intel UHD 610 / Whiskey Lake-U), Mesa iHD VAAPI driver
• anthias-viewer (Qt 6.8.2 QMediaPlayer, FFmpeg backend) running under cage (Wayland)
• Symptom visible in /data/.anthias/playback-stats.log:

  LOADFILE uri=rtsp://<host>:8554/hevc ...
  ERROR uri=rtsp://<host>:8554/hevc code=2 message=Input/output error
  INVALID_MEDIA uri=...
  

It is NOT a decode-capability problem

Each layer was ruled out:

Check	Result
vainfo on the iGPU	✅ lists VAProfileHEVCMain / Main10 (VAEntrypointVLD)
ffmpeg software decode of the stream	✅ decodes
ffmpeg -hwaccel vaapi decode	✅ decodes (VAAPI HEVC works standalone)
Offscreen AnthiasViewer (QT_QPA_PLATFORM=offscreen, no RHI/GL) via VAAPI	✅ 743 frames decoded (verified with QT_FFMPEG_DEBUG=1 QT_LOGGING_RULES="*.ffmpeg.*=true")
Real viewer (cage Wayland + GL/RHI) via VAAPI	❌ FormatError / "Input/output error", 0 frames

The only failing configuration is VAAPI-decoding HEVC while QMediaPlayer runs inside cage's EGL/GL context. H.264 over the same VAAPI-in-cage path works, so it is codec-specific. Mechanistically, Qt appears to derive its VAAPI device from the active EGL/Wayland display, and for HEVC the iHD surface handling in that derived context errors out with no software fallback.

Workarounds tested

• QT_DISABLE_HW_TEXTURES_CONVERSION=1 — ❌ does not help (so it is not just the zero-copy texture upload).
• QT_FFMPEG_DECODING_HW_DEVICE_TYPES=, (disable HW decode) — ✅ HEVC then plays cleanly via software (2164 frames, ~6 dropped, near-full framerate on UHD 610).

Proposed direction (surgical fix wanted)

Blanket-disabling HW decode on x86 also drops H.264 to software (more CPU), so a less blunt fix is preferable, e.g.:

• Detect the VAAPI HEVC failure at QMediaPlayer::errorOccurred and retry once with HW decode disabled (per-asset SW fallback), or
• Set a curated QT_FFMPEG_DECODING_HW_DEVICE_TYPES only when needed (board/codec conditional in start_viewer.sh), or
• Investigate why Qt's VAAPI-via-EGL HEVC surface path fails under cage on iHD (possible upstream Qt/driver bug) and address at that layer.

Repro

1. On an x86 Anthias device (display via cage), add an HEVC video/stream asset.
2. Observe black screen; playback-stats.log shows ERROR ... code=2 ... Input/output error, 0 frames.
3. Add QT_FFMPEG_DECODING_HW_DEVICE_TYPES=, to the anthias-viewer service env and recreate it → HEVC plays.

🤖 Generated with Claude Code