Skip to content

README.md

Latest commit

 

History

History
71 lines (51 loc) · 2.59 KB

README.md

File metadata and controls

71 lines (51 loc) · 2.59 KB
Raw

FlowAvatar

Real-Time Full-Body Avatars from Sparse Egocentric Inputs on Consumer XR Devices IEEE ISMAR 2026

FlowAvatar is a unified system for real-time personalized look-alike avatars with full-body pose, hand articulation, facial expression, and eye gaze tracking from sparse egocentric inputs (head + hands) on consumer XR devices (Meta Quest Pro).

Repository layout

Directory Content
flowavatar/ Shared Python package: pose network definitions, checkpoint loading, rotation/FK utilities
streaming/ Streaming configuration: Python inference server that receives Quest tracking over TCP and streams SMPL-X pose back to Unity
training/ Training and evaluation on AMASS (coming soon)
unity/ Unity (Meta Quest) client: streaming and on-device (Sentis) configurations (coming soon)
avatar-pipeline/ Personalized avatar enrollment: phone video → UV texture + shape (coming soon)
docs/ Protocol and architecture documentation

Streaming demo quick start

  1. Install dependencies (Python ≥ 3.10):

    pip install -r requirements.txt

  2. Download the SMPL-X body model. Register at smpl-x.is.tue.mpg.de, download SMPLX_NEUTRAL.npz, and place it at body_models/smplx/SMPLX_NEUTRAL.npz (or set the SMPLX_MODEL_PATH environment variable). The SMPL-X model is licensed separately by MPI-IS and cannot be redistributed here.

  3. Download checkpoints — see checkpoints/README.md.

  4. Run the server:

    python -m streaming.live_demo --model gru

    The server waits for the Unity client on TCP ports 8888 (tracking in) and 8889 (pose out); see docs/streaming_protocol.md.

  5. Start the Unity client (see unity/, coming soon) on the same machine or point it at the server's IP.

Model options: gru (default, best accuracy) and lstm (the on-device variant used for Unity Sentis export).

On-device (Unity Sentis) export

Unity Sentis has no GRU operator, so the on-device configuration uses the LSTM variant:

pip install onnx onnxruntime onnx-simplifier
python -m streaming.export_onnx --model_path checkpoints/LSTM/baseline.pt

License

Code license TBD. Note that the SMPL-X body model, the AMASS dataset, and the human_body_prior package are licensed separately by MPI-IS for non-commercial scientific research and are not covered by this repository's license.