FlowAvatar

Real-Time Full-Body Avatars from Sparse Egocentric Inputs on Consumer XR Devices IEEE ISMAR 2026

FlowAvatar is a unified system for real-time personalized look-alike avatars with full-body pose, hand articulation, facial expression, and eye gaze tracking from sparse egocentric inputs (head + hands) on consumer XR devices (Meta Quest Pro).

Repository layout

Directory	Content
flowavatar/	Shared Python package: pose network definitions, checkpoint loading, rotation/FK utilities
streaming/	Streaming configuration: Python inference server that receives Quest tracking over TCP and streams SMPL-X pose back to Unity
training/	Training and evaluation on AMASS (coming soon)
unity/	Unity (Meta Quest) client: streaming and on-device (Sentis) configurations (coming soon)
avatar-pipeline/	Personalized avatar enrollment: phone video → UV texture + shape (coming soon)
docs/	Protocol and architecture documentation

Streaming demo quick start

1. Install dependencies (Python ≥ 3.10):

   pip install -r requirements.txt

2. Download the SMPL-X body model. Register at smpl-x.is.tue.mpg.de, download SMPLX_NEUTRAL.npz, and place it at body_models/smplx/SMPLX_NEUTRAL.npz (or set the SMPLX_MODEL_PATH environment variable). The SMPL-X model is licensed separately by MPI-IS and cannot be redistributed here.

3. Download checkpoints — see checkpoints/README.md.

4. Run the server:

   python -m streaming.live_demo --model gru

   The server waits for the Unity client on TCP ports 8888 (tracking in) and 8889 (pose out); see docs/streaming_protocol.md.

5. Start the Unity client (see unity/, coming soon) on the same machine or point it at the server's IP.

Model options: gru (default, best accuracy) and lstm (the on-device variant used for Unity Sentis export).

On-device (Unity Sentis) export

Unity Sentis has no GRU operator, so the on-device configuration uses the LSTM variant:

pip install onnx onnxruntime onnx-simplifier
python -m streaming.export_onnx --model_path checkpoints/LSTM/baseline.pt

License

Code license TBD. Note that the SMPL-X body model, the AMASS dataset, and the human_body_prior package are licensed separately by MPI-IS for non-commercial scientific research and are not covered by this repository's license.