FocalCap : Compressed Video Captioning

Note: FocalCap is an extension to the official CoCap (Compressed Video Captioning) project.

Introduction

FocalCap extends the end-to-end video captioning method based on compressed domain information from the encoded H.264 videos. This approach aims to accurately generate captions for compressed videos in a fast and efficient manner. The key extensions include a new 2-phase training procedure (Distilled Motion MAE) for pretraining, and the integration of an AGDTR module to include selective patches that enrich the caption generation.

Note: The current architecture does not include a CLAP audio encoder.

Preparation

1. Install the Requirements

To run the code, please install the dependency libraries by using the following command:

sudo apt update && sudo apt install default-jre -y  # required by pycocoevalcap
pip3 install -e . # See `requirements.txt` for exact versions used in development

Additionally, you will need to install the compressed video reader as described in the README.md of Compressed-Video-Reader.

2. Prepare the Pretrained Models

The model requires pretrained CLIP weights and optionally phase-1 motion module checkpoints. You can run the following script to download the weights before training to avoid network issues:

sudo apt update && sudo apt install aria2 -y  # install aria2
bash model_zoo/download_model.sh

This will download the CLIP model to model_zoo/clip_model. Note that this directory is hard-coded in our code.

3. Prepare the Data

Datasets are stored in the dataset folder under the project root. For detailed instructions on downloading and preparing the training data, please refer to dataset/README.md.

Training & Evaluation

The training is configured using YAML, and all the configurations are listed in configs/compressed_video.

Phase 1: Distilled Motion MAE (Pretraining)

FocalCap introduces a phase-1-only pretraining project for distilled motion learning from compressed video GOPs.

• MotionStudent with output fixed at 8 motion tokens per GOP
• Distillation decoder for CLIP-delta prediction
• Kendall uncertainty weighting in loss via log_var_cos and log_var_mse

Train Phase 1:

python tools/train_net.py # Ensure phase 1 config is set

Useful overrides for Phase 1:

Use pre-extracted .pt features:

python tools/train_net.py model.use_preextracted_features=true

Change batch size/workers:

python tools/train_net.py train_dataloader.batch_size=8 train_dataloader.num_workers=4

Change output directory:

python tools/train_net.py trainer.default_root_dir=./logs/my_pretrain_run

Continue from checkpoint (Full-state resume):

python tools/train_net.py ckpt_path=./logs/vatex_pretrain/checkpoints/best/last.ckpt

Phase 2: Captioning

You can use the following commands to run the captioning experiments:

# msrvtt
python3 tools/train_net.py --config-name=exp/train/msrvtt_captioning
# msvd
python3 tools/train_net.py --config-name=exp/train/msvd_captioning
# vatex
python3 tools/train_net.py --config-name=exp/train/vatex_captioning

Logs and results will be saved to ./logs/<experiment_name>/. The loss and metrics are visualized using tensorboard.

Checkpoint Locations

Default root for phase-1: ./logs/vatex_pretrain

Motion module checkpoints are saved to:

• ./logs/vatex_pretrain/checkpoints/modules/motion_student_last.pt
• ./logs/vatex_pretrain/checkpoints/modules/motion_student_best.pt

Compatibility aliases:

• motion_encoder_last.pt
• motion_encoder_best.pt

Citation

@inproceedings{cocap,
      title={Accurate and Fast Compressed Video Captioning}, 
      author={Yaojie Shen and Xin Gu and Kai Xu and Heng Fan and Longyin Wen and Libo Zhang},
      booktitle={Proceedings of the IEEE/CVF International Conference on Computer Vision},
      year={2023}
}

License

This project is licensed under the MIT License - see the LICENSE file for details.