ViFT: Do we Really Need Visual Instructions? Towards Visual Instruction-Free Fine-tuning for Large Vision-Language Models
-
ViFT is the first instruction-free fine-tuning method with comparable performance to SOTA LVLMs.
-
We specially designed the training and inference methods for disentangling and combining natural language task-solving and visual perception abilities, to efficiently improve the multimodal capabilities of LVLMs.
-
Our ViFT is a low-cost approach for scaling data to improve LVLMs. Experimental results demonstrate the effectiveness of our approach on several benchmarks.
- python
- transformers
- pytorch
- accelerate
- openai==0.28.0
We prepare all the training datasets here:
-
first_stage_captions.json contains 1m web captions sampled from laion, which is used for first-stage training(or pretraining, which can be skipped with compromised performance)
-
second_stage_captions.json contains 1.7M high-quality captions collected or synthesized via Qwen2-VL-72B. We collect image from different data source, encompassing a broad range of visual domain and visual tasks.
-
text_data_magpie.json and text_data_qwen_distill.json contain text-only data distilled from Qwen2.5-72B for inheriting and enhancing task-solving ability from LLM. They are mixed with second_stage_captions for second-stage training.
Please refer to lmm-evals.
