MiniMedGPT: Efficient Large Vision-Language Model For Medical Visual Question Answering

VQA System for Neuroradiology, Brain Tumors, and Lung Cancer Screening and More Using MRI and CT Scans

MiniMedGPT is designed to assist radiologists by using AI to interpret MRI and CT scans, specifically for brain tumors and lung cancer. This AI-driven chatbot aims to provide real-time answers to clinical questions, improve diagnostic accuracy, and enhance the efficiency of medical imaging workflows. By integrating such technology, the goal is to reduce the heavy workload on radiologists, expedite the diagnostic process, and ultimately improve patient outcomes through earlier and more precise detection of critical health issues.

Installation

1- Prepare the code and the environment
Git clone our repository, create a Python environment, and activate it via the following commands:

git clone 
cd MiniMedGPT
conda env create -f environment.yml
conda activate MiniMedGPT

2- Prepare the pretrained LLM weights
MiniMedGPT is a finetuned model of MiniGPT-v2, which is based on Llama2 Chat 7B. Download the LLM weights from the following Hugging Face space via cloning the repository using git-lfs. Download
3- Prepare the pretrained model checkpoints
Download the pretrained model checkpoints.
Download
Set the path to the pretrained checkpoint in the evaluation config file eval_configs/minigptv2_eval.yaml at Line 8.
Download our pretrained model.

Launching Demo Locally

Run:

python demo_v2.py --cfg-path eval_configs/minigptv2_eval.yaml --gpu-id 0

To save GPU memory, LLMs load as 8-bit by default, with a beam search width of 1. This configuration requires about 23G GPU memory for 13B LLM and 11.5G GPU memory for 7B LLM. For more powerful GPUs, you can run the model in 16-bit by setting low_resource to False in the relevant config file: minigptv2_eval.yaml

Training

You can download the data we used here.
In the train_configs/minigptv2_finetune.yaml, you need to set up the following paths:

llama_model checkpoint path: ```/path/to/llama_checkpoint```
ckpt: ```/path/to/pretrained_checkpoint```
ckpt save path: ```/path/to/save_checkpoint```

For ckpt, you may load from our pretrained model checkpoints:
Download

torchrun --nproc-per-node NUM_GPU train.py --cfg-path train_configs/minigptv2_finetune.yaml

Evaluation

Evaluation dataset structure

├── Slake
│   └── imgs
│   ├── train.json
│   ├── validate.json
│   ├── test.json

Requirements

• python 3.8
• scikit-image (visit this page for installation)
• matplotlib (visit this page for installation)

Examples

Check out the examples below:

Acknowledgement

Thanks to the creators of MiniGPT-v2, check out their website here.