Can the Stage 1 Trained Weight (efficient_sam3_repvit_m0_9_mobileclip_s0.pth) Be Used for Inference?

[Lali-Lali]

I have completed stage 1 training and obtained efficient_sam3_repvit_m0_9_mobileclip_s0.pth in efficientsam3/stage1/output/. I have not yet performed Stage 1 Fine-tuning — Geometry-Aware Distillation. However, I am curious: can the efficient_sam3_repvit_m0_9_mobileclip_s0.pth I temporarily obtained be used for inference? I did not find the usage instructions for efficient_sam3_repvit_m0_9_mobileclip_s0.pth in the stage 1 README file. I want to use this weight to do something, such as perform inference testing on the COCO dataset. How should I do that?

Additionally, I would like to know if the size of this weight is correct — it is actually 2.11GB, while sam3.pt is 3.21GB.

I look forward to your reply. Thank you very much.

[colinlin1982]

sample code for inference single image:
https://github.com/SimonZeng7108/efficientsam3/blob/main/sam3/efficientsam3_examples/run_sam3_text_prompt.py

sample code for coco eval:
https://github.com/SimonZeng7108/efficientsam3/blob/main/sam3/evaluation/eval_coco.py

code for trim weights:
https://github.com/SimonZeng7108/efficientsam3/blob/main/stage1/trim_weights.py

[Lali-Lali]

This is very, very helpful. However, I've encountered another issue. I used 'shoe' as the text prompt to perform inference on the image at efficientsam3/sam3/assets/images/test_image.jpg. It detected 0 masks. When I used a point prompt, it only utilized the visual encoder weights, and the result was acceptable.

After that, I used the text encoder weights from your GitHub repository that were trained on 1% of the Recap-DataComp-1B dataset: MobileCLIP-S1 and MobileCLIP-S0. Using 'shoe' as the text prompt again, the results were acceptable. This indicates that there is some problem with the text encoder weights I trained myself. This is precisely what confuses me, because the loss during my distillation training of the text encoder stabilized around 0.2. Given the magnitude of this loss, it shouldn't be causing any issues. Where did I go wrong? Looking forward to your reply.

Below is my train.sh and log for distilling the text encoder.

Below is the comparison of weight loading when I ran efficientsam3/sam3/efficientsam3_examples/run_sam3_text_prompt.py.

[SimonZeng7108]

Hi was this following the stage 1 readme to train the text encoder?

[Lali-Lali]

Yes. I followed the README_stage1.md to train the text encoder. The only changes I made were: I used 8 GPUs, changed the batch size from 64 to 256, and doubled the BASE_LR. In the end, I completed 100 epochs of training, which took about 2 days, and the final loss was 0.18. The training ran smoothly without any errors.

Below is the content of the corresponding YAML file for text encoder training:

MODEL:
NAME: EfficientSAM3_Text_S_Pretrained
BACKBONE: MobileCLIP-S0
# Download pretrained weights:
# mkdir -p checkpoints/mobileclip
# wget https://docs-assets.developer.apple.com/ml-research/datasets/mobileclip/mobileclip_s0.pt -P checkpoints/mobileclip
PRETRAINED: 'efficientsam3/checkpoints/mobileclip/mobileclip_s0.pt'
RESUME: ''

TRAIN:
START_EPOCH: 0
EPOCHS: 100 # Increased from 50
# WARMUP_EPOCHS: 10 # Increased from 5
WARMUP_EPOCHS: 0 # Increased from 5
WEIGHT_DECAY: 0.05 # Increased from 0.01
# BASE_LR: 5e-3 # Increased from 1e-3
BASE_LR: 1e-2 # Increased from 1e-3
WARMUP_LR: 1e-6
MIN_LR: 5e-6 # Increased from 1e-5
CLIP_GRAD: 1.0 # Reduced from 5.0
AUTO_RESUME: True
ACCUMULATION_STEPS: 2 # Increased from 1
USE_CHECKPOINT: False
LAYER_LR_DECAY: 0.9 # Reduced from 1.0
EVAL_BN_WHEN_TRAINING: False
FIND_UNUSED_PARAMETERS: True

LR_SCHEDULER:
NAME: 'cosine'
DECAY_EPOCHS: 30
DECAY_RATE: 0.1

OPTIMIZER:
NAME: 'adamw'
EPS: 1e-8
BETAS: (0.9, 0.999)
MOMENTUM: 0.9

AMP_ENABLE: True
# OUTPUT: output/stage1_text/mobileclip_s0_pretrained
OUTPUT: efficientsam3/output/stage1_text/mobileclip_s0_pretrained_100epochs
TAG: pretrained_init
SAVE_FREQ: 1
PRINT_FREQ: 10
SEED: 0
EVAL_MODE: False
THROUGHPUT_MODE: False
LOCAL_RANK: 0

DATA:
IMG_SIZE: 1008
DATASET: recap_datacomp
# DATA_PATH: data/
DATA_PATH: Datasets/recap_subset
BATCH_SIZE: 64
NUM_WORKERS: 8

DISTILL:
ENABLED: True
ENCODER_ONLY: True
EMBED_DIM: 256
EMBED_SIZE: 0
NUM_EMBED: 32
PIXEL_WISE: 1.0
CHANNEL_WISE: 0.0
CORRELATION: 0.0
COSINE: 2.0 # Increased from 1.0
# TEACHER_EMBED_PATH: 'output/stage1_text_teacher/embeddings'
TEACHER_EMBED_PATH: 'efficientsam3/output/stage1_text_teacher/embeddings'
SAVE_TEACHER_EMBED: False
NO_RAND: True
MAX_ALLOWED_PROMPTS: -1

[SimonZeng7108]

Yes. I followed the README_stage1.md to train the text encoder. The only changes I made were: I used 8 GPUs, changed the batch size from 64 to 256, and doubled the BASE_LR. In the end, I completed 100 epochs of training, which took about 2 days, and the final loss was 0.18. The training ran smoothly without any errors.

Below is the content of the corresponding YAML file for text encoder training:

MODEL: NAME: EfficientSAM3_Text_S_Pretrained BACKBONE: MobileCLIP-S0 # Download pretrained weights: # mkdir -p checkpoints/mobileclip # wget https://docs-assets.developer.apple.com/ml-research/datasets/mobileclip/mobileclip_s0.pt -P checkpoints/mobileclip PRETRAINED: 'efficientsam3/checkpoints/mobileclip/mobileclip_s0.pt' RESUME: ''

TRAIN: START_EPOCH: 0 EPOCHS: 100 # Increased from 50 # WARMUP_EPOCHS: 10 # Increased from 5 WARMUP_EPOCHS: 0 # Increased from 5 WEIGHT_DECAY: 0.05 # Increased from 0.01 # BASE_LR: 5e-3 # Increased from 1e-3 BASE_LR: 1e-2 # Increased from 1e-3 WARMUP_LR: 1e-6 MIN_LR: 5e-6 # Increased from 1e-5 CLIP_GRAD: 1.0 # Reduced from 5.0 AUTO_RESUME: True ACCUMULATION_STEPS: 2 # Increased from 1 USE_CHECKPOINT: False LAYER_LR_DECAY: 0.9 # Reduced from 1.0 EVAL_BN_WHEN_TRAINING: False FIND_UNUSED_PARAMETERS: True

LR_SCHEDULER: NAME: 'cosine' DECAY_EPOCHS: 30 DECAY_RATE: 0.1

OPTIMIZER: NAME: 'adamw' EPS: 1e-8 BETAS: (0.9, 0.999) MOMENTUM: 0.9

AMP_ENABLE: True # OUTPUT: output/stage1_text/mobileclip_s0_pretrained OUTPUT: efficientsam3/output/stage1_text/mobileclip_s0_pretrained_100epochs TAG: pretrained_init SAVE_FREQ: 1 PRINT_FREQ: 10 SEED: 0 EVAL_MODE: False THROUGHPUT_MODE: False LOCAL_RANK: 0

DATA: IMG_SIZE: 1008 DATASET: recap_datacomp # DATA_PATH: data/ DATA_PATH: Datasets/recap_subset BATCH_SIZE: 64 NUM_WORKERS: 8

DISTILL: ENABLED: True ENCODER_ONLY: True EMBED_DIM: 256 EMBED_SIZE: 0 NUM_EMBED: 32 PIXEL_WISE: 1.0 CHANNEL_WISE: 0.0 CORRELATION: 0.0 COSINE: 2.0 # Increased from 1.0 # TEACHER_EMBED_PATH: 'output/stage1_text_teacher/embeddings' TEACHER_EMBED_PATH: 'efficientsam3/output/stage1_text_teacher/embeddings' SAVE_TEACHER_EMBED: False NO_RAND: True MAX_ALLOWED_PROMPTS: -1

The Coco eval code is in here. Did you merge the trained weights after training?

[Lali-Lali]

Yes. I used C. Merge Both Encoders (Recommended) from README_stage1.md. I ran the merge operation again, but the result is still the same: zero masks are detected.

I am particularly concerned about the training loss of the text encoder, which ended up being 0.18. If this loss value seems normal and indicates that training proceeded without issues, I will then look elsewhere to figure out where the problem might be.

[SimonZeng7108]

Yes. I used C. Merge Both Encoders (Recommended) from README_stage1.md. I ran the merge operation again, but the result is still the same: zero masks are detected.

I am particularly concerned about the training loss of the text encoder, which ended up being **0.18**. If this loss value seems normal and indicates that training proceeded without issues, I will then look elsewhere to figure out where the problem might be.

Hey, a good way to figure our which part of the trained model has issue is to merge seperately with the origianl sam3 model, for example if you have a trained student text encoder then merge the rest with the sam3 including the original sam3 image encoder, then if your text prompt works you will be more confident that the image encoder you trained has issues. If the trained text encoder does not produce a good mask even with the original sam3 image encoder, then it has a big underfitting issue. In my experience, the text encoder is so easy to train and can fit a range of vocab very easily. Also to check if the text encoder training script works, you can just feed in one word, like 'shoe', train it for 1000 epochs or something to make sure it overfits, if it does not even overfit on one word, then the training script has issues.