Is it possible that stage3 has fine-tuned other components？Can the tracker be trained using the checkpoint from stage 3?

[TroubleMaker33]

Nice work!

I compared the efficienciesam3_tinyvit.pt released by you with the original sam3.pt in terms of architecture, parameters, and the actual corresponding values. What I want to know is whether you made a little fine-tuning for the detector during the stage 3 fine-tuning phase (some statistical indicators will be provided later).

Because I read your development notes where you mentioned that the parameters of the teacher model were kept frozen, but I found that their values were different (through metrics such as mean and std), but these differences were very small.

Additionally, I would like to ask, I tested the detection effect of your released "efficientsam3_tinyvit.pt" on my data for each frame. The overall prediction confidence decreased by about 0.5 compared to "sam3.pt". I need a lower threshold to obtain a good result. I want to know why this is the case. Also, I would like to know if it is feasible to train the tracker based on the stage 3 checkpoint you released.

I look forward to and thank you for your response！

detector_weights_comparison.log

[SimonZeng7108]

Hi, we only fintuned the image encoder and the FPNs. When you say ' the parameters of the teacher model were kept frozen', what do you mean by this? the teacher model is only used in stage 1. The lower confidence prediction is expected, you could further finetune tracker for your own task but in our ablation study, training the encoder and FPN gave the best results.

[TroubleMaker33]

Thank you very much for your reply. Today, I tested and found that it was probably because my code failed to handle the case of multiple numbers that caused some differences in the precision of the values.

Additionally, I noticed that the checkpoint released in stage 3 performed poorly in my dataset (in very dense scenarios). The detector_scores of the objects were very low (the original sam3.pt results were quite high), while the presence_scores were very high. Should I train the image encoder in this case? You mentioned that the features obtained by the image encoder are related to the entire subsequent process. Do you have any better suggestions for this situation?

Thank you again for your response.

{
"frame_idx": 61,
"num_detections": 20,
"scores": [
0.37764206528663635,
0.5240384340286255,
0.3015645146369934,
0.3170602023601532,
0.43611425161361694,
0.3022215664386749,
0.3536682426929474,
0.3013399541378021,
0.3676304817199707,
0.4678803086280823,
0.3803555965423584,
0.3394852578639984,
0.3035842180252075,
0.41608500480651855,
0.3147321045398712,
0.36269310116767883,
0.3304760158061981,
0.5560706853866577,
0.47903385758399963,
0.5137956142425537
],
"detector_scores": [
0.3777868151664734,
0.5242393016815186,
0.3016800880432129,
0.3171817362308502,
0.4362814128398895,
0.3023374080657959,
0.353803813457489,
0.30145546793937683,
0.3677713871002197,
0.46805962920188904,
0.380501389503479,
0.3396153748035431,
0.3037005662918091,
0.4162444770336151,
0.3148527443408966,
0.36283212900161743,
0.3306026756763458,
0.5562838315963745,
0.4792174696922302,
0.5139925479888916
],
"presence_scores": [
0.9996168613433838,
0.9996168613433838,
0.9996168613433838,
0.9996168613433838,
0.9996168613433838,
0.9996168613433838,
0.9996168613433838,
0.9996168613433838,
0.9996168613433838,
0.9996168613433838,
0.9996168613433838,
0.9996168613433838,
0.9996168613433838,
0.9996168613433838,
0.9996168613433838,
0.9996168613433838,
0.9996168613433838,
0.9996168613433838,
0.9996168613433838,
0.9996168613433838
],

[colinlin1982]

@TroubleMaker33 can you paste your image(frame_idx=61), so I can try it with my model?

[TroubleMaker33]

你能粘贴你的图片（frame_idx=61），这样我就能用模型试试吗？
Thank you for your willingness to offer assistance. I'm very sorry that due to copyright issues, I cannot directly provide you with the original image but have instead sent you the version with a mosaic added. I have tested it myself and the result is similar. Once again, thank you for your help.

[colinlin1982]

I inferenced the whole image and got 33 masks by 0.3~0.52 scores with my model.

Then I split image as left and right part, the inference result seems better(conf thresh=0.5):

note: my model is difference from efficientsam3.

[SimonZeng7108]

I inferenced the whole image and got 33 masks by 0.3~0.52 scores with my model.

Then I split image as left and right part, the inference result seems better(conf thresh=0.5):

note: my model is difference from efficientsam3.

this makes sense, for fine granularity segmentation I recommand sahi

[colinlin1982]

你能粘贴你的图片（frame_idx=61），这样我就能用模型试试吗？
Thank you for your willingness to offer assistance. I'm very sorry that due to copyright issues, I cannot directly provide you with the original image but have instead sent you the version with a mosaic added. I have tested it myself and the result is similar. Once again, thank you for your help.

Hi, @TroubleMaker33 .

I think about your image again, and I found the reason.

EfficientSAM3 is trained with letter box input, which means every image is pad to 1008*1008.

While Sam3Processor use resize mode, which means every image is resized to 1008*1008.

Your sample image's W-H ratio is about 4:1, which result in a large distortion‌.

I pad your sample image to 1:1 and run my model again, the result is better: