replacement sometimes works perfect sometimes not

[OnurTheBruh]

as you know your input photo needs to be same position of the person on the first frame of the video for the perfect resulsts but i had 2 photos with diffrent man and same position. one of photo is worked perfect but other one didnt work very well even it was litrally same position like %95. also didnt try relightning or dpo loras

[fengjia-guo]

Hi, thanks for sharing this. We've also received other feedbacks regarding the instability of replacement mode, and here's a recommended recipe based on experiences from the community:

• character reference image(s) on a black background (btw, it is better to have the reference image aligned with the first frame in pose, so you're on the right track)
• plus a LightX2V T2V LoRA
• and using SCAIL-2 animation mode (set replacement mode OFF in the node)
• (Optional) use relighting lora, as it is trained on replacement tasks only

This tends to be more stable in some cases than directly relying on replacement mode.

[bondobrus]

Hi, thanks for sharing this. We've also received other feedbacks regarding the instability of replacement mode, and here's a recommended recipe based on experiences from the community:

• character reference image(s) on a black background (btw, it is better to have the reference image aligned with the first frame in pose, so you're on the right track)
• plus a LightX2V T2V LoRA
• and using SCAIL-2 animation mode (set replacement mode OFF in the node)
• (Optional) use relighting lora, as it is trained on replacement tasks only

This tends to be more stable in some cases than directly relying on replacement mode.

I want to follow-up regarding black background in replacement mode, as I understand the background in this case is used from the video or only based on the prompt? Thanks.

[fengjia-guo]

Hi, thanks for sharing this. We've also received other feedbacks regarding the instability of replacement mode, and here's a recommended recipe based on experiences from the community:

• character reference image(s) on a black background (btw, it is better to have the reference image aligned with the first frame in pose, so you're on the right track)
• plus a LightX2V T2V LoRA
• and using SCAIL-2 animation mode (set replacement mode OFF in the node)
• (Optional) use relighting lora, as it is trained on replacement tasks only

This tends to be more stable in some cases than directly relying on replacement mode.

I want to follow-up regarding black background in replacement mode, as I understand the background in this case is used from the video or only based on the prompt? Thanks.

The background in this case inherits from the driving video, as in replacement mode. So this is a trick of running replacement in animation mode, and the results are often better than running in replacement mode.

[OnurTheBruh]

Hi, thanks for sharing this. We've also received other feedbacks regarding the instability of replacement mode, and here's a recommended recipe based on experiences from the community:

• character reference image(s) on a black background (btw, it is better to have the reference image aligned with the first frame in pose, so you're on the right track)
• plus a LightX2V T2V LoRA
• and using SCAIL-2 animation mode (set replacement mode OFF in the node)
• (Optional) use relighting lora, as it is trained on replacement tasks only

This tends to be more stable in some cases than directly relying on replacement mode.

I want to follow-up regarding black background in replacement mode, as I understand the background in this case is used from the video or only based on the prompt? Thanks.

The background in this case inherits from the driving video, as in replacement mode. So this is a trick of running replacement in animation mode, and the results are often better than running in replacement mode.

hmm figure out something i think. if person on the photo is occupies a large space , it generate person big too idk how to fix it tbh also The camera angle in the photo and the camera angle in the video must be the same if you want perfect results otherwise looks a bit weird

[fengjia-guo]

hmm figure out something i think. if person on the photo is occupies a large space , it generate person big too idk how to fix it tbh also The camera angle in the photo and the camera angle in the video must be the same if you want perfect results otherwise looks a bit weird

Yes, although SCAIL-2 has the capability of replacing a character into the video when the reference image differs in pose and position with the driving video, we still recommend aligning the reference image with the first frame of the driving video in pose, scale and camera position for best performance.

[OnurTheBruh]

hmm figure out something i think. if person on the photo is occupies a large space , it generate person big too idk how to fix it tbh also The camera angle in the photo and the camera angle in the video must be the same if you want perfect results otherwise looks a bit weird

Yes, although SCAIL-2 has the capability of replacing a character into the video when the reference image differs in pose and position with the driving video, we still recommend aligning the reference image with the first frame of the driving video in pose, scale and camera position for best performance.

thank you so much for your reply. last question, Is the aspect ratio of the photo important? for example video that i uploaded is 16:19 but referance image is 9:16

[fengjia-guo]

Most workflows resize and crop driving video and reference image to a unified resolution, e.g. 704x1280 or 512x896. If you're concerned about the cropping (e.g. your character is on the edge of the image and may be cropped apart), you can manually resize and crop the inputs to the expected resolution.

[OnurTheBruh]

Most workflows resize and crop driving video and reference image to a unified resolution, e.g. 704x1280 or 512x896. If you're concerned about the cropping (e.g. your character is on the edge of the image and may be cropped apart), you can manually resize and crop the inputs to the expected resolution.

last shot, what can I do to keep the person in the reference photo in the same position, and I don't want him/her to take the same position as the person in the reference video? my reference video is about a person enters right side of the frame and show the middle finger to person and person is shocked. i just want the motion that who enters right side of the frame and show finger the the person , I don't want him to take the position of the person he's giving the middle finger to and i want the person in the reference photo to stay in the same position.

[fengjia-guo]

last shot, what can I do to keep the person in the reference photo in the same position, and I don't want him/her to take the same position as the person in the reference video? my reference video is about a person enters right side of the frame and show the middle finger to person and person is shocked. i just want the motion that who enters right side of the frame and show finger the the person , I don't want him to take the position of the person he's giving the middle finger to and i want the person in the reference photo to stay in the same position.

SCAIL-2 uses colored mask to locate individuals. Please make sure that the person (in the driving video) that you want to replace, and the person in the reference photo, have masks of same color. As the person in your case enters the frame in the middle, the tracking and coloring process may lead to unexpected results and you may need to check the colors more carefully.

[OnurTheBruh]

last shot, what can I do to keep the person in the reference photo in the same position, and I don't want him/her to take the same position as the person in the reference video? my reference video is about a person enters right side of the frame and show the middle finger to person and person is shocked. i just want the motion that who enters right side of the frame and show finger the the person , I don't want him to take the position of the person he's giving the middle finger to and i want the person in the reference photo to stay in the same position.

SCAIL-2 uses colored mask to locate individuals. Please make sure that the person (in the driving video) that you want to replace, and the person in the reference photo, have masks of same color. As the person in your case enters the frame in the middle, the tracking and coloring process may lead to unexpected results and you may need to check the colors more carefully.

i think my prompt was too long so i tried to use very short prompt and worked. didnt even mention person on the photo. thanks