Image resize issue

[DamienGaullier]

Hi,
I have an issue with my output image which does not have the same dimensions as the input one.
Original image: 640x480
Output image: 576x512
Do you know why ?

here the sample :

import numpy

from controlnet_aux.processor import MidasDetector
from PIL import Image

image_path = "d:/temp/cat.png"
model_path = "d:/model/dreamshaper-v8.0"

convertedImage = Image.open(image_path)
resolution = min(convertedImage.size)

processor = MidasDetector.from_pretrained(model_path)
controlnet_image = processor(
    input_image=convertedImage,
    a=numpy.pi * 2.0,
    bg_th=0.1,
    depth_and_normal=False,
    detect_resolution=resolution,
    image_resolution=resolution,
    output_type='pil'
)

controlnet_image.save("d:/temp/cat_result.png") 

Thank you!

[nullarch]

same issue

[patrickvonplaten]

Could you maybe open a PR to fix it? More than happy to review & merge

[lerignoux]

@patrickvonplaten Do we want to "fix" this ?

I was checking the issue:
This cut is coming from the util method resize_image, directly coming from illasviel Controlnet util resize_image.
It happens when the resize is requested on a resolution which is not a multiple of 64.
From what I see this is linked to the model convolution size (to confirm ?)

While it works to generate images with different size. I guess it is not optimal in this case the better solution is to resize the image to 512 (64*8) for instance.

We could:

• Add some documentation. Which I tried to do in Adding a note on image resolutions to preserve ration in the README. #89
• Make the resolutions parameter optional and not resize if None (I actually hesitated with this, cf MR )
• Raise an exception when resolution is not a multiple of 64
• Accept custom resolutions (just need to remove the line above-mentioned) but would probably be better done in ControlNet library also

What do you wish to go for ?

[patrickvonplaten]

I'm actually happy to accept custom resolution. Diffusers unets can work with any multiple of 8

[lerignoux]

Hey

Ok I tested a bit (allowing a None resolution to bypass resizing).
It seems it only works if image sizes are multiple of 32.
I tried to go lower but always ended up on a RuntimeError.

I would then suggest to fix it like this: cf MR

@patrickvonplaten tell me if this sounds ok for you.

I see there were no unit tests on the project. Do you think it is worth adding one for this ?

[lerignoux]

Hello @patrickvonplaten Do you have an issue with the above-mentioned suggestion ?

Tell us if you see anything you would like to add.

Regards

[lordsoffallen]

Any update on this one? How can this work with non square images? I am also confused by detection_size and image_resolution parameter as it does not take a h, w but a single value. What happens to 1360 X 768 image size?

[lordsoffallen]

For some reason this is happening in each control class, i don't know. Just removed the whole resizing from Canny detector and I am able to get matching sizes.

[pdoane]

What I do:

• Set the detect_resolution and image_resolution to min(req.width, req.height)
• After running the detector, resize back to the requested size

https://github.com/pdoane/seed-alchemy/blob/main/backend/image_generator.py#L77

There are some subtle reasons you might want to set the resolution sizes differently, but the above works well enough that i didn't find it useful to expose in a UI.

[lordsoffallen]

That does not work because we're losing information where we should not. I understand some processor may require this but i don't think all of them need it and copy pasting the same code does not help. I simply wrote my canny detector code and it works fine without any resizing. Not to mention that landscape images won't work well with this logic. (1360x768)

[asomoza]

I agree, this is kind of old and based on the original controlnet while now we have a lot more preprocessors and ones that actually need more resolution than the min required but this depends also in the trained controlnet.

I want to port all the relevant ones to the new library which doesn't have this restriction.

I'll test and review #90 but it will take a while since I have to ensure that all of the preprocessors work correctly and I'm working on other things at the moment.