Request for Details on Fine-Tuning and High-Resolution Optical Image Generation Pipeline

[Cfty-R]

I noticed in the supplementary information that you mentioned fine-tuning the Stable Diffusion model for artwork generation and employing a slightly more advanced digital encoder for the higher-resolution optical image generation pipeline.

I was wondering if you could kindly provide more details on the following:

Which publicly available Stable Diffusion model did you use for fine-tuning, and what dataset was utilized? Could you share more specifics about the training process?
How did you use the Stable Diffusion model as a teacher to train the optical generative model for producing high-resolution images? I would greatly appreciate it if you could provide some additional technical details about this process.
If it’s possible to share the original code, it would be incredibly helpful for understanding and learning from your work.

Thank you for your time and for contributing to the community. I look forward to your response!

[TanGeeGo]

Hi, thanks for your advice!
We have another ongoing project with the high resolution image generation pipeline, so we hope we could quickly release that.

We used Stable Diffusion v1.5 for fine-tuning. The dataset is VangoghToPhoto dataset, and we caption each image with a LLM, you can use ChatGPT API or open sourced QWEN to generate the text description of each image. The finetuning tutorial is easy to be find by google, you can also follow this link: https://huggingface.co/docs/diffusers/v0.14.0/en/training/text2image (but pay attention to not finetuning too much, some issues like catastrophic forgetting will happen when training step goes up).

[Cfty-R]

Hi, I really appreciate your generous and detailed response! I still have a few follow-up questions and would be grateful if you could help clarify them:

1. I wonder whether a single RTX 4090 GPU could handle this fine-tuning task within a reasonable training time (around 100 hours), if I relax some settings such as reducing the batch size or the number of epochs.

2. As I understand, fine-tuning an SD model usually requires more computational resources than training a DDPM from scratch. May I ask why you chose fine-tuning SD instead? Is it because the VanGoghToPhoto dataset is not large enough to support DDPM in generating high-quality and diverse images?

Thanks again for your time and help!

[TanGeeGo]

Hi, sorry for the late reply, I was busy on some other experiments recently.

1. For single RTX 4090 GPU, I recommend you use LoRA (low rank approximation) to finetuning the pretrained latent diffusion model. You can use some python package like "peft" to impelement the finetuning procedure. Here are some resources: https://huggingface.co/docs/diffusers/training/lora and https://huggingface.co/blog/flux-qlora?utm_source=chatgpt.com#flux-architecture
2. Yes, because the digital Vangogh image only has limited image numbers, so just finetune a pretrained latent diffusion model can retain high quality semantic generation quality and easily transfer the generation style. If you train the diffusion from scratch, limited dataset is hard for the diffusion to preserve the semantic information in global level, because the data point is so sparse for normal distribution.