Fine-tuning Stable Diffusion with KerasCV 🔥

[Sayak_Paul]

Hi folks,

I hope y’all are doing well. Happy new year!

Delighted to present a project that @deep-diver and I have been working on for a while now – Fine-tuning Stable Diffusion with KerasCV. The project shows how to fine-tune the encoder part of Stable Diffusion on a custom image-caption dataset. It supports model checkpointing, mixed-precision training and is fairly customizable. Following are the main links:

• GitHub Repository
• Blog post on keras.io
• Hugging Face Space
• Deployment
• Fine-tuned model parameters

We hope y’all will like the project and have fun playing with it.

[8bitmp3]

Thanks @Sayak_Paul !

There is also:

• High-performance image generation using Stable Diffusion in KerasCV  |  TensorFlow Core
  • Same as High-performance image generation using Stable Diffusion in KerasCV (by @fchollet and Keras team)
• A walk through latent space with Stable Diffusion
• Teach StableDiffusion new concepts via Textual Inversion
• Denoising Diffusion Implicit Models

Amazing stuff

[Sayak_Paul]

You missed this one

Keras documentation: Denoising Diffusion Probabilistic Model

[aseem_khanduja]

Can this model be used for Inpainting by simply adding init and mask images?

[Sayak_Paul]

Yup. There is already a class for that.

@Ian_Stenbit