[Community Pipeline] Imagic: Text-Based Real Image Editing with Diffusion Models

[apolinario]

Intro

Community Pipelines are introduced in diffusers==0.4.0 with the idea of allowing the community to quickly add, integrate, and share their custom pipelines on top of diffusers.

You can find a guide about Community Pipelines here. You can also find all the community examples under examples/community/. If you have questions about the Community Pipelines feature, please head to the parent issue.

Idea: Imagic: Text-based Real Image Editing with Diffusion Models

This pipeline aims to implement this paper to Stable Diffusion, allowing for real-world image editing. Example from the paper:

[Alx-AI]

Would love to see this added, notebook implementation here for reference https://github.com/justinpinkney/stable-diffusion/blob/main/notebooks/imagic.ipynb

[asofiaoliveira]

I would like to work on this

[patrickvonplaten]

Awesome @asofiaoliveira !

Feel free to open a PR and to attach it here

[MarkRich]

tried my hand at this here: #958 please let me know if there's any comments!

[asofiaoliveira]

I guess I'll leave it to @MarkRich then 😅

[0xdevalias]

FYI, it looks like that PR has been merged now:

• Add imagic to community pipelines #958

And the implementation is available here:

• https://github.com/huggingface/diffusers/tree/main/examples/community#imagic-stable-diffusion

Can this issue be closed now?

[njucckevin]

FYI

Is there someone try the effect with this code? I failed to achieve the effect in the paper (for example, let a doy playing with a toy) with the Imagic Stable Diffusion.

[askerlee]

@0xdevalias It seems there is some issue with the implementation? In train(), the text embedding is optimized first. But prior to that, unet and text_encoder are set to disable BP:

    self.unet.requires_grad_(False)
    self.text_encoder.requires_grad_(False)

Does this mean there won't be valid gradients back propagated from the loss to the text embedding?
I'm not very sure. Thanks.

[0xdevalias]

@askerlee I don't know anything about the implementation, nor really used it. I just noticed the PR and figured I'd link it here

[ShaoTengLiu]

@njucckevin I also get wrong results.
For example, here are the results for "A photo of a bird spreading wings":

Can anyone give me some hints?