Hunyuan Video Framepack F1

[a-r-r-o-w]

Example:

import torch
from diffusers import HunyuanVideoFramepackPipeline, HunyuanVideoFramepackTransformer3DModel
from diffusers.hooks import apply_group_offloading
from diffusers.utils import export_to_video, load_image
from transformers import SiglipImageProcessor, SiglipVisionModel

transformer = HunyuanVideoFramepackTransformer3DModel.from_pretrained(
    "lllyasviel/FramePack_F1_I2V_HY_20250503", torch_dtype=torch.bfloat16
)
feature_extractor = SiglipImageProcessor.from_pretrained(
    "lllyasviel/flux_redux_bfl", subfolder="feature_extractor"
)
image_encoder = SiglipVisionModel.from_pretrained(
    "lllyasviel/flux_redux_bfl", subfolder="image_encoder", torch_dtype=torch.float16
)
pipe = HunyuanVideoFramepackPipeline.from_pretrained(
    "hunyuanvideo-community/HunyuanVideo",
    transformer=transformer,
    feature_extractor=feature_extractor,
    image_encoder=image_encoder,
    torch_dtype=torch.float16,
)

onload_device = torch.device("cuda")
offload_device = torch.device("cpu")
list(map(
    lambda x: apply_group_offloading(x, onload_device, offload_device, offload_type="leaf_level", use_stream=True, low_cpu_mem_usage=True),
    [pipe.text_encoder, pipe.text_encoder_2, pipe.transformer]
))
pipe.image_encoder.to(onload_device)
pipe.vae.to(onload_device)
pipe.vae.enable_tiling()

image = load_image(
    "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/penguin.png"
)
output = pipe(
    image=image,
    prompt="A penguin dancing in the snow",
    height=832,
    width=480,
    num_frames=91,
    num_inference_steps=30,
    guidance_scale=9.0,
    generator=torch.Generator().manual_seed(0),
    sampling_type="vanilla",
).frames[0]
print(f"Max memory: {torch.cuda.max_memory_allocated() / 1024**3:.3f} GB")
export_to_video(output, "output.mp4", fps=30)

output.mp4

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[a-r-r-o-w]

Verified that the previous inference code results in the same generated video for the inverted anti-drifting model

[nitinmukesh]

@a-r-r-o-w

As mentioned in code
apply_group_offloading

Does this still need a lot of RAM? I remember last in one of the topics there was discussion to reduce RAM requirements. Sorry but a little behind.

Is it possible to separate components and then do inference, will this require less RAM.

text_encoder -> prompt_embeds (with apply_group_offloading)

transformer -> latents (with apply_group_offloading)

vae -> video output (without apply_group_offloading)

Just some random thought it may be stupid though.

[a-r-r-o-w]

@nitinmukesh It still needs a lot of RAM but the usage can be significantly lowered if you set low_cpu_mem_usage=True when using cuda streams (use_stream=True). See PR #11106 for some numbers.

You can combine group offloading (which is used to tradeoff RAM for lower VRAM usage) with separate-component-inference as you mention. In the example, we apply group offloading to all models at once -- this will use a lot of RAM. But what you mentioned is correct and possibly the best way to run inference when both resources are limited. Many UIs do this automatically under the hood but since diffusers is a library, these things have to be manually implemented by users for it to be runnable on consumer GPUs

[nitinmukesh]

@a-r-r-o-w

Thank you. I am gonna try this and see the feasibility to use this feature on low VRAM. I hope this works on quantized model as well.

[yiyixuxu]

thanks! @a-r-r-o-w

[tin2tin]

@a-r-r-o-w Does the F1 implementation replace the first and reverse(from end to start) FramePack pipeline? To my knowledge, doesn't F1 support first frame / last frame, so has the feature been removed with this implementation?

[a-r-r-o-w]

@tin2tin The first model that was released supported inverted anti-drifting (which means, it generates the last frame first and then fills in the blanks between the first and last frame) sampling. For this model, the community found that providing the end frame as target latent worked well in using first-last-frame interpolation case. With the new model (F1), it is based on the vanilla forward technique. Here, I don't think there's a trivial way you can inject last frame control. So, until further investigation, end frame support does not exist with the F1 model but should work as expected for the first model

[tin2tin]

Is it the scheduler which tells diffusers which way to handle the model(forwards or backwards)?

[a-r-r-o-w]

It's the new sampling_type parameter (which should be "vanilla" for the second model [F1] and "inverted_anti_drifting" for the first model).

Schedulers are completely different here in that they only handle the denoising aspect i.e. computing a less noisy sample from a more noisy sample given the model prediction. Both models use the same scheduler but different preprocessing/postprocessing of the history latents that Framepack uses.