Diffusers documentation

SANA-WM

Diffusers

You are viewing main version, which requires installation from source. If you'd like regular pip install, checkout the latest stable version (v0.41.0).

Join the Hugging Face community

and get access to the augmented documentation experience

Collaborate on models, datasets and Spaces

Faster examples with accelerated inference

Switch between documentation themes

to get started

SANA-WM

SANA-WM is a camera-controlled image-to-video world model built on top of SANA. Given a first-frame image, a text prompt, and a camera trajectory (either explicit c2w poses or a WASD/IJKL action string), it generates a video whose motion follows the requested camera path.

Inference runs in two stages:

Stage 1 — SANA-WM DiT. A 1.6B-parameter bidirectional DiT with GDN-Triton linear attention and a UCPE camera-control branch. Sampling uses an LTX-style flow-matching Euler scheduler with per-token timesteps; the first latent frame is the conditioning anchor.
Stage 2 — LTX-2 refiner (optional). A separate sink-bidirectional Euler refiner pipeline (SanaWMLTX2Refiner) that wraps SanaWMLTX2RefinerTransformer3DModel + LTX2TextConnectors and a Gemma-3 text encoder, run for 3 distilled sigma steps.

Both stages decode through the AutoencoderKLLTX2Video VAE.

Available models:

Model	Recommended dtype
`Efficient-Large-Model/SANA-WM_bidirectional-diffusers`	`torch.bfloat16`
`Efficient-Large-Model/SANA-WM_bidirectional-diffusers-refiner`	`torch.bfloat16`

SANA-WM is trained at a fixed 704×1280 resolution. The recommended dtype is for the transformer weights — keep the text encoder in torch.bfloat16 and the VAE in torch.float32 for best numerics. The pipeline expects camera intrinsics [fx, fy, cx, cy] in original-image pixel coordinates; the resize-and-center-crop transform is applied internally.

Inference

import torch
from PIL import Image

from diffusers import SanaWMPipeline
from diffusers.utils import export_to_video

pipe = SanaWMPipeline.from_pretrained(
    "Efficient-Large-Model/SANA-WM_bidirectional-diffusers",
    torch_dtype=torch.bfloat16,
)
pipe.enable_model_cpu_offload()  # ~45 GB of weights — offload between stages

# SANA-WM was trained on the LTX-2 VAE in framewise mode with tiling enabled. Without these
# settings the VAE encodes the whole (B, C, T, H, W) clip in one shot, which gives subtly
# different numerics from the released checkpoint.
pipe.vae.enable_tiling()
pipe.vae.use_framewise_encoding = True
pipe.vae.use_framewise_decoding = True
pipe.vae.tile_sample_stride_num_frames = 64
pipe.vae.tile_sample_min_num_frames = 96

prompt = "A car driving across a vast desert plain at golden hour."
output = pipe(
    image=Image.open("input.png").convert("RGB"),
    prompt=prompt,
    action="w-80,jw-40,w-40",        # WASD-style action DSL: forward 80f, jump+forward 40f, forward 40f
    intrinsics=[800.0, 800.0, 845.0, 464.0],  # fx, fy, cx, cy in original-image pixels
    num_frames=161,
    num_inference_steps=60,
    guidance_scale=5.0,
    generator=torch.Generator(device="cuda").manual_seed(42),
    output_type="latent",            # hand the latents to the refiner below
)

Pass action=None and supply your own c2w poses ((F, 4, 4) numpy array) to drive the camera trajectory explicitly. Drop output_type="latent" to get video straight out of stage 1 and skip the refiner.

Stage 2 — the LTX-2 refiner

SanaWMLTX2Refiner ships as its own repository, the way SDXL splits base and refiner. Pass the base pipeline’s VAE so the weights are shared rather than loaded twice:

from diffusers import SanaWMLTX2Refiner

refiner = SanaWMLTX2Refiner.from_pretrained(
    "Efficient-Large-Model/SANA-WM_bidirectional-diffusers-refiner",
    vae=pipe.vae,
    torch_dtype=torch.bfloat16,
)
refiner.enable_model_cpu_offload()

frames = refiner(output.latent, prompt, fps=16)
export_to_video(list(frames), "sana_wm.mp4", fps=16)

Without a vae the refiner returns refined latents instead of video, which is useful if you want to decode yourself.

Enable offloading on both pipelines, or free stage 1 before stage 2 (pipe.transformer.to("cpu")). The two stages together are around 45 GB in torch.bfloat16, and keeping both resident on one 80 GB card leaves too little room for activations. Note also that stage 2 honours torch_dtype: loading the refiner in torch.float32 roughly doubles its memory and changes the output slightly.

If you don’t have camera intrinsics, a hosted modular block can estimate them from a single frame. It lives outside diffusers because it pulls in Pi3X — an extra dependency and a second checkpoint — so nothing is downloaded until you ask for it:

from diffusers import ModularPipeline

# One-time per image. Requires `pip install pi3-vision`.
estimator = ModularPipeline.from_pretrained(
    "Efficient-Large-Model/pi3x-intrinsics-estimator", trust_remote_code=True
)
estimator.load_components(dtype=torch.bfloat16)
intrinsics = estimator(image=Image.open("input.png").convert("RGB"), output="intrinsics")

Converting the released checkpoint

If you have the source SANA-WM release (not the pre-converted diffusers snapshot), run the conversion script once:

python scripts/convert_sana_wm_to_diffusers.py \
    --src Efficient-Large-Model/SANA-WM_bidirectional \
    --dst ./SANA-WM_bidirectional-diffusers

This writes two directories: the base pipeline at --dst, and the stage-2 refiner alongside it at ./SANA-WM_bidirectional-diffusers-refiner (override with --dst-refiner). Then load each from its local path as usual.

Components

tokenizer — GemmaTokenizerFast
text_encoder — Gemma-2 (returns decoder hidden states)
vae — AutoencoderKLLTX2Video (LTX-2, spatial ×32 / temporal ×8)
transformer — SanaWMTransformer3DModel, 1.6B-parameter bidirectional DiT
scheduler — FlowMatchEulerDiscreteScheduler

The stage-2 refiner is a separate repository with its own transformer (SanaWMLTX2RefinerTransformer3DModel), connectors (LTX2TextConnectors), tokenizer, Gemma-3 text_encoder and scheduler. It has no vae of its own — pass the base pipeline’s.

SanaWMPipeline

class diffusers.SanaWMPipeline

< source >

( tokenizer: GemmaTokenizer | GemmaTokenizerFasttext_encoder: Gemma2PreTrainedModelvae: AutoencoderKLLTX2Videotransformer: SanaWMTransformer3DModelscheduler: FlowMatchEulerDiscreteScheduler )

Parameters

tokenizer ([GemmaTokenizer] or [GemmaTokenizerFast]) — The Gemma-2 tokenizer.
text_encoder ([Gemma2PreTrainedModel]) — The Gemma-2 text encoder.
vae ([AutoencoderKLLTX2Video]) — The LTX-2 VAE.
transformer ([SanaWMTransformer3DModel]) — The 1600M bidirectional SANA-WM DiT.
scheduler ([FlowMatchEulerDiscreteScheduler]) — Flow-matching Euler scheduler (LTX-style per-token timesteps). latents directly.

SANA-WM camera-controlled image-to-video pipeline.

Generates a video from a first-frame image, a text prompt, and a camera trajectory (explicit c2w poses or a WASD/IJKL action string). Uses the 1600M bidirectional SANA DiT for stage-1 sampling and the LTX-2 sink-bidirectional Euler refiner for stage-2 polish; both decode through the LTX-2 VAE.

call

< source >

( image: PIL.Image.Image | str | Pathprompt: strc2w: np.ndarray | None = Noneaction: str | None = Noneintrinsics: np.ndarray | list[float] | None = Noneheight: int = 704width: int = 1280num_frames: int = 161fps: int = 16num_inference_steps: int = 60guidance_scale: float = 5.0negative_prompt: str = ''generator: torch.Generator | list[torch.Generator] | None = Noneprompt_embeds: torch.Tensor | None = Noneprompt_attention_mask: torch.Tensor | None = Nonenegative_prompt_embeds: torch.Tensor | None = Nonenegative_prompt_attention_mask: torch.Tensor | None = Nonemax_sequence_length: int = 300chi_prompt: list[str] | None = Noneoutput_type: Literal['np', 'pil', 'pt', 'latent'] = 'np'callback_on_step_end: Callable[[DiffusionPipeline, int, int, dict], dict] | None = Nonecallback_on_step_end_tensor_inputs: list[str] = ['latents']return_dict: bool = True )

Parameters

image (PIL.Image.Image or str) — First-frame image (PIL or path).
prompt (str) — Text prompt.
c2w (np.ndarray, optional) — (F, 4, 4) camera-to-world poses. Mutually exclusive with action.
action (str, optional) — Action-DSL string e.g. "w-80,jw-40,w-40". Mutually exclusive with c2w.
intrinsics (np.ndarray or list[float]) — [fx, fy, cx, cy] in original-image pixel coordinates. The pipeline applies the resize+crop transform internally.
height (int, defaults to 704) — Output frame height (fixed for the public model).
width (int, defaults to 1280) — Output frame width (fixed for the public model).
num_frames (int, defaults to 161) — Target frame count; snapped to 8k+1 (LTX-2 VAE constraint).
fps (int, defaults to 16) — Output frame rate (also fed to the refiner).
num_inference_steps (int, defaults to 60) — Number of stage-1 DiT sampling steps.
guidance_scale (float, defaults to 5.0) — Classifier-free guidance scale.
negative_prompt (str, defaults to "") — Optional negative prompt.
generator (torch.Generator or list[torch.Generator], optional) — One or more torch generators to make the noise sampling deterministic. If None, sampling is non-deterministic.
prompt_embeds (torch.Tensor, optional) — Pre-computed text embeddings, to skip the text encoder. Must be passed together with prompt_attention_mask.
prompt_attention_mask (torch.Tensor, optional) — Attention mask for prompt_embeds.
negative_prompt_embeds (torch.Tensor, optional) — Pre-computed negative text embeddings. Must be passed together with negative_prompt_attention_mask.
negative_prompt_attention_mask (torch.Tensor, optional) — Attention mask for negative_prompt_embeds.
max_sequence_length (int, defaults to 300) — Max prompt tokens.
chi_prompt (list[str], optional) — Override the chi-prompt prefix (default mirrors the public release).
output_type (“np”, “pil”, “pt”, or “latent”, defaults to “np”) — Output format.
callback_on_step_end (Callable, optional) — Called at the end of each denoising step as callback_on_step_end(self, step, timestep, callback_kwargs), where callback_kwargs holds the tensors named in callback_on_step_end_tensor_inputs. Tensors in the returned dict replace the pipeline’s.
callback_on_step_end_tensor_inputs (list[str], defaults to [“latents”]) — Which tensors to pass to callback_on_step_end; must be a subset of ._callback_tensor_inputs.
return_dict (bool, defaults to True) — Return [SanaWMPipelineOutput] vs tuple.

Generate a SANA-WM camera-controlled video.

Examples:

>>> import torch
>>> from PIL import Image
>>> from diffusers import SanaWMPipeline

>>> pipe = SanaWMPipeline.from_pretrained(
...     "Efficient-Large-Model/SANA-WM_bidirectional-diffusers", torch_dtype=torch.bfloat16
... ).to("cuda")

>>> output = pipe(
...     image=Image.open("input.png").convert("RGB"),
...     prompt="A car driving across a vast desert plain at golden hour.",
...     action="w-80,jw-40,w-40",
...     intrinsics=[800.0, 800.0, 845.0, 464.0],  # fx, fy, cx, cy in original-image pixels
...     num_inference_steps=60,
... )
>>> # output.frames is (T, H, W, 3) float np.ndarray in [0, 1] (diffusers convention).

check_inputs

< source >

Validate __call__ inputs. Raises on bad input and returns nothing.

encode_prompt

< source >

( prompt: strnegative_prompt: str = ''device: torch.device | None = Nonemax_sequence_length: int = 300chi_prompt: list[str] | None = Noneprompt_embeds: torch.Tensor | None = Noneprompt_attention_mask: torch.Tensor | None = Nonenegative_prompt_embeds: torch.Tensor | None = Nonenegative_prompt_attention_mask: torch.Tensor | None = None )

Encode prompt + negative prompt through Gemma-2.

Mirrors the SANA chi-prompt-prefix trick: the chi prompt is prepended to the user prompt, then a select_index = [0, -L+1, ..., -1] slice takes the BOS token plus the last max_sequence_length - 1 tokens.

prepare_camera_trajectory

< source >

( c2w: np.ndarray | Noneaction: str | Noneintrinsics: np.ndarray | list[float]num_frames: int )

Normalize the camera inputs to (c2w_(F, 4, 4), intrinsics_(F, 4)).

Also snaps num_frames to the VAE-friendly 8k+1 and trims both arrays to match. The cropped image and rescaled intrinsics come later, once the target resolution is known.

prepare_latents

< source >

( first_latent: torch.Tensornum_frames: intheight: intwidth: intdtype: torch.dtypedevice: torch.devicegenerator: torch.Generator )

Sample initial latents and pin the first frame as the conditioning anchor.

Returns (latents, condition_mask) where condition_mask has ones on the first frame’s tokens (they are held clean throughout sampling) and zeros elsewhere.

SanaWMLTX2Refiner

The LTX-2 stage-2 refiner is a standalone DiffusionPipeline that takes stage-1 latents. Give it a vae (the base pipeline’s, so the weights are shared) to have it decode to video; without one it returns refined latents.

class diffusers.SanaWMLTX2Refiner

< source >

( transformer: SanaWMLTX2RefinerTransformer3DModelconnectors: LTX2TextConnectorstokenizer: GemmaTokenizer | GemmaTokenizerFasttext_encoder: Gemma3ForConditionalGenerationscheduler: FlowMatchEulerDiscreteSchedulervae: AutoencoderKLLTX2Video | None = Nonetext_max_sequence_length: int = 1024 )

Parameters

transformer ([SanaWMLTX2RefinerTransformer3DModel]) — The LTX-2 video DiT with the chunk-causal sliding-window KV cache.
connectors ([LTX2TextConnectors]) — LTX-2 text connectors.
tokenizer — Gemma-3 tokenizer.
text_encoder — Gemma-3 text encoder.
scheduler ([FlowMatchEulerDiscreteScheduler]) — Flow-matching Euler scheduler. Constructed with shift=1.0 so the distilled sigmas pass through unmodified.
vae ([AutoencoderKLLTX2Video], optional) — The same VAE used by [SanaWMPipeline]; pass vae=pipe.vae to share the weights. When given, the refiner decodes to video, otherwise it returns refined latents.
text_max_sequence_length (int, defaults to 1024) — Maximum tokens passed to the Gemma-3 tokenizer.

LTX-2 sink-bidirectional Euler refiner — SANA-WM stage 2, as a standalone pipeline.

Wraps the LTX-2 components (refiner transformer + text connectors + Gemma-3 text encoder + tokenizer) plus a [FlowMatchEulerDiscreteScheduler] that carries the distilled sigma schedule and performs the Euler steps. It is registered as an optional component of [SanaWMPipeline] and can also be used on its own to refine stage-1 latents.

call

< source >

( sana_latent: torch.Tensorprompt: strfps: floatsink_size: int = 1generator: torch.Generator | None = Noneblock_size: int = 3kv_max_frames: int = 11sigmas: tuple[float, ...] = (0.909375, 0.725, 0.421875, 0.0)output_type: str = 'np' ) → torch.Tensor

Parameters

sana_latent — (B, C, F, H, W) stage-1 latent.
prompt — text prompt.
fps — video frame rate (drives LTX-2 RoPE temporal scaling).
sink_size — how many leading raw z_sana frames to anchor as the attention sink (canonical: 1).
generator — torch.Generator for the FM endpoint noise. Defaults to a generator seeded with 42 so results are reproducible out of the box.
block_size — latent frames per AR block (canonical: 3).
kv_max_frames — maximum context+active frames retained in the sliding window (canonical: 11 = 1 sink + 10 recent).
sigmas — descending Euler schedule terminating at 0.0 (canonical 3-step distilled: (0.909375, 0.725, 0.421875, 0.0)). Fed to self.scheduler (minus the trailing 0.0, which the scheduler appends itself).
output_type — “latent” returns the refined latents. Anything else decodes through self.vae and post-processes to that type (“np”, “pt”, “pil”); without a vae the latents are returned regardless.

Returns

torch.Tensor

Refined VAE latents of shape (B, C, F, H, W) — the first sink_size frames carry the raw stage-1 sink latents unchanged, the rest carry the refined output.

Run the LTX-2 refiner and return refined VAE latents.

Uses the chunk-causal AR recipe the model was trained on (block_size=3, kv_max_frames=11): a sliding window of [source_sink + recent_history + active_block] K/V is fed to the transformer one block at a time, so per-block compute is bounded and total refinement cost scales linearly with video length.

SanaWMPipelineOutput

class diffusers.SanaWMPipelineOutput

< source >

( frames: typing.Union[torch.Tensor, numpy.ndarray, list[list[PIL.Image.Image]]]c2w: numpy.ndarray | None = Nonelatent: typing.Optional[torch.Tensor] = None )

Parameters

frames (torch.Tensor, np.ndarray, or list[PIL.Image.Image]) — Generated video: a float np.ndarray of shape (T, H, W, 3) in [0, 1] for output_type="np", a (T, 3, H, W) tensor for "pt", a list of T `PIL.Image for "pil", or the latent tensor (B, C, T_lat, H_lat, W_lat) for "latent".
c2w (np.ndarray) — Camera-to-world poses (T, 4, 4) aligned with frames (the refiner drops the sink anchor frame; this array is realigned accordingly when the refiner ran).
latent (torch.Tensor, optional) — Latent tensor in LTX-2 VAE space, shape (B, C, T_lat, H_lat, W_lat). Returned when `output_type=“latent”“.

Output class for SANA-WM image-to-video pipeline.

Update on GitHub

←Sana Video Shap-E→