docs/source/en/api/pipelines/joyimage_edit.md
JoyAI-Image is a unified multimodal foundation model for image understanding, text-to-image generation, and instruction-guided image editing. It combines an 8B Multimodal Large Language Model (MLLM) with a 16B Multimodal Diffusion Transformer (MMDiT). A central principle of JoyAI-Image is the closed-loop collaboration between understanding, generation, and editing.
JoyAI-Image-Edit supports general image editing as well as spatial editing capabilities including object move, object rotation, and camera control.
| Model | Description | Download |
|---|---|---|
| JoyAI-Image-Edit | Instruction-guided image editing with precise and controllable spatial manipulation | Hugging Face |
import torch
from diffusers import JoyImageEditPipeline
from diffusers.utils import load_image
pipeline = JoyImageEditPipeline.from_pretrained(
"jdopensource/JoyAI-Image-Edit-Diffusers", torch_dtype=torch.bfloat16
)
pipeline.to("cuda")
image = load_image("https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/astronaut.jpg")
prompt = "Add wings to the astronaut."
output = pipeline(
image=image,
prompt=prompt,
num_inference_steps=40,
guidance_scale=4.0,
generator=torch.Generator("cuda").manual_seed(0),
).images[0]
output.save("joyimage_edit_output.png")
JoyAI-Image supports three spatial editing prompt patterns: Object Move, Object Rotation, and Camera Control. For best results, follow the prompt templates below as closely as possible. For more information, refer to SpatialEdit.
Move a target object into a specified region marked by a red box in the input image.
Move the <object> into the red box and finally remove the red box.
Rotate an object to a specific canonical view. Supported <view> values: front, right, left, rear, front right, front left, rear right, rear left.
Rotate the <object> to show the <view> side view.
Change the camera viewpoint while keeping the 3D scene unchanged.
Move the camera.
- Camera rotation: Yaw {y_rotation}°, Pitch {p_rotation}°.
- Camera zoom: in/out/unchanged.
- Keep the 3D scene static; only change the viewpoint.
[[autodoc]] JoyImageEditPipeline
[[autodoc]] pipelines.joyimage.pipeline_output.JoyImageEditPipelineOutput