docs/source/package_reference/unilora.md
Uni-LoRA is a PEFT method that shares a compact trainable
vector bank across low-rank adapter weights. Instead of learning every LoRA matrix element independently, UniLoRA
deterministically projects entries into shared theta_d values and learns the shared parameters used by the adapter
update.
from peft import UniLoraConfig, get_peft_model
from transformers import AutoModelForCausalLM
model = AutoModelForCausalLM.from_pretrained("meta-llama/Llama-3.2-3B")
config = UniLoraConfig(
r=32,
theta_d_length=256,
proj_seed=42,
target_modules=["q_proj", "v_proj"],
unilora_dropout=0.0,
init_weights=True,
task_type="CAUSAL_LM",
)
peft_model = get_peft_model(model, config)
peft_model.print_trainable_parameters()
r controls the low-rank adapter dimension. Larger values increase adapter capacity and memory use.
theta_d_length controls the length of the shared UniLoRA vector bank. This is the main trainable storage shared by
the projected adapter entries.
proj_seed controls deterministic index generation for the fixed projections into theta_d. Reusing the same seed and
configuration makes the generated adapter indices reproducible.
target_modules selects which modules receive UniLoRA adapters. Use module suffixes such as ["q_proj", "v_proj"], a
regex string, or "all-linear" when supported by the model architecture.
unilora_dropout applies dropout inside UniLoRA adapter layers during training.
init_weights controls UniLoRA parameter initialization. Set it to False to keep a random theta_d
initialization when you need to manage initialization manually.
save_indices controls whether UniLoRA checkpoints save the generated index and scale tensors together with the
shared theta_d parameters. Keeping this disabled gives smaller checkpoints and regenerates indices from
proj_seed; enabling it makes saved adapters independent from future index-generation changes.
[[autodoc]] tuners.unilora.config.UniLoraConfig
[[autodoc]] tuners.unilora.model.UniLoraModel