docs/source/en/model_doc/higgs_audio_v2.md
This model was released on 2025-07-22 and added to Hugging Face Transformers on 2026-02-19.
</div>
Higgs Audio V2 is a powerful audio foundation model developed by Boson AI. The model was pretrained on over 10 million hours of audio data and a diverse set of text data. Despite having no post-training or fine-tuning, Higgs Audio v2 excels in expressive audio generation, thanks to its deep language and acoustic understanding.
Model Architecture:
<div class="flex justify-center"> </div> Higgs Audio v2 adopts the "generation variant" depicted in the architecture figure above. Its strong performance is driven by three key technical innovations:All of the snippets below mirror the integration tests in test_higgs_audio.py, ensuring the doc stays in sync with the officially supported workflows.
from transformers import AutoProcessor, HiggsAudioV2ForConditionalGeneration
model_id = "eustlb/higgs-audio-v2-generation-3B-base"
processor = AutoProcessor.from_pretrained(model_id, device_map="auto")
model = HiggsAudioV2ForConditionalGeneration.from_pretrained(model_id, device_map="auto")
conversation = [
{
"role": "system",
"content": [
{
"type": "text",
"text": "Generate audio following instruction."
}
],
},
{
"role": "scene",
"content": [
{
"type": "text",
"text": "Audio is recorded from a quiet room."
}
]
},
{
"role": "user",
"content": [
{
"type": "text",
"text": "The sun rises in the east and sets in the west. This simple fact has been observed by humans for thousands of years."
}
]
}
]
inputs = processor.apply_chat_template(
conversation,
add_generation_prompt=True,
tokenize=True,
return_dict=True,
sampling_rate=24000,
return_tensors="pt"
).to(model.device)
outputs = model.generate(**inputs, max_new_tokens=1000, do_sample=False)
decoded = processor.batch_decode(outputs)
processor.save_audio(decoded, "output_single_speaker_smart_voice.wav")
from transformers import AutoProcessor, HiggsAudioV2ForConditionalGeneration
model_id = "eustlb/higgs-audio-v2-generation-3B-base"
processor = AutoProcessor.from_pretrained(model_id, device_map="auto")
model = HiggsAudioV2ForConditionalGeneration.from_pretrained(model_id, device_map="auto")
system_message = """You are an AI assistant designed to convert text into speech.
If the user's message includes a [SPEAKER*] tag, do not read out the tag and generate speech for the following text, using the specified voice.
If no speaker tag is present, select a suitable voice on your own."""
user_message = """[SPEAKER0] I can't believe you did that without even asking me first!
[SPEAKER1] Oh, come on! It wasn't a big deal, and I knew you would overreact like this.
[SPEAKER0] Overreact? You made a decision that affects both of us without even considering my opinion!
[SPEAKER1] Because I didn't have time to sit around waiting for you to make up your mind! Someone had to act."""
conversation = [
{
"role": "system",
"content": [
{
"type": "text",
"text": system_message
}
]
},
{
"role": "scene",
"content": [
{
"type": "text",
"text": "Audio is recorded from a quiet room."
},
{
"type": "text",
"text": "SPEAKER0: feminine"
},
{
"type": "text",
"text": "SPEAKER1: masculine"
},
]
},
{
"role": "user",
"content": [
{
"type": "text",
"text": user_message
}
]
}
]
inputs = processor.apply_chat_template(
conversation,
add_generation_prompt=True,
tokenize=True,
return_dict=True,
sampling_rate=24000,
return_tensors="pt"
).to(model.device)
outputs = model.generate(**inputs, max_new_tokens=50, do_sample=False)
decoded = processor.batch_decode(outputs)
processor.save_audio(decoded, "output_multi_speaker_smart_voice.wav")
from transformers import AutoProcessor, HiggsAudioV2ForConditionalGeneration
model_id = "eustlb/higgs-audio-v2-generation-3B-base"
processor = AutoProcessor.from_pretrained(model_id, device_map="auto")
model = HiggsAudioV2ForConditionalGeneration.from_pretrained(model_id, device_map="auto")
conversation = [
{
"role": "system",
"content": [
{
"type": "text",
"text": "Generate audio following instruction."
}
]
},
{
"role": "scene",
"content": [
{
"type": "text",
"text": "Audio is recorded from a quiet room."
}
]
},
{
"role": "user",
"content": [
{
"type": "text",
"text": "It was the night before my birthday. Hooray! It's almost here! It may not be a holiday, but it's the best day of the year."
}
]
},
{
"role": "assistant",
"content": [
{
"type": "audio",
"url": "https://huggingface.co/datasets/eustlb/dummy-audio-samples-higgs/resolve/main/belinda.wav"
}
]
},
{
"role": "user",
"content": [
{
"type": "text",
"text": "The sun rises in the east and sets in the west. This simple fact has been observed by humans for thousands of years."
}
]
}
]
inputs = processor.apply_chat_template(
conversation,
add_generation_prompt=True,
tokenize=True,
return_dict=True,
sampling_rate=24000,
return_tensors="pt"
).to(model.device)
outputs = model.generate(**inputs, max_new_tokens=1000, do_sample=False)
decoded = processor.batch_decode(outputs)
processor.save_audio(decoded, "output_zero_shot_voice_cloning.wav")
from transformers import AutoProcessor, HiggsAudioV2ForConditionalGeneration
model_id = "eustlb/higgs-audio-v2-generation-3B-base"
processor = AutoProcessor.from_pretrained(model_id, device_map="auto")
model = HiggsAudioV2ForConditionalGeneration.from_pretrained(model_id, device_map="auto")
user_message = """[SPEAKER0] I can't believe you did that without even asking me first!
[SPEAKER1] Oh, come on! It wasn't a big deal, and I knew you would overreact like this.
[SPEAKER0] Overreact? You made a decision that affects both of us without even considering my opinion!
[SPEAKER1] Because I didn't have time to sit around waiting for you to make up your mind! Someone had to act."""
conversation = [
{
"role": "system",
"content": [
{
"type": "text",
"text": "Generate audio following instruction."
}
]
},
{
"role": "scene",
"content": [
{
"type": "text",
"text": "Audio is recorded from a quiet room."
},
{
"type": "text",
"text": "SPEAKER0:"
},
{
"type": "audio",
"url": "https://qianwen-res.oss-cn-beijing.aliyuncs.com/Qwen2-Audio/audio/guess_age_gender.wav"
},
{
"type": "text",
"text": "SPEAKER1:"
},
{
"type": "audio",
"url": "https://qianwen-res.oss-cn-beijing.aliyuncs.com/Qwen2-Audio/audio/1272-128104-0000.flac"
},
]
},
{
"role": "user",
"content": [
{
"type": "text",
"text": user_message
}
]
},
]
inputs = processor.apply_chat_template(
conversation,
add_generation_prompt=True,
tokenize=True,
return_dict=True,
sampling_rate=24000,
return_tensors="pt"
).to(model.device)
outputs = model.generate(**inputs, max_new_tokens=1000, do_sample=False)
decoded = processor.batch_decode(outputs)
processor.save_audio(decoded, "output_multi_speaker_voice_cloning.wav")
from transformers import AutoProcessor, HiggsAudioV2ForConditionalGeneration
model_id = "eustlb/higgs-audio-v2-generation-3B-base"
processor = AutoProcessor.from_pretrained(model_id, device_map="auto")
model = HiggsAudioV2ForConditionalGeneration.from_pretrained(model_id, device_map="auto")
conversation1 = [
{
"role": "system",
"content": [
{
"type": "text",
"text": "Generate audio following instruction."
}
]
},
{
"role": "scene",
"content": [
{
"type": "text",
"text": "Audio is recorded from a quiet room."
}
]
},
{
"role": "user",
"content": [
{
"type": "text",
"text": "It was the night before my birthday. Hooray! It's almost here! It may not be a holiday, but it's the best day of the year."
}
]
},
{
"role": "assistant",
"content": [
{
"type": "audio",
"url": "https://huggingface.co/datasets/eustlb/dummy-audio-samples-higgs/resolve/main/belinda.wav"
}
]
},
{
"role": "user",
"content": [
{
"type": "text",
"text": "The sun rises in the east and sets in the west. This simple fact has been observed by humans for thousands of years."
}
]
}
]
conversation2 = [
{
"role": "system",
"content": [
{
"type": "text",
"text": "Generate audio following instruction."
}
]
},
{
"role": "scene",
"content": [
{
"type": "text",
"text": "Audio is recorded from a quiet room."
}
]
},
{
"role": "user",
"content": [
{
"type": "text",
"text": " It's super important to assess fairly the fact that our former model is over. And this is not a question of adjustment. This is not the same world, 2024, 2025. And on top of that, we are making the same mistakes, on top of the key elements I mentioned. We are over-regulating and under-investing. So just if, in the two to three years to come, if we follow our classical agenda, we will be out of the market. I have no doubts."
}
]
},
{
"role": "assistant",
"content": [
{
"type": "audio",
"url": "https://huggingface.co/datasets/eustlb/dummy-audio-samples-higgs/resolve/main/macron.wav"
}
]
},
{
"role": "user",
"content": [
{
"type": "text",
"text": "Hey, here is a clone from the given voice."
}
]
}
]
inputs = processor.apply_chat_template(
[conversation1, conversation2],
add_generation_prompt=True,
tokenize=True,
return_dict=True,
sampling_rate=24000,
return_tensors="pt"
).to(model.device)
outputs = model.generate(**inputs, max_new_tokens=1000, do_sample=False)
decoded = processor.batch_decode(outputs)
processor.save_audio(decoded, ["output_batched_1.wav", "output_batched_2.wav"])
[!TIP] By default, the model does not load the text language modeling head to save memory (~1.5GiB reduction), as it's not required for generation. However, when training the model, you need the text head to compute loss on text tokens. To enable it, set
use_text_head=Truewhen instantiating the model (see example below).
from transformers import AutoProcessor, HiggsAudioV2ForConditionalGeneration
model_id = "eustlb/higgs-audio-v2-generation-3B-base"
processor = AutoProcessor.from_pretrained(model_id, device_map="auto")
model = HiggsAudioV2ForConditionalGeneration.from_pretrained(model_id, device_map="auto", use_text_head=True)
conversation1 = [
{
"role": "system",
"content": [
{
"type": "text",
"text": "Generate audio following instruction."
}
]
},
{
"role": "scene",
"content": [
{
"type": "text",
"text": "Audio is recorded from a quiet room."
}
]
},
{
"role": "user",
"content": [
{
"type": "text",
"text": "It was the night before my birthday. Hooray! It's almost here! It may not be a holiday, but it's the best day of the year."
}
]
},
{
"role": "assistant",
"content": [
{
"type": "audio",
"url": "https://huggingface.co/datasets/eustlb/dummy-audio-samples-higgs/resolve/main/belinda.wav"
}
]
}
]
conversation2 = [
{
"role": "system",
"content": [
{
"type": "text",
"text": "Generate audio following instruction."
}
]
},
{
"role": "scene",
"content": [
{
"type": "text",
"text": "Audio is recorded from a quiet room."
}
]
},
{
"role": "user",
"content": [
{
"type": "text",
"text": " I would imagine so. A wand with a dragon heartstring core is capable of dazzling magic, and the bond between you and your wand should only grow stronger. Do not be surprised at your new wand's ability to perceive your intentions, particularly in a moment of need"
}
]
},
{
"role": "assistant",
"content": [
{
"type": "audio",
"url": "https://huggingface.co/datasets/eustlb/dummy-audio-samples-higgs/resolve/main/broom_salesman.wav"
}
]
}
]
inputs = processor.apply_chat_template(
[conversation1, conversation2],
add_generation_prompt=True,
tokenize=True,
return_dict=True,
sampling_rate=24000,
return_tensors="pt",
output_labels=True,
).to(model.device)
outputs = model(**inputs)
outputs.loss.backward()
This model was contributed by Shuai Zheng and Eustache Le Bihan. The original code can be found here.
[[autodoc]] HiggsAudioV2Config
[[autodoc]] HiggsAudioV2Processor - call - decode
[[autodoc]] HiggsAudioV2Model - forward
[[autodoc]] HiggsAudioV2ForConditionalGeneration - forward - generate