ContextQMD
Back to Transformers

FalconH1

docs/source/en/model_doc/falcon_h1.md

5.8.03.3 KB
Original Source
<!--Copyright 2025 The HuggingFace Team. All rights reserved. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. ⚠️ Note that this file is in Markdown but contain specific syntax for our doc-builder (similar to MDX) that may not be rendered properly in your Markdown viewer. -->

This model was released on 2025-05-21 and added to Hugging Face Transformers on 2025-05-21.

FalconH1

Overview

The FalconH1 model was developed by the TII Pretraining team. A comprehensive research paper covering the architecture, pretraining dynamics, experimental results, and conclusions is forthcoming. You can read more about this series in this website.

Contributors

This model was contributed by DhiyaEddine, ybelkada, JingweiZuo, IlyasChahed, and MaksimVelikanov. The original code can be found here.

FalconH1Config

ModelDepthDimAttn HeadsKVMamba Headsd_headd_stateCtx Len
H1 0.5B361024822464 / 641284K, 16K-SFT
H1 1.5B2420488248128 / 64256128K
H1 1.5B-d6612806224128 / 64256128K
H1 3B32256010232128 / 128256128K
H1 7B44307212224128 / 128256256K
H1 34B72512020432128 / 128256256K

[[autodoc]] FalconH1Config

<!--- ## Usage Tips Tips: - The architecture is based on Mamba-2 models. ## FalconH1Model [[autodoc]] FalconH1Model - forward -->

FalconH1ForCausalLM

python
from transformers import AutoModelForCausalLM, AutoTokenizer


model = AutoModelForCausalLM.from_pretrained("tiiuae/Falcon-H1-7B-Instruct", device_map="auto")
tokenizer = AutoTokenizer.from_pretrained("tiiuae/Falcon-H1-7B-Instruct")

message = ["Mamba is a snake with following properties  "]
inputs = tokenizer(message, return_tensors='pt', return_token_type_ids=False).to(model.device)
response = model.generate(**inputs, max_new_tokens=64)
print(tokenizer.batch_decode(response, skip_special_tokens=True)[0])

[[autodoc]] FalconH1ForCausalLM - forward

This HF implementation is contributed by younesbelkada and DhiaEddineRhaiem.