examples/beft_finetuning/README.md
Fine-tuning the bias terms of large language models (LLMs) has the potential to achieve unprecedented parameter efficiency while maintaining competitive performance, particularly in low-data regimes. In this paper, we investigate the link between fine-tuning b<sub>q</sub>, b<sub>k</sub>, and b<sub>v</sub> with the performance of the downstream task, both analytically and empirically. We study and shed light on the expressive power of bias terms b<sub>q</sub>, b<sub>k</sub>, and b<sub>v</sub> in the query, key, or value projections of LLMs including bias-term-free LLMs. Our key finding is that directly fine-tuning b<sub>v</sub> generally leads to higher downstream performance in low-data regimes, in comparison to b<sub>q</sub> and b<sub>k</sub>.
You can try target_modules=["v"], or ["q"], or ["k"] in beft_finetuning.py to see the downstream accuracy.
@inproceedings{huang2026beft,
title={BEFT: Bias-Efficient Fine-Tuning of Language Models in Low-Data Regimes},
author={Huang, Baichuan and Balashankar, Ananth and Aminifar, Amir},
booktitle={Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics},
year={2026}
}