SavedAdd your journal

Journals / Gazi University Journal of Science Part A: Engineering and Innovation / 2026 / Volume: 13 Issue: 3

QLoRA vs. LoRA: An Empirical Study of Performance–Efficiency Trade-offs in Fine-Tuning Mistral‑7B on Instruction‑Following Data

Abdullah Talha Kabakuş(DUZCE UNIVERSITY)0000-0003-2181-4292
Pages
—
Publication Date
2026-09-30
🏛️Bibliographic metadata and full text access for this article are provided via DergiPark.
DergiPark Record ↗
✨

AI Summary & Key Takeaways

AI Insight
⚡ TL;DR(Too Long; Didn't Read)

The results show that QLoRA not only reduces memory usage by approximately 70% but also achieves superior performance compared to LoRA, with improvements in token‑level F1 and BERTScore F1 reaching statistical significance (p < 0.05), while other lexical metrics showed consistent but non‑significant improvements.

📌 3 Key Takeaways

🎯
Objective & Hypothesis:Parameter‑Efficient Fine‑Tuning (PEFT) approaches have emerged as a key strategy for efficiently adapting Large Language Models (LLMs) in resource‑constrained environments.
📊
Key Findings & Data:Both approaches are evaluated across multiple dimensions, including generation quality (ROUGE‑L, Exact Match, token‑level F1, and the semantic metric BERTScore), training efficiency, and GPU memory consumption.
💡
Conclusion & Impact:The results show that QLoRA not only reduces memory usage by approximately 70% but also achieves superior performance compared to LoRA, with improvements in token‑level F1 and BERTScore F1 reaching statistical significance (p < 0.05), while other lexical metrics showed consistent but non‑significant improvements.
🔬Design:Modeling & Simulation
📍Scope:Comprehensive Sample
⚙️Variables:Large Language Model · Fine-Tuning · Low-Rank Adaptation

Abstract

Parameter‑Efficient Fine‑Tuning (PEFT) approaches have emerged as a key strategy for efficiently adapting Large Language Models (LLMs) in resource‑constrained environments. Within this family of methods, LoRA and its quantized extension QLoRA provide effective alternatives by substantially decreasing the number of trainable parameters required for adaptation. In this study, a systematic comparison of LoRA and QLoRA is presented using a controlled experimental setup with identical base models and training data, with all experiments repeated across three random seeds and statistical significance assessed via paired t‑tests. Both approaches are evaluated across multiple dimensions, including generation quality (ROUGE‑L, Exact Match, token‑level F1, and the semantic metric BERTScore), training efficiency, and GPU memory consumption. Baseline (untrained) model performance is reported to quantify the gain from fine‑tuning, and loss curves are provided to confirm convergence. The results show that QLoRA not only reduces memory usage by approximately 70% but also achieves superior performance compared to LoRA, with improvements in token‑level F1 and BERTScore F1 reaching statistical significance (p < 0.05), while other lexical metrics showed consistent but non‑significant improvements. Furthermore, the performance–efficiency trade‑offs are analyzed, and it is demonstrated that QLoRA offers a more favorable balance between computational cost and model quality. These findings highlight QLoRA as a highly effective and practical fine‑tuning strategy for resource‑constrained environments, supported by a reproducible, statistically robust evaluation pipeline.

🔖

Cite This Article

Kabakuş, A. T. (2026). QLoRA vs. LoRA: An Empirical Study of Performance–Efficiency Trade-offs in Fine-Tuning Mistral‑7B on Instruction‑Following Data. Gazi University Journal of Science Part A: Engineering and Innovation, Volume: 13 Issue: 3. https://doi.org/10.54287/gujsa.1937530

📄 Full Text PDF Viewer

📄

PDF görüntüleyici hazırlanıyor...

References (30)

  1. Ben Zaken, E., Goldberg, Y., & Ravfogel, S. (2022). BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-models. In: S. Muresan, P. Nakov, & A. Villavicencio (Eds.), Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Vol. 2, pp. 1–9), (22-27 May 2022), Dublin, Ireland. https://doi.org/10.18653/v1/2022.acl-short.1
  2. Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D. M., Wu, J., Winter, C., … Amodei, D. (2020). Language models are few-shot learners. In: H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, H. Lin (Eds.), Proceedings of the 34th International Conference on Neural Information Processing Systems (NeurIPS 2020) (pp. 1877–1901), (6-12 December 2020), Vancouver BC Canada. https://doi.org/10.48550/arXiv.2005.14165
  3. Chung, H. W., Hou, L., Longpre, S., Zoph, B., Tai, Y., Fedus, W., Li, Y., Wang, X., Dehghani, M., Brahma, S., Webson, A., Gu, S. S., Dai, Z., Suzgun, M., Chen, X., Chowdhery, A., Castro-Ros, A., Pellat, M., Robinson, K., … Wei, J. (2024). Scaling Instruction-Finetuned Language Models. Journal of Machine Learning Research, 25(1), 3381–3433. https://doi.org/10.48550/arXiv.2210.11416
  4. Dettmers, T., Lewis, M., Belkada, Y., & Zettlemoyer, L. (2022). LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale. In: S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, & A. Oh (Eds.), Proceedings of the 36th International Conference on Neural Information Processing Systems (NeurIPS 2022) (pp. 30318 - 30332), (28 November - 9 December 2022), New Orleans LA USA. https://doi.org/10.48550/arXiv.2208.07339
  5. Dettmers, T., Pagnoni, A., Holtzman, A., & Zettlemoyer, L. (2023). QLoRA: Efficient Finetuning of Quantized LLMs. In: A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, & S. Levine (Eds.), Proceedings of the 37th International Conference on Neural Information Processing Systems (NeurIPS 2023), (pp. 10088–10115), (10-16 December 2023), New Orleans LA USA. https://doi.org/10.48550/arXiv.2305.14314
  6. Ding, N., Qin, Y., Yang, G., Wei, F., Yang, Z., Su, Y., Hu, S., Chen, Y., Chan, C.-M., Chen, W., Yi, J., Zhao, W., Wang, X., Liu, Z., Zheng, H.-T., Chen, J., Liu, Y., Tang, J., Li, J., & Sun, M. (2023). Parameter-efficient fine-tuning of large-scale pre-trained language models. Nature Machine Intelligence, 5(3), 220–235. https://doi.org/10.1038/s42256-023-00626-4
  7. Frantar, E., Ashkboos, S., Hoefler, T., & Alistarh, D. (2023). OPTQ: Accurate Quantization for Generative Pre-trained Transformers. In: Proceedings of the 11th International Conference on Learning Representations (ICLR 2023) (pp. 1–16), (1-5 May 2023), Kigali Rwanda.
  8. Guan, Z. K., & Wang, Y. (2025). Evolving Paradigms in Task-Based Search and Learning: A Comparative Analysis of Traditional Search Engine with LLM-Enhanced Conversational Search System. https://doi.org/10.48550/arXiv.2512.00313
  9. Hayou, S., Ghosh, N., & Yu, B. (2024). LoRA+: Efficient Low Rank Adaptation of Large Models. In: R. Salakhutdinov, Z. Kolter, K. Heller, A. Weller, N. Oliver, J. Scarlett, & F. Berkenkamp (Eds.), Proceedings of the 41st International Conference on Machine Learning (ICML’24), (pp. 17783–17806), (21-27 July 2024), Vienna Austria. https://doi.org/10.48550/arXiv.2402.12354
  10. He, J., Zhou, C., Ma, X., Berg-Kirkpatrick, T., & Neubig, G. (2022). Towards a unified view of parameter-efficient transfer learning. In: Proceedings of the 10th International Conference on Learning Representations (ICLR 2022) (pp. 1–15), (25-29 April 2022). https://doi.org/10.48550/arXiv.2110.04366
  11. Houlsby, N., Giurgiu, A., Jastrzebski, S., Morrone, B., de Laroussilhe, Q., Gesmundo, A., Attariyan, M., & Gelly, S. (2019). Parameter-efficient transfer learning for NLP. In: K. Chaudhuri, & R. Salakhutdinov (Eds.), Proceedings of the 36th International Conference on Machine Learning (ICML 2019) (pp. 1–10), (9-15 June 2019), Long Beach, California, USA. https://doi.org/10.48550/arXiv.1902.00751
  12. Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., & Chen, W. (2022). LoRA: Low-Rank Adaptation of Large Language Models. In: Proceedings of the 10th International Conference on Learning Representations (ICLR 2022) (pp. 1–13), (25-29 April 2022). https://doi.org/10.48550/arXiv.2106.09685
  13. Hu, K. (2023, February 2). ChatGPT sets record for fastest-growing user base - analyst note. Reuters. https://www.reuters.com/technology/chatgpt-sets-record-fastest-growing-user-base-analyst-note-2023-02-01/
  14. Jiang, A. Q., Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D. S., Casas, D. de las, Bressand, F., Lengyel, G., Lample, G., Saulnier, L., Lavaud, L. R., Lachaux, M.-A., Stock, P., Scao, T. L., Lavril, T., Wang, T., Lacroix, T., & Sayed, W. El. (2023). Mistral 7B. https://doi.org/10.48550/arXiv.2310.06825
  15. Lester, B., Al-Rfou, R., & Constant, N. (2021). The Power of Scale for Parameter-Efficient Prompt Tuning. In: M.-F. Moens, X. Huang, L. Specia, & S. W.-t. Yih (Eds.), Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (EMNLP 2021) (pp. 3045–3059), (7-11 November 2021), Punta Cana, Dominican Republic. https://doi.org/10.18653/v1/2021.emnlp-main.243
  16. Li, X. L., & Liang, P. (2021). Prefix-tuning: Optimizing continuous prompts for generation. In: C. Zong, F. Xia, W. Li, & R. Navigli (Eds.), Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (ACL-IJCNLP 2021) (Vol. 1, pp. 4582–4597), (1-6 August 2021). https://doi.org/10.18653/v1/2021.acl-long.353
  17. Liao, B., Meng, Y., & Monz, C. (2023). Parameter-Efficient Fine-Tuning without Introducing New Latency. In: A. Rogers, J. Boyd-Graber, & N. Okazaki (Eds.), Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (ACL 2023) (Volume 1: Long Papers), 1, (pp. 4242–4260), (10-12 July 2023), Toronto, Canada. https://doi.org/10.18653/v1/2023.acl-long.233
  18. Lin, C.-Y. (2004). ROUGE: A Package for Automatic Evaluation of Summaries. In: Proceedings of the Workshop on Text Summarization Branches Out (WAS 2004), (pp. 74–81), (25–26 July 2004) Barcelona, Spain.
  19. Loshchilov, I., & Hutter, F. (2019). Decoupled Weight Decay Regularization. In: Proceedings of the Seventh International Conference on Learning Representations (ICLR 2019), (pp. 1–18), (6-9 May 2019), New Orleans. https://doi.org/10.48550/arXiv.1711.05101
  20. Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., Schulman, J., Hilton, J., Kelton, F., Miller, L., Simens, M., Askell, A., Welinder, P., Christiano, P., Leike, J., & Lowe, R. (2022). Training language models to follow instructions with human feedback. In: S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, & A. Oh (Eds.), Proceedings of the 36th Conference on Neural Information Processing Systems (NeurIPS 2022) (pp. 27730 - 27744), (28 November - 9 December 2022), New Orleans. https://doi.org/10.48550/arXiv.2203.02155
  21. Pfeiffer, J., Kamath, A., Rücklé, A., Cho, K., & Gurevych, I. (2021). AdapterFusion: Non-destructive task composition for transfer learning. In: P. Merlo, J. Tiedemann, & R. Tsarfaty (Eds.), Proceedings of 16th Conference of the European Chapter of the Association for Computational Linguistics, Proceedings of the Conference (EACL 2021) (pp. 487–503), (19-23 April 2021). https://doi.org/10.18653/v1/2021.eacl-main.39
  22. Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., & Liu, P. J. (2020). Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer. Journal of Machine Learning Research, 21(1), 5485–5551. https://doi.org/10.48550/arXiv.1910.10683
  23. Rajpurkar, P., Zhang, J., Lopyrev, K., & Liang, P. (2016). SQuAD: 100,000+ Questions for Machine Comprehension of Text. In: J. Su, K. Duh, & X. Carreras (Eds.), Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP 2016) (pp. 2383–2392), (1-5 November 2016), Austin, Texas, USA. https://doi.org/10.18653/v1/d16-1264
  24. Taori, R., Gulrajani, I., Zhang, T., Dubois, Y., Li, X., Guestrin, C., Liang, P., & Hashimoto, T. B. (2023). Stanford Alpaca: An Instruction-following LLaMA Model. In GitHub repository. GitHub. https://github.com/tatsu-lab/stanford_alpaca
  25. Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., Rodriguez, A., Joulin, A., Grave, E., & Lample, G. (2023). LLaMA: Open and Efficient Foundation Language Models. https://doi.org/10.48550/arXiv.2302.13971
  26. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., & Polosukhin, I. (2017). Attention Is All You Need. In: U. von Luxburg, I. Guyon, S. Bengio, H. Wallach, & R. Fergus (Eds.), Proceedings of the 31st Annual Conference on Neural Information Processing Systems (NIPS 2017), (4-9 December 2017), Long Beach California USA. https://doi.org/10.48550/arXiv.1706.03762
  27. Xiao, G., Lin, J., Seznec, M., Wu, H., Demouth, J., & Han, S. (2023). SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models. In: A. Krause, E. Brunskill, K. Cho, B. Engelhardt, S. Sabato, & J. Scarlett (Eds.), Proceedings of the 40th International Conference on Machine Learning (ICML’23) (pp. 38087 – 38099), (23-29 July 2023), Honolulu, Hawaii, USA. https://doi.org/10.48550/arXiv.2211.10438
  28. Xu, L., Xie, H., Qin, S.-Z. J., Tao, X., & Wang, F. L. (2026). Parameter-Efficient Fine-Tuning Methods for Pretrained Language Models: A Critical Review and Assessment. IEEE Transactions on Pattern Analysis and Machine Intelligence, 48(6), 6107–6126. https://doi.org/10.1109/TPAMI.2026.3657354
  29. Zhang, Q., Chen, M., Bukharin, A., He, P., Cheng, Y., Chen, W., & Zhao, T. (2023). Adaptive Budget Allocation for Parameter-Efficient Fine-Tuning. In: Proceedings of the 11th International Conference on Learning Representations (ICLR 2023) (pp. 1–17), (1-5 May 2023), Kigali Rwanda. https://doi.org/10.48550/arXiv.2303.10512
  30. Zhang, T., Kishore, V., Wu, F., Weinberger, K. Q., & Artzi, Y. (2020). BERTScore: Evaluating Text Generation with BERT. In: Proceedings of the 8th International Conference on Learning Representations (ICLR 2020) (pp. 1–43), (26-30 April 2020), Addis Ababa, Ethiopia. https://doi.org/10.48550/arXiv.1904.09675