Balancing Efficiency and Performance: A Comparative Study of LoRA, Adapters, and Prompt Tuning

Abstract

The rapid growth of pre-trained language models has substantially improved performance across Natural Language Processing (NLP) tasks. However, full fine-tuning remains computationally expensive because it requires updating and storing all model parameters for each downstream task. This limitation is particularly important in resource-constrained environments where access to high-performance computing infrastructure is limited. Parameter-efficient fine-tuning (PEFT) methods, including Low-Rank Adaptation (LoRA), Adapters, and Prompt Tuning, offer practical alternatives by updating only a small subset of parameters while keeping most of the pre-trained model frozen. This study presents a controlled and reproducible comparison of LoRA, Adapters, and Prompt Tuning under lightweight experimental conditions. Using SST-2 and MRPC from the GLUE benchmark with the BERT-base model, we evaluate predictive performance using accuracy and F1-score, and computational efficiency using trainable parameter ratio, training time per epoch, and peak GPU memory usage. The experimental design emphasizes accessibility by using modest hardware settings such as a single-GPU Google Colab environment. The results show that LoRA provides the strongest efficiency–performance balance, achieving performance close to full fine-tuning while substantially reducing the number of trainable parameters. Adapters demonstrate stable performance and modular flexibility but introduce moderate computational overhead. Prompt Tuning requires the fewest trainable parameters, but its performance is more sensitive to dataset size and task complexity. These findings provide practical guidance for selecting PEFT methods in resource-constrained NLP applications and highlight the importance of evaluating model adaptation strategies through both accuracy and computational cost.