Efficient Fine-Tuning Strategies for Domain-Specific Large Language Models

Abstract

Fine-tuning pre-trained large language models (LLMs) for domain-specific applications -- adapting general-purpose models to clinical, legal, financial, or scientific language domains -- is the primary practical methodology for deploying LLM capabilities in specialised contexts. The computational cost and data efficiency of fine-tuning are critical practical constraints: full fine-tuning of 7B-70B parameter models requires substantial compute and risks catastrophic forgetting of general capabilities, while naive instruction tuning on small domain datasets risks overfitting and poor out-of-distribution generalisation. This paper presents a systematic comparative evaluation of six parameter-efficient fine-tuning (PEFT) strategies -- LoRA, QLoRA, prefix tuning, prompt tuning, IA3, and full fine-tuning with selective layer unfreezing -- across four domain adaptation benchmarks: clinical NLP (MedQA), legal reasoning (LegalBench), financial analysis (FinBench), and scientific literature understanding (SciQ). Evaluation covers five criteria: domain task performance, general capability retention, training compute efficiency, memory efficiency, and catastrophic forgetting susceptibility. Results demonstrate that QLoRA achieves the best overall efficiency-performance profile: matching full fine-tuning domain performance within 2.1% on average while reducing training memory by 73.4% and training time by 61.8%. Domain-specific analysis reveals that the optimal PEFT strategy varies by domain: LoRA outperforms QLoRA in legal reasoning (precision-sensitive tasks), while prefix tuning achieves the lowest catastrophic forgetting in scientific NLP. The study contributes a systematic PEFT strategy evaluation benchmark, domain-specific fine-tuning recommendations, and an open-source fine-tuning toolkit implementing all six strategies on a unified codebase.