parameter

**Parameter-Efficient Fine-Tuning (PEFT) and LoRA** is **a family of techniques that adapt large pretrained models to downstream tasks by training a small number of additional parameters rather than fine-tuning the entire model — reducing memory requirements, storage costs, and computational overhead while maintaining competitive performance**. Parameter-Efficient Fine-Tuning emerged from the practical challenge of fine-tuning billion-parameter models on memory-constrained hardware. Low-Rank Adaptation (LoRA) is the most prominent PEFT technique, introducing small, trainable rank-decomposed matrices that modify the weight matrices of pretrained models. In LoRA, for each weight matrix W, trainable matrices A and B are added where the weight update is computed as ΔW = AB^T, with A having shape d×r and B having shape k×r, where r is a small rank (typically 4-64). Since only A and B are trained while W remains frozen, the number of trainable parameters scales linearly with model size rather than quadratically. LoRA can be applied selectively to specific layers (typically attention layers show best results) and different tasks can share the base model with task-specific LoRA modules, enabling efficient multitask learning. The technique achieves remarkable efficiency gains — adapting a 7B parameter model requires training only millions rather than billions of parameters. Other PEFT approaches include adapter modules that insert small bottleneck layers, prompt tuning that learns task-specific tokens, prefix tuning that prepends learnable embeddings, and selective fine-tuning of specific layer types. QLoRA combines LoRA with quantization, reducing memory requirements further by quantizing the base model to 4-bit precision while keeping LoRA adapters in higher precision. Many PEFT techniques have been unified under frameworks that allow composable combinations of different parameter-efficient modules. The effectiveness of PEFT is particularly striking in few-shot scenarios where task-specific data is limited, sometimes matching or exceeding standard fine-tuning. Research shows that LoRA's effectiveness stems from the low intrinsic dimensionality of task-specific adaptation — the actual changes needed for downstream tasks lie in a low-rank subspace. The techniques generalize across different model architectures and modalities, working effectively for vision, language, and multimodal models. Infrastructure benefits include faster training, reduced storage for multiple adapted models, and enabling deployment on edge devices. **Parameter-efficient fine-tuning techniques like LoRA democratize adaptation of large models by dramatically reducing computational and storage requirements while maintaining state-of-the-art performance.**