parameter-efficient fine-tuning lora
# Parameter-Efficient Fine-Tuning (LoRA)
## Introduction & Motivation
LoRA: low-rank adaptation for efficient fine-tuning. Train small adapters instead of full model. Applications: memory-efficient training, multiple task adapters.
Motivation: Fine-tune large models with minimal parameters.
Applications: Multi-task adaptation, resource-constrained training.
---
## Core Concepts & Theory
### Low-Rank Decomposition
Factor weight updates into low-rank matrices.
### Adapter Layers
Lightweight trainable modules.
### Frozen Base
Keep pretrained weights frozen.
### Linear Projection
Project to and from low-rank space.
---
## Mathematical Formulation
Weight Update:
$$W' = W + \Delta W = W + BA$$
Where B ∈ R^{d×r}, A ∈ R^{r×k}, r << min(d,k)
LoRA Loss:
$$\mathcal{L} = ext{loss}(y, f_W(x) + f_{BA}(x))$$
---
## Advanced Theory & Extensions
### QLoRA
Quantized LoRA.
### Multi-Task LoRA
Shared and task-specific.
### Parameterized Scaling
Scale by module.
---
## Computational Considerations
Parameters: 0.1-1% of model.
Training: 10-20% of full fine-tuning.
Inference: Negligible overhead.
---
## Practical Implementation Strategies
### Rank Selection
r = 8, 16, 32 typical.
### Module Selection
Which layers to adapt.
### Initialization
Random or structured.
---
## Benchmark Datasets & Evaluation
GLUE: Text understanding tasks.
SuperGLUE: Challenging tasks.
Domain Adaptation: Task-specific evaluation.
---
## Key Challenges & Limitations
### Capacity
Limited by rank.
### Task Similarity
Sharing hurts dissimilar tasks.
### Tuning
Rank requires experimentation.
---
## Hyperparameter Tuning
Rank: 8-64.
Learning rate: 1e-4 to 1e-3.
Alpha: 16-32.
---
## Real-World Applications & Case Studies
Multi-Task: Single model, many tasks.
Budget-Constrained: Limited memory.
Rapid Adaptation: Quick fine-tuning.
---
## Integration with Other Methods
LoRA + quantization for QLoRA; + other adapters.
---
## Summary & Key Takeaways
LoRA enables efficient fine-tuning.
Principles:
1. Low-rank: BA decomposition.
2. Efficient: 0.1% additional parameters.
3. Frozen: Keep pretrained weights.
4. Flexible: Easy multi-task adaptation.
5. Performance: Match full fine-tuning.
---
## Appendix: Practical Labs
### Lab 1: LoRA Update
import numpy as np
def lora_forward(x, weight, B, A, alpha=16):
"""Forward pass with LoRA"""
base_out = x @ weight.T
lora_out = (x @ A.T) @ B.T
output = base_out + (alpha / 16) * lora_out
return output
np.random.seed(42)
x = np.random.randn(10, 768)
W = np.random.randn(1000, 768)
B = np.random.randn(768, 8)
A = np.random.randn(8, 768)
out = lora_forward(x, W, B, A)
assert out.shape == (10, 1000)
print("✓ LoRA forward working")### Lab 2: Parameter Efficiency
def compute_lora_efficiency(original_params, rank, output_dim, input_dim):
"""Compute parameter efficiency"""
lora_params = rank * input_dim + rank * output_dim
efficiency = lora_params / original_params
return efficiency
original = 768 * 3072
efficiency = compute_lora_efficiency(original, 16, 3072, 768)
assert efficiency < 0.01
print(f"✓ Parameter efficiency: {efficiency:.2%}")### Lab 3: Multi-Task LoRA
import numpy as np
def multi_task_lora(x, base_weight, task_loras, task_id):
"""Multi-task LoRA inference"""
base_out = x @ base_weight.T
B, A = task_loras[task_id]
task_out = (x @ A.T) @ B.T
output = base_out + task_out
return output
np.random.seed(42)
x = np.random.randn(10, 768)
W = np.random.randn(1000, 768)
task_loras = [(np.random.randn(768, 8), np.random.randn(8, 768))
for _ in range(5)]
out = multi_task_lora(x, W, task_loras, 0)
assert out.shape == (10, 1000)
print("✓ Multi-task LoRA working")### Lab 4: Rank Analysis
import numpy as np
def analyze_rank_effect(x, weight, ranks=[4, 8, 16, 32, 64]):
"""Analyze effect of LoRA rank"""
results = []
for r in ranks:
B = np.random.randn(768, r)
A = np.random.randn(r, 768)
base = x @ weight.T
lora = (x @ A.T) @ B.T
error = np.mean((lora) ** 2)
results.append({'rank': r, 'error': error})
return results
np.random.seed(42)
x = np.random.randn(10, 768)
W = np.random.randn(1000, 768)
results = analyze_rank_effect(x, W)
print(f"✓ Rank analysis: {len(results)} configurations")---