Parameter-Efficient Fine-Tuning LoRA

# Parameter-Efficient Fine-Tuning (LoRA)

## Introduction & Motivation

LoRA: low-rank adaptation for efficient fine-tuning. Train small adapters instead of full model. Applications: memory-efficient training, multiple task adapters.

Motivation: Fine-tune large models with minimal parameters.

Applications: Multi-task adaptation, resource-constrained training.

---

## Core Concepts & Theory

### Low-Rank Decomposition

Factor weight updates into low-rank matrices.

### Adapter Layers

Lightweight trainable modules.

### Frozen Base

Keep pretrained weights frozen.

### Linear Projection

Project to and from low-rank space.

---

## Mathematical Formulation

Weight Update:
$$W' = W + \Delta W = W + BA$$

Where B ∈ R^{d×r}, A ∈ R^{r×k}, r << min(d,k)

LoRA Loss:
$$\mathcal{L} = ext{loss}(y, f_W(x) + f_{BA}(x))$$

---

## Advanced Theory & Extensions

### QLoRA

Quantized LoRA.

### Multi-Task LoRA

Shared and task-specific.

### Parameterized Scaling

Scale by module.

---

## Computational Considerations

Parameters: 0.1-1% of model.

Training: 10-20% of full fine-tuning.

Inference: Negligible overhead.

---

## Practical Implementation Strategies

### Rank Selection

r = 8, 16, 32 typical.

### Module Selection

Which layers to adapt.

### Initialization

Random or structured.

---

## Benchmark Datasets & Evaluation

GLUE: Text understanding tasks.

SuperGLUE: Challenging tasks.

Domain Adaptation: Task-specific evaluation.

---

## Key Challenges & Limitations

### Capacity

Limited by rank.

### Task Similarity

Sharing hurts dissimilar tasks.

### Tuning

Rank requires experimentation.

---

## Hyperparameter Tuning

Rank: 8-64.

Learning rate: 1e-4 to 1e-3.

Alpha: 16-32.

---

## Real-World Applications & Case Studies

Multi-Task: Single model, many tasks.

Budget-Constrained: Limited memory.

Rapid Adaptation: Quick fine-tuning.

---

## Integration with Other Methods

LoRA + quantization for QLoRA; + other adapters.

---

## Summary & Key Takeaways

LoRA enables efficient fine-tuning.

Principles:
1. Low-rank: BA decomposition.
2. Efficient: 0.1% additional parameters.
3. Frozen: Keep pretrained weights.
4. Flexible: Easy multi-task adaptation.
5. Performance: Match full fine-tuning.

---

## Appendix: Practical Labs

### Lab 1: LoRA Update

import numpy as np

def lora_forward(x, weight, B, A, alpha=16):
 """Forward pass with LoRA"""
 base_out = x @ weight.T
 lora_out = (x @ A.T) @ B.T
 output = base_out + (alpha / 16) * lora_out
 return output

np.random.seed(42)
x = np.random.randn(10, 768)
W = np.random.randn(1000, 768)
B = np.random.randn(768, 8)
A = np.random.randn(8, 768)
out = lora_forward(x, W, B, A)
assert out.shape == (10, 1000)
print("✓ LoRA forward working")

### Lab 2: Parameter Efficiency

def compute_lora_efficiency(original_params, rank, output_dim, input_dim):
 """Compute parameter efficiency"""
 lora_params = rank * input_dim + rank * output_dim
 efficiency = lora_params / original_params
 return efficiency

original = 768 * 3072
efficiency = compute_lora_efficiency(original, 16, 3072, 768)
assert efficiency < 0.01
print(f"✓ Parameter efficiency: {efficiency:.2%}")

### Lab 3: Multi-Task LoRA

import numpy as np

def multi_task_lora(x, base_weight, task_loras, task_id):
 """Multi-task LoRA inference"""
 base_out = x @ base_weight.T
 
 B, A = task_loras[task_id]
 task_out = (x @ A.T) @ B.T
 
 output = base_out + task_out
 return output

np.random.seed(42)
x = np.random.randn(10, 768)
W = np.random.randn(1000, 768)
task_loras = [(np.random.randn(768, 8), np.random.randn(8, 768)) 
 for _ in range(5)]
out = multi_task_lora(x, W, task_loras, 0)
assert out.shape == (10, 1000)
print("✓ Multi-task LoRA working")

### Lab 4: Rank Analysis

import numpy as np

def analyze_rank_effect(x, weight, ranks=[4, 8, 16, 32, 64]):
 """Analyze effect of LoRA rank"""
 results = []
 for r in ranks:
 B = np.random.randn(768, r)
 A = np.random.randn(r, 768)
 
 base = x @ weight.T
 lora = (x @ A.T) @ B.T
 error = np.mean((lora) ** 2)
 
 results.append({'rank': r, 'error': error})
 
 return results

np.random.seed(42)
x = np.random.randn(10, 768)
W = np.random.randn(1000, 768)
results = analyze_rank_effect(x, W)
print(f"✓ Rank analysis: {len(results)} configurations")

---