In-Context Learning Adaptation

# In-Context Learning & Adaptation

## Introduction & Motivation

In-context learning: adapt models via demonstration in prompt. Few-shot adaptation without training. Applications: rapid task adaptation, versatile models.

Motivation: Enable quick adaptation to new tasks.

Applications: Few-shot learning, zero-shot generalization, task switching.

---

## Core Concepts & Theory

### Demonstration

Show task via examples.

### Adaptation

Learn from demonstrations in-context.

### Rapid Transfer

No parameter updates needed.

### Emergent Abilities

Scale enables new capabilities.

---

## Mathematical Formulation

In-Context Learning:
$$P(y|x, E) = P(y|\{(x_1, y_1), ..., (x_k, y_k), x\})$$

Context Compression:
$$ ext{Performance} \propto ext{quality}(E)$$

---

## Advanced Theory & Extensions

### Instruction Tuning

Optimize for instruction following.

### Calibration

Control output confidence.

### Context Distillation

Extract key patterns.

---

## Computational Considerations

Context processing: O(k·T).

Inference: Standard forward pass.

Memory: Stored in context.

---

## Practical Implementation Strategies

### Example Ordering

Order affects performance.

### Demonstration Quality

Choose representative examples.

### Prompt Engineering

Optimize context formulation.

---

## Benchmark Datasets & Evaluation

Few-Shot Benchmarks: GLUE-X, SuperGLUE-X.

Zero-Shot: Direct generalization.

Domain Transfer: Cross-domain adaptation.

---

## Key Challenges & Limitations

### Instability

Sensitive to demonstration order.

### Context Length

Limited by window size.

### Performance Ceiling

Below task-specific training.

---

## Hyperparameter Tuning

Number of examples: 1-32.

Example selection: Semantic or random.

Prompt format: Task-dependent.

---

## Real-World Applications & Case Studies

Code Synthesis: Generate programs.

Translation: Cross-lingual transfer.

Classification: Rapid adaptation.

---

## Integration with Other Methods

In-context + prompt engineering; + retrieval for example selection.

---

## Summary & Key Takeaways

In-context learning enables rapid adaptation.

Principles:
1. Demonstration: Show via examples.
2. Context: Provide in prompt.
3. Adaptation: Learn without updates.
4. Scalability: Improves with model size.
5. Versatility: Broad task applicability.

---

## Appendix: Practical Labs

### Lab 1: Construct Context

def construct_in_context_examples(examples, instructions=""):
 """Build in-context learning prompt"""
 context = instructions + "

"
 for ex in examples:
 context += f"Input: {ex['input']}
"
 context += f"Output: {ex['output']}

"
 return context

examples = [
 {"input": "hello", "output": "HELLO"},
 {"input": "world", "output": "WORLD"}
]
ctx = construct_in_context_examples(examples, "Convert to uppercase")
assert "HELLO" in ctx
print("✓ Context construction working")

### Lab 2: Order Sensitivity

def evaluate_order_impact(examples):
 """Test sensitivity to example ordering"""
 import numpy as np
 
 scores = []
 for _ in range(5):
 shuffled = examples.copy()
 np.random.shuffle(shuffled)
 # Simulate evaluation
 score = len(shuffled)
 scores.append(score)
 
 std = np.std(scores)
 return std

examples = ["A", "B", "C", "D"]
sensitivity = evaluate_order_impact(examples)
assert sensitivity >= 0
print(f"✓ Order sensitivity: std={sensitivity:.2f}")

### Lab 3: Context Compression

import numpy as np

def compress_context(examples, compression_ratio=0.5):
 """Compress examples to key information"""
 num_keep = max(1, int(len(examples) * compression_ratio))
 # Select most important examples
 importance = np.random.rand(len(examples))
 indices = np.argsort(importance)[-num_keep:]
 return [examples[i] for i in indices]

examples = ["Ex1", "Ex2", "Ex3", "Ex4"]
compressed = compress_context(examples, 0.5)
assert len(compressed) <= len(examples)
print(f"✓ Compression: {len(examples)} → {len(compressed)}")

### Lab 4: Task Switching

def switch_task_context(current_task, new_task, examples):
 """Switch context for different task"""
 context = f"Switching task from {current_task} to {new_task}

"
 for ex in examples:
 context += f"{ex}
"
 return context

ctx = switch_task_context("classification", "generation", ["Ex1", "Ex2"])
assert "generation" in ctx
print("✓ Task switching working")

---