GPT - Generative Pre-Training

# GPT - Generative Pre-Training

## Introduction & Motivation

GPT: autoregressive language modeling. Left-to-right generation, decoder-only architecture. Applications: text generation, language understanding.

Motivation: Generate coherent text via next-token prediction.

Applications: Content generation, conversational AI.

---

## Core Concepts & Theory

### Autoregressive Language Modeling

Predict next token from context.

### Decoder-Only Architecture

Single-direction transformer.

### In-Context Learning

Few-shot adaptation capability.

### Scaling Laws

Performance with model size.

---

## Mathematical Formulation

Autoregressive Loss:
$$\mathcal{L} = -\sum_t \log p(w_t | w_{<t})$$

Next-Token Prediction:
$$p(w_t) = ext{softmax}(W f_ heta(w_{<t}))$$

Perplexity:
$$ ext{PPL} = \exp(\mathcal{L})$$

---

## Advanced Theory & Extensions

### GPT-2

Larger scale, improved results.

### GPT-3

Few-shot in-context learning.

### Instruction Tuning

Fine-tune on tasks.

---

## Computational Considerations

Forward pass: O(T·D·H).

Decoding: O(T²·D) (autoregressive).

Training: O(E·B·T·D).

---

## Practical Implementation Strategies

### Temperature Sampling

Control output diversity.

### Top-k Sampling

Restrict to top candidates.

### Beam Search

Parallel hypothesis exploration.

---

## Benchmark Datasets & Evaluation

LAMBADA: Long-range dependencies.

HellaSwag: Commonsense reasoning.

MMLU: Multitask language understanding.

---

## Key Challenges & Limitations

### Computational Cost

Large model requirements.

### Factuality Issues

Hallucinated information.

### Token Limit

Fixed context window.

---

## Hyperparameter Tuning

Learning rate: 1e-4 to 5e-4.

Temperature: 0.5-1.5.

Top-k: 40-50.

---

## Real-World Applications & Case Studies

Text Generation: Creative writing assistance.

Code Generation: Programming task automation.

Conversation: Interactive dialogue systems.

---

## Integration with Other Methods

GPT + RLHF for alignment; + RAG for grounding.

---

## Summary & Key Takeaways

GPT uses autoregressive generation for coherent text synthesis.

Principles:
1. Autoregressive: Token-by-token generation.
2. Decoder-only: Unidirectional architecture.
3. Scaling: Performance with model size.
4. In-context learning: Few-shot adaptation.
5. Generation: Diverse sampling strategies.

---

## Appendix: Practical Labs

### Lab 1: Temperature Sampling

import numpy as np

def temperature_sampling(logits, temperature=1.0):
 """Sample from logits with temperature control"""
 scaled = logits / temperature
 probs = np.exp(scaled - np.max(scaled))
 probs /= np.sum(probs)
 
 sample = np.random.choice(len(probs), p=probs)
 return sample

np.random.seed(42)
logits = np.random.randn(1000)
sample_t1 = temperature_sampling(logits, temperature=1.0)
sample_t10 = temperature_sampling(logits, temperature=10.0)
assert 0 <= sample_t1 < 1000, "Valid sample"
print("✓ Temperature sampling working")

### Lab 2: Top-K Sampling

import numpy as np

def top_k_sampling(logits, k=40):
 """Sample from top-k most likely tokens"""
 top_k_indices = np.argsort(logits)[-k:]
 top_k_logits = logits[top_k_indices]
 
 probs = np.exp(top_k_logits - np.max(top_k_logits))
 probs /= np.sum(probs)
 
 sample_idx = np.random.choice(len(probs), p=probs)
 return top_k_indices[sample_idx]

np.random.seed(42)
logits = np.random.randn(1000)
sample = top_k_sampling(logits, k=40)
assert 0 <= sample < 1000, "Valid sample"
print("✓ Top-k sampling working")

### Lab 3: Beam Search

import numpy as np

def beam_search(start_token, vocab_size, beam_width=5, max_length=10):
 """Simple beam search for token generation"""
 sequences = [[start_token]]
 scores = [0.0]
 
 for _ in range(max_length - 1):
 new_sequences = []
 new_scores = []
 
 for seq, score in zip(sequences, scores):
 logits = np.random.randn(vocab_size)
 top_indices = np.argsort(logits)[-beam_width:]
 
 for idx in top_indices:
 new_seq = seq + [idx]
 new_score = score + logits[idx]
 new_sequences.append(new_seq)
 new_scores.append(new_score)
 
 # Keep top-k sequences
 top_k = np.argsort(new_scores)[-beam_width:]
 sequences = [new_sequences[i] for i in top_k]
 scores = [new_scores[i] for i in top_k]
 
 return sequences[0]

np.random.seed(42)
sequence = beam_search(101, vocab_size=1000, beam_width=5, max_length=5)
assert len(sequence) == 5, "Correct sequence length"
print("✓ Beam search working")

### Lab 4: Perplexity Calculation

import numpy as np

def calculate_perplexity(losses):
 """Compute perplexity from log losses"""
 avg_loss = np.mean(losses)
 perplexity = np.exp(avg_loss)
 return perplexity

np.random.seed(42)
losses = np.random.uniform(0, 3, 1000)
ppl = calculate_perplexity(losses)
assert ppl > 1, "Valid perplexity"
print(f"✓ Perplexity calculation working: {ppl:.2f}")

---