T5 - Text-to-Text Transfer Transformer

# T5 - Text-to-Text Transfer Transformer

## Introduction & Motivation

T5: unified text-to-text framework. Encoder-decoder architecture, diverse NLP tasks. Applications: translation, summarization, QA.

Motivation: Single architecture for multiple NLP tasks.

Applications: Machine translation, abstractive summarization.

---

## Core Concepts & Theory

### Text-to-Text Framework

Convert all tasks to text generation.

### Encoder-Decoder Architecture

Separate encoding and decoding.

### Task-Specific Prefixes

Distinguish task types.

### Multi-Task Training

Joint optimization across tasks.

---

## Mathematical Formulation

Sequence-to-Sequence Loss:
$$\mathcal{L} = -\log p(y_1...y_n | ext{enc}(x))$$

Cross-Attention:
$$ ext{Attention}(Q_{ ext{dec}}, K_{ ext{enc}}, V_{ ext{enc}})$$

Task Prefix:
$$ ext{input} = " ext{task}: " + ext{original\_input}$$

---

## Advanced Theory & Extensions

### T5-Base to T5-11B

Scaling across sizes.

### Multitask Transfer

Pre-training on diverse tasks.

### Fine-tuning Strategies

Task-specific adaptation.

---

## Computational Considerations

Encoder: O(T·D·H).

Decoder: O(T²·D) (autoregressive).

Cross-attention: O(T·D·H).

---

## Practical Implementation Strategies

### Prefix Tokens

Task identification.

### Beam Search Decoding

Multi-hypothesis generation.

### Length Constraints

Output length control.

---

## Benchmark Datasets & Evaluation

GLUE: General understanding.

SQuAD: Question answering.

WMT: Machine translation.

---

## Key Challenges & Limitations

### Decoder Length

Sequential generation speed.

### Task Diversity

Balancing multiple objectives.

### Fine-tuning Convergence

Task-specific optimization difficulty.

---

## Hyperparameter Tuning

Learning rate: 1e-4 to 5e-4.

Batch size: 64-128.

Beam size: 4-10.

---

## Real-World Applications & Case Studies

Machine Translation: Multilingual translation.

Summarization: Document condensation.

Question Answering: Extractive and generative QA.

---

## Integration with Other Methods

T5 + data augmentation; + domain adaptation.

---

## Summary & Key Takeaways

T5 unifies diverse NLP tasks in encoder-decoder text-to-text framework.

Principles:
1. Text-to-text: Unified formulation.
2. Encoder-decoder: Separation of concerns.
3. Task prefixes: Explicit task signaling.
4. Sequence-to-sequence: Autoregressive generation.
5. Multi-task: Joint training across domains.

---

## Appendix: Practical Labs

### Lab 1: Task Prefix Formatting

import numpy as np

def format_with_task_prefix(text, task="translate English to German"):
 """Format input with T5-style task prefix"""
 prefixed = f"{task}: {text}"
 return prefixed

text = "The quick brown fox"
prefixed = format_with_task_prefix(text, task="summarize")
assert prefixed.startswith("summarize:"), "Prefix added"
print(f"✓ Task prefix formatting: {prefixed[:30]}...")

### Lab 2: Encoder-Decoder Forward

import numpy as np

def encoder_decoder_forward(input_ids, decoder_input_ids, encoder_weights, decoder_weights):
 """Simplified encoder-decoder forward pass"""
 # Encode
 encoder_output = input_ids @ encoder_weights
 encoder_output = np.maximum(encoder_output, 0)
 
 # Decode with attention to encoder
 decoder_output = decoder_input_ids @ decoder_weights
 attention_output = encoder_output.mean(axis=0, keepdims=True) * decoder_output
 
 return attention_output

np.random.seed(42)
input_ids = np.random.randn(10, 256)
decoder_ids = np.random.randn(1, 256)
enc_w = np.random.randn(256, 512) * 0.01
dec_w = np.random.randn(256, 512) * 0.01
output = encoder_decoder_forward(input_ids, decoder_ids, enc_w, dec_w)
assert output.shape == (1, 512), "Correct output shape"
print("✓ Encoder-decoder forward working")

### Lab 3: Beam Search with Constraints

import numpy as np

def constrained_beam_search(logits, min_length=5, max_length=20, beam_width=4):
 """Beam search with length constraints"""
 sequences = [[]]
 scores = [0.0]
 
 for step in range(max_length):
 new_sequences = []
 new_scores = []
 
 for seq, score in zip(sequences, scores):
 if step < min_length:
 # Avoid EOS token
 eos_logit = logits[2]
 logits = logits.copy()
 logits[2] = -np.inf
 
 top_indices = np.argsort(logits)[-beam_width:]
 for idx in top_indices:
 new_seq = seq + [idx]
 new_score = score + logits[idx]
 new_sequences.append(new_seq)
 new_scores.append(new_score)
 
 top_k = np.argsort(new_scores)[-beam_width:]
 sequences = [new_sequences[i] for i in top_k]
 scores = [new_scores[i] for i in top_k]
 
 return sequences[0]

np.random.seed(42)
logits = np.random.randn(1000)
seq = constrained_beam_search(logits, min_length=3, max_length=10)
assert len(seq) <= 10, "Length constraint respected"
print("✓ Constrained beam search working")

### Lab 4: Cross-Attention Computation

import numpy as np

def compute_cross_attention(decoder_states, encoder_states, num_heads=8):
 """Compute cross-attention between decoder and encoder"""
 batch_size, seq_len, hidden_dim = decoder_states.shape
 head_dim = hidden_dim // num_heads
 
 # Compute attention scores
 scores = decoder_states @ encoder_states.T / np.sqrt(head_dim)
 attention_weights = np.exp(scores - np.max(scores, axis=-1, keepdims=True))
 attention_weights /= np.sum(attention_weights, axis=-1, keepdims=True)
 
 # Apply attention
 output = attention_weights @ encoder_states
 return output, attention_weights

np.random.seed(42)
decoder = np.random.randn(4, 10, 512)
encoder = np.random.randn(4, 15, 512)
output, weights = compute_cross_attention(decoder, encoder)
assert output.shape == decoder.shape, "Correct output shape"
print("✓ Cross-attention computation working")

---