Paraphrase Generation Text Generation

# Paraphrase Generation & Text Generation

## Introduction & Motivation

Paraphrase Generation: generate semantically similar text. Diverse rewording; data augmentation. Applications: text augmentation, document simplification, machine translation.

Motivation: Create text variations; preserve meaning; improve robustness.

Applications: Data augmentation, document simplification, quality improvement.

---

## Core Concepts & Theory

### Semantic Similarity

Preserve meaning across paraphrases.

### Lexical Diversity

Vary vocabulary and structure.

### Back-Translation

Translate and translate back.

### Controlled Generation

Generate with specific attributes.

---

## Mathematical Formulation

Seq2Seq Loss:
$$L = -\sum_t \log P(y_t | y_{<t}, x)$$

Diversity Penalty:
$$L_ ext{div} = -\lambda \log(1 - \cos( ext{emb}(y_i), ext{emb}(y_j))) ext{ for } i eq j$$

Similarity Metric:
$$ ext{BLEU}(x, y) = \exp(\sum_{n=1}^{N} w_n \log p_n)$$

---

## Advanced Theory & Extensions

### Variational Autoencoders (VAE)

Latent space paraphrase generation.

### Reinforcement Learning

Reward diverse paraphrases.

### Plug-and-Play Language Models

Control generation attributes.

---

## Computational Considerations

Seq2seq: O(n·m·d²).

Diversity computation: O(N²).

Beam search: O(beam_size·seq_len).

---

## Practical Implementation Strategies

### Beam Search Diversity

Avoid similar hypotheses in beam.

### Temperature Sampling

Control randomness in generation.

### Diverse Beam Search

Maximize diversity across beams.

---

## Benchmark Datasets & Evaluation

PAWS: 108K paraphrase pairs.

STS Benchmark: Semantic textual similarity.

ParaNMT: Paraphrase generation dataset.

---

## Key Challenges & Limitations

### Semantic Preservation

Maintaining meaning accuracy.

### Diversity-Fidelity Trade-off

Balancing variation and meaning.

### Rare Phenomenon Generation

Uncommon paraphrase patterns.

---

## Hyperparameter Tuning

Beam size: 4-10.

Length penalty: 0.6-1.2.

Diversity strength: 0.1-0.5.

---

## Real-World Applications & Case Studies

Question Paraphrasing: Diversity in QA datasets.

Document Simplification: Readability improvement.

Augmentation: Increase training data variety.

---

## Integration with Other Methods

Paraphrase generation + data augmentation for robustness; + summarization for document processing.

---

## Summary & Key Takeaways

Paraphrase Generation via seq2seq and controlled generation enables diverse text creation.

Principles:
1. Semantic preservation: Meaning consistency.
2. Seq2seq architecture: Encoder-decoder.
3. Diverse sampling: Varied hypotheses.
4. Back-translation: Indirect augmentation.
5. Reinforcement learning: Reward optimization.

---

---

## Appendix: Practical Labs

### Lab 1: Back-Translation

import numpy as np

def back_translate(text, forward_model, backward_model):
 """Generate paraphrase via back-translation"""
 # Simulate forward translation
 intermediate = forward_model(text)
 
 # Simulate backward translation
 paraphrase = backward_model(intermediate)
 
 return paraphrase

# Test
def mock_forward(text):
 return "mocked translation"

def mock_backward(text):
 return "translated back paraphrase"

original = "Hello world"
paraphrase = back_translate(original, mock_forward, mock_backward)

assert isinstance(paraphrase, str), "Paraphrase string"
print("✓ Back-translation working")

if __name__ == "__main__":
 print("Lab 1: BackTranslation - PASSED")

### Lab 2: Diversity Penalty

import numpy as np

def compute_diversity_penalty(candidates, lambda_div=0.1):
 """Compute penalty for similar candidates"""
 n = len(candidates)
 diversity_loss = 0
 
 for i in range(n):
 for j in range(i + 1, n):
 # Simplified: compute embedding similarity
 sim = np.random.rand()
 
 # Penalty for similar outputs
 if sim > 0.8:
 diversity_loss += lambda_div * sim
 
 return diversity_loss

# Test
candidates = ["Hello", "Hi", "Hey there"]

penalty = compute_diversity_penalty(candidates)

assert penalty >= 0, "Penalty non-negative"
print("✓ Diversity penalty working")

if __name__ == "__main__":
 print("Lab 2: DiversityPenalty - PASSED")

### Lab 3: Temperature Sampling

import numpy as np

def sample_with_temperature(logits, temperature=1.0, k=10):
 """Sample from logits with temperature"""
 # Scale logits by temperature
 scaled_logits = logits / temperature
 
 # Compute probabilities
 probs = np.exp(scaled_logits - np.max(scaled_logits))
 probs = probs / probs.sum()
 
 # Top-k sampling
 top_k_indices = np.argsort(probs)[-k:]
 top_k_probs = probs[top_k_indices]
 top_k_probs = top_k_probs / top_k_probs.sum()
 
 # Sample
 sampled = np.random.choice(top_k_indices, p=top_k_probs)
 
 return sampled

# Test
logits = np.random.randn(1000)

sample_low_temp = sample_with_temperature(logits, temperature=0.5)
sample_high_temp = sample_with_temperature(logits, temperature=2.0)

assert 0 <= sample_low_temp < 1000, "Valid sample"
assert 0 <= sample_high_temp < 1000, "Valid sample"
print("✓ Temperature sampling working")

if __name__ == "__main__":
 print("Lab 3: TemperatureSampling - PASSED")

### Lab 4: Beam Search Diversity

import numpy as np

def diverse_beam_search(scores, diversity_strength=0.5, beam_size=5):
 """Diverse beam search"""
 beams = []
 
 # First beam: greedy
 top_idx = np.argmax(scores)
 beams.append(top_idx)
 
 # Remaining beams with diversity penalty
 for b in range(1, beam_size):
 penalized_scores = scores.copy()
 
 # Penalize similar to existing beams
 for existing_idx in beams:
 similarity = np.abs(existing_idx - np.arange(len(scores))) / (len(scores) + 1e-8)
 penalized_scores -= diversity_strength * similarity
 
 top_idx = np.argmax(penalized_scores)
 beams.append(top_idx)
 
 return beams

# Test
scores = np.random.rand(100)

beams = diverse_beam_search(scores, beam_size=5)

assert len(beams) == 5, "Correct beam count"
assert len(set(beams)) == 5, "Diverse beams"
print("✓ Diverse beam search working")

if __name__ == "__main__":
 print("Lab 4: DiverseBeamSearch - PASSED")