Text Summarization
# Text Summarization
## Introduction & Motivation
Text Summarization: generate concise versions of documents. Extractive and abstractive approaches. Applications: news summarization, document understanding, information retrieval.
Motivation: Reduce information overload; extract key information.
Applications: News digests, research paper abstracts.
---
## Core Concepts & Theory
### Extractive Summarization
Select important sentences from original text.
### Abstractive Summarization
Generate new summary via generation.
### Document Representation
Encode document for summary extraction.
### Attention Mechanisms
Weight important words and sentences.
---
## Mathematical Formulation
TF-IDF Scoring:
$$ ext{score}(s_i) = \sum_{w \in s_i} ext{TF}(w, d) \cdot ext{IDF}(w)$$
Seq2Seq Loss:
$$L = -\sum_t \log P(y_t | y_{<t}, x)$$
ROUGE Score:
$$ ext{ROUGE-N} = \frac{\sum ext{max(count}_n(w))}{\sum ext{count}_n(w)}$$
---
## Advanced Theory & Extensions
### Hierarchical Attention
Document → sentence → word levels.
### Pointer-Generator Networks
Copy mechanism + generation.
### Pre-trained Models (BERT, T5)
Fine-tune for summarization.
---
## Computational Considerations
Extractive: O(N·S·W).
Abstractive: O(N·summary_len).
Attention: O(N²).
---
## Practical Implementation Strategies
### Sentence Tokenization
Split documents into sentences.
### Position Encoding
Bias toward early/important sentences.
### Length Constraint
Control summary length.
---
## Benchmark Datasets & Evaluation
CNN/DailyMail: 300K news articles and summaries.
XSum: 226K BBC articles, abstractive.
SAMSum: Dialogue summarization dataset.
---
## Key Challenges & Limitations
### Factual Consistency
Errors in abstractive summaries.
### Coherence
Maintaining logical flow.
### Domain Specificity
Varying document types.
---
## Hyperparameter Tuning
Compression ratio: 0.1-0.3.
Max summary length: 100-150 tokens.
Temperature (sampling): 0.7-1.0.
---
## Real-World Applications & Case Studies
News Aggregation: Automated news digest.
Legal Document Review: Contract summarization.
Medical Records: Patient summary generation.
---
## Integration with Other Methods
Summarization + retrieval for query-focused summaries; + NER for entity-based summaries.
---
## Summary & Key Takeaways
Text Summarization via extractive and abstractive methods enables document compression.
Principles:
1. Importance scoring: Identify key content.
2. Extraction: Select representative sentences.
3. Abstraction: Generate new text.
4. Sequence-to-sequence: Encoder-decoder.
5. Evaluation: ROUGE metrics.
---
---
## Appendix: Practical Labs
### Lab 1: TF-IDF Scoring
import numpy as np
def compute_tfidf_summary(sentences, num_sentences=3):
"""Extractive summarization using TF-IDF"""
from collections import Counter
# Compute TF
all_words = []
for sent in sentences:
all_words.extend(sent.lower().split())
tf = Counter(all_words)
# Compute IDF
doc_freq = Counter()
for sent in sentences:
words_in_sent = set(sent.lower().split())
doc_freq.update(words_in_sent)
idf = {word: np.log(len(sentences) / (count + 1)) for word, count in doc_freq.items()}
# Score sentences
scores = []
for sent in sentences:
score = 0
for word in sent.lower().split():
score += tf[word] * idf.get(word, 0)
scores.append(score)
# Select top sentences
top_indices = np.argsort(scores)[-num_sentences:]
summary_sentences = [sentences[i] for i in sorted(top_indices)]
return summary_sentences
# Test
sentences = ["This is first.", "Second sentence here.", "Third important point.", "Fourth sentence."]
summary = compute_tfidf_summary(sentences, num_sentences=2)
assert len(summary) == 2, "Correct summary length"
print("✓ TF-IDF summarization working")
if __name__ == "__main__":
print("Lab 1: TFIDFSummarization - PASSED")### Lab 2: ROUGE Score
import numpy as np
def compute_rouge_score(predicted, reference, n=1):
"""Compute ROUGE-N score"""
# Extract n-grams
pred_ngrams = set()
ref_ngrams = set()
pred_words = predicted.lower().split()
ref_words = reference.lower().split()
for i in range(len(pred_words) - n + 1):
pred_ngrams.add(' '.join(pred_words[i:i+n]))
for i in range(len(ref_words) - n + 1):
ref_ngrams.add(' '.join(ref_words[i:i+n]))
# Compute overlap
overlap = len(pred_ngrams & ref_ngrams)
recall = overlap / (len(ref_ngrams) + 1e-8)
precision = overlap / (len(pred_ngrams) + 1e-8)
f1 = 2 * (precision * recall) / (precision + recall + 1e-8)
return f1
# Test
predicted = "The quick brown fox"
reference = "A quick brown fox"
score = compute_rouge_score(predicted, reference, n=1)
assert 0 <= score <= 1, "Score in range"
print("✓ ROUGE score working")
if __name__ == "__main__":
print("Lab 2: ROUGEScore - PASSED")### Lab 3: Position Bias
import numpy as np
def apply_position_bias(sentence_scores, decay=0.1):
"""Apply position bias to sentence scores"""
n_sents = len(sentence_scores)
# Earlier sentences ranked higher
position_scores = np.array([1 - i * decay for i in range(n_sents)])
position_scores = np.clip(position_scores, 0, 1)
# Combine with original scores
combined_scores = sentence_scores * (1 + position_scores)
return combined_scores
# Test
scores = np.array([0.5, 0.6, 0.7, 0.4])
biased = apply_position_bias(scores, decay=0.1)
assert len(biased) == len(scores), "Same length"
assert np.isfinite(biased).all(), "All finite"
print("✓ Position bias working")
if __name__ == "__main__":
print("Lab 3: PositionBias - PASSED")### Lab 4: Length Constraint
import numpy as np
def enforce_summary_length(sentences, max_length=100, word_count=None):
"""Enforce maximum summary length"""
if word_count is None:
word_count = {}
summary = []
total_length = 0
for sent in sentences:
sent_length = len(sent.split())
if total_length + sent_length <= max_length:
summary.append(sent)
total_length += sent_length
else:
break
return summary
# Test
sentences = ["Short.", "Another short.", "This is longer sentence here.", "Last one."]
summary = enforce_summary_length(sentences, max_length=20)
total = sum(len(s.split()) for s in summary)
assert total <= 20, "Length constraint satisfied"
print("✓ Length constraint working")
if __name__ == "__main__":
print("Lab 4: LengthConstraint - PASSED")