Sequence Labeling Tagging

# Sequence Labeling & Tagging

## Introduction & Motivation

Sequence labeling: assign labels to sequence elements. POS tagging, NER. Applications: text understanding, information extraction.

Motivation: Tag individual tokens in sequences.

Applications: Part-of-speech tagging, chunking, slot filling.

---

## Core Concepts & Theory

### POS Tagging

Part-of-speech categorization.

### BIO Tagging

Begin-Inside-Outside scheme.

### CRF Modeling

Structured prediction.

### Token Classification

Per-token predictions.

---

## Mathematical Formulation

Sequence Score:
$$ ext{score}(y|x) = \sum_t W_{y_{t-1}, y_t} + \sum_t ext{emit}(x_t, y_t)$$

CRF Loss:
$$\mathcal{L} = -\log \frac{\exp( ext{score}(y^*|x))}{\sum_y \exp( ext{score}(y|x))}$$

Viterbi Decoding:
$$y^* = \arg\max_y ext{score}(y|x)$$

---

## Advanced Theory & Extensions

### Structured Output

Joint prediction of sequences.

### Transfer Learning

Pre-trained token classifiers.

### Multi-Task Tagging

Simultaneous label prediction.

---

## Computational Considerations

Emission: O(T·D).

Transition: O(T·K²).

Viterbi: O(T·K²).

---

## Practical Implementation Strategies

### BIO Encoding

Standard tagging scheme.

### Softmax over Tags

Per-position classification.

### Greedy Decoding

Fast inference.

---

## Benchmark Datasets & Evaluation

Penn Treebank: POS tagging.

CoNLL: NER and chunking.

ACE: Entity and relation tagging.

---

## Key Challenges & Limitations

### Long-Range Dependencies

Context limitations.

### Tag Imbalance

Rare label handling.

### Cross-Domain Transfer

Domain-specific patterns.

---

## Hyperparameter Tuning

Sequence length: 32-512.

Dropout: 0.3-0.5.

Learning rate: 1e-4 to 1e-3.

---

## Real-World Applications & Case Studies

NER: Entity extraction.

POS Tagging: Syntactic analysis.

Slot Filling: Information extraction.

---

## Integration with Other Methods

Tagging + embeddings; + CRF for structure.

---

## Summary & Key Takeaways

Sequence labeling assigns per-token labels to sequences.

Principles:
1. Token classification: Per-position prediction.
2. BIO scheme: Standard encoding.
3. CRF: Structured modeling.
4. Viterbi: Optimal decoding.
5. Transfer learning: Pre-trained features.

---

## Appendix: Practical Labs

### Lab 1: BIO Encoding

def bio_encode(entities, seq_len):
 """Encode entities in BIO format"""
 tags = ['O'] * seq_len
 for start, end, label in entities:
 tags[start] = f'B-{label}'
 for i in range(start + 1, end):
 tags[i] = f'I-{label}'
 return tags

entities = [(0, 2, 'PER'), (5, 7, 'LOC')]
tags = bio_encode(entities, 10)
assert tags[0] == 'B-PER', "BIO encoding correct"
print("✓ BIO encoding working")

### Lab 2: CRF Transition Matrix

import numpy as np

def compute_transition_scores(prev_tag_idx, curr_tag_idx, transition_matrix):
 """Score transition between tags"""
 score = transition_matrix[prev_tag_idx, curr_tag_idx]
 return score

np.random.seed(42)
num_tags = 10
trans_matrix = np.random.randn(num_tags, num_tags)
score = compute_transition_scores(2, 3, trans_matrix)
assert isinstance(score, (int, float, np.number)), "Valid score"
print("✓ Transition scoring working")

### Lab 3: Greedy Decoding

import numpy as np

def greedy_decode(emission_scores):
 """Greedy tag prediction"""
 predictions = []
 for t in range(len(emission_scores)):
 tag = np.argmax(emission_scores[t])
 predictions.append(tag)
 return predictions

np.random.seed(42)
emissions = np.random.randn(10, 5)
preds = greedy_decode(emissions)
assert len(preds) == 10, "Correct predictions"
print("✓ Greedy decoding working")

### Lab 4: Tag Accuracy

import numpy as np

def compute_tag_accuracy(predictions, gold_labels):
 """Compute per-tag accuracy"""
 correct = np.sum(predictions == gold_labels)
 total = len(gold_labels)
 accuracy = correct / total
 return accuracy

np.random.seed(42)
preds = np.random.randint(0, 5, 100)
gold = np.random.randint(0, 5, 100)
acc = compute_tag_accuracy(preds, gold)
assert 0 <= acc <= 1, "Valid accuracy"
print(f"✓ Tag accuracy: {acc:.2%}")

---