conditional random fields crf structured prediction

# Conditional Random Fields: CRF & Structured Prediction

## Introduction & Motivation

Conditional Random Fields: discriminative graphical model. Model P(Y|X) directly. Undirected graphs; no independence assumptions on input. Applications: sequence labeling, named entity recognition, structured prediction.

Motivation: Discriminative model avoids modeling input distribution.

Applications: Tagging, structured prediction.

---

## Core Concepts & Theory

### Potential Functions

Unnormalized factor scores.

### Clique Potentials

Local compatibility functions.

### Partition Function

Normalizing constant Z(x).

---

## Mathematical Formulation

CRF probability:
$$P(y|x) = \frac{1}{Z(x)} \exp\left(\sum_c \psi_c(y_c, x) ight)$$

Linear-chain CRF:
$$P(y|x) = \frac{1}{Z(x)} \exp\left(\sum_t (\mathbf{w} \cdot \mathbf{f}(y_t, y_{t-1}, x_t)) ight)$$

Loss (negative log-likelihood):
$$L = -\log P(y|x)$$

---

## Advanced Theory & Extensions

### Structured Perceptron

Online discriminative learning.

### Skip-Chain CRF

Long-range dependencies.

### Semi-CRF

Segmental structure.

---

## Computational Considerations

Inference: O(T·|Y|²) Viterbi.

Learning: O(T·|Y|²) per gradient computation.

Decoding: O(T·|Y|) with dynamic programming.

---

## Practical Implementation Strategies

### Feature Engineering

Hand-crafted features for domain.

### Regularization

L2 penalty on weights.

### Beam Search

Approximate inference for speedup.

---

## Benchmark Datasets & Evaluation

CoNLL 2003: NER benchmark.

POS Tagging: Sequence labeling task.

BIO Tagging: Entity boundaries.

---

## Key Challenges & Limitations

### Feature Sparsity

Manual feature engineering needed.

### Computational Cost

Normalizing constant expensive.

### Label Imbalance

Common in tagging tasks.

---

## Hyperparameter Tuning

Regularization: L2 penalty 0.01-1.0.

Learning rate: 0.001-0.1.

Beam size: 1-10; speed-accuracy tradeoff.

---

## Real-World Applications & Case Studies

Named Entity Recognition: CoNLL benchmark.

Part-of-Speech Tagging: Syntax annotation.

Slot Filling: Dialogue systems.

---

## Integration with Other Methods

CRF + Neural Network → neural CRF.

CRF + Attention → attended structure.

---

## Summary & Key Takeaways

Conditional Random Fields via structured prediction enable sequence labeling through discriminative potential functions and graph-based inference.

Principles:
1. Discriminative: P(Y|X) directly.
2. Potentials: local compatibility.
3. Partition function: normalization.
4. Viterbi: optimal path decoding.
5. Feature engineering: domain knowledge.

---

---

## Appendix: Practical Labs

### Lab 1: Linear-Chain CRF Scoring

import numpy as np

def crf_score(sequence, transitions, emissions, weights):
 """Compute CRF score for sequence"""
 score = 0.0
 
 for t in range(len(sequence) - 1):
 # Transition score
 prev_state = sequence[t]
 curr_state = sequence[t+1]
 score += transitions[prev_state, curr_state]
 
 # Emission scores
 for t in range(len(sequence)):
 state = sequence[t]
 score += emissions[t, state]
 
 return score

# Test
np.random.seed(42)
sequence = [0, 1, 0, 1]
transitions = np.random.randn(2, 2)
emissions = np.random.randn(4, 2)

score = crf_score(sequence, transitions, emissions, None)

assert np.isfinite(score), "Score finite"
print("✓ CRF scoring working")

if __name__ == "__main__":
 print("Lab 1: CRFScoring - PASSED")

### Lab 2: Partition Function (Forward Algorithm)

import numpy as np

def partition_function(transitions, emissions, num_states):
 """Compute partition function Z(x)"""
 T = emissions.shape[0]
 
 # Forward table
 alpha = np.zeros((T, num_states))
 
 # Initialize
 alpha[0] = emissions[0]
 
 # Forward pass
 for t in range(1, T):
 for j in range(num_states):
 alpha[t, j] = np.sum(alpha[t-1] * np.exp(transitions[:, j])) * np.exp(emissions[t, j])
 
 # Partition function
 Z = np.sum(alpha[-1])
 
 return Z

# Test
np.random.seed(42)
transitions = np.random.randn(2, 2)
emissions = np.random.randn(4, 2)

Z = partition_function(transitions, emissions, 2)

assert Z > 0, "Partition function positive"
assert np.isfinite(Z), "Partition function finite"
print("✓ Partition function working")

if __name__ == "__main__":
 print("Lab 2: PartitionFunction - PASSED")

### Lab 3: CRF Inference (Viterbi)

import numpy as np

def crf_viterbi(emissions, transitions, num_states):
 """Viterbi inference for CRF"""
 T = emissions.shape[0]
 
 # Viterbi table
 viterbi = np.zeros((T, num_states))
 backpointer = np.zeros((T, num_states), dtype=int)
 
 # Initialize
 viterbi[0] = emissions[0]
 
 # Forward pass
 for t in range(1, T):
 for j in range(num_states):
 # Find best previous state
 temp = viterbi[t-1] + transitions[:, j]
 backpointer[t, j] = np.argmax(temp)
 viterbi[t, j] = np.max(temp) + emissions[t, j]
 
 # Backtrack
 path = [np.argmax(viterbi[-1])]
 for t in range(T-1, 0, -1):
 path.append(backpointer[t, path[-1]])
 
 return list(reversed(path))

# Test
np.random.seed(42)
emissions = np.random.randn(4, 2)
transitions = np.random.randn(2, 2)

path = crf_viterbi(emissions, transitions, 2)

assert len(path) == 4, "Path length"
assert all(0 <= s < 2 for s in path), "Valid states"
print("✓ CRF Viterbi working")

if __name__ == "__main__":
 print("Lab 3: CRFViterbi - PASSED")

### Lab 4: Marginal Probabilities

import numpy as np

def crf_marginals(emissions, transitions, num_states):
 """Compute marginal probabilities at each position"""
 T = emissions.shape[0]
 
 # Forward pass
 alpha = np.zeros((T, num_states))
 alpha[0] = np.exp(emissions[0])
 
 for t in range(1, T):
 for j in range(num_states):
 alpha[t, j] = np.sum(alpha[t-1] * np.exp(transitions[:, j])) * np.exp(emissions[t, j])
 
 # Backward pass
 beta = np.zeros((T, num_states))
 beta[-1] = 1.0
 
 for t in range(T-2, -1, -1):
 for i in range(num_states):
 beta[t, i] = np.sum(np.exp(transitions[i]) * np.exp(emissions[t+1]) * beta[t+1])
 
 # Marginals
 Z = np.sum(alpha[-1])
 marginals = (alpha * beta) / Z
 
 return marginals

# Test
np.random.seed(42)
emissions = np.random.randn(4, 2)
transitions = np.random.randn(2, 2)

marginals = crf_marginals(emissions, transitions, 2)

assert marginals.shape == (4, 2), "Marginals shape"
assert np.allclose(marginals.sum(axis=1), 1.0), "Marginals sum to 1"
print("✓ CRF marginals working")

if __name__ == "__main__":
 print("Lab 4: CRFMarginals - PASSED")