Unilm - Unified Language Model

# UniLM - Unified Language Model

## Introduction & Motivation

UniLM: unified pre-training for diverse NLP tasks via shared architecture. Supports understanding, generation, machine translation. Applications: versatile NLP systems.

Motivation: Single model for multiple task types (NLU, NLG, MT).

Applications: Multitask learning, universal NLP systems, transfer learning.

---

## Core Concepts & Theory

### Unified Masking

Flexible masking for different task types.

### Shared Parameters

Single model for diverse tasks.

### Attention Masking Strategy

Control information flow for each task.

### Task-Specific Fine-tuning

Adapt shared model per task.

---

## Mathematical Formulation

Bidirectional Attention (NLU):
$$ ext{mask}_{ij} = 0 ext{ for all } i, j$$

Causal Attention (NLG):
$$ ext{mask}_{ij} = \begin{cases} 0 & ext{if } j \leq i \\ -\infty & ext{if } j > i \end{cases}$$

Prefix Attention (MT):
$$ ext{mask}_{ij} = 0 ext{ for } j \in ext{source}$$

---

## Advanced Theory & Extensions

### Multi-Stage Training

Sequential task training.

### Intermediate Task Tuning

Bridge NLU and NLG.

### Prompt-Based Control

Task specification via prompts.

---

## Computational Considerations

Parameter sharing: O(D²) shared weights.

Task masking: O(T²) attention.

Batch diversity: Mixed task batches.

---

## Practical Implementation Strategies

### Attention Mask Design

Task-specific masking patterns.

### Fine-tuning Strategy

Sequential or joint training.

### Prompt Engineering

Task specification.

---

## Benchmark Datasets & Evaluation

GLUE: Text understanding.

CNN/DailyMail: Summarization.

WMT14: Machine translation.

---

## Key Challenges & Limitations

### Task Interference

Negative transfer between tasks.

### Optimization Complexity

Multi-task optimization difficult.

### Memory Requirements

Larger model for versatility.

---

## Hyperparameter Tuning

Mask ratio: 0.1-0.3.

Task mixing ratio: Equal or weighted.

Learning rate: 1e-4 to 1e-3.

---

## Real-World Applications & Case Studies

Question Answering: Fine-tune for QA.

Text Summarization: Abstractive generation.

Machine Translation: Bilingual NMT.

---

## Integration with Other Methods

UniLM + task adapters; + prompt learning for zero-shot.

---

## Summary & Key Takeaways

UniLM enables unified pre-training across diverse NLP tasks.

Principles:
1. Unified architecture: Single shared model.
2. Task masking: Flexible attention patterns.
3. Shared parameters: Efficient scaling.
4. Multi-task learning: Diverse pre-training.
5. Versatility: NLU, NLG, MT in one model.

---

## Appendix: Practical Labs

### Lab 1: Attention Mask Design

import numpy as np

def create_attention_masks(seq_len, task_type='bidirectional'):
 """Create task-specific attention masks"""
 mask = np.zeros((seq_len, seq_len))
 
 if task_type == 'bidirectional':
 pass # All 0s, fully visible
 elif task_type == 'causal':
 for i in range(seq_len):
 mask[i, i+1:] = -np.inf
 elif task_type == 'encoder':
 # Prefix mask for encoder-decoder
 source_len = seq_len // 2
 mask[:source_len, source_len:] = -np.inf
 
 return mask

mask_bi = create_attention_masks(10, 'bidirectional')
mask_causal = create_attention_masks(10, 'causal')
assert np.sum(mask_causal == -np.inf) > 0
print("✓ Attention mask design working")

### Lab 2: Task Masking Strategy

import numpy as np

def apply_task_masking(tokens, task_id, mask_prob=0.15):
 """Apply task-specific masking"""
 masked = tokens.copy()
 n_mask = int(len(tokens) * mask_prob)
 
 if task_id == 0: # NLU - random masking
 indices = np.random.choice(len(tokens), n_mask, replace=False)
 elif task_id == 1: # NLG - causal masking
 indices = np.arange(n_mask)
 elif task_id == 2: # MT - source-aware masking
 indices = np.random.choice(len(tokens)//2, n_mask, replace=False)
 
 masked[indices] = '[MASK]'
 return masked

np.random.seed(42)
tokens = ['hello', 'world', 'test', 'model']
masked = apply_task_masking(tokens, 0)
assert '[MASK]' in masked
print("✓ Task masking working")

### Lab 3: Unified Loss

import numpy as np

def unified_loss(nlu_loss, nlu_weight, nlg_loss, nlg_weight):
 """Combine losses from different tasks"""
 total_loss = (nlu_loss * nlu_weight + nlg_loss * nlg_weight) / (nlu_weight + nlg_weight)
 return total_loss

nlu_l = 2.5
nlg_l = 1.8
total = unified_loss(nlu_l, 1.0, nlg_l, 1.0)
assert total > 0
print(f"✓ Unified loss: {total:.3f}")

### Lab 4: Task Switching

def rotate_task_batch(task_ids, batch_size):
 """Rotate through tasks in batch"""
 num_tasks = len(set(task_ids))
 samples_per_task = batch_size // num_tasks
 
 batch_task_ids = []
 for task_id in range(num_tasks):
 batch_task_ids.extend([task_id] * samples_per_task)
 
 return batch_task_ids[:batch_size]

batch_ids = rotate_task_batch([0, 1, 2], 9)
assert len(batch_ids) == 9
print(f"✓ Task batch rotation: {batch_ids}")

---