Image Captioning Visual Description

# Image Captioning & Visual Description

## Introduction & Motivation

Image captioning: generate natural language descriptions. Encoder-decoder models, visual grounding. Applications: accessibility, image understanding.

Motivation: Describe images in natural language.

Applications: Accessibility tools, content understanding.

---

## Core Concepts & Theory

### Visual Encoder

Image feature extraction.

### Language Decoder

Caption generation.

### Attention Mechanisms

Attend to relevant image regions.

### Training Objectives

Cross-entropy loss for generation.

---

## Mathematical Formulation

Encoder:
$$v = ext{CNN}(I)$$

Decoder with Attention:
$$p(w_t | w_{<t}, I) = ext{softmax}(W[ ext{attn}(h_t, v)])$$

Attention:
$$\alpha_i = \frac{\exp(h_t \cdot v_i)}{\sum_j \exp(h_t \cdot v_j)}$$

---

## Advanced Theory & Extensions

### Bottom-Up Attention

Region-based visual features.

### Transformer-Based Captioning

Self-attention for generation.

### Reinforcement Learning

Optimize for metrics.

---

## Computational Considerations

CNN encoding: O(H·W·C).

RNN decoding: O(T·H²).

Attention: O(T·R·D).

---

## Practical Implementation Strategies

### Beam Search

Generate multiple captions.

### Scheduled Sampling

Training stability.

### Metric Optimization

CIDEr, BLEU, METEOR.

---

## Benchmark Datasets & Evaluation

COCO Captions: Large-scale captioning.

Flickr30K: Image-text pairs.

Flickr8K: Smaller benchmark.

---

## Key Challenges & Limitations

### Caption Diversity

Avoid repetitive descriptions.

### Long Captions

Extended sequence generation.

### Evaluation Metrics

Imperfect correlation with human judgment.

---

## Hyperparameter Tuning

Attention heads: 4-12.

Decoder hidden: 256-512.

Beam size: 3-5.

---

## Real-World Applications & Case Studies

Accessibility: Image description for blind users.

Content Moderation: Automated caption verification.

Image Search: Semantic indexing.

---

## Integration with Other Methods

Captioning + VQA for understanding; + grounding for precision.

---

## Summary & Key Takeaways

Image captioning generates natural language descriptions of images.

Principles:
1. Visual encoding: CNN feature extraction.
2. Language decoding: RNN caption generation.
3. Attention: Region-specific focus.
4. Beam search: Multiple hypothesis generation.
5. Evaluation: Similarity metrics.

---

## Appendix: Practical Labs

### Lab 1: Attention Weights

import numpy as np

def compute_attention_weights(decoder_state, visual_features):
 """Compute spatial attention weights"""
 scores = visual_features @ decoder_state
 weights = np.exp(scores) / np.sum(np.exp(scores))
 return weights

np.random.seed(42)
decoder = np.random.randn(512)
features = np.random.randn(49, 512)
weights = compute_attention_weights(decoder, features)
assert np.isclose(weights.sum(), 1.0), "Normalized weights"
print("✓ Attention weights working")

### Lab 2: Attended Features

import numpy as np

def apply_attention(visual_features, attention_weights):
 """Weight visual features by attention"""
 attended = attention_weights @ visual_features
 return attended

np.random.seed(42)
features = np.random.randn(49, 2048)
weights = np.random.rand(49)
weights /= weights.sum()
attended = apply_attention(features, weights)
assert attended.shape == (2048,), "Correct attended shape"
print("✓ Attention application working")

### Lab 3: Decoder One-Step

import numpy as np

def decoder_step(prev_word, prev_hidden, attended_features):
 """One step of caption decoding"""
 combined = np.concatenate([prev_word, attended_features])
 new_hidden = np.tanh(combined @ np.random.randn(len(combined), 512) * 0.01)
 logits = new_hidden @ np.random.randn(512, 10000) * 0.01
 return logits, new_hidden

np.random.seed(42)
word = np.random.randn(256)
hidden = np.random.randn(512)
attended = np.random.randn(2048)
logits, new_h = decoder_step(word, hidden, attended)
assert logits.shape == (10000,), "Correct logits shape"
print("✓ Decoder step working")

### Lab 4: BLEU Score

import numpy as np

def compute_bleu_score(hypothesis, reference, n=4):
 """Simplified BLEU score"""
 scores = []
 for gram_size in range(1, n+1):
 h_grams = set()
 r_grams = set()
 for i in range(len(hypothesis) - gram_size + 1):
 h_grams.add(tuple(hypothesis[i:i+gram_size]))
 r_grams.add(tuple(reference[i:i+gram_size]))
 overlap = len(h_grams & r_grams)
 total = len(h_grams)
 score = overlap / total if total > 0 else 0
 scores.append(score)
 return np.mean(scores)

hyp = [1, 2, 3, 4, 5]
ref = [1, 2, 3, 4, 5]
bleu = compute_bleu_score(hyp, ref)
assert 0 <= bleu <= 1, "Valid BLEU"
print(f"✓ BLEU score: {bleu:.3f}")

---