Speech Recognition Acoustic Modeling

# Speech Recognition & Acoustic Modeling

## Introduction & Motivation

Speech Recognition: convert audio to text. Acoustic models, language models, end-to-end systems. Applications: voice assistants, transcription, accessibility.

Motivation: Enable voice interfaces; improve accessibility.

Applications: Voice assistants, transcription services, real-time translation.

---

## Core Concepts & Theory

### Acoustic Features

MFCC, spectrogram, mel-scale features.

### Acoustic Model

Map audio features to phonemes.

### Language Model

Predict text sequences.

### Connectionist Temporal Classification (CTC)

Alignment-free sequence modeling.

---

## Mathematical Formulation

MFCC Computation:
$$ ext{MFCC} = ext{DCT}(\log( ext{Mel-filter bank outputs}))$$

CTC Loss:
$$L = -\log P(Y|X) = -\log \sum_{A} \prod_t P(a_t|X)$$

Beam Search Decoding:
$$P( ext{sequence}) = \prod_t P( ext{word}_t | ext{context})$$

---

## Advanced Theory & Extensions

### RNN-T (Transducer)

Online streaming recognition.

### Conformer

Convolution + Transformer hybrid.

### Wav2Vec 2.0

Self-supervised speech representation.

---

## Computational Considerations

MFCC extraction: O(frames·filters).

Acoustic model: O(time·feature_dim²).

Language model: O(vocab²).

---

## Practical Implementation Strategies

### Feature Normalization

Standardize acoustic features.

### Data Augmentation

SpecAugment for robustness.

### Multi-Task Learning

Joint phoneme and text prediction.

---

## Benchmark Datasets & Evaluation

LibriSpeech: 1000 hours English speech.

Common Voice: Multilingual open dataset.

TIMIT: Phoneme recognition benchmark.

---

## Key Challenges & Limitations

### Noise Robustness

Background noise degradation.

### Accents & Dialects

Variability in speech.

### Long-Range Dependencies

Capturing sentence context.

---

## Hyperparameter Tuning

MFCC coefficients: 13-40.

Frame size: 20-50 ms.

Learning rate: 1e-4 to 1e-3.

---

## Real-World Applications & Case Studies

Voice Assistants: Alexa, Google Assistant.

Transcription: Automated meeting notes.

Call Centers: Automated customer service.

---

## Integration with Other Methods

Speech recognition + language model for accuracy; + noise suppression for robustness.

---

## Summary & Key Takeaways

Speech Recognition via CTC and Transformer-based models enables robust audio-to-text conversion.

Principles:
1. Acoustic features: Audio representation.
2. Acoustic model: Sound-to-phoneme mapping.
3. Language model: Sequence probability.
4. CTC: Alignment-free learning.
5. Decoding: Beam search inference.

---

---

## Appendix: Practical Labs

### Lab 1: MFCC Extraction (Simplified)

import numpy as np

def extract_mfcc(audio_signal, sample_rate=16000, n_mfcc=13):
 """Extract MFCC features"""
 # Simplified MFCC (assuming pre-filtered mel spectrogram)
 spectrogram = np.random.rand(100, 128)
 
 # Apply DCT
 from scipy.fftpack import dct
 mfcc = dct(spectrogram, axis=1, type=2)[:, :n_mfcc]
 
 return mfcc

# Test
np.random.seed(42)
audio = np.random.randn(16000)

mfcc = extract_mfcc(audio)

assert mfcc.shape[1] == 13, "Correct MFCC dimension"
print("✓ MFCC extraction working")

if __name__ == "__main__":
 print("Lab 1: MFCCExtraction - PASSED")

### Lab 2: CTC Loss

import numpy as np

def ctc_loss_simplified(predictions, targets, input_length, target_length):
 """Simplified CTC loss"""
 # Predictions: (batch, time, vocab)
 batch_size = predictions.shape[0]
 
 # Take log for numerical stability
 log_probs = np.log(predictions + 1e-7)
 
 # Simplified: sum of log probabilities
 loss = 0
 for b in range(batch_size):
 # Get target sequence
 target_seq = targets[b, :target_length[b]]
 
 # Sum log probability of correct labels
 for t in range(input_length[b]):
 if t < len(target_seq):
 loss -= log_probs[b, t, target_seq[t]]
 
 return loss / batch_size

# Test
np.random.seed(42)
preds = np.random.rand(2, 100, 50)
preds = preds / preds.sum(axis=2, keepdims=True)
targets = np.array([[1, 2, 3, 0, 0], [4, 5, 0, 0, 0]])
in_lens = np.array([100, 100])
tgt_lens = np.array([3, 2])

loss = ctc_loss_simplified(preds, targets, in_lens, tgt_lens)

assert np.isfinite(loss), "Loss finite"
print("✓ CTC loss working")

if __name__ == "__main__":
 print("Lab 2: CTCLoss - PASSED")

### Lab 3: Beam Search Decoding

import numpy as np

def beam_search_decode(probabilities, beam_width=5, vocab_size=1000):
 """Simple beam search decoding"""
 time_steps = probabilities.shape[0]
 
 # Initialize beam
 beams = [([], 0.0)]
 
 for t in range(time_steps):
 new_beams = []
 
 for prefix, score in beams:
 for word_idx in range(vocab_size):
 new_score = score + np.log(probabilities[t, word_idx] + 1e-10)
 new_beams.append((prefix + [word_idx], new_score))
 
 # Keep top-k
 new_beams.sort(key=lambda x: x[1], reverse=True)
 beams = new_beams[:beam_width]
 
 return beams[0][0]

# Test
np.random.seed(42)
probs = np.random.rand(50, 100)
probs = probs / probs.sum(axis=1, keepdims=True)

decode = beam_search_decode(probs, beam_width=3, vocab_size=100)

assert isinstance(decode, list), "Decoded sequence"
print("✓ Beam search decoding working")

if __name__ == "__main__":
 print("Lab 3: BeamSearchDecoding - PASSED")

### Lab 4: Audio Preprocessing

import numpy as np

def preprocess_audio(audio, sample_rate=16000, target_db=-20):
 """Preprocess audio signal"""
 # Normalization
 audio_normalized = audio / (np.max(np.abs(audio)) + 1e-8)
 
 # Compute loudness (simplified)
 rms = np.sqrt(np.mean(audio_normalized ** 2))
 db = 20 * np.log10(rms + 1e-10)
 
 # Normalize to target level
 gain = target_db - db
 audio_normalized = audio_normalized * (10 ** (gain / 20))
 
 return audio_normalized

# Test
np.random.seed(42)
audio = np.random.randn(16000)

processed = preprocess_audio(audio)

assert processed.shape == audio.shape, "Shape preserved"
print("✓ Audio preprocessing working")

if __name__ == "__main__":
 print("Lab 4: AudioPreprocessing - PASSED")