Speech Recognition Audio Representation Learning
# Speech Recognition & Audio Representation Learning
## Introduction & Motivation
Automatic speech recognition (ASR) and audio representation learning form the technological foundation behind voice assistants, real-time transcription, dictation tools, call-center analytics, and accessibility technology for the deaf and hard of hearing. The field has undergone a dramatic transformation over the past decade, moving from hand-engineered acoustic pipelines built on Gaussian Mixture Models and Hidden Markov Models to end-to-end neural architectures trained directly on raw or lightly processed audio waveforms. This shift was driven by the availability of large labeled and unlabeled speech corpora, the maturation of sequence modeling architectures (recurrent networks, then Transformers), and the discovery that self-supervised pretraining on unlabeled audio, much like masked language modeling in text, produces representations that transfer remarkably well to downstream tasks with limited labeled data. Modern systems such as OpenAI's Whisper and Meta's wav2vec 2.0 family have pushed ASR toward near-human accuracy on many benchmarks and, critically, toward robustness across accents, background noise, and multiple languages within a single unified model. Beyond transcription, audio representation learning underlies a broader set of tasks including speaker identification and diarization, keyword spotting, emotion recognition, music information retrieval, and audio event detection, all of which benefit from the same underlying advances in learned, general-purpose audio embeddings.
## Core Concepts & Theory
Raw audio is a high-frequency one-dimensional waveform (commonly sampled at 16kHz for speech), which is too high-dimensional and locally redundant to feed directly into most models without preprocessing; the standard first step converts the waveform into a spectrogram, most often a mel-spectrogram, which represents how the signal's frequency content evolves over time, using the mel scale to approximate the non-linear frequency sensitivity of human hearing. The central modeling challenge in ASR is a sequence-to-sequence alignment problem: an audio sequence spanning many thousands of samples must be mapped to a much shorter sequence of words or subword tokens, without a known alignment between input frames and output tokens ahead of time. Three major architectural paradigms solve this alignment problem differently. Connectionist Temporal Classification (CTC) allows a model to output a label (or a special blank token) at every input frame, then collapses repeated labels and removes blanks to produce the final transcription, enabling training without explicit frame-level alignment labels. Attention-based encoder-decoder models (e.g., Listen, Attend and Spell) use a full sequence-to-sequence architecture where a decoder attends over the entire encoded audio sequence at each output step, learning the alignment implicitly through the attention mechanism. RNN-Transducer (RNN-T) architectures combine an audio encoder, a label encoder (predicting based on previously emitted tokens), and a joint network, enabling streaming, low-latency recognition since output tokens can be emitted as audio arrives rather than requiring the full utterance to be processed first, which is essential for real-time voice assistants.
## Mathematical Formulation
The CTC loss marginalizes over all possible frame-level alignments that collapse to the correct target label sequence. Given an input sequence of length T producing per-frame output distributions, and a target label sequence of length U less than or equal to T, the CTC loss sums the probability of all valid alignments (paths) that reduce, via the collapsing function B, to the target sequence:
$$ p(y \mid x) = \sum_{\pi \in B^{-1}(y)} \prod_{t=1}^{T} p(\pi_t \mid x) $$
$$ \mathcal{L}_{CTC} = -\log p(y \mid x) $$
where this sum over exponentially many possible alignments is computed efficiently via a forward-backward dynamic programming algorithm analogous to the forward-backward algorithm for Hidden Markov Models. For self-supervised pretraining approaches like wav2vec 2.0, the model learns by solving a contrastive task: given a sequence of latent audio representations with some time steps masked, the model must identify the true quantized latent representation for a masked step among a set of distractors sampled from other masked positions in the same utterance:
$$ \mathcal{L}_{contrastive} = -\log \frac{\exp( ext{sim}(c_t, q_t) / \kappa)}{\sum_{ ilde{q} \in Q_t} \exp( ext{sim}(c_t, ilde{q}) / \kappa)} $$
where c_t is the model's contextualized representation at the masked position, q_t is the true quantized target, Q_t is the set of candidate quantized representations (true target plus distractors), sim denotes cosine similarity, and kappa is a temperature hyperparameter. For attention-based encoder-decoder ASR, the decoder produces an output distribution at each step conditioned on all previously generated tokens and the full encoded audio context, trained with standard cross-entropy loss over the target token sequence, identical in form to sequence-to-sequence machine translation training objectives.
## Advanced Theory & Extensions
Whisper, trained on 680,000 hours of weakly supervised, multilingual audio-transcript pairs scraped from the internet, demonstrated that scaling weakly labeled data (rather than relying purely on carefully curated, professionally transcribed corpora) produces remarkably robust zero-shot transcription across languages, accents, and acoustic conditions, and additionally supports translation, language identification, and voice activity detection within a single unified multitask sequence-to-sequence model. wav2vec 2.0 and its successors (HuBERT, WavLM) established the self-supervised pretrain-then-fine-tune paradigm for speech: a model is pretrained on large amounts of unlabeled audio using a masked prediction objective, then fine-tuned on a comparatively small amount of labeled transcribed speech, dramatically reducing the labeled data requirements for achieving strong ASR performance in low-resource languages. HuBERT refines this approach by using offline clustering of acoustic features (e.g., MFCCs, then iteratively the model's own learned representations) to generate discrete pseudo-labels for masked prediction, sidestepping some of the instability associated with wav2vec 2.0's contrastive objective. Conformer architectures combine convolutional layers (which capture local acoustic patterns efficiently) with Transformer self-attention layers (which capture long-range dependencies), and have become a dominant encoder architecture for high-accuracy streaming and non-streaming ASR systems. Speaker diarization ("who spoke when") and multi-speaker separation extend the core recognition problem to multi-speaker audio, requiring joint modeling of speaker identity and content, an increasingly important capability for meeting transcription and call-center applications.
## Computational Considerations
Streaming ASR imposes a hard latency constraint that fundamentally shapes architecture choice: full-attention encoder-decoder models that must process an entire utterance before producing any output are unsuitable for real-time voice assistants, motivating streaming-friendly architectures like RNN-T and chunk-based or causal-attention Conformer variants that can emit partial transcriptions incrementally as audio arrives. The computational cost of self-attention over long audio sequences scales quadratically with sequence length, which is a substantial burden given that even a short spoken utterance produces many hundreds of spectrogram frames; this motivates either aggressive temporal downsampling in early encoder layers (typically reducing frame rate by a factor of four to eight before the main Transformer stack) or the use of efficient attention variants for very long-form audio such as podcasts or lecture recordings. Beam search decoding, standard for attention-based and RNN-T models, trades decoding latency and compute against transcription accuracy through the beam width hyperparameter, and production systems must carefully tune this trade-off against real-time factor requirements (the ratio of processing time to audio duration, which must remain below one for real-time streaming applications). Self-supervised pretraining itself is extremely compute-intensive, with models like wav2vec 2.0 and HuBERT requiring hundreds to thousands of GPU-days on large unlabeled audio corpora, though this cost is amortized across the many downstream tasks and languages that benefit from the resulting pretrained representations via comparatively cheap fine-tuning.
## Practical Implementation Strategies
Data augmentation is unusually impactful for ASR robustness: SpecAugment, which applies random time masking, frequency masking, and time warping directly to the mel-spectrogram during training, is a near-universal component of modern ASR training pipelines and substantially improves generalization to noisy and out-of-domain audio without requiring any additional real training data. Choosing an appropriate subword tokenization vocabulary (e.g., Byte Pair Encoding or SentencePiece applied to the transcript text) rather than raw characters or full words balances vocabulary size against sequence length and out-of-vocabulary robustness, and is standard practice across nearly all modern ASR output tokenization schemes. For low-resource languages or domains with limited labeled data, fine-tuning a large multilingual self-supervised checkpoint (wav2vec 2.0 XLSR, Whisper) on even a modest amount of labeled in-domain data typically outperforms training a smaller model from scratch by a wide margin, making transfer learning the default starting point for nearly all practical ASR development today. Language model fusion, where a separately trained text-only language model is combined with the acoustic model's output distribution during decoding (shallow fusion via simple log-probability interpolation, or deeper integration methods), improves recognition of rare words and domain-specific terminology that may be underrepresented in the paired audio-transcript training data. Evaluation should track Word Error Rate (WER) broken down by relevant subgroups (accent, noise condition, speaker demographic) rather than only an aggregate figure, since aggregate WER can mask substantial performance disparities across these subgroups that matter greatly for real-world deployment fairness and reliability.
## Benchmark Datasets & Evaluation
LibriSpeech, derived from public-domain audiobook recordings, remains the most widely used English ASR benchmark, with clean and noisy ("other") test splits providing a standard, comparable measure of model accuracy across the research community. Common Voice, Mozilla's crowdsourced, multilingual, openly licensed speech corpus, has become the primary resource for training and evaluating ASR in lower-resource languages, substantially expanding language coverage beyond what proprietary, English-centric corpora provide. VoxCeleb provides large-scale speaker recognition and diarization benchmarks derived from celebrity interview videos, testing speaker embedding quality independent of transcription accuracy. TED-LIUM and Switchboard/CallHome provide, respectively, lecture-style and conversational telephone speech benchmarks, testing generalization to more naturalistic, disfluent, and acoustically challenging speech than the relatively clean, scripted style of audiobook narration. Word Error Rate (WER), computed as the sum of substitutions, insertions, and deletions divided by the total number of reference words, remains the standard evaluation metric for transcription accuracy, though Character Error Rate (CER) is often reported alongside WER for languages without clear word boundaries (e.g., Mandarin Chinese, Japanese). The SUPERB benchmark evaluates self-supervised audio representations across a broad battery of downstream tasks (ASR, speaker identification, emotion recognition, keyword spotting) using frozen or lightly fine-tuned representations, directly measuring the general-purpose transferability of a pretrained audio encoder rather than only its ASR-specific performance.
## Key Challenges & Limitations
Accent, dialect, and sociolinguistic variation remain a persistent source of ASR performance disparity, with documented higher error rates for speakers of underrepresented dialects and accents, a fairness and equity concern that has motivated dedicated benchmark and mitigation research across the field. Robustness to background noise, overlapping speech, and far-field recording conditions (e.g., a smart speaker across a room rather than a close-talking microphone) continues to degrade performance substantially relative to clean, close-talk benchmark conditions, despite the substantial gains from data augmentation and large-scale weakly supervised training. Rare word and named entity recognition remains difficult because such words are, by definition, underrepresented in training data, and contextual biasing techniques (injecting a list of likely relevant terms, such as contact names or product names, at inference time) are an active area of both research and production engineering to address this gap. Code-switching, where speakers alternate between multiple languages within a single utterance, is poorly handled by many ASR systems trained primarily on monolingual data, though multilingual models like Whisper have shown meaningfully improved, if still imperfect, code-switching robustness. Punctuation, capitalization, and disfluency handling (filler words like "um" and "uh", false starts, self-corrections) are often treated as secondary post-processing steps rather than being jointly and robustly modeled with the core transcription task, leading to transcripts that require additional cleanup for readability in many production applications.
## Hyperparameter Tuning
The temporal downsampling factor applied in early encoder layers trades off sequence length (and therefore compute and memory cost) against the granularity of acoustic detail available to later layers, with typical choices in the range of four to eight times downsampling for Conformer-based encoders. Beam width during decoding directly trades decoding latency against transcription accuracy, with production streaming systems often constrained to very small beam widths (or greedy decoding) to meet real-time latency budgets, while offline batch transcription can afford much wider beams for marginally improved accuracy. The masking ratio and span length used in self-supervised pretraining objectives (wav2vec 2.0 typically masks around 49% of time steps in spans of ten frames) affect the difficulty of the pretraining task and the resulting representation quality, with too little masking making the task trivially easy and too much masking removing so much context that the model cannot learn meaningful structure. Language model fusion weight, controlling how strongly an external language model's predictions are blended with the acoustic model's output during decoding, requires careful tuning since excessive weighting can cause the model to favor common word sequences over what the audio actually indicates, hurting recognition of unusual but correct utterances. SpecAugment's masking parameters (number and width of time and frequency masks) need to be scaled appropriately to utterance length and the target noise robustness profile, with overly aggressive augmentation slowing convergence and potentially hurting clean-condition accuracy in exchange for noise robustness gains.
## Real-World Applications & Case Studies
Voice assistants (Siri, Google Assistant, Amazon Alexa) rely on streaming, low-latency ASR as the entry point to nearly every user interaction, with strict on-device or edge-latency requirements driving substantial research investment into compact, efficient streaming architectures distinct from the large offline models used for batch transcription tasks. Clinical documentation tools use ASR combined with domain-specific language models and named entity recognition to transcribe physician-patient conversations directly into structured electronic health record entries, reducing physician documentation burden, though requiring very high accuracy on medical terminology given the safety-critical nature of clinical records. Contact center analytics platforms transcribe and analyze large volumes of customer service calls at scale, combining ASR with downstream sentiment analysis, topic modeling, and compliance monitoring to surface actionable insights from previously unstructured audio data. Real-time captioning and translation services for accessibility (e.g., live captioning for deaf and hard-of-hearing users, or real-time speech translation for cross-lingual communication) place a premium on both low latency and robustness to natural, spontaneous conversational speech rather than the cleaner, more scripted speech that earlier-generation systems were often optimized for. Media and content platforms use ASR at scale to auto-generate subtitles and searchable transcripts for podcasts, videos, and lecture archives, making previously inaccessible audio content searchable and indexable by text-based systems.
## Integration with Other Methods
Self-supervised audio pretraining objectives (masked prediction, contrastive learning) are directly analogous to masked language modeling and contrastive objectives used in text and vision self-supervised learning, and cross-pollination of techniques between these modalities has driven much of the recent progress in audio representation learning. Multimodal models increasingly integrate audio directly alongside text and vision within a single architecture (e.g., audio-visual speech recognition, which uses lip-reading video to improve ASR robustness in noisy acoustic environments), connecting speech processing to the broader multimodal learning literature. Text-to-speech (TTS) synthesis and ASR are increasingly trained with shared or jointly optimized components, since both tasks require a similar underlying alignment between acoustic and textual representations, and some unified speech-language models now handle both recognition and synthesis within a single architecture. Large language models are increasingly used downstream of ASR output for tasks like meeting summarization, action-item extraction, and conversational question answering over transcribed audio, making ASR transcription quality a critical upstream bottleneck for the overall accuracy of these LLM-powered audio applications. Knowledge distillation is commonly used to compress large, accurate offline ASR models into smaller, faster models suitable for on-device or real-time streaming deployment, applying the same compression principles used broadly across deep learning to the specific constraints of speech processing pipelines.
## Future Research Directions
Improving robustness and fairness across accents, dialects, and acoustic conditions remains a central open problem, with ongoing research into more representative training data collection, fairness-aware evaluation protocols, and architectural or training interventions that explicitly reduce subgroup performance disparities rather than only improving aggregate accuracy. Extending self-supervised pretraining to cover a much broader and more balanced set of the world's languages, most of which currently have little to no labeled or even unlabeled digital speech data available, is essential for closing the substantial performance gap between high-resource and low-resource language ASR. Joint and unified modeling of speech and text within a single foundation model, handling recognition, translation, synthesis, and language understanding within one architecture rather than a pipeline of separately trained components, is an active direction connecting speech research more tightly with the broader large language model ecosystem. Improving robustness to overlapping speech, far-field recording, and real-world acoustic conditions beyond what current augmentation-based approaches achieve remains an open engineering and research challenge, particularly for ambient, always-listening deployment scenarios. Finally, privacy-preserving and on-device speech processing, enabling accurate ASR without transmitting raw audio to remote servers, is an increasingly important direction driven both by user privacy expectations and by regulatory requirements, motivating continued research into efficient, compact model architectures suitable for local, resource-constrained inference.
## Summary & Key Takeaways
Modern speech recognition has shifted from hand-engineered acoustic pipelines to end-to-end neural architectures, with CTC, attention-based encoder-decoder, and RNN-Transducer representing three distinct solutions to the core sequence-alignment problem between long audio inputs and short text outputs. Self-supervised pretraining approaches (wav2vec 2.0, HuBERT) learn general-purpose audio representations from unlabeled speech via masked prediction objectives, dramatically reducing the labeled data required for strong downstream ASR performance, while weakly supervised, large-scale multitask training (Whisper) has demonstrated remarkable robustness and multilingual capability by trading labeling precision for massive data scale. Practical ASR systems depend heavily on data augmentation (SpecAugment), appropriate subword tokenization, and transfer learning from large pretrained checkpoints, evaluated primarily via Word Error Rate but increasingly with attention to subgroup fairness across accents and demographics. Persistent challenges include accent and dialect robustness disparities, rare word and code-switching handling, and noisy or far-field acoustic conditions, while future research is oriented toward broader language coverage, unified speech-text foundation models, and privacy-preserving on-device processing.
Keywords: automatic speech recognition, ASR, wav2vec 2.0, HuBERT, Whisper, Connectionist Temporal Classification, CTC, RNN-Transducer, Conformer, self-supervised audio pretraining, mel-spectrogram, SpecAugment, word error rate, speaker diarization, streaming ASR, beam search decoding, language model fusion, multilingual speech recognition, audio representation learning, contrastive audio pretraining
---
## Appendix: Practical Labs
### Lab 1: Mel-Spectrogram Feature Extraction from a Synthetic Waveform
import numpy as np
def generate_synthetic_speech_like_signal(duration=1.0, sample_rate=16000, seed=0):
"""Generates a synthetic waveform with a time-varying fundamental
frequency plus harmonics and noise, standing in for a real speech
signal for demonstration purposes."""
rng = np.random.RandomState(seed)
t = np.linspace(0, duration, int(duration * sample_rate))
f0 = 120 + 30 * np.sin(2 * np.pi * 2 * t) # time-varying pitch
signal = np.sin(2 * np.pi * f0 * t)
signal += 0.5 * np.sin(2 * np.pi * 2 * f0 * t) # first harmonic
signal += 0.05 * rng.randn(len(t)) # additive noise
return signal, sample_rate
def compute_power_spectrogram(signal, sample_rate, frame_length=400, hop_length=160):
"""Short-time Fourier transform magnitude spectrogram via simple framing
and windowing (no external audio library dependency)."""
n_frames = 1 + (len(signal) - frame_length) // hop_length
window = np.hanning(frame_length)
spectrogram = []
for i in range(n_frames):
start = i * hop_length
frame = signal[start:start + frame_length] * window
spectrum = np.abs(np.fft.rfft(frame)) ** 2
spectrogram.append(spectrum)
return np.array(spectrogram).T # (freq_bins, n_frames)
def hz_to_mel(hz):
return 2595 * np.log10(1 + hz / 700)
def mel_to_hz(mel):
return 700 * (10 ** (mel / 2595) - 1)
def mel_filterbank(n_filters, n_fft_bins, sample_rate, frame_length):
low_mel, high_mel = hz_to_mel(0), hz_to_mel(sample_rate / 2)
mel_points = np.linspace(low_mel, high_mel, n_filters + 2)
hz_points = mel_to_hz(mel_points)
bin_points = np.floor((frame_length + 1) * hz_points / sample_rate).astype(int)
filters = np.zeros((n_filters, n_fft_bins))
for i in range(1, n_filters + 1):
left, center, right = bin_points[i - 1], bin_points[i], bin_points[i + 1]
for j in range(left, center):
if center > left:
filters[i - 1, j] = (j - left) / (center - left)
for j in range(center, right):
if right > center:
filters[i - 1, j] = (right - j) / (right - center)
return filters
def test_mel_spectrogram_pipeline():
signal, sr = generate_synthetic_speech_like_signal()
power_spec = compute_power_spectrogram(signal, sr)
n_fft_bins = power_spec.shape[0]
filterbank = mel_filterbank(n_filters=40, n_fft_bins=n_fft_bins, sample_rate=sr, frame_length=400)
mel_spec = filterbank @ power_spec
log_mel_spec = np.log(mel_spec + 1e-8)
print(f"Raw waveform length: {len(signal)} samples ({len(signal)/sr:.2f}s)")
print(f"Power spectrogram shape: {power_spec.shape} (freq_bins, frames)")
print(f"Log-mel spectrogram shape: {log_mel_spec.shape} (mel_filters, frames)")
assert log_mel_spec.shape[0] == 40, "Should have 40 mel filter channels"
assert not np.isnan(log_mel_spec).any(), "Log-mel spectrogram should not contain NaNs"
print("Mel-spectrogram feature extraction test passed.")
if __name__ == "__main__":
test_mel_spectrogram_pipeline()### Lab 2: CTC Loss Forward Algorithm (Simplified)
import numpy as np
def ctc_forward_score(log_probs, target_sequence, blank_id=0):
"""Computes the CTC forward-algorithm log-probability of a target label
sequence given per-frame output log-probabilities, using the standard
blank-augmented alignment lattice.
log_probs: (T, n_classes) log-probabilities per time step.
target_sequence: list of label ids (not including blanks).
"""
T, n_classes = log_probs.shape
# Build the blank-augmented label sequence: blank, l1, blank, l2, ..., blank
ext_labels = [blank_id]
for label in target_sequence:
ext_labels.append(label)
ext_labels.append(blank_id)
S = len(ext_labels)
neg_inf = -1e10
alpha = np.full((T, S), neg_inf)
alpha[0, 0] = log_probs[0, ext_labels[0]]
if S > 1:
alpha[0, 1] = log_probs[0, ext_labels[1]]
def log_sum_exp(a, b):
m = max(a, b)
if m == neg_inf:
return neg_inf
return m + np.log(np.exp(a - m) + np.exp(b - m))
for t in range(1, T):
for s in range(S):
label = ext_labels[s]
score = alpha[t - 1, s]
if s > 0:
score = log_sum_exp(score, alpha[t - 1, s - 1])
if s > 1 and label != blank_id and ext_labels[s - 2] != label:
score = log_sum_exp(score, alpha[t - 1, s - 2])
alpha[t, s] = score + log_probs[t, label]
final_score = log_sum_exp(alpha[T - 1, S - 1], alpha[T - 1, S - 2] if S > 1 else neg_inf)
return final_score
def test_ctc_forward():
rng = np.random.RandomState(0)
T, n_classes = 10, 5 # blank=0, labels 1..4
logits = rng.randn(T, n_classes)
log_probs = logits - np.log(np.exp(logits).sum(axis=1, keepdims=True)) # log-softmax
target = [1, 2, 1] # a short target label sequence
score = ctc_forward_score(log_probs, target)
print(f"CTC log-probability of target sequence {target}: {score:.4f}")
assert score < 0, "Log-probability should be negative (probability <= 1)"
assert np.isfinite(score), "CTC score should be a finite value"
# A longer, less likely target sequence should generally score lower
# (more negative) under the same random logits.
longer_target = [1, 2, 1, 3, 2, 1, 4]
longer_score = ctc_forward_score(log_probs, longer_target)
print(f"CTC log-probability of longer sequence {longer_target}: {longer_score:.4f}")
assert longer_score <= score, "Longer, more constrained sequences should not have higher probability"
print("CTC forward algorithm test passed.")
if __name__ == "__main__":
test_ctc_forward()### Lab 3: Self-Supervised Contrastive Pretraining Objective for Audio
import torch
import torch.nn as nn
import torch.nn.functional as F
class AudioEncoder(nn.Module):
"""Toy convolutional audio encoder mapping raw frame features to
contextualized latent representations."""
def __init__(self, input_dim=40, hidden_dim=64):
super().__init__()
self.conv = nn.Sequential(
nn.Conv1d(input_dim, hidden_dim, kernel_size=3, padding=1),
nn.ReLU(),
nn.Conv1d(hidden_dim, hidden_dim, kernel_size=3, padding=1),
)
def forward(self, x):
# x: (batch, input_dim, time)
return self.conv(x) # (batch, hidden_dim, time)
def mask_time_steps(features, mask_prob=0.3, seed=None):
"""Randomly masks a fraction of time steps by zeroing them, returning
the masked feature tensor and a boolean mask of which steps were hidden."""
if seed is not None:
torch.manual_seed(seed)
batch, dim, time = features.shape
mask = torch.rand(batch, time) < mask_prob
masked_features = features.clone()
for b in range(batch):
masked_features[b, :, mask[b]] = 0.0
return masked_features, mask
def wav2vec_style_contrastive_loss(context_reps, target_reps, mask, n_distractors=10, temperature=0.1, seed=0):
"""For each masked position, contrasts the true target representation
against distractor representations sampled from other masked positions
in the batch, mirroring the wav2vec 2.0 pretraining objective."""
torch.manual_seed(seed)
batch, dim, time = context_reps.shape
losses = []
all_masked_targets = []
for b in range(batch):
for t in range(time):
if mask[b, t]:
all_masked_targets.append(target_reps[b, :, t])
if len(all_masked_targets) < n_distractors + 1:
return torch.tensor(0.0)
all_masked_targets = torch.stack(all_masked_targets) # (n_masked_total, dim)
for b in range(batch):
for t in range(time):
if not mask[b, t]:
continue
c_t = context_reps[b, :, t]
true_target = target_reps[b, :, t]
distractor_idx = torch.randperm(len(all_masked_targets))[:n_distractors]
distractors = all_masked_targets[distractor_idx]
candidates = torch.cat([true_target.unsqueeze(0), distractors], dim=0)
sims = F.cosine_similarity(c_t.unsqueeze(0), candidates, dim=1) / temperature
labels = torch.zeros(1, dtype=torch.long) # true target is always index 0
loss = F.cross_entropy(sims.unsqueeze(0), labels)
losses.append(loss)
return torch.stack(losses).mean() if losses else torch.tensor(0.0)
def test_contrastive_audio_pretraining():
torch.manual_seed(0)
batch, input_dim, time = 4, 40, 20
features = torch.randn(batch, input_dim, time)
encoder = AudioEncoder(input_dim=input_dim, hidden_dim=32)
masked_features, mask = mask_time_steps(features, mask_prob=0.4, seed=1)
context_reps = encoder(masked_features)
target_reps = encoder(features) # "clean" pass provides pretraining targets
loss = wav2vec_style_contrastive_loss(context_reps, target_reps.detach(), mask)
print(f"Number of masked positions: {mask.sum().item()}")
print(f"Contrastive pretraining loss: {loss.item():.4f}")
assert loss.item() >= 0, "Cross-entropy-based contrastive loss should be non-negative"
print("Self-supervised audio contrastive pretraining test passed.")
if __name__ == "__main__":
test_contrastive_audio_pretraining()### Lab 4: Greedy and Beam Search Decoding over CTC Output
import numpy as np
def greedy_ctc_decode(log_probs, blank_id=0):
"""Greedy CTC decoding: take the argmax label at each frame, then
collapse repeated labels and remove blanks."""
best_path = np.argmax(log_probs, axis=1)
decoded = []
prev = None
for label in best_path:
if label != prev and label != blank_id:
decoded.append(label)
prev = label
return decoded
def beam_search_ctc_decode(log_probs, blank_id=0, beam_width=3):
"""Simplified CTC beam search: maintains a set of candidate prefixes
and their accumulated log-probabilities, extending each by one label
per time step and pruning to the top beam_width candidates."""
T, n_classes = log_probs.shape
# Each beam entry: (prefix_tuple, log_prob)
beams = {(): 0.0}
for t in range(T):
new_beams = {}
for prefix, score in beams.items():
for label in range(n_classes):
lp = log_probs[t, label]
if label == blank_id:
new_prefix = prefix
elif len(prefix) > 0 and prefix[-1] == label:
new_prefix = prefix # repeated non-blank label collapses
else:
new_prefix = prefix + (label,)
new_score = score + lp
if new_prefix not in new_beams or new_beams[new_prefix] < new_score:
new_beams[new_prefix] = new_score
# Keep only the top beam_width candidates by score.
beams = dict(sorted(new_beams.items(), key=lambda kv: -kv[1])[:beam_width])
best_prefix = max(beams.items(), key=lambda kv: kv[1])[0]
return list(best_prefix), beams
def test_ctc_decoding():
rng = np.random.RandomState(0)
T, n_classes = 8, 4 # blank=0, labels 1,2,3
logits = rng.randn(T, n_classes) * 2
logits[:, 1] += 3 # bias toward label 1 to create a clear expected decode
log_probs = logits - np.log(np.exp(logits).sum(axis=1, keepdims=True))
greedy_result = greedy_ctc_decode(log_probs)
beam_result, beams = beam_search_ctc_decode(log_probs, beam_width=5)
print(f"Greedy decode: {greedy_result}")
print(f"Beam search decode: {beam_result}")
print(f"Number of final beam candidates: {len(beams)}")
assert isinstance(greedy_result, list), "Greedy decode should return a list of labels"
assert isinstance(beam_result, list), "Beam search decode should return a list of labels"
# Beam search explores more candidates, so its best score should be at
# least as good as the single greedy path's implied score.
print("CTC decoding test passed.")
if __name__ == "__main__":
test_ctc_decoding()