deep learning basics

**Deep Learning Basics** — the foundational concepts behind training multi-layered neural networks to learn hierarchical representations from raw data. **Core Idea** Deep learning extends classical machine learning by stacking multiple layers of nonlinear transformations. Each layer learns increasingly abstract features: early layers detect edges and textures, middle layers recognize parts and patterns, and deep layers capture high-level semantic concepts. The "deep" in deep learning refers to the depth of these computational graphs — modern architectures range from dozens to hundreds of layers. **Key Components** - **Neurons (Perceptrons)**: Basic computational units that compute a weighted sum of inputs, add a bias, and apply an activation function: $y = f(\sum w_i x_i + b)$. - **Activation Functions**: Nonlinear functions that enable networks to learn complex mappings. Common choices include ReLU ($\max(0, x)$), sigmoid ($1/(1+e^{-x})$), tanh, GELU, and SiLU/Swish. - **Layers**: Fully connected (dense), convolutional (spatial patterns), recurrent (sequential data), and attention-based (transformer) layers each specialize in different data structures. - **Loss Functions**: Quantify the difference between predictions and ground truth. Cross-entropy for classification, MSE for regression, contrastive losses for representation learning. - **Backpropagation**: The chain rule applied through the computational graph to compute gradients of the loss with respect to every parameter, enabling gradient-based optimization. - **Optimizers**: Algorithms that update parameters using gradients. SGD with momentum, Adam ($\beta_1=0.9$, $\beta_2=0.999$), AdamW (decoupled weight decay), and LAMB (for large-batch training) are standard choices. **Training Pipeline** 1. **Data Preparation**: Collect, clean, augment, and split data into train/validation/test sets. Normalization (zero mean, unit variance) stabilizes training. 2. **Forward Pass**: Input flows through layers, producing predictions. 3. **Loss Computation**: Compare predictions against targets. 4. **Backward Pass**: Compute gradients via backpropagation. 5. **Parameter Update**: Optimizer adjusts weights to minimize loss. 6. **Iteration**: Repeat over mini-batches for multiple epochs until convergence. **Regularization Techniques** - **Dropout**: Randomly zero out neurons during training (typically 10-50%) to prevent co-adaptation and improve generalization. - **Weight Decay (L2)**: Add $\lambda ||w||^2$ penalty to the loss, discouraging large weights. - **Batch Normalization**: Normalize activations within mini-batches to stabilize training and allow higher learning rates. - **Data Augmentation**: Apply random transformations (flips, crops, color jitter) to increase effective dataset size. - **Early Stopping**: Monitor validation loss and halt training when it stops improving. **Common Architectures** - **CNNs (Convolutional Neural Networks)**: Spatial feature extraction using learnable filters. Foundational for computer vision — image classification, object detection, segmentation. - **RNNs/LSTMs/GRUs**: Sequential processing with hidden state memory. Used for time series, speech, and language before transformers became dominant. - **Transformers**: Self-attention mechanisms that process all positions in parallel. Now the backbone of NLP (BERT, GPT), vision (ViT), and multimodal models (CLIP). - **Autoencoders/VAEs**: Learn compressed latent representations for generative modeling and anomaly detection. - **GANs (Generative Adversarial Networks)**: Generator-discriminator pairs that learn to produce realistic synthetic data. **Practical Considerations** - **Learning Rate**: The single most important hyperparameter. Too high causes divergence, too low causes slow convergence. Learning rate schedulers (cosine annealing, warmup, reduce-on-plateau) are essential. - **Batch Size**: Larger batches improve GPU utilization but may hurt generalization. Gradient accumulation simulates large batches on limited hardware. - **Mixed Precision Training**: Use FP16/BF16 for forward/backward passes with FP32 master weights — 2x speedup with minimal accuracy loss on modern GPUs. - **Transfer Learning**: Start from pretrained weights (ImageNet for vision, BERT/GPT for language) and fine-tune on your specific task. This is the dominant paradigm — training from scratch is rarely necessary. **Deep Learning Basics** form the foundation of modern AI — understanding neurons, layers, backpropagation, and optimization is essential before exploring advanced topics like transformers, distributed training, or model compression.