Longformer
foundation model
Sliding-window and sparse attention are techniques that cut the cost of the Transformer's attention by computing only a chosen subset of query-key pairs instead of all of them.…
974 technical terms and definitions
foundation model
Sliding-window and sparse attention are techniques that cut the cost of the Transformer's attention by computing only a chosen subset of query-key pairs instead of all of them.…
architecture
Longformer attention is the sparse attention mechanism combining sliding-window local attention with selected global attention tokens for long-sequence processing - it enables…
optimization
Longformer Attention is a sparse-attention pattern combining local windows with selected global tokens - It is a core method in modern semiconductor AI serving…
optimization
Lookahead Optimizer is a meta-optimizer that wraps around any base optimizer (SGD, Adam) — maintaining two sets of weights: "fast weights" updated by the inner optimizer…
Lookahead-AdamW Hybrid Optimization Architecture Introduction & Motivation Lookahead-AdamW Hybrid Optimization Architecture addresses a central problem in optimization algorithm…
speculative, parallel, draft, speedup, inference
Lookahead decoding is a speculative decoding technique that generates multiple tokens in parallel — using n-gram patterns or draft models to predict likely continuations, then…
speculative decoding, llm acceleration
Lookahead decoding is an inference acceleration technique that generates multiple tokens in parallel using speculative execution — predicting future tokens speculatively…
optimization
Lookahead Decoding is a decoding method that evaluates multiple future token candidates in parallel within one planning step - It is a core method in modern semiconductor AI…
Lookahead Optimizer with Slow and Fast Weight Updates Introduction & Motivation Lookahead Optimizer with Slow and Fast Weight Updates addresses a central problem in optimization…
robotics
Loop closure detection is the SLAM process of recognizing previously visited places and adding constraints that correct accumulated trajectory drift - it turns local odometry into…
packaging
Loop height control is the process of setting and maintaining bonded wire loop vertical profile within specified limits for clearance and reliability - it is critical for avoiding…
loop nest tiling and blocking in ml compiler, loop nest tiling, nest tiling blocking, tiling blocking ml, blocking ml compiler, loop nest, nest tiling, tiling blocking, blocking ml
Loop Nest Tiling and Blocking in ML Compiler Introduction & Motivation Loop Nest Tiling and Blocking in ML Compiler addresses a central problem in ml-compiler process control…
model optimization
Loop Optimization is transforming loop structure to improve instruction efficiency and memory access behavior - It is central to compiler-level acceleration of numeric kernels.…
loop tiling and data reuse estimation for cache optimization, loop tiling data, tiling data reuse, data reuse estimation, reuse estimation cache, estimation cache optimization, loop tiling
Loop Tiling and Data Reuse Estimation for Cache Optimization Introduction & Motivation Loop Tiling and Data Reuse Estimation for Cache Optimization addresses a central problem…
loop transformation and optimization, loop transformation optimization, loop transformation, transformation optimization
Loop Transformation and Optimization Introduction & Motivation Loop Transformation and Optimization addresses a central problem in compiler-optimization process control…
model optimization
Loop Unrolling is a compiler optimization that replicates loop bodies to reduce branch overhead and increase instruction-level parallelism - It improves throughput…
loop unrolling and register pressure trade-offs, loop unrolling register, unrolling register pressure, register pressure trade-offs, loop unrolling, unrolling register
Loop Unrolling and Register Pressure Trade-Offs Introduction & Motivation Loop Unrolling and Register Pressure Trade-Offs addresses a central problem in compute process control…
loopy belief propagation inference prediction, loopy belief propagation, belief propagation inference, propagation inference prediction, loopy belief, belief propagation
Loopy Belief Propagation Inference Prediction with Machine Learning Introduction & Motivation Loopy Belief Propagation Inference Prediction with Machine Learning addresses…
rslora, lora rank, lora alpha, lora scaling, lora adapter, lora placement, lora target modules, lora memory, lora serving, lora quantization, adapter rank
The LoRA paper's headline is that it trains 10,000 times fewer parameters, and the number is honest — GPT-3 175B with rank-4 adapters on the query and value projections has 18.9M…
low rank adaptation, qlora, parameter efficient fine tuning, peft adapter
LoRA (Low-Rank Adaptation) is the parameter-efficient fine-tuning method that injects small trainable low-rank matrices into frozen pretrained model layers — enabling fine-tuning…
parameter efficient fine tuning, peft, qlora, adapter fine tuning, low rank adaptation
LoRA (Low-Rank Adaptation) is the parameter-efficient fine-tuning technique that injects trainable low-rank decomposition matrices into frozen pretrained model weights — enabling…
dreambooth, customize
LoRA for Diffusion Models enables efficient customization of Stable Diffusion and similar image generators — using Low-Rank Adaptation to fine-tune large diffusion models on just…
multimodal ai
LoRA Fine-Tuning is parameter-efficient adaptation using low-rank update matrices inserted into pretrained model layers - It enables fast customization with small trainable…
low rank adaptation, lora adapter, peft lora, lora rank selection
Low-Rank Adaptation (LoRA) is the parameter-efficient fine-tuning technique that adds small, trainable low-rank decomposition matrices to frozen pretrained weights — factoring…
generative models
LoRA for diffusion enables efficient fine-tuning to learn specific styles, subjects, or concepts with minimal resources. Application: Customize Stable Diffusion for particular…
generative models
LoRA for diffusion is the parameter-efficient fine-tuning method that trains low-rank adapter matrices instead of full model weights - it enables fast customization with smaller…
peft lora fine tuning, lora adapters, parameter efficient fine tuning, qlora workflow, adapter based llm customization
LoRA (Low-Rank Adaptation) is a parameter-efficient fine-tuning method that freezes the original model weights and trains small low-rank adapter matrices inserted into selected…
parameter efficient fine tuning peft, lora adapter training, qlora quantized lora, lora rank alpha
LoRA (Low-Rank Adaptation) is the parameter-efficient fine-tuning technique that adapts a large pre-trained model to new tasks by injecting small, trainable low-rank decomposition…
peft parameter efficient, adapter fine tuning, qlora quantized lora, fine tuning efficient
LoRA (Low-Rank Adaptation) is the parameter-efficient fine-tuning technique that adapts large language models to specific tasks by injecting small trainable low-rank matrices into…
Low-Rank Adaptation LoRA for Parameter-Efficient Tuning Introduction & Motivation Low-Rank Adaptation LoRA for Parameter-Efficient Tuning addresses a central problem in transfer…
generative models
LoRA merging is the process of combining one or more LoRA adapter weights into a base model or composite adapter set - it creates reusable model variants without retraining…
loss function computation precision and gradient flow, loss function computation, function computation precision, computation precision gradient, precision gradient flow
Loss Function Computation Precision and Gradient Flow Introduction & Motivation Loss Function Computation Precision and Gradient Flow addresses a central problem in training…
objective, minimize
Loss Functions for Language Models Cross-Entropy Loss The standard loss for language modeling: Where is the correct token and is the model's predicted probability. How It Works…
cross entropy, objective
Cross-entropy loss is the standard objective function for language model training, measuring the difference between predicted token probability distributions and actual (one-hot)…
quality
The Taguchi Loss Function is a revolutionary quality engineering philosophy formulated by Genichi Taguchi that fundamentally destroyed the prevailing industrial "goalpost"…
cost function, objective function
Loss Function — the mathematical function that measures how wrong the model's predictions are, providing the signal that guides training through gradient descent. Classification…
cross entropy loss, focal loss, triplet loss, contrastive loss function
Loss Functions are the mathematical objectives that quantify the discrepancy between model predictions and desired outputs, guiding the optimization process through gradient…
optimization objectives, custom loss functions, training objectives, loss landscape analysis
Loss Function Design and Optimization — Loss functions define the mathematical objective that neural networks minimize during training, translating task requirements into…
loss function design and selection, loss function design, function design selection, loss function, function design, design selection
Loss Function Design and Selection Introduction & Motivation Loss Function Design and Selection addresses a central problem in ml-training process control and loss-expressiveness…
loss function design, loss function, function design
Loss Function Design Introduction & Motivation Loss Function Design addresses a central problem in systems process control and loss-type-regression-classification prediction: how…
ml loss, training objective, cross entropy, focal loss, mse, mae, huber, contrastive loss
Loss function maps predictions and targets or preferences to an optimization signal. The loss defines what learning rewards and penalizes, so it shapes convergence, calibration…
quality & reliability
Loss Function Quality is a quality-economics model that maps deviation from target to monetary or operational loss - It is a core method in modern semiconductor quality…
loss functions cross entropy focal loss class imbalance, loss functions, cross entropy focal, ap ce cifar coco l1 nlp, loss functions cross-entropy focal loss & class imbalance
Loss Functions: Cross-Entropy, Focal Loss & Class Imbalance Introduction & Motivation Loss function: optimization objective. Cross-entropy: standard classification. Focal loss…
loss functions cross entropy focal loss contrastive, loss functions, cross entropy focal, cifar nt passed, loss functions cross-entropy focal loss & contrastive, cross-entropy focal loss
Loss Functions: Cross-Entropy, Focal Loss & Contrastive Introduction & Motivation Loss functions: model training objectives. Cross-entropy: classification standard. Focal loss…
loss functions different tasks, loss functions, functions different, loss functions different, functions different tasks, different tasks
Loss Functions for Different Tasks Introduction & Motivation Loss Functions: measure prediction error. Cross-entropy, MSE, Triplet loss. Applications: objective definition…
hidden loss manufacturing, manufacturing operations, efficiency loss
Hidden Loss is productivity loss not visible in standard reports due to data granularity or classification gaps - It conceals real capacity constraints and improvement…
theory
Loss Landscape Analysis is the study of the geometry of a neural network's loss function in parameter space — visualizing and characterizing the shape of the high-dimensional loss…
theory
Loss Landscape Smoothness refers to the geometric properties of the loss function surface in parameter space — smooth landscapes (low curvature, wide minima) correlate with better…
model training
Mixed-precision training is the standard recipe that lets modern models train in half the memory and roughly twice the throughput without losing accuracy. The idea is simple…
dynamic loss scaling, gradient scaling fp16, loss scale overflow, gradient underflow prevention
Loss Scaling Techniques are the numerical methods for preventing gradient underflow in FP16 training by multiplying the loss by a large scale factor (1024-65536) before…