Upscaling

Upscaling, or super-resolution, is the task of turning a low-resolution image or video into a higher-resolution one with more pixels, by estimating the detail that the smaller version does not contain. Simply enlarging pixel dimensions is easy; the hard part is that fine detail (hair strands, text strokes, brick texture) was lost when the image was captured or downsampled, and many different sharp images shrink to the very same blurry one. Upscaling is therefore an ill-posed problem, and the central distinction in the field is between methods that interpolate existing pixel values and methods that learn what high-resolution images typically look like and synthesize plausible detail.

Classical interpolation computes each new pixel as a fixed weighted average of nearby original pixels, and it adds no real information. Nearest-neighbor copies the closest pixel and gives a blocky result; bilinear averages four neighbors and gives a soft one; bicubic uses a 4 × 4 neighborhood and is the long-time default in image editors; Lanczos resampling uses a windowed sinc function and keeps edges somewhat crisper. These are fast, deterministic and predictable, and they never invent content, but they cannot restore edges or textures, so a 4× enlargement looks smooth and blurry. Bicubic is also the usual way benchmark datasets create the low-resolution inputs, and it serves as the baseline that learned methods must beat.

Learned super-resolution began in earnest with SRCNN (Dong, Loy, He and Tang, 2014), a three-layer convolutional network trained to map bicubically enlarged low-resolution images to their high-resolution originals. It beat the best classical sparse-coding methods of its day while being simple and fast, and it started a quick succession of designs: VDSR and EDSR deepened the networks with residual learning, and sub-pixel convolution (Shi and colleagues, 2016) let the network do its work at low resolution and rearrange channels into a larger image at the end. These models are trained on pairs of images with a pixel-wise loss such as mean squared error, which maximizes measures like PSNR and SSIM but averages over the many possible sharp answers, so outputs tend to look smooth and slightly waxy.

Generative approaches accept that the goal is realism, not pixel-exact recovery, which is why they look sharper. SRGAN (Ledig and colleagues, 2016) trained a generator against a discriminator plus a perceptual loss computed on features from a pretrained VGG network; ESRGAN (Wang and colleagues, 2018) improved the architecture with residual-in-residual dense blocks and a relativistic discriminator and won the PIRM 2018 perceptual super-resolution challenge. Real-ESRGAN (Wang, Xie, Dong and Shan, 2021) trained on synthetic degradations that mimic real-world blur, noise and compression, which made it popular for restoring old photos and anime, and SwinIR (2021) brought transformer blocks to the task. Diffusion models then took over the quality frontier: SR3 (Saharia and colleagues, 2021) conditions a denoising diffusion model on the low-resolution image, cascaded diffusion systems upscale base images in stages, and Stable Diffusion's 4× upscaler applies a latent diffusion model for the same purpose. The perception-distortion trade-off proved by Blau and Michaeli (2018) states that no method can maximize both pixel fidelity and perceptual quality at once, which formalizes the choice.

The key pitfall is hallucination: a generative upscaler adds detail that looks right but was never in the original, so it must not be treated as evidence. A face may get invented pores, a blurry license plate may be rewritten into crisp but wrong characters, and a medical or satellite image may acquire structure that was not there; for forensic, scientific or diagnostic work, interpolation or clearly labeled enhancement is the defensible choice. Other pitfalls include temporal flicker when images are upscaled frame by frame in video, over-sharpened or plasticky skin and textures, artifacts when the input's degradation differs from the training assumptions, and the mistake of equating upscaling with resolution: a 4× enlarged photo has more pixels but not more measured information.

Upscaling also exists as a real-time rendering feature in games, which brings the idea to consumer hardware. NVIDIA's DLSS, introduced in 2018 with RTX GPUs, renders frames at a lower internal resolution and uses a neural network, fed with motion vectors and previous frames, to reconstruct a higher-resolution picture; AMD's FSR and Intel's XeSS follow similar goals, with FSR initially using hand-designed spatial algorithms before adding temporal and learned variants in later versions. Using temporal information is the advantage of video and game upscalers: successive frames sample the scene at slightly different sub-pixel offsets, which gives genuinely extra information that a single still image lacks.

Read upscaling as an educated guess about missing detail: interpolation guesses with fixed arithmetic and invents nothing, learned and generative upscalers guess with a model of what the world usually looks like, producing sharper results at the price of plausible but unverifiable detail.