No single microscope sees a semiconductor defect in all the ways that matter. Secondary electrons reveal surface form, diffraction reveals crystal orientation, EDS or EELS reveals chemistry, cathodoluminescence reveals radiative pathways, EBIC reveals charge collection, and scanning probes reveal topography or local electrical response. Multimodal microscopy connects these partial views at the same feature so that structure, composition, strain, optical behavior, and device function can test one another instead of becoming separate stories.

Multimodal microscopy begins with a shared specimen question, not a stack of attractive images. The experiment should specify the latent property or mechanism to constrain—such as whether a dark electrical defect is a dislocation decorated by an impurity—and assign each modality a distinct evidentiary role. One channel may locate morphology, another measure composition, another test electrical consequence, and another bound a competing explanation. Collecting more channels without defining this logic increases dose, registration complexity, and false-correlation opportunities without necessarily increasing information.

Registration is a measurement with uncertainty, not a cosmetic overlay. A coordinate (\mathbf x_A) in modality A is mapped into modality B by a transform (T) estimated from landmarks, stage coordinates, or shared image structure:

The transform may be rigid, affine, projective, elastic, or a chain across intermediate scales. The residual ε includes landmark localization, drift, lens distortion, sectioning deformation, stage repeatability, and genuine specimen change. A low registration residual on the landmarks does not guarantee accuracy between them, especially with an overly flexible warp. Fiducials should span the region of interest, held-out landmarks should test generalization, and local registration uncertainty should accompany any claim that two nanoscale features coincide.

For (N) validation landmarks, a simple residual summary is

That scalar should be compared with pixel size, point-spread widths, feature dimensions, and the separation relevant to the hypothesis. Residual vectors and spatial maps can reveal systematic shear or local deformation hidden by one average. When the claimed offset is comparable to registration uncertainty, the correct conclusion is unresolved—not coincident or separated.

Common pixels do not imply common spatial resolution or sampling volume. A modality records a specimen property after convolution with its own point-spread or interaction function (h_m), plus noise and artifacts:

Resampling a coarse chemical map onto a fine SEM grid creates more pixels, not more chemical resolution. Pixelwise correlation after interpolation can inflate the apparent sample size and assign sharp boundaries to a diffuse signal. Comparisons should use a common physical support: degrade higher-resolution data to a justified effective response, aggregate within independent regions, or forward-model each modality at its native grid. The claimed correlation scale cannot be finer than the registration and response functions support.

Sequential measurements can observe different specimen states. Air exposure grows oxides and adsorbates; vacuum changes volatile species and charging; FIB sectioning removes material and introduces damage; ion sputtering mixes and reduces surfaces; electron or photon dose heats, charges, deposits carbon, and creates defects; electrical bias and temperature alter carrier populations. Acquisition order is therefore part of the causal record. Non-destructive, low-dose, and ambient-sensitive measurements are usually scheduled before destructive preparation, while repeated reference measurements test whether the region changed between modalities.

question[State mechanism and distinct role of each modality] --> specimen[Design specimen, fiducials, coordinate hierarchy, and dose order]
specimen --> acquire[Acquire native data plus calibration and state references]
acquire --> qa{Same region and acceptably unchanged state?}
qa -- no --> revise[Re-register, bound state change, or reject correlation]
revise --> acquire
qa -- yes --> register[Estimate transform with held-out landmark validation]
register --> support[Propagate uncertainty and harmonize physical support]
support --> compare[Compare native measurements and explicit hypotheses]
compare --> fuse{Does a justified joint model add information?}
fuse -- no --> evidence[Keep registered modalities as separate evidence]
fuse -- yes --> validate[Test fusion on simulations, residuals, and withheld data]
validate --> evidence
evidence --> report[Report provenance, transforms, resolution, uncertainty, and alternatives]

Correlation is weaker than a mechanism and can be driven by shared morphology. Two channels may covary because both respond to thickness, surface tilt, contamination, or the same segmentation boundary. Spatial autocorrelation makes conventional pixelwise p-values invalid because neighboring pixels are not independent. Test competing explanations, use region- or feature-level statistics, include negative controls, and ask whether one modality adds predictive information beyond morphology and acquisition geometry. A chemical hotspot aligned with an EBIC-dark region supports a hypothesis only if topography, preparation, and registration error cannot explain both signals.

Mutual information is useful for multimodal registration because it can align images whose intensities are not linearly related:

Yet an optimizer can find a numerically high value at a physically wrong alignment when fields of view repeat, overlap is small, contrast is dominated by borders, or one modality has artifacts. Initialization from stage coordinates or landmarks, masks, multiscale optimization, transform regularization, and held-out visual features remain necessary. The similarity metric is evidence for a transform, not proof of correspondence.

Data fusion requires a generative relationship between modalities. Early fusion concatenates registered features, intermediate fusion learns shared representations, and late fusion combines modality-specific decisions. In hypermodal electron microscopy, data blocks can share spatial factors while retaining distinct spectral or diffraction loadings. A schematic block model is

where (W) represents shared spatial factors and (H_m) modality-specific signatures. Block scaling is consequential: a high-count or high-dimensional modality can dominate the objective even when it is less relevant. Shared factors can improve sensitivity, but they can also impose a structure from a strong channel onto a weak channel that never independently measured it.

Fusion should be tested against an unfused baseline, synthetic or reference data with known truth, withheld regions, perturbations to registration, alternate ranks and weights, and modality-dropout analysis. Residuals must be inspected separately for every block. If a fused chemical feature disappears when the morphology block is removed, the method may be sharpening by prior correlation rather than recovering independent chemistry. A reconstructed high-resolution map is a model output and must not be labeled as direct measurement.

Uncertainty has modality-specific, registration, and model components. Shot noise, calibration, segmentation, peak fitting, indexing, cross-sections, and detector response differ by technique. Registration adds coordinate covariance; resolution harmonization adds response uncertainty; fusion adds parameter and structural-model uncertainty. Monte Carlo propagation can sample plausible transforms and modality parameters, rerun the comparison, and show whether the mechanism survives. An uncertainty band around a joint parameter is incomplete if it conditions on one exact alignment and one exact fusion rank.

A Bayesian evidence model can make assumptions explicit:

where (z) is a latent mechanism and (D_m) are modality data. Replacing the joint likelihood with a product assumes conditional independence; that is often false when modalities share dose history, morphology, calibration, or preprocessing. Double-counting correlated evidence produces unjustified certainty. A causal diagram or dependency audit is often more valuable than a sophisticated fusion algorithm because it reveals shared error sources before they enter the model.

Provenance is the backbone of reproducible correlation. Archive raw native data, coordinate systems, units, stage and specimen orientation, timestamps, acquisition order, beam or probe conditions, environmental state, calibration, dose, preparation history, fiducial definitions, transforms, software versions, masks, and analysis parameters. Store transforms as data rather than baking them into screenshots. Every derived map should trace back to a native modality, a processing step, and an uncertainty estimate. This enables later re-registration when a better landmark or physical model becomes available.

For semiconductor failure analysis, a strong multimodal chain might proceed from device-scale electrical localization to SEM morphology, EBIC collection contrast, CL recombination behavior, FIB cross-section coordinates, and TEM/EELS structure and chemistry. Each transition narrows the region while risking preparation or registration error. The conclusion becomes persuasive when the proposed mechanism predicts all channels, contradicts plausible alternatives, and survives the uncertainty accumulated across scales.

For semiconductor process learning, the central question is not “how many modalities agree visually?” It is “which mechanism remains supported after coordinate uncertainty, resolution mismatch, specimen-state change, shared confounders, and fusion assumptions are tested?” Reading multimodal microscopy through that registered-independent-evidence-and-state-provenance lens turns an overlay montage into a defensible structure–property argument.