se-transformers

**SE(3)-Transformers** are **attention-based neural architectures that achieve equivariance to the Special Euclidean group SE(3) — the group of 3D rotations and translations — by combining the transformer's attention mechanism with geometric features based on spherical harmonics** — enabling powerful, long-range attention over 3D point clouds and molecular structures while guaranteeing that predictions are independent of the arbitrary choice of coordinate system. **What Are SE(3)-Transformers?** - **Definition**: An SE(3)-Transformer (Fuchs et al., 2020) replaces the standard transformer's attention and value computations with SE(3)-equivariant versions. The attention weights depend only on invariant quantities (pairwise distances, angles), ensuring that the same attention pattern emerges regardless of how the 3D structure is oriented. The value vectors carry geometric information using type-$l$ spherical harmonic features that transform predictably under rotation. - **Geometric Attention**: In a standard transformer, attention weights are computed from key-query dot products on abstract embeddings. In an SE(3)-Transformer, attention weights are computed from invariant features — pairwise distances $|x_i - x_j|$, scalar node features, and angle-based geometric features — ensuring the "who attends to whom" decision is rotation-independent. - **Spherical Harmonic Features**: Features at each node are organized by their rotation order $l$ — type-0 (scalars, invariant), type-1 (vectors, rotate as 3D vectors), type-2 (matrices, rotate as rank-2 tensors). The transformer's value computation uses Clebsch-Gordan coefficients to combine features of different types while maintaining equivariance, propagating both scalar and geometric information through attention layers. **Why SE(3)-Transformers Matter** - **Protein Structure Prediction**: AlphaFold2's success demonstrated that SE(3)-aware attention is essential for protein structure prediction — the 3D coordinates of amino acid residues must be predicted in a rotation-equivariant manner. SE(3)-Transformers provide the theoretical framework for this type of geometric attention, and AlphaFold2's Invariant Point Attention is a practical variant of this approach. - **Long-Range 3D Interactions**: Graph neural networks propagate information locally through edges, requiring many message-passing layers to capture long-range interactions. SE(3)-Transformers use attention to compute direct long-range interactions between distant atoms or residues, capturing non-local effects (electrostatic interactions, allosteric regulation) in fewer layers. - **Expressiveness**: By incorporating higher-order spherical harmonic features (type-1 vectors, type-2 tensors), SE(3)-Transformers can represent directional information — bond angles, torsional angles, dipole moments — that scalar-only models like EGNNs cannot capture. This additional expressiveness is critical for tasks requiring angular sensitivity (predicting force directions, molecular conformations). - **Unified Architecture**: SE(3)-Transformers provide a single architecture that handles both invariant tasks (energy prediction) and equivariant tasks (force prediction, structure generation) by selecting the appropriate output feature type — type-0 for invariant outputs, type-1 for vector outputs, type-2 for tensor outputs. **SE(3)-Transformer Architecture** | Component | Function | Geometric Property | |-----------|----------|-------------------| | **Invariant Attention** | Compute attention weights from distances and scalar features | SE(3)-invariant (same weights under rotation) | | **Type-$l$ Features** | Spherical harmonic features at each node | Transform as irreps of SO(3) | | **Tensor Product** | Combine features of different types via Clebsch-Gordan | Maintains equivariance during feature interaction | | **Equivariant Value** | Attention-weighted aggregation of geometric features | SE(3)-equivariant output | **SE(3)-Transformers** are **rotating attention heads** — applying the full power of transformer-style attention to 3D point clouds and molecular structures while respecting the fundamental geometry of 3D space, enabling long-range interactions that preserve rotational and translational symmetry.