neural module composition
**Neural Module Composition** is the **architectural paradigm where neural network layouts are dynamically assembled at inference time by selecting and connecting specialized computational modules based on the structure of the input query** — enabling Visual Question Answering (VQA) systems to parse a natural language question into a symbolic program and then wire together the corresponding neural modules into a custom computation graph that executes against the visual input.
**What Is Neural Module Composition?**
- **Definition**: Neural Module Composition refers to Neural Module Networks (NMNs) and their descendants — models that maintain a library of specialized neural modules (e.g., "Locate," "Describe," "Count," "Compare") and compose them into question-specific computation graphs at inference time. Rather than processing all questions through a fixed architecture, each question generates a unique program that determines which modules execute in what order.
- **Dynamic Assembly**: A semantic parser analyzes the input question ("What color is the large sphere left of the cube?") and produces a symbolic program: `Describe(Color, Filter(Large, Relate(Left, Locate(Sphere), Locate(Cube))))`. The system retrieves the neural weights for each module and wires them into a custom feedforward network that processes the image.
- **Module Library**: Each module is a small neural network specialized for a specific visual reasoning operation — spatial filtering, attribute extraction, counting, comparison, or relationship detection. Modules are trained jointly across all questions, learning reusable visual primitives.
**Why Neural Module Composition Matters**
- **Compositional Generalization**: Fixed-architecture VQA models memorize question-answer patterns and fail on novel compositions. Module composition generalizes systematically — if "red" and "sphere" modules work individually, "red sphere" works automatically by composing them, even if that exact combination never appeared in training.
- **Interpretability**: The program trace provides a complete, human-readable explanation of the reasoning process. For "How many red objects are bigger than the blue cylinder?", the trace shows: Filter(red) → FilterBigger(Filter(blue) → Filter(cylinder)) → Count — each step is inspectable and verifiable.
- **Data Efficiency**: Because modules learn reusable primitives rather than holistic pattern matching, new concepts can be learned from fewer examples. A new color module can be trained on a handful of examples and immediately composed with all existing shape, size, and relation modules.
- **Scalability**: The number of answerable questions scales combinatorially with the module library size. Adding one new module (e.g., "Behind") immediately enables all compositions involving spatial behind-relations without retraining existing modules.
**Key Architectures**
| Architecture | Innovation | Key Property |
|-------------|-----------|--------------|
| **NMN (Andreas et al.)** | First neural module networks with parser-generated layouts | Proved compositional VQA feasibility |
| **N2NMN** | End-to-end learned program generation replacing external parser | Removed dependency on symbolic parser |
| **Stack-NMN** | Soft module selection via attention over module library | Fully differentiable, no discrete program |
| **NS-VQA** | Neuro-symbolic: neural perception + symbolic program execution | Perfect accuracy on CLEVR via hybrid approach |
**Neural Module Composition** is **on-the-fly neural circuit compilation** — building a custom computation graph for every input by assembling specialized modules into question-specific reasoning pipelines that generalize compositionally to novel combinations.