Neuromorphic Vision

Neuromorphic Vision is a paradigm for artificial visual perception that draws inspiration from biological sensory systems, combining event-based cameras (Dynamic Vision Sensors) with neuromorphic processors and spiking neural networks to achieve sub-millisecond latency, extreme power efficiency, and high dynamic range that conventional frame-based cameras and standard neural networks cannot match. The core insight: biological vision doesn't process full frames — it responds asynchronously to changes, computing only when something moves or changes, consuming milliwatts instead of watts.

Event-Based Cameras: The Neuromorphic Sensor

Conventional cameras capture full frames at fixed intervals (30-120 fps). Event cameras (Dynamic Vision Sensors, DVS) operate fundamentally differently:

- Each pixel independently and asynchronously fires an event when its log-luminance changes by a threshold:
- Positive event (+1): Brightness increased at pixel $(x, y)$ at time $t$
- Negative event (-1): Brightness decreased at pixel $(x, y)$ at time $t$
- Output: A stream of events $(x, y, t, p)$ — position, microsecond timestamp, polarity
- Static scenes: No output (nothing to report)
- Moving objects: High event density along motion boundaries

Key Properties vs. Conventional Cameras

| Property | Frame Camera | Event Camera |
|----------|-------------|-------------|
| Temporal resolution | 1-120 fps (8-33ms) | 1 microsecond |
| Latency | 1 frame (8-33ms) | ~1 microsecond |
| Dynamic range | 60-80 dB | 120-140 dB |
| Data rate | Fixed (always full frame) | Sparse (only on change) |
| Power (sensor) | 100-500mW | 1-10mW |
| Motion blur | Significant at high speed | None |
| Low light performance | Noisy | Good (high dynamic range) |

Leading Event Camera Hardware

- Sony IMX636: 1280×720 resolution, 120 dB dynamic range, QVGA to HD — commercially available in industrial machine vision
- iniVation DAVIS346: Combined event + frame camera (346×260 pixels), popular in research
- Prophesee EVK4: High-resolution (1280×720), automotive and industrial focus
- Samsung DVS: Research prototypes with higher resolution targets

Neuromorphic Processors

Processing event streams efficiently requires neuromorphic processors that handle sparse, asynchronous spike data:

Intel Loihi 2 (2021):
- 1 million neurons, 120 million synapses per chip
- On-chip learning via spike-timing-dependent plasticity (STDP)
- ~0.5W per chip at full load
- Loihi 2 improves on-chip learning; Intel's Hala Point system (2024) uses 1,152 Loihi 2 chips = 1.15B neurons
- Not yet production-deployed at scale; primary use: research

IBM TrueNorth (2014):
- 4096 neurosynaptic cores, 1M neurons, 256M programmable synapses
- 70mW at 1 billion synaptic events/second — orders of magnitude below GPU
- Fixed function: not reconfigurable like Loihi

BrainScaleS (Heidelberg/Human Brain Project):
- Analog computation — physical circuits implement neuronal dynamics
- 10,000x faster than biological brain (extreme temporal compression)
- Research platform for neuroscience-inspired AI

Spiking Neural Networks (SNNs)

Spiking Neural Networks are the computational model for neuromorphic hardware:
- Neurons: Leaky integrate-and-fire (LIF) model accumulates input voltage, fires when threshold is reached, resets
- Spikes: Binary events (0 or 1) replacing the continuous activations of standard ANNs
- Temporal coding: Information encoded in spike timing, not just spike rate
- Energy: Computation happens only when spikes occur (sparse, event-driven)

SNN training challenges:
- Non-differentiable: Spike generation is a step function — cannot backpropagate through it directly
- Surrogate gradients: Approximate the spike derivative with smooth surrogates (sigmoid, piecewise linear)
- ANN-to-SNN conversion: Train a standard ANN, then convert to SNN by replacing activations with neurons

Current Performance Gap: State-of-art SNNs on ImageNet reach ~70-75% top-1 accuracy vs 80%+ for equivalent ANNs. Closing this gap is an active research area.

Applications

Autonomous Vehicles and Robotics:
- Event cameras detect fast-moving objects (pedestrians, vehicles) with μs latency — critical for emergency braking
- Motor control: Drone flight stabilization with event cameras at <1ms response vs >30ms for frame cameras
- Prophesee partnered with Stellantis for automotive event camera integration

Edge AI and IoT:
- Smart surveillance: Motion detection at milliwatts — sensors running on harvested energy
- Industrial inspection: Detection of high-speed defects (production lines running at 10m/s)
- Wearables: Always-on gesture recognition, eye tracking for AR/VR

Space and Defense:
- Satellite tracking: High dynamic range handles Sun glare and dark space simultaneously
- Drone detection: μs latency event streams enable tracking fast-moving UAVs

Robotics: Event cameras now appear in research robots at MIT, ETH Zurich, and DARPA programs for agile, low-power perception.

The Road Ahead

Neuromorphic vision represents a different computing philosophy than the GPU-dominated AI stack:
- Physics-limited latency (speed of light through silicon) vs. frame-rate limited conventional
- Linear energy scaling with scene complexity vs. fixed full-frame energy
- Not yet competitive with CNNs on standard benchmarks — but for applications requiring <1ms latency at <10mW, nothing else comes close

The convergence of improving SNN training algorithms, commercial event cameras, and dedicated neuromorphic chips (Loihi 2, commercial successors) is moving neuromorphic vision from research curiosity to production-viable technology in specific verticals.