State Space Models Mamba
# State Space Models & Mamba
## Introduction & Motivation
State Space Models: linear recurrent neural networks. Efficient sequence processing. Applications: long sequences, efficient inference.
Motivation: Combine efficiency of linear models with sequence capability.
Applications: Long sequences, efficient processing.
---
## Core Concepts & Theory
### Linear RNN
Recurrent computation in linear space.
### State Equation
dx/dt = Ax + Bu.
### Output Equation
y = Cx + Du.
### Discretization
Convert continuous to discrete.
---
## Mathematical Formulation
Continuous SSM:
$$\frac{dx}{dt} = Ax(t) + Bu(t)$$
$$y(t) = Cx(t) + Du(t)$$
Discrete:
$$x_k = \bar{A} x_{k-1} + \bar{B} u_k$$
---
## Advanced Theory & Extensions
### Structured Matrices
Parameterization strategies.
### Selective State Spaces
Learn which to update.
### Hardware Efficiency
Implement on GPUs/TPUs.
---
## Computational Considerations
Inference: O(L).
Training: O(L log L) with FFT.
Memory: O(L) not O(L²).
---
## Practical Implementation Strategies
### Initialization
Careful matrix setup.
### Discretization Method
Bilinear transform.
### Layer Stacking
Cascade SSM layers.
---
## Benchmark Datasets & Evaluation
Long Range Arena: Sequence tasks.
Language Modeling: Perplexity.
Speed: Inference latency.
---
## Key Challenges & Limitations
### Hyperparameter Tuning
Matrix A parameterization.
### Hardware Support
Limited implementations.
### Theoretical Understanding
Ongoing research.
---
## Hyperparameter Tuning
State dimension: 64-256.
Discretization: Bilinear, ZOH.
Learning rate: 1e-4 to 1e-3.
---
## Real-World Applications & Case Studies
Long Documents: Process full papers.
DNA Sequences: Handle long biology.
Time Series: Efficient forecasting.
---
## Integration with Other Methods
SSM + transformers for hybrid; + attention layers.
---
## Summary & Key Takeaways
State Space Models enable efficient sequence processing.
Principles:
1. Linear recurrence: Efficient.
2. State equation: Ax + Bu.
3. Discretization: Bilinear transform.
4. Efficiency: O(L) complexity.
5. Scalability: Long sequences.
---
## Appendix: Practical Labs
### Lab 1: Discretize SSM
import numpy as np
def discretize_ssm(A, B, dt=0.001):
"""Convert continuous SSM to discrete"""
# Bilinear transform
I = np.eye(A.shape[0])
A_discrete = (I + dt/2 * A) @ np.linalg.inv(I - dt/2 * A)
B_discrete = np.linalg.inv(I - dt/2 * A) @ B * dt
return A_discrete, B_discrete
np.random.seed(42)
A = np.random.randn(4, 4)
B = np.random.randn(4, 1)
A_d, B_d = discretize_ssm(A, B)
assert A_d.shape == A.shape
print("✓ SSM discretization working")### Lab 2: Forward Pass
import numpy as np
def ssm_forward(u, A, B, C, D):
"""Forward pass through SSM"""
state = np.zeros(A.shape[0])
outputs = []
for u_t in u:
state = A @ state + B @ u_t.reshape(-1, 1)
y_t = C @ state + D @ u_t.reshape(-1, 1)
outputs.append(y_t.squeeze())
return np.array(outputs)
np.random.seed(42)
u = np.random.randn(10, 1)
A = np.random.randn(4, 4)
B = np.random.randn(4, 1)
C = np.random.randn(1, 4)
D = np.zeros((1, 1))
y = ssm_forward(u, A, B, C, D)
assert y.shape[0] == 10
print("✓ SSM forward pass working")### Lab 3: Selective State Update
import numpy as np
def selective_ssm(u, A, B, C, D, gate_fn):
"""SSM with selective state updates"""
state = np.zeros(A.shape[0])
outputs = []
for u_t in u:
# Gating decides update
gate = gate_fn(u_t)
if gate > 0.5:
state = A @ state + B @ u_t.reshape(-1, 1)
y_t = C @ state + D @ u_t.reshape(-1, 1)
outputs.append(y_t.squeeze())
return np.array(outputs)
np.random.seed(42)
u = np.random.randn(10, 1)
A = np.random.randn(4, 4)
B = np.random.randn(4, 1)
C = np.random.randn(1, 4)
D = np.zeros((1, 1))
gate = lambda x: np.random.rand()
y = selective_ssm(u, A, B, C, D, gate)
assert y.shape[0] == 10
print("✓ Selective SSM working")### Lab 4: Efficiency Comparison
def compare_ssm_transformer(seq_len, d_model, num_heads=8):
"""Compare SSM vs transformer complexity"""
# Transformer
transformer_flops = seq_len * seq_len * d_model
# SSM
ssm_flops = seq_len * d_model * d_model
speedup = transformer_flops / ssm_flops
return speedup
speedup = compare_ssm_transformer(1024, 768)
assert speedup > 1
print(f"✓ SSM speedup: {speedup:.1f}x")---