Graph Neural Networks for Engineering Systems
# Graph Neural Networks for Engineering Systems
## Introduction & Motivation
Many engineering systems naturally represent as graphs: molecular structures, process flowcharts, power grids, supply chains. Graph neural networks (GNNs) leverage this structure for predictions, optimizations, and simulations on relational data, enabling advances in materials discovery, process design, and system analysis.
Motivation: Apply GNNs to graph-structured engineering data.
Applications: Molecular property prediction, network optimization, chemical structure analysis, process flow optimization.
---
## Core Concepts & Theory
### Graph Structure
Nodes, edges, and attributes.
### Message Passing
Neighborhood aggregation.
### Node Embeddings
Learned representations.
### Graph Pooling
Global graph representations.
---
## Mathematical Formulation
Message Passing:
$$\mathbf{h}_v^{(k+1)} = ext{Update}(\mathbf{h}_v^{(k)}, ext{Aggregate}(\{\mathbf{h}_u^{(k)} : u \in N(v)\}))$$
Graph Convolution:
$$Z^{(k+1)} = \sigma(\hat{D}^{-1/2}\hat{A}\hat{D}^{-1/2}Z^{(k)}W^{(k)})$$
Graph Pooling:
$$\mathbf{s} = ext{ReadOut}(\{\mathbf{h}_v : v \in V\})$$
---
## Advanced Theory & Extensions
### Attention Mechanisms
Learnable neighbor weighting.
### Heterogeneous Graphs
Multi-type nodes and edges.
### Temporal Graphs
Dynamic edge/node changes.
---
## Computational Considerations
Message Passing: O(|E|·D) per layer for D features.
Training: O(N·L·D²) for N nodes, L layers.
Inference: O(|E|·D) per forward pass.
---
## Practical Implementation Strategies
### Graph Construction
Creating graph representations.
### Feature Encoding
Node and edge features.
### Normalization
Adjacency matrix preprocessing.
---
## Benchmark Datasets & Evaluation
OGB: Open Graph Benchmark.
Molecular Graphs: Chemical databases.
Citation Networks: Publication relationships.
---
## Key Challenges & Limitations
### Over-smoothing
Information loss with depth.
### Scalability
Large graph processing.
### Graph Heterogeneity
Mixed node/edge types.
---
## Hyperparameter Tuning
Hidden dimensions: 32-256.
Number of layers: 2-4.
Dropout rate: 0.0-0.5.
---
## Real-World Applications & Case Studies
Molecule Design: Property prediction.
Protein Folding: Structure prediction.
Chemical Reactions: Outcome prediction.
---
## Integration with Other Methods
GNNs + physics constraints; + molecular descriptors; + optimization.
---
## Summary & Key Takeaways
GNNs enable learning on graph-structured data.
Principles:
1. Structure: Leverage graph topology.
2. Message Passing: Aggregate neighbor information.
3. Learning: Train end-to-end representations.
4. Pooling: Create graph-level predictions.
5. Application: Solve domain problems.
---
## Appendix: Practical Labs
### Lab 1: Basic Graph Construction
import numpy as np
class GraphRepresentation:
def __init__(self, n_nodes):
self.n_nodes = n_nodes
self.adj_matrix = np.zeros((n_nodes, n_nodes))
self.node_features = np.random.randn(n_nodes, 10)
def add_edge(self, i, j, weight=1.0):
"""Add edge between nodes i and j"""
self.adj_matrix[i, j] = weight
self.adj_matrix[j, i] = weight # Undirected
def neighbors(self, node_id):
"""Get neighbors of a node"""
return np.where(self.adj_matrix[node_id] > 0)[0]
def degree(self, node_id):
"""Get degree of a node"""
return np.sum(self.adj_matrix[node_id] > 0)
# Create graph
graph = GraphRepresentation(n_nodes=5)
graph.add_edge(0, 1)
graph.add_edge(1, 2)
graph.add_edge(2, 3)
graph.add_edge(3, 4)
graph.add_edge(4, 0)
print(f"✓ Graph created:")
print(f" Adjacency matrix shape: {graph.adj_matrix.shape}")
print(f" Node 0 neighbors: {graph.neighbors(0)}")
print(f" Node degrees: {[graph.degree(i) for i in range(5)]}")### Lab 2: Message Passing Layer
import numpy as np
class MessagePassingLayer:
def __init__(self, input_dim, output_dim):
self.input_dim = input_dim
self.output_dim = output_dim
# Learnable weights
self.W_self = np.random.randn(input_dim, output_dim) * 0.1
self.W_neighbor = np.random.randn(input_dim, output_dim) * 0.1
def forward(self, node_features, adj_matrix):
"""Forward pass"""
n_nodes = len(node_features)
# Aggregate neighbor features
neighbor_sum = adj_matrix @ node_features # [N, D]
# Self transformation
self_transformed = node_features @ self.W_self
# Neighbor transformation
neighbor_transformed = neighbor_sum @ self.W_neighbor
# Combine
output = self_transformed + neighbor_transformed
output = np.tanh(output) # Activation
return output
# Test
node_features = np.random.randn(5, 10)
adj_matrix = np.array([
[0, 1, 0, 0, 1],
[1, 0, 1, 0, 0],
[0, 1, 0, 1, 0],
[0, 0, 1, 0, 1],
[1, 0, 0, 1, 0]
], dtype=float)
layer = MessagePassingLayer(input_dim=10, output_dim=8)
output = layer.forward(node_features, adj_matrix)
print(f"✓ Message passing layer:")
print(f" Input shape: {node_features.shape}")
print(f" Output shape: {output.shape}")### Lab 3: Graph Classification
import numpy as np
class GraphClassifier:
def __init__(self, hidden_dim=16):
self.hidden_dim = hidden_dim
self.conv_weights = np.random.randn(10, hidden_dim) * 0.1
self.pool_weights = np.random.randn(hidden_dim, 2) * 0.1
def forward(self, node_features, adj_matrix):
"""Classify entire graph"""
# Graph convolution
hidden = np.tanh(node_features @ self.conv_weights)
# Attention-based pooling
attention = np.softmax(hidden @ np.ones((self.hidden_dim, 1)), axis=0)
graph_embedding = np.sum(hidden * attention, axis=0)
# Classification
logits = graph_embedding @ self.pool_weights
probabilities = 1 / (1 + np.exp(-logits)) # Sigmoid
return probabilities
# Test on multiple graphs
classifier = GraphClassifier(hidden_dim=16)
graphs = [
(np.random.randn(5, 10), np.random.randint(0, 2, (5, 5))),
(np.random.randn(4, 10), np.random.randint(0, 2, (4, 4))),
(np.random.randn(6, 10), np.random.randint(0, 2, (6, 6)))
]
print(f"✓ Graph classification:")
for i, (features, adj) in enumerate(graphs):
prob = classifier.forward(features, adj)
print(f" Graph {i}: Class probabilities: [{prob[0]:.3f}, {prob[1]:.3f}]")### Lab 4: Molecular Property Prediction
import numpy as np
class MolecularGNN:
def __init__(self, atom_feature_dim=11, hidden_dim=64):
self.atom_feat_dim = atom_feature_dim
self.hidden_dim = hidden_dim
# GNN weights
self.atom_embedding = np.random.randn(atom_feature_dim, hidden_dim) * 0.1
self.bond_transform = np.random.randn(hidden_dim, hidden_dim) * 0.1
# Property prediction head
self.property_head = np.random.randn(hidden_dim, 1) * 0.01
def encode_atoms(self, atomic_numbers, formal_charges):
"""Encode atom features"""
features = np.zeros((len(atomic_numbers), self.atom_feat_dim))
for i, (atom, charge) in enumerate(zip(atomic_numbers, formal_charges)):
features[i, 0] = atom / 100 # Normalize atomic number
features[i, 1] = charge
features[i, 2:] = np.random.randn(self.atom_feat_dim - 2) * 0.1
return features
def message_passing_step(self, atom_features, bond_matrix):
"""One GNN step"""
# Node update
node_messages = bond_matrix @ atom_features @ self.bond_transform
atom_features = np.tanh(atom_features @ self.atom_embedding + node_messages)
return atom_features
def predict_property(self, atomic_numbers, formal_charges, bond_matrix, n_steps=3):
"""Predict molecular property"""
atom_features = self.encode_atoms(atomic_numbers, formal_charges)
# Multiple message passing steps
for _ in range(n_steps):
atom_features = self.message_passing_step(atom_features, bond_matrix)
# Graph pooling and prediction
graph_representation = np.mean(atom_features, axis=0)
property_value = graph_representation @ self.property_head
return property_value[0]
# Test on molecules
gnn = MolecularGNN(atom_feature_dim=11, hidden_dim=64)
# Molecule 1: Simple
atoms1 = np.array([1, 6, 8]) # H, C, O
charges1 = np.array([0, 0, 0])
bonds1 = np.array([[0, 1, 0], [1, 0, 1], [0, 1, 0]], dtype=float)
prop1 = gnn.predict_property(atoms1, charges1, bonds1)
print(f"✓ Molecular property prediction:")
print(f" Molecule 1 property: {prop1:.3f}")
# Molecule 2: Different
atoms2 = np.array([6, 6, 8, 8])
charges2 = np.array([0, 0, 0, -1])
bonds2 = np.random.rand(4, 4) > 0.5
bonds2 = bonds2.astype(float)
prop2 = gnn.predict_property(atoms2, charges2, bonds2)
print(f" Molecule 2 property: {prop2:.3f}")---