Spectroscopy Analysis with Machine Learning
# Spectroscopy Analysis with Machine Learning
## Introduction & Motivation
Spectroscopic data contains rich information about material composition and structure. ML models accelerate analysis of spectroscopy (NMR, XRD, IR, UV-Vis, Raman) for rapid material identification, quality control, and process monitoring in pharmaceutical, chemical, and materials industries.
Motivation: Rapid spectroscopy interpretation using ML models.
Applications: Material identification, contamination detection, structural analysis, process monitoring.
---
## Core Concepts & Theory
### Spectral Features
Peak positions, intensities, and linewidths.
### Fingerprint Regions
Material-specific spectral signatures.
### Peak Assignment
Functional group and molecular structure identification.
### Signal Noise
Measurement uncertainty and baseline effects.
---
## Mathematical Formulation
Spectrum Model:
$$I(\lambda) = \sum_i A_i g(\lambda - \lambda_i, \sigma_i) + B(\lambda) + \epsilon$$
Baseline Correction:
$$I_{corrected}(\lambda) = I_{raw}(\lambda) - B(\lambda)$$
Peak Detection:
$$ ext{peaks} = \{\lambda : dI/d\lambda = 0, d²I/d\lambda² < 0\}$$
---
## Advanced Theory & Extensions
### 2D Spectroscopy
Correlation techniques and correlation maps.
### Time-Resolved Spectroscopy
Kinetic measurements and dynamics.
### Hyperspectral Analysis
Spatial and spectral information integration.
---
## Computational Considerations
Spectral Processing: O(N) for N wavelengths.
Feature Extraction: O(N·F) for F features.
Model Training: O(M·N²) for M spectra.
---
## Practical Implementation Strategies
### Data Preprocessing
Baseline subtraction and normalization.
### Feature Engineering
Peak statistics and spectral moments.
### Classification**
Material type and contamination detection.
---
## Benchmark Datasets & Evaluation
NIST Spectral Database: Reference spectra.
SDBS: Spectroscopy databases.
Research Datasets: Laboratory spectroscopy.
---
## Key Challenges & Limitations
### Spectral Variability
Measurement conditions and instrument variation.
### Peak Overlap
Overlapping features and resolution limits.
### Quantification Accuracy
Concentration determination uncertainty.
---
## Hyperparameter Tuning
Peak detection threshold: 0.01-0.1 (normalized).
Smoothing window: 3-11 points.
Feature normalization: L2 or L∞.
---
## Real-World Applications & Case Studies
Pharmaceutical QC: API and excipient identification.
Environmental Monitoring: Pollutant detection.
Metabolomics: Biomarker identification.
---
## Integration with Other Methods
Spectroscopy ML + chemometrics; + signal processing; + domain expertise.
---
## Summary & Key Takeaways
ML accelerates spectroscopy analysis and interpretation.
Principles:
1. Preprocessing: Clean and normalize spectra.
2. Features: Extract meaningful spectral characteristics.
3. Classification: Identify materials and anomalies.
4. Quantification: Predict concentrations.
5. Validation: Ground truth comparison.
---
## Appendix: Practical Labs
### Lab 1: Spectrum Preprocessing
import numpy as np
def baseline_correction_polynomial(wavelengths, intensities, poly_order=3):
"""Remove baseline using polynomial fitting"""
# Fit polynomial to intensity data
coeffs = np.polyfit(wavelengths, intensities, poly_order)
baseline = np.polyval(coeffs, wavelengths)
corrected = intensities - baseline
corrected = np.maximum(corrected, 0) # Ensure non-negative
return corrected, baseline
def normalize_spectrum(intensities, method='l2'):
"""Normalize spectrum"""
if method == 'l2':
norm = np.linalg.norm(intensities)
return intensities / (norm + 1e-10)
elif method == 'max':
return intensities / (np.max(intensities) + 1e-10)
elif method == 'area':
area = np.trapz(intensities)
return intensities / (area + 1e-10)
else:
return intensities
def smooth_spectrum(intensities, window_size=5):
"""Smooth spectrum using moving average"""
kernel = np.ones(window_size) / window_size
smoothed = np.convolve(intensities, kernel, mode='same')
return smoothed
# Test
wavelengths = np.linspace(400, 800, 100)
# Simulate spectrum with peaks and noise
intensities = (100 * np.exp(-((wavelengths - 500)**2) / 5000) +
50 * np.exp(-((wavelengths - 650)**2) / 4000) +
np.random.randn(100) * 5)
corrected, baseline = baseline_correction_polynomial(wavelengths, intensities, poly_order=2)
smoothed = smooth_spectrum(corrected, window_size=5)
normalized = normalize_spectrum(smoothed)
print(f"✓ Spectrum preprocessing complete")
print(f" Original max: {intensities.max():.1f}")
print(f" Corrected max: {corrected.max():.1f}")
print(f" Normalized integral: {np.trapz(normalized):.3f}")### Lab 2: Peak Detection
import numpy as np
class PeakDetector:
def __init__(self, threshold=0.05, min_distance=5):
self.threshold = threshold
self.min_distance = min_distance
def find_peaks(self, spectrum):
"""Detect peaks in spectrum"""
peaks = []
# Compute first derivative
derivative = np.diff(spectrum)
# Find zero crossings (peaks)
for i in range(1, len(derivative)-1):
if derivative[i-1] > 0 and derivative[i] < 0:
# Local maximum
if spectrum[i] > self.threshold * np.max(spectrum):
peaks.append(i)
# Apply minimum distance constraint
filtered_peaks = []
for peak in peaks:
if not filtered_peaks or peak - filtered_peaks[-1] > self.min_distance:
filtered_peaks.append(peak)
return filtered_peaks
def peak_properties(self, spectrum, peak_indices):
"""Compute peak properties"""
properties = []
for idx in peak_indices:
# Position, intensity, width (simplified)
position = idx
intensity = spectrum[idx]
# Width at half height (simplified)
half_height = intensity / 2
width = 1
for i in range(idx-1, -1, -1):
if spectrum[i] < half_height:
width = 2 * (idx - i)
break
properties.append({
'position': position,
'intensity': intensity,
'width': width,
'area': intensity * width
})
return properties
# Test
spectrum = 100*np.exp(-((np.arange(100)-30)**2)/100) + 80*np.exp(-((np.arange(100)-70)**2)/100)
spectrum += np.random.randn(100) * 3
detector = PeakDetector(threshold=0.1, min_distance=5)
peaks = detector.find_peaks(spectrum)
properties = detector.peak_properties(spectrum, peaks)
print(f"✓ Detected {len(peaks)} peaks")
for i, prop in enumerate(properties):
print(f" Peak {i}: position={prop['position']}, intensity={prop['intensity']:.1f}")### Lab 3: Spectral Classification
import numpy as np
class SpectralClassifier:
def __init__(self, n_materials=3, n_features=10):
self.n_materials = n_materials
self.n_features = n_features
# Template spectra for each material
self.templates = np.random.randn(n_materials, n_features)
def extract_features(self, spectrum):
"""Extract features from spectrum"""
features = np.zeros(self.n_features)
# Simple features: moments
features[0] = np.mean(spectrum)
features[1] = np.std(spectrum)
features[2] = np.max(spectrum)
features[3] = np.min(spectrum)
features[4] = np.median(spectrum)
# Higher order moments
features[5] = np.sum(spectrum ** 2)
features[6] = np.sum(spectrum ** 3)
# Gradient features
grad = np.diff(spectrum)
features[7] = np.mean(np.abs(grad))
features[8] = np.max(np.abs(grad))
return features
def classify(self, spectrum):
"""Classify material from spectrum"""
features = self.extract_features(spectrum)
# Compute distances to templates
distances = np.linalg.norm(self.templates - features, axis=1)
# Classify as nearest template
material_class = np.argmin(distances)
confidence = 1.0 / (1.0 + distances[material_class])
return material_class, confidence
def batch_classification(self, spectra):
"""Classify multiple spectra"""
results = []
for spectrum in spectra:
mat_class, conf = self.classify(spectrum)
results.append((mat_class, conf))
return results
classifier = SpectralClassifier(n_materials=3, n_features=10)
# Test spectra
test_spectra = [np.random.randn(50) for _ in range(5)]
results = classifier.batch_classification(test_spectra)
print(f"✓ Classified {len(results)} spectra")
for i, (mat_class, conf) in enumerate(results):
print(f" Spectrum {i}: Material {mat_class}, confidence {conf:.2f}")### Lab 4: Integrated Spectroscopy Analysis System
import numpy as np
class SpectrometryAnalysisSystem:
def __init__(self, n_reference_spectra=50):
self.n_ref = n_reference_spectra
# Reference database
self.reference_spectra = np.random.randn(n_reference_spectra, 100)
self.reference_labels = np.random.randint(0, 5, n_reference_spectra)
def process_raw_spectrum(self, raw_spectrum):
"""Full preprocessing pipeline"""
# Baseline correction
coeffs = np.polyfit(np.arange(len(raw_spectrum)), raw_spectrum, 2)
baseline = np.polyval(coeffs, np.arange(len(raw_spectrum)))
corrected = raw_spectrum - baseline
corrected = np.maximum(corrected, 0)
# Smoothing
window = 5
smoothed = np.convolve(corrected, np.ones(window)/window, mode='same')
# Normalization
normalized = smoothed / (np.max(smoothed) + 1e-10)
return normalized
def identify_material(self, sample_spectrum):
"""Identify material using spectral matching"""
processed = self.process_raw_spectrum(sample_spectrum)
# Compare to references
similarities = []
for ref in self.reference_spectra:
# Cosine similarity
dot = np.dot(processed, ref)
similarity = dot / (np.linalg.norm(processed) * np.linalg.norm(ref) + 1e-10)
similarities.append(similarity)
similarities = np.array(similarities)
best_match_idx = np.argmax(similarities)
best_similarity = similarities[best_match_idx]
identified_material = self.reference_labels[best_match_idx]
return identified_material, best_similarity
def quality_check(self, sample_spectrum, tolerance=0.15):
"""Quality control: check for contaminants"""
processed = self.process_raw_spectrum(sample_spectrum)
# Find peaks
grad = np.diff(processed)
peaks = []
for i in range(1, len(grad)-1):
if grad[i-1] > 0 and grad[i] < 0:
peaks.append(i)
# Check for unexpected peaks
anomalies = []
for peak_idx in peaks:
if processed[peak_idx] > tolerance:
anomalies.append(peak_idx)
quality_flag = 'PASS' if len(anomalies) <= 2 else 'FAIL'
return quality_flag, len(anomalies)
system = SpectrometryAnalysisSystem(n_reference_spectra=50)
# Test
sample = np.random.randn(100)
material, similarity = system.identify_material(sample)
quality, n_anomalies = system.quality_check(sample)
print(f"✓ Identified material: {material}, similarity: {similarity:.2f}")
print(f"✓ Quality control: {quality}, anomalies: {n_anomalies}")---