What is Probability?
When you flip a fair coin, it has an equal chance of landing on Heads or Tails. Because there are two equally likely outcomes, the chance of Heads is 1 out of 2, or 50%.
Probability is always a number between 0 and 1. A probability of 0 means something is completely impossible (like rolling a 7 on a standard 6-sided die), while a probability of 1 means it is guaranteed to happen.
- Favorable Outcomes: The specific events you are counting or hoping for.
- Sample Space ($S$): The set of all possible outcomes that could occur.
Counting Possibilities: Permutations & Combinations
If you have 3 different shirts and 2 different hats, how many outfits can you make? You multiply $3 imes 2 = 6$ outfits! This is the fundamental counting principle.
When the order of items matters (like a locker passcode), we call it a permutation. When order doesn't matter (like choosing 3 ice cream flavors in a bowl), we call it a combination.
- Factorial ($n!$): Product of all integers down to 1 ($3! = 3 imes 2 imes 1 = 6$).
- Rule of Product: If event A has $m$ outcomes and B has $n$, together they have $m imes n$ outcomes.
The Law of Averages
If you flip a coin 4 times, you might get 4 Heads in a row. Does that mean the coin is broken? No! Short-term streaks happen all the time.
However, if you flip that coin 1,000 times, the percentage of Heads will get closer and closer to exactly 50%. This is the famous Law of Large Numbers.
- Short-Term Fluctuation: Small sample sizes exhibit high random variation.
- Long-Term Convergence: As trials approach infinity, the sample mean converges to true expected value.
Level 1 Completed: Junior Probability & Combinatorics Certificate
Conferred for foundational competence in probability axioms, sample spaces, permutations, and empirical law of large numbers.
Variance & Standard Deviation
Two datasets can have the exact same mean but look completely different. For example, {50, 50, 50} and {0, 50, 100} both average 50, but the second set has huge variability.
Variance ($\sigma^2$) measures the average squared deviation of each data point from the mean. Taking its square root yields standard deviation ($\sigma$), which is expressed in the original physical units of measurement.
- Population Variance ($\sigma^2$): $ rac{1}{N} \sum (x_i - \mu)^2$.
- Bessel's Correction: Dividing by $n-1$ for sample variance ($s^2$) to eliminate downward estimation bias.
The Gaussian (Normal) Distribution
The normal distribution (bell curve) is ubiquitous in nature, describing human heights, measurement errors, and thermal electronic noise in transistors.
It is completely defined by two parameters: its center mean ($\mu$) and spread ($\sigma$). The Empirical 68-95-99.7 Rule states that 68.2% of data falls within $\pm 1\sigma$, 95.4% falls within $\pm 2\sigma$, and 99.7% falls within $\pm 3\sigma$.
- Probability Density Function (PDF): $f(x) = rac{1}{\sigma \sqrt{2\pi}} e^{- rac{1}{2}\left( rac{x-\mu}{\sigma} ight)^2}$.
- 68-95-99.7 Rule: Percentage of probability mass contained within 1, 2, and 3 standard deviations.
Discrete Distributions: Binomial & Poisson
The Binomial distribution models the number of successes in $n$ independent Bernoulli trials with success probability $p$.
When events occur independently at a known constant average rate $\lambda$ per interval of time or space (like radioactive decays, cosmic ray strikes on a chip, or website visits per second), they follow the Poisson distribution.
- Binomial PMF: $P(X=k) = inom{n}{k} p^k (1-p)^{n-k}$.
- Poisson PMF: $P(X=k) = rac{\lambda^k e^{-\lambda}}{k!}$, where mean and variance both equal $\lambda$.
Level 2 Completed: Probability Distributions & Dispersion Analyst
Conferred for competence in sample variance, Gaussian probability densities, the empirical 68-95-99.7 rule, and discrete Poisson arrival processes.
The Central Limit Theorem (CLT)
The Central Limit Theorem is one of the most astonishing theorems in all of mathematics. It states that if you take independent and identically distributed (i.i.d.) random variables from ANY distribution (uniform, skewed, bimodal) with finite variance, the sum or average of those variables will approach a Gaussian normal distribution as sample size $n$ grows!
This explains why the bell curve is everywhere: macroscopic physical quantities are sums of billions of microscopic atomic fluctuations.
- CLT Convergence: $\frac{\bar{X}_n - \mu}{\sigma / \sqrt{n}} \xrightarrow{d} \mathcal{N}(0, 1)$.
- Sample Size Rule of Thumb: $n \ge 30$ is often sufficient for practical normality.
Point Estimation & Desirable Properties
An estimator $\hat{\theta}$ is a rule or formula calculated from sample data to guess an unknown population parameter $\theta$.
We judge estimators by three mathematical properties: Unbiasedness (expected value equals the true parameter), Consistency (converges to truth in probability as $n \to \infty$), and Efficiency (achieves the minimum possible variance, bounded by the Cramér-Rao lower bound).
- Unbiased Estimator: $\text{Bias}(\hat{\theta}) = \mathbb{E}[\hat{\theta}] - \theta = 0$.
- Mean Squared Error: $\text{MSE}(\hat{\theta}) = \text{Var}(\hat{\theta}) + (\text{Bias}(\hat{\theta}))^2$.
Maximum Likelihood Estimation (MLE)
How do we derive optimal formulas for parameters? Ronald Fisher created Maximum Likelihood Estimation (MLE). We ask: Which parameter value makes the observed sample data most probable?
We define the likelihood function $L(\theta) = \prod_{i=1}^n f(x_i \mid \theta)$. In practice, we take the natural logarithm to turn products into sums, obtaining the log-likelihood $\ell(\theta) = \ln L(\theta)$, and find where its derivative equals zero.
- Log-Likelihood: $\ell(\theta) = \sum_{i=1}^n \ln f(x_i \mid \theta)$.
- Score Function: Gradient of the log-likelihood function ($S(\theta) = \nabla_\theta \ell(\theta) = 0$).
Level 3 Completed: Sampling Theory & Maximum Likelihood Specialist
Conferred for mastery of the Central Limit Theorem, standard error scaling, unbiased estimation, and Maximum Likelihood optimization.
The Neyman-Pearson Framework & Power
Jerzy Neyman and Egon Pearson formalized statistical decision theory around two competitive states: the null hypothesis $H_0$ and the alternative hypothesis $H_1$.
Statistical Power ($1 - eta$) is the probability of correctly rejecting the null hypothesis when $H_1$ is true (avoiding a Type II false negative). Power depends on sample size $n$, effect size $d$ (Cohen's $d$), and significance level $lpha$.
- Neyman-Pearson Lemma: Proves that the Likelihood Ratio Test (LRT) is the most powerful test for simple hypotheses.
- Statistical Power ($1 - eta$): Probability of detecting a true effect of specified magnitude.
Chi-Square Tests for Categorical Counts
When data consists of discrete category counts rather than continuous measurements, Pearson's Chi-Square ($\chi^2$) test compares observed frequencies ($O_i$) to expected theoretical frequencies ($E_i$).
Under $H_0$, the test statistic asymptotically follows a $\chi^2$ distribution with $(r-1)(c-1)$ degrees of freedom. It is widely used for Goodness-of-Fit and Contingency Table Independence testing.
- Chi-Square Statistic: $\chi^2 = \sum_{i=1}^k rac{(O_i - E_i)^2}{E_i}$.
- Contingency Table: Cross-tabulation matrix assessing independence between two categorical attributes.
Non-Parametric Rank Tests
Parametric tests (t-tests, ANOVA) assume underlying normal distributions. When data is severely skewed or contains severe outliers, non-parametric rank tests provide robust inference without distributional assumptions.
The Wilcoxon Signed-Rank test evaluates paired ordinal differences, while the Mann-Whitney U test evaluates whether one distribution is stochastically greater than another by ranking all pooled observations.
- Mann-Whitney U Test: Non-parametric counterpart to the two-sample independent t-test.
- Asymptotic Relative Efficiency (ARE): Non-parametric tests retain ~95% efficiency even if normality actually holds.
Level 4 Completed: Mathematical Hypothesis Testing & Experimental Power Engineer
Conferred for proficiency in Neyman-Pearson likelihood ratio testing, Chi-Square contingency tables, non-parametric rank tests, and power calculations.
Bayes' Theorem & Conjugate Pointers
Frequentist statistics treats parameters as fixed unknown constants and data as random. Bayesian statistics treats observed data as fixed and parameters $\theta$ as random variables described by probability distributions.
Bayes' theorem updates our prior belief $p(\theta)$ in light of new evidence $p(X \mid \theta)$ to produce the posterior distribution $p(\theta \mid X) \propto p(X \mid \theta) p(\theta)$. When prior and posterior belong to the same probability family, the prior is called conjugate (e.g. Beta-Binomial, Normal-Normal).
- Posterior Distribution: $p(\theta \mid X) = \frac{p(X \mid \theta) p(\theta)}{\int p(X \mid \theta') p(\theta') d\theta'}$.
- Credible Interval: An interval $[a, b]$ containing the parameter with exact subjective probability (e.g. $P(\theta \in [a, b] \mid X) = 0.95$).
Markov Chain Monte Carlo (MCMC)
In multi-parameter models, computing the marginal likelihood evidence integral $p(X) = \int p(X \mid \theta) p(\theta) d\theta$ is analytically impossible. MCMC bypasses this normalization denominator by drawing samples directly from the unnormalized posterior.
A Markov chain is designed whose stationary equilibrium distribution is exactly the target posterior distribution $p(\theta \mid X)$.
- Stationary Distribution: The equilibrium probability distribution preserved by Markov transition kernel.
- Burn-In Period: Initial iterations discarded while the Markov chain travels from starting state to target distribution.
Metropolis-Hastings & Gibbs Sampling
The Metropolis-Hastings algorithm proposes candidate state $\theta^*$ from proposal $q(\theta^* \mid \theta)$ and accepts it with probability $\alpha = \min\left(1, \frac{p(X \mid \theta^*)p(\theta^*) q(\theta \mid \theta^*)}{p(X \mid \theta)p(\theta) q(\theta^* \mid \theta)}\right)$.
Gibbs sampling is a special case: when conditional distributions $p(\theta_i \mid \theta_{-i}, X)$ are known, each parameter is sampled sequentially from its exact conditional, yielding an acceptance probability of identically 1.
- Acceptance Probability ($lpha$): Decides whether to step to candidate state or stay at current state.
- Hamiltonian Monte Carlo (HMC): Uses gradient physics to propose distant states with high acceptance rates (implemented in Stan/PyMC).
Level 5 Completed: Bayesian Computational Statistics & MCMC Specialist
Conferred for mastery of conjugate priors, posterior density derivation, MCMC convergence diagnostics, Metropolis-Hastings, and Gibbs sampling.
The Multivariate Normal & Mahalanobis Distance
When modeling $p$ correlated variables simultaneously, the scalar variance is replaced by a $p imes p$ symmetric positive semi-definite Covariance Matrix $\Sigma$.
Standard Euclidean distance fails because it ignores variable correlations and differing scales. Mahalanobis distance measures distance between a point $\mathbf{x}$ and mean $oldsymbol{\mu}$ in standard deviation units along principal covariance axes.
- Covariance Matrix: $\Sigma = \mathbb{E}[(\mathbf{X} - oldsymbol{\mu})(\mathbf{X} - oldsymbol{\mu})^T]$.
- Mahalanobis Distance: $D_M(\mathbf{x}) = \sqrt{(\mathbf{x} - oldsymbol{\mu})^T \Sigma^{-1} (\mathbf{x} - oldsymbol{\mu})}$.
Stationary Time Series & ARIMA Models
Time series data violates independence assumptions because sequential observations exhibit temporal autocorrelation. Weak (covariance) stationarity requires constant mean, constant variance, and autocovariance that depends solely on time lag $k$.
AutoRegressive Integrated Moving Average (ARIMA($p, d, q$)) models difference non-stationary series $d$ times, combining $p$ autoregressive lags with $q$ lagged moving-average white noise forecast errors.
- Autocorrelation Function (ACF): Correlation between $X_t$ and $X_{t-k}$.
- Dickey-Fuller Test: Tests for presence of a unit root (non-stationarity).
Heteroskedasticity & GARCH Volatility
In financial time series and high-speed telemetry, variance is not constant over time: tranquil periods alternate with bursts of high volatility (volatility clustering).
Engle and Bollerslev introduced Generalized AutoRegressive Conditional Heteroskedasticity (GARCH($p, q$)), modeling conditional variance $\sigma_t^2$ as a dynamic linear combination of past squared residuals and past variances.
- ARCH Effect: Autoregressive conditional variance driven by shocks.
- GARCH(1,1): $\sigma_t^2 = \omega + lpha \epsilon_{t-1}^2 + eta \sigma_{t-1}^2$ (stationary if $lpha + eta < 1$).
Level 6 Completed: Multivariate Analysis & Time Series Scientist
Conferred for advanced research mastery of multivariate Gaussian geometry, Mahalanobis distances, ARIMA forecasting, and GARCH volatility modeling.
High-Dimensionality & Marchenko-Pastur Law
When dimension $p$ grows proportionally with sample size $n$ ($p/n o \gamma \in (0, \infty)$), classical asymptotic statistics completely breaks down. The sample covariance matrix eigenvalues no longer converge to population values.
Random Matrix Theory and the Marchenko-Pastur law describe the asymptotic spectral distribution of singular values, allowing statisticians to filter true signal eigenvalues from pure high-dimensional noise bulk.
- Marchenko-Pastur Law: Bounds the spectral support of noise eigenvalues between $\lambda_\pm = \sigma^2(1 \pm \sqrt{\gamma})^2$.
- Curse of Dimensionality: High-dimensional unit spheres concentrate all their volume in an infinitely thin shell near the surface.
Information Theory & Entropy
Claude Shannon founded information theory by quantifying uncertainty with Shannon Entropy: $H(X) = -\sum p(x) \log_2 p(x)$. Entropy measures the minimum expected number of bits required to encode outcomes.
Mutual Information $I(X; Y) = H(X) - H(X \mid Y)$ measures non-linear statistical dependence between variables without assuming linear correlation, providing the foundational metric for modern feature selection and deep neural bottlenecks.
- Kullback-Leibler (KL) Divergence: Non-symmetric measure of information loss when approximating true distribution $P$ with $Q$.
- Information Bottleneck Principle: $\min_{\hat{X}} [I(X; \hat{X}) - eta I(\hat{X}; Y)]$.
Causal Inference & Judea Pearl's Do-Calculus
Observational data can never distinguish correlation from causation without structural assumptions. Judea Pearl developed the Structural Causal Model (SCM) and Directed Acyclic Graph (DAG) framework to elevate statistics to causality.
The $do$-operator, $P(Y \mid do(X = x))$, represents an active physical intervention rather than passive observation. The Backdoor Criterion and Pearl's 3 rules of $do$-calculus prove when causal effects can be identified purely from non-interventional observational data.
- Backdoor Criterion: De-confounding an observational study by conditioning on a set of variables blocking all backdoor paths.
- Counterfactual Equation: $Y_x(u) = Y_{M_x}(u)$ (evaluating what would have occurred had an alternative action been taken).
Level 7 Completed: Distinguished Mathematical Statistics & Causal Inference Fellow
Conferred for lifetime visionary contributions to mathematical probability, high-dimensional random matrix theory, information entropy, and structural causal do-calculus.