Data Analysis

Data analysis is the process of inspecting, cleaning, and modeling data to surface conclusions that support a decision, and it sits one step narrower than data science: where data science covers the full discipline — collection, engineering, modeling, communication — data analysis is specifically the reasoning step that turns prepared data into an answer. John Tukey's distinction, laid out across his career and crystallized in his 1977 book Exploratory Data Analysis, is the field's foundational reference point: Tukey separated exploratory data analysis, which looks at data without a fixed hypothesis to find patterns worth investigating, from confirmatory data analysis, which tests a specific hypothesis using the formal machinery of statistical inference. Most real analysis work moves between the two — explore first to find what's there, then confirm formally before trusting a conclusion enough to act on it.

The core discipline of data analysis is asking what a number is actually measuring before asking what it means. A rising average order value could mean customers are buying more, or it could mean a change in the customer mix — fewer small orders, not more large ones — and those two explanations call for opposite business responses. This is why analysts spend disproportionate time on definitions: what counts as a "user," what window a metric is computed over, whether an outlier should be excluded or investigated. Getting the definition wrong produces an analysis that is internally consistent and completely misleading, which is a harder failure to catch than a calculation error because nothing about the output looks obviously broken.

Quantitative and qualitative analysis answer different kinds of questions, and conflating them is a common source of overreach. Quantitative analysis works on numerical data — counts, measurements, rates — and produces conclusions a statistical test can validate: whether a difference is larger than chance would produce. Qualitative analysis works on non-numerical material — interview transcripts, open-ended survey responses, support tickets — and produces conclusions about meaning and category rather than magnitude: what themes recur, what language customers use to describe a problem. Neither substitutes for the other; a quantitative result can tell you churn rose 4 percentage points without telling you why, and a qualitative read of exit interviews can surface the why without telling you how widespread it is — which is why mature analysis functions routinely pair both on the same question rather than treating them as competing methods.

The most common technical failure in data analysis is not a wrong formula but an unexamined assumption about the data's structure. Analyzing a non-random sample as though it were representative, treating correlated observations as independent, or comparing two time periods without accounting for a known external event (a product launch, a pricing change, a seasonal cycle) all produce numerically correct calculations that answer a question the data can't actually support. This is distinct from — and arguably harder to catch than — a coding bug, because the output is plausible-looking and internally consistent; it fails a domain-knowledge check, not a syntax check, which is exactly why Conway's "substantive expertise" leg of data science matters as much to analysis as the statistical leg does.

Confirmation bias is the specific cognitive failure data analysis methodology is built to guard against: the tendency to notice evidence that supports a belief already held and discount evidence that doesn't. Pre-registering a hypothesis before looking at the data — common practice in clinical trials and increasingly adopted in applied analytics — forces the analyst to commit to what counts as confirming or disconfirming evidence before the data can influence that judgment. Blind or masked analysis, where the analyst doesn't know which group is the treatment and which is the control until after the analysis is run, serves the same purpose. Both practices exist because data analysis gives a skilled analyst enough flexibility in how to define a metric, choose a comparison window, or handle outliers that an analyst motivated to find a particular result usually can.

Modern data analysis tooling has consolidated around a small set of standard environments — spreadsheets for small-scale or ad hoc work, SQL for querying structured data at scale, and Python or R with their associated statistical and visualization libraries for more involved analysis — but the tool choice is secondary to the discipline it's used inside. The same confirmatory-versus-exploratory distinction, the same scrutiny of what a metric actually measures, and the same guard against confirmation bias apply whether the analysis runs in a spreadsheet or a notebook; the tooling has gotten faster and more scalable over the decades since Tukey's original work, but the reasoning discipline underneath it has not materially changed.

Read data analysis as the reasoning layer that sits inside the larger practice of data science: Tukey's split between exploring data without a hypothesis and confirming one formally gives the field its basic workflow, and the recurring failure modes — unexamined assumptions about structure, confirmation bias, a metric that doesn't measure what it's assumed to measure — are failures of judgment about the data's meaning, not failures of the tool used to compute the number.