Semi-Structured Data

Semi-structured data sits between structured and unstructured: it has organizational markers — tags, keys, nested fields — but no fixed, uniform schema enforced across every record, so unlike a relational database table where every row shares exactly the same columns, two semi-structured records can have different fields, different nesting depth, or optional attributes present in one and absent in another. JSON, XML, and YAML are the dominant formats in this category, and so, less obviously, is a CSV file in practice — it looks tabular but typically has no enforced types or constraints, which is why CSV is more accurately classified as semi-structured despite its superficial row-and-column appearance.

The defining practical feature of semi-structured data is self-description: each record carries its own field names inline rather than relying on an external schema definition the way a database table does, which is what allows the structure to vary from record to record without anything breaking. A JSON object for one customer might include a "loyalty_tier" field while another customer's record omits it entirely, and both are perfectly valid — nothing forces every record in a JSON dataset to share the same set of keys, in contrast to a SQL table, where every row is schema-bound to the same fixed columns whether or not a given row actually needs all of them. This flexibility is the entire value proposition: it accommodates records that genuinely vary in shape (a product catalog where different product categories have different relevant attributes) without the structured-data problem of mostly-empty optional columns.

NoSQL document databases, most notably MongoDB, were built specifically to store semi-structured data natively — persisting JSON-like documents without requiring every document in a collection to share an identical schema — a deliberate departure from the relational model Edgar F. Codd proposed in 1970, which requires exactly that uniformity. This trades consistency guarantees for flexibility: a document database won't reject a record for having an unexpected field or missing an expected one the way a strict relational schema would, which is useful when the underlying real-world entities genuinely vary in shape, but pushes validation responsibility onto application code rather than the database engine, a tradeoff that became central to the "schema-on-read versus schema-on-write" debate in data architecture through the 2010s.

APIs are the other major source of semi-structured data in practice, because most modern web APIs return JSON responses, and JSON's nested, self-describing structure naturally accommodates the kind of variable, hierarchical data that APIs tend to expose — a user object with a nested array of addresses, each address itself a nested object with its own fields. Consuming this data for structured analysis typically requires a flattening step — unpacking nested arrays and objects into the flat rows and columns a relational database or a dataframe expects — and that flattening is itself a common source of data-quality problems, since a decision has to be made about how to handle records where a nested field is missing, duplicated across multiple rows, or nested more deeply in some records than others.

Schema-on-read, the practice of applying structure to semi-structured data at the moment it's queried rather than at the moment it's written (schema-on-write, the relational-database default), is the architectural pattern that modern data lakes are built around, letting an organization ingest JSON, logs, or other semi-structured formats as-is and defer the decision of how to structure them until a specific analytical need defines what structure is actually required. This defers cost rather than eliminating it: every query against schema-on-read data has to reparse and validate structure that a schema-on-write system would have already enforced once, at ingestion — a genuine tradeoff between ingestion-time rigor and query-time flexibility, not a free upgrade over the relational approach.

Read semi-structured data as the pragmatic middle format that accepts some structure without demanding all of it: self-describing fields let records vary in shape without breaking, document databases and data lakes exist specifically to store and query that variability natively, and the recurring cost of this flexibility — pushed-downstream validation, a required flattening step before structured analysis, repeated schema-on-read overhead — is the price paid for not having committed to a single fixed shape up front.