property-based test generation

**Property-Based Test Generation** is the **AI task of identifying and generating invariants, algebraic laws, and universal properties that a function must satisfy for all valid inputs** — rather than specific example-based tests (`assert sort([3,1,2]) == [1,2,3]`), property-based tests define rules (`assert len(sort(x)) == len(x)` for all x) that testing frameworks like Hypothesis, QuickCheck, or ScalaCheck verify by generating thousands of random inputs, finding the minimal failing case when a property is violated. **What Is Property-Based Test Generation?** Properties are universal truths about function behavior: - **Round-Trip Properties**: `assert decode(encode(x)) == x` — encoding then decoding recovers the original. - **Invariant Properties**: `assert len(sort(x)) == len(x)` — sorting preserves list length. - **Idempotency Properties**: `assert sort(sort(x)) == sort(x)` — sorting an already-sorted list changes nothing. - **Commutativity Properties**: `assert add(a, b) == add(b, a)` — addition order doesn't matter. - **Monotonicity Properties**: `if a <= b then f(a) <= f(b)` — monotone functions preserve ordering. **Why Property-Based Testing Matters** - **Edge Case Discovery Power**: A property test with 1,000 random examples explores the input space far more thoroughly than 10 hand-written example tests. Hypothesis (Python's property testing library) found bugs in Python's standard library `datetime` module within minutes of applying property tests — bugs that had survived years of example-based testing. - **Minimal Counterexample Shrinking**: When a property fails, frameworks like Hypothesis automatically find the smallest input that causes the failure. If `sort()` fails on a list of 1,000 elements, Hypothesis shrinks the counterexample to the minimal list that reproduces the bug — often revealing exactly which edge case was missed. - **Mathematical Thinking Scaffold**: Writing meaningful properties requires thinking about functions in mathematical terms — what relationships must hold? What operations should be inverse? AI assistance bridges this gap for developers who are not trained in formal methods but can recognize suggested properties as correct. - **Specification Documentation**: Properties serve as executable specifications. `assert decode(encode(x)) == x` formally specifies that the codec is lossless. `assert checksum(data) != checksum(corrupt(data))` specifies that the checksum detects corruption. These properties document guarantees in the strongest possible terms. - **Regression Safety**: Properties catch regressions that example tests miss. If a refactoring introduces a subtle edge case for inputs with Unicode characters, the property test will find it in the next random generation cycle even if no existing example test covers Unicode. **AI-Specific Challenges and Approaches** **Property Identification**: The hardest part is identifying what properties to test. AI models trained on code and mathematics can recognize common algebraic structures (monoids, functors, idempotent functions) and suggest applicable properties from function signatures and documentation. **Domain Constraint Generation**: Property tests require knowing the valid input domain. AI generates appropriate type strategies for Hypothesis: `@given(st.lists(st.integers(), min_size=1))` for a sort function that requires non-empty lists, `@given(st.text(alphabet=st.characters(whitelist_categories=("L",))))` for a function expecting only letters. **Counterexample Analysis**: When AI-generated properties fail, LLMs can explain why the failing case violates the property and suggest whether the property is itself incorrect or reveals a genuine bug in the implementation. **Tools and Frameworks** - **Hypothesis (Python)**: The gold standard Python property-based testing library. `@given` decorator, automatic shrinking, database of previously found failures. - **QuickCheck (Haskell)**: The original property-based testing system (1999) that all others have been inspired by. - **fast-check (JavaScript)**: QuickCheck-style property testing for JavaScript/TypeScript with full shrinking support. - **ScalaCheck**: Property-based testing for Scala, deeply integrated with ScalaTest. - **PropEr (Erlang)**: Property-based testing for Erlang with stateful testing support. Property-Based Test Generation is **software verification through mathematics** — replacing the finite safety net of example tests with universal laws that must hold for all inputs, catching the unexpected edge cases that live in the vast space between the specific examples developers think to write.