# Testing and CI gates # MEP-45 research note 11, Testing and CI gates Author: research pass for MEP-45. Date: 2026-05-22 (GMT+7). The single most important property a transpiler can have is *output correctness*. This note describes how MEP-45 proves it. ## 1. Differential testing against the VM Every fixture under `tests/vm/valid/` (and the `examples/v0.3/` ... `examples/v0.5/` corpora) has a recorded expected stdout produced by the VM. The transpiler's output must: 1. Compile under tier-1 toolchains. 2. Run the produced binary under each tier-1 target (via qemu-user for cross-arch on Linux CI). 3. Produce **byte-equal** stdout. This is the master gate. A regression here blocks the PR. The VM and the transpiler share the same parser and type-checker output (MIR). The codegen is the only thing under test. Any divergence is either a codegen bug or a *language-level* ambiguity worth documenting. ## 2. The BG corpus The "byte-equal goldens" corpus lives under `tests/cross-aot/bg//`. Each fixture is a complete program plus an `expect.txt` (stdout) and an optional `expect.exit` (exit code). Phasing per the MEP body: - Phase 1: `tests/cross-aot/bg/host/` only. - Phase 2: tier-1 cross-arch added. - Phase 3: tier-2 added (BSDs, riscv, armv7). - Phase 5.2 (already shipped per git log): BG fixtures on every cross-target triple. - Phase 5.2.1 (already shipped): Linux + wasm run-gates. The MEP-45 phasing extends this by *introducing* the C-target fixtures alongside the existing VM ones. ## 3. Run gate per target For each (target, profile) pair: ``` mochi build --target=$T --profile=$P fixture.mochi ./out | diff - expect.txt ``` Targets currently on tier 1 (see note 07 ยง3). Profiles in the gate: `--dev` and `--release` for every fixture; `--debug` (sanitisers) for a curated 10% subset (full corpus would be slow under ASan). ## 4. Sanitiser matrix The nightly job runs the corpus under: | Sanitiser | Build flag | Expected detected | |---|---|---| | ASan | `-fsanitize=address` | use-after-free, double free, OOB | | UBSan | `-fsanitize=undefined` | signed overflow, alignment, oob shifts, null deref | | TSan | `-fsanitize=thread` | data races in streams/agents | | MSan | `-fsanitize=memory` | uninit reads | | LeakSan | bundled with ASan | runtime leaks above BDWGC floor | Failure on any sanitiser blocks merge. The intent: the transpiler must produce sanitiser-clean code on the entire fixture corpus. ## 5. Property tests Domain-specific properties: - **Pattern-matcher**: for a random MIR pattern set, the generated decision tree must classify every value identically to a reference naive matcher. (Counter-example shrinking via `theft`.) - **Sort**: `mochi_sort__T_by(xs, cmp)` is stable and total under any consistent `cmp`. Random inputs of length 0..1000. - **Swiss-table**: insert/erase/get sequence against `std::unordered_map` reference. 1M ops per run. - **JSON round-trip**: parse(serialise(x)) == x for random records. - **YAML round-trip**: same. - **CSV round-trip**: same, modulo non-string column types. - **Stream fan-out**: every emit reaches every subscriber exactly once, in emit order, under random scheduling. ## 6. Fuzzing libFuzzer + ASan/UBSan harnesses for: - The parser: any input must not crash; either parses or returns an error span. - The type-checker: same. - The JSON loader: yyjson is fuzzed upstream, but we add a Mochi- facing harness because we lower JSON values to typed records. - The pattern matcher: any program must produce a deterministic match outcome that agrees with the reference matcher. The fuzzing corpus seeds from the BG corpus and from a `corpora/` directory grown by OSS-Fuzz reports. ## 7. Differential vs other backends Where the language has another backend that has shipped a feature (currently Go is the most mature), the gate runs the same fixture through both backends and diffs stdout. Useful for tracking inadvertent divergence. ## 8. Reproducibility check The gate rebuilds each release-profile fixture twice, on the same machine and on a second build host, and asserts SHA-256 equality of the binary. If reproducibility breaks, the PR is rejected. ## 9. Spec-in-sync gate Per the memory note `feedback_spec_in_sync`: any PR that lands a codegen change must update the MEP file (or its referenced research notes) in the same PR. A bot enforces this by checking that any change under `internal/aot/c/` is accompanied by a change under `spec/0045/` or `spec/MEP-45.md`. ## 10. Phasing gates The MEP body defines phases. Each phase has a *measurable gate* matching the umbrella-phase coverage rule (`feedback_umbrella_phase_targets`): | Phase | Gate description | Targets in scope | |-------|------------------|------------------| | 1 | Compile + run hello-world; produce stdout "hello, mochi!" | host only | | 2 | Full primitives + records + lists + maps; arithmetic suite passes | host | | 3 | Sum types + pattern matching; option corpus passes | host | | 4 | Closures + higher-order functions; map/filter/fold corpus | host | | 5 | Strings + I/O + error model; stdlib suite passes | host | | 6 | Query DSL; query corpus passes byte-equal vs VM | host | | 7 | Streams + agents + concurrency; stream corpus passes | host | | 8 | FFI shells (C direct, Go via RPC); FFI corpus passes | host | | 9 | Cross-compile tier-1 architectures | tier-1 triples | | 10 | WASM/WASI; wasi corpus passes | wasi | | 11 | APE / Cosmopolitan; ape corpus passes | one-binary all-OS | | 12 | LLM bindings + generate; replay-mode tests pass | host | | 13 | Datalog; logic corpus passes | host | | 14 | Sanitiser matrix clean across full corpus | tier-1 triples | | 15 | Reproducible builds; SHA-256 stable across two CI hosts | tier-1 triples | | 16 | Performance: median fixture within 2x of Go backend wall-time | host | Each phase becomes a sub-PR. Auto-merge applies per `feedback_auto_ship_phases`. ## 11. Performance gates Phase 16 is a soft gate (warn on regression > 10%). The benchmark suite uses the BG corpus plus the "perf" subset of fixtures (long- running compute, query-heavy, stream-heavy). We track: - Wall-clock time. - Peak RSS. - Binary size (release stripped). - Compile time. Per-release reports go to a static page. ## 12. Stress tests A nightly stress run does: - Build the entire example corpus under `--debug` (sanitisers). - Run the streams fixture suite under a 10x message rate. - Run the agents fixture suite with 4x worker threads and ramped CPU load. - Run the datalog suite with a 100x fact count. Failures don't block merge but file an automatic issue. ## 13. Goal alignment per phase Per the memory note `feedback_goal_alignment_audit`: before each phase starts, the MEP gets a one-paragraph audit confirming the phase's gate ties to the user-facing goal ("produce a working C executable from a Mochi program") rather than spec-internal scaffolding. Example audit (phase 6, query DSL): "Query DSL is the highest-value language feature for the dataset/AI workflows the docs target. Without it, even simple ETL fixtures fail. The gate (byte-equal stdout) is end-user-observable. Aligns." ## 14. CI infrastructure GitHub Actions matrix: - Linux x86_64 host (ubuntu-24.04 runner): tier-1 triple builds via zig cc, qemu-user-static for cross-arch run. - macOS arm64 host (macos-15): native, plus zig cc cross. - Windows x86_64 host (windows-2025): clang-cl native, plus zig cc cross. A nightly self-hosted bare-metal runner (rented hardware) handles the sanitiser matrix, the stress suite, the reproducibility check, and the performance report. ## 15. Bug bounty entry points The MEP body recommends listing the following as bounty-eligible: - Codegen producing UB on a valid Mochi program. - Codegen producing different stdout from the VM on a valid program. - Runtime leaking memory above the BDWGC floor on a finite-time program. - Type-checker accepting a program that crashes the codegen. This sets a clear contract for what the transpiler guarantees. ## 16. Open questions 1. Whether sanitiser matrix is per-PR or only per-merge. 2. Whether the reproducibility gate runs on every PR or only on release branches. 3. Whether the LLM tests use real provider credentials in CI (cost, flakiness) or only the replay cassettes. 4. Whether stress tests should block release or only file issues. 5. Whether the Go-backend differential gate stays as a permanent CI line or sunsets once both backends are at parity.