# gopy v0.5 testing strategy

# 1625. v0.5 testing strategy

This spec is the test-side companion to 1620 and 1665. The checklist in
1620 names every Go file that must land for v0.5; this spec names the
test that proves each file matches CPython 3.14.

The rule is one-to-one: every implementation checkbox in 1620 has a
test checkbox here. A v0.5 release ships only when every box on both
specs is ticked.

## Layered gates

The v0.5 test pyramid has four layers, each strictly stricter than the
one above. Code lands at the bottom and climbs.

1. **Unit**: package-local Go tests against hand-built inputs. Catches
   bad refactors and obvious arity errors.
2. **Cross-check**: Go output diffed against CPython output for the
   same input, where the input is built in-process (no parser).
3. **Marshal parity**: gopy emits `co_code`, `co_linetable`,
   `co_exceptiontable` byte-equal to CPython for the same source.
4. **Run parity**: a future v0.6 layer; out of scope for v0.5.

The cross-check layer needs a host CPython 3.14 on `$PATH`. Tests that
need it are tagged with `//go:build cpython` and skipped on CI without
it; the gate jobs run with the tag.

## How to test each kind of port

### Pure data types (Seq, Pos, Token)

Table-driven equality tests. No CPython invocation. Cover empty,
single, many, mutation, out-of-bounds. Locks shape; the per-field
parity is in 1601.

### Hand-written nodes (ast/nodes.go)

Construction, field access, `Position()` round-trip, `IsDocString`
truth table. No CPython needed; the asdl source-of-truth is in 1620.

### Generated nodes (ast/nodes_gen.go)

Generator-output parity: a `tools/asdl_go_test` round-trips
`Python.asdl` and asserts the emitted Go file is byte-equal to the
checked-in `nodes_gen.go`. This catches generator drift without
re-running the generator on every CI.

### Validators (ast/validate.go, ast/preprocess.go)

Two-axis tests:
- Acceptance: every node kind that CPython accepts must round-trip nil.
- Rejection: every error string in `Python/ast.c` matches verbatim. The
  test asserts both `errors.Is`-style class and `err.Error()` substring.

For PEP 765 (try/finally with break/continue/return), the rejection
panel mirrors `Lib/test/test_syntax.py` line-for-line.

### Future flags (future/future.go)

One test per `__future__` name (recognised, no-op, or rejected). Plus
the four wrong-shape rejections (braces, unknown, after non-import,
relative). Plus location tracking on the `from` keyword.

### Symtable

Cross-check against `symtable.symtable(src, '<test>', 'exec')` in
CPython. The Go test reads CPython's JSON dump of the symbol table for
each fixture and asserts identical scope tree, identical name set per
scope, identical flags per name. Fixtures live in
`symtable/testdata/*.py` and `symtable/testdata/*.json`; the JSON is
regenerated by `tools/symtable_golden`.

Coverage panel:
- Module / function / class / lambda / comprehension scopes.
- Free variable capture (closure cells).
- `global` and `nonlocal` declarations.
- Walrus inside comprehension.
- `import a.b.c` name binding.
- PEP 695 type-parameter scope.

### Codegen (compile/codegen.go)

Per statement and per expression. Each test:
1. Builds the AST in Go (no parser).
2. Runs codegen.
3. Asserts the emitted opcode sequence matches a hand-curated golden.
4. Cross-checks (with `//go:build cpython`) that CPython emits the same
   sequence for the equivalent source.

Statement panel mirrors the codegen visitor list in 1620 §6.

### Flowgraph (compile/flowgraph.go)

One test per optimization pass. Each pass has:
- A "minimal positive" case (input that triggers it, output that
  applies it).
- A "negative" case (input where the pass must not fire).
- A "fixed-point" case (running the pass twice equals once).

Plus a property test: random instruction sequences round-trip through
`FromSequence -> Optimize -> ToSequence` without losing reachable
instructions.

### Assemble (compile/assemble.go)

Byte-exact tests. For each artefact (`co_code`, `co_linetable`,
`co_exceptiontable`):
- Hand-built varint encodings for every form (short, one-line, long,
  no-location for PEP 626; full form for PEP 657).
- Round-trip: assemble then disassemble the bytes back into the
  in-memory representation.
- Cross-check (cpython tag): identical bytes from CPython `compile()`
  for a panel of source files.

### Compiler driver (compile/compiler.go)

End-to-end on hand-built ASTs. The test panel here is the union of
every other panel: assignment, if/while, try/except, def, class,
comprehension, async function, match, type alias.

### Tokenize (1665)

Iterator contract: `New(src, false).Next()` ends in `io.EOF`, never
panics on empty input, never yields a zero-Type token.

Token-stream parity (cpython tag): for every fixture in
`tokenize/testdata/*.py`, the Go iterator's full output (Type, Value,
Start, End, Line) matches `tokenize.tokenize(io.BytesIO(src).readline)`
in the host CPython.

Error parity: each lexer error string matches CPython verbatim.

## Per-checkbox test mapping

The table below mirrors the order of 1620 §1 through §14 plus 1665.
Every implementation checkbox has its test counterpart here. When the
implementation lands, both boxes are ticked in the same commit.

### 1. ast package: asdl runtime

* [x] `Seq[T].Len/Get/Set` table test.
* [x] `NewSeq(0)` returns non-nil empty sequence.

### 2. ast package: hand-written nodes

* [x] `Module`/`Interactive`/`Expression`/`FunctionType` construction.
* [x] `Position()` returns the embedded `Pos` for every node.
* [x] `IsDocString` truth table (string Constant yes; non-string no;
  non-ExprStmt no).

### 3. ast package: validate

* [x] Negative position rejected; `NoPos` accepted.
* [x] `end_lineno < lineno` rejected.
* [x] `end_col_offset < col_offset` on same line rejected.
* [x] Negative ImportFrom level rejected with exact CPython text.
* [x] Constant: every accepted scalar kind, plus tuple, plus
  frozenset, plus invalid-type rejection.
* [x] Nil module / nil statement / nil expression rejected.

### 4. ast package: preprocess (constant fold + PEP 765)

* [ ] UnaryOp folding panel: `-1`, `not True`, `~0xFF`.
* [ ] BinOp folding panel: every accepted (op, lhs-type, rhs-type)
  triple.
* [ ] BoolOp folding (`True and X`, `False or X`).
* [ ] Compare folding when both sides are constants.
* [ ] Tuple/Frozenset/Set literal folding.
* [ ] PEP 765: `break` / `continue` / `return` inside `finally`
  rejected with verbatim string.

### 5. ast package: unparse

* [ ] Per node kind: round-trip
  `Unparse(parser.Parse(src)) == normalised(src)` for a curated panel.
* [ ] Operator precedence parens preserved.
* [ ] f-string and t-string round-trip.

### 6. ast package: asdl-driven generator

* [ ] `tools/asdl_go` parses `Python.asdl` without error.
* [ ] Generator output is byte-equal to checked-in `nodes_gen.go`
  (the regen-and-diff test).

### 7. future package

* [x] Annotations recognised; bit set; location captured.
* [x] barry_as_FLUFL recognised.
* [x] No-op features (division, generators, ...) accepted.
* [x] `from __future__ import braces` rejected with "not a chance".
* [x] Unknown name rejected.
* [x] Docstring is skipped before scanning.
* [x] Stop at first non-future statement.
* [x] Relative `from . import x` ignored.
* [x] `Expression` mode: no future flags.

### 8. symtable

* [ ] Module scope name set parity with CPython.
* [ ] Function scope: locals, args, free vars.
* [ ] Class scope: `__class__` cell, MRO entries.
* [ ] Lambda scope.
* [ ] Comprehension scope (free var capture).
* [ ] `global` / `nonlocal` flag bits.
* [ ] Walrus rebinds outer scope.
* [ ] PEP 695 type-parameter scope.
* [ ] All symtable error strings verbatim.

### 9. compile/instrseq

* [x] `NewLabel` IDs are 1-based and monotonic.
* [x] `Addop` rejects opcode > MaxOpcode and oparg >= MaxOparg.
* [x] `Insert` shifts existing labels.
* [x] `ApplyLabelMap` is idempotent.
* [x] Non-jump opargs are not rewritten.
* [x] ExceptHandlerInfo.Label is resolved.
* [x] `SetAnnotationsCode` panics on second call.
* [x] `AddNested` attaches child sequence.

### 10. compile/opcodes_gen

* [x] Opcode numbers cross-check against CPython 3.14 opmap.
* [x] Per-flag predicates for a sample (LOAD_CONST, LOAD_FAST,
  LOAD_NAME, JUMP_FORWARD, NOP).
* [x] `HasTarget` matches `HasJump` for the jump opcodes.
* [x] Out-of-range Name returns "".
* [ ] Generator round-trip: regenerate and diff against checked-in
  `opcodes_gen.go`.

### 11. compile/codegen

Statements:
* [ ] Assign / AugAssign / AnnAssign.
* [ ] If / While / For / AsyncFor.
* [ ] Try / TryStar / Raise.
* [ ] With / AsyncWith.
* [ ] FunctionDef / AsyncFunctionDef.
* [ ] ClassDef.
* [ ] Match (per pattern kind: literal, capture, sequence, mapping,
  class, or, as, value).
* [ ] Import / ImportFrom.
* [ ] Global / Nonlocal.
* [ ] Return / Break / Continue.
* [ ] Pass.
* [ ] PEP 695 TypeAlias / TypeVar.
* [ ] Assert.
* [ ] Delete.

Expressions:
* [ ] BoolOp / BinOp / UnaryOp / Lambda / IfExp.
* [ ] Dict / Set / List / Tuple displays.
* [ ] ListComp / SetComp / DictComp / GeneratorExp.
* [ ] Await / Yield / YieldFrom.
* [ ] Compare (chained).
* [ ] Call (star, starstar, keyword).
* [ ] FormattedValue / Interpolation / JoinedStr / TemplateStr.
* [ ] Constant.
* [ ] Attribute (LOAD_SUPER_ATTR super-instruction).
* [ ] Subscript (load/store/delete).
* [ ] Starred.
* [ ] Name (LOAD_FAST/DEREF/GLOBAL/NAME by scope).
* [ ] Slice.

Block / unwind:
* [ ] fblock_push / pop / unwind equivalents.
* [ ] Try/except/finally exception-table emission.
* [ ] With unwinding calls __exit__.
* [ ] Async-with unwinding awaits __aexit__.
* [ ] Generator return-value path.
* [ ] Coroutine close path.

### 12. compile/flowgraph

* [ ] LOAD_CONST chain folding (positive, negative, fixed-point).
* [ ] Jump threading.
* [ ] Conditional-jump propagation.
* [ ] Unreachable block elimination.
* [ ] Dead-code after unconditional terminator.
* [ ] Stack-effect verification: every block's net push - pop matches
  the recorded entry depth.
* [ ] RESUME insertion at entry.
* [ ] Implicit `LOAD_CONST None`/`RETURN_VALUE` for fall-through.
* [ ] EXTENDED_ARG insertion for wide opargs.
* [ ] Push-null fix-up at super-instruction call sites.
* [ ] Property test: random sequences round-trip through Optimize.

### 13. compile/assemble

* [ ] EXTENDED_ARG widening for opargs > 0xFF, > 0xFFFF, > 0xFFFFFF.
* [ ] PEP 626 location-table varint encoder per form.
* [ ] PEP 657 exception-table varint encoder.
* [ ] Constants table dedup parity.
* [ ] Names / Varnames / Freevars / Cellvars population.
* [ ] `co_flags` per kind: function, async, generator, coroutine,
  async generator.
* [ ] `co_qualname` for nested defs.
* [ ] `co_code` byte emission.
* [ ] cpython-tag: byte-equal `co_code` for assignment, if, while,
  try/except, def, comprehension, async function.

### 14. compile/compiler

* [ ] End-to-end `Compile(mod, "<test>", 0)` for the full statement
  panel.
* [ ] `optimize` levels -1, 0, 1, 2 each suppress the right opcodes.
* [ ] cpython-tag: `marshal.dumps(gopy_code) == marshal.dumps(cpython_code)`
  for the v05test gate panel.

### 15. compile/dis

* [ ] `Format` output line-for-line equal to CPython 3.14 `dis.dis()`
  on the v05test panel.

### 16. tokenize (1665)

* [ ] `Type` constant numeric values match CPython 3.14 `token.h`.
* [ ] `New("", false).Next()` returns `io.EOF` immediately.
* [ ] Iterator never yields a zero `Type`.
* [ ] cpython-tag: token-stream byte-parity on every fixture in
  `tokenize/testdata/`.
* [ ] Error parity: every documented `SyntaxError` string is verbatim.

### 17. v05test cross-cut gate

* [ ] `TestGateAssign`: `a = 1 + 2`.
* [ ] `TestGateIfWhile`: if/while panel.
* [ ] `TestGateTryExcept`: try/except, finally, exception table.
* [ ] `TestGateDef`: `def f(x): return x + 1`.
* [ ] `TestGateComprehension`: `[x*x for x in range(3)]`.
* [ ] `TestGateAsyncFunction`: `async def f(): pass`.
* [ ] `TestGateMarshalRoundtrip`: byte-equal `co_code`,
  `co_linetable`, `co_exceptiontable` against CPython marshal output.
* [ ] `tools/golden/`: regeneration script for golden blobs under
  `v05test/golden/`.

## Test-only utilities

* `internal/parityhelpers` (out of scope per "no internal/" rule;
  instead a flat `testparity/` package): wrappers around
  `exec.Command("python3.14", "-c", ...)` for cpython-tag tests.
* `tools/golden/`: regenerate captured CPython output. Stamps the
  CPython version into the file header so a host upgrade does not
  silently invalidate the golden.
* `tools/symtable_golden/`: dump CPython symtables to JSON.

## Coverage policy

Every Go file under v0.5 has its own `_test.go` covering at minimum:
- One success path per exported symbol.
- One rejection path per exported error.
- One regression hook for every CPython quirk preserved in the port.

A symbol with no tests is a release blocker. Lint rule (manual): grep
for `^func [A-Z]` in implementation files; cross-reference against
`_test.go` in the same package; unmatched names fail the gate.

## Out of scope for v0.5 testing

* Run-parity against CPython under the real VM. v0.6.
* JIT / specialization parity. v0.11+.
* Stdlib import behaviour. v0.8.