TEST-002D defines seen-test-leak-soak-v1. A canonical snapshot registers bounded providers for allocations, file descriptors, threads, child processes, mapped windows, asynchronous I/O requests and completions, persistent workers and tasks, GPU objects, staging bytes, and committed and resident VRAM. Each provider reports baseline, peak, final, acquired, and released values after at least 10,000 iterations.
Available providers must balance their lifecycle totals and return to the baseline. Retained resources fail with test.002d.leak; inconsistent peaks, ordering, unsupported-provider measurements, and VRAM residency above committed memory fail closed. A provider for a capability not yet implemented is retained in canonical order with available=false and zero metrics—never silently substituted. The JSON fixtures are synthetic contract data, not I/O or GPU hardware certification.
The native policy is testing.leak; serialized acceptance uses scripts/check_leak_soak_evidence.py. Run tests/misc_root_tests/seen_leak_soak_contract.sh, setting SEEN_TEST_002D_FUZZ_SECONDS=60 for the required seed-1101 fuzz duration. The compiling example is compiler_seen/examples/test_leak_soak.seen, and the native 10,000-validation soak is tests/fixtures/soak/test_002d_leak_soak.seen.
TEST-002C defines seen-benchmark-evidence-v1, a strict, canonical evidence tuple for Colibri performance claims. Each record binds the hardware, toolchain, source and baseline commits, exact command, read-back cgroup limits, five warmups, thirty sorted nanosecond samples, their median, bandwidth, peak RSS and VRAM, transfer volume, correctness, fallback count, backend maturity, and whether real hardware executed. Unknown or duplicate fields fail closed.
Designated kernels have a hard five-percent median regression ceiling. They must execute on hardware, may not use fallback, and cannot use unsupported or compile-only evidence. CPU records cannot claim VRAM, transfers, or GPU execution. The RTX 4090 JSON under tests/fixtures/test-002c is synthetic contract data only; it is not hardware certification.
The native policy lives in testing.benchmark; the Python checker only validates serialized acceptance evidence. Run tests/misc_root_tests/seen_benchmark_evidence_contract.sh, setting SEEN_TEST_002C_FUZZ_SECONDS=60 for required seed-1101 fuzz evidence. The compiling example is compiler_seen/examples/test_benchmark_evidence.seen. Legacy scripts/perf_gate.sh suite baselines remain useful general regression signals, but cannot substitute for this Colibri evidence tuple.
TEST-002B defines seen-test-fuzz-corpus-v1. A corpus manifest pins its seed, target, maximum input size, required replay count, and canonical byte-ordered entries. Each entry records its original and minimized sizes, lowercase SHA-256 content address, stable test.* failure code, and the failure code observed by every replay. Unknown fields, duplicate identities or payloads, unstable replays, non-canonical order, and oversized input fail closed.
Minimized payloads live in cases/<sha256>.bin beside the manifest. Validation requires contained, non-symlink regular files whose length and digest match the manifest. The minimizer uses deterministic chunk deletion and accepts a change only while the caller's failure predicate continues to reproduce. Canonical output is written with an fsync-and-rename transaction; cancellation and write failures remove temporary output. Corpus parsing, ordering, bounds, and replay policy are native Seen; the Python checker is acceptance tooling for serialized fixtures and never supplies production policy.
The compiling example is compiler_seen/examples/test_fuzz_corpus.seen. The acceptance command is tests/misc_root_tests/seen_fuzz_corpus_contract.sh; set SEEN_TEST_002B_FUZZ_SECONDS=60 for the required seed-1101 fuzz duration.
TEST-002A defines the seen-test-instrumentation-v1 contract with default, ci, coverage, sanitizer-undefined, and sanitizer-thread profiles. The current unreleased compiler exposes these through seen test <path> --profile <profile>; the published 0.22.11 compiler does not include that command. Maintained acceptance harnesses include tests/misc_root_tests/seen_test_instrumentation_contract.sh and tests/misc_root_tests/seen_test_runner_contract.sh. Compiler-capable invocations must use the verified containment described in bootstrap.md.
The source runner compiles Seen tests with the selected instrumentation, per-test timeout, and serialized compiler path. Reports and raw coverage profiles stay below .seen_cache/test/run. Coverage post-processing requires llvm-profdata and llvm-cov; missing tools are infrastructure failures, not an implicit downgrade. On LLD targets, no-fork mode also bounds linker and ThinLTO workers. AddressSanitizer and MemorySanitizer compiler flags do not imply runnable source-runner profiles; unsupported profile execution fails closed rather than weakening the mandatory memory limits.
Acceptance evidence names compiler host code, GPU emitters, Seen modules, the native runtime, and ledgered ABI shims separately. Each must be compiled and software-executed. GPU emitter execution proves the compiler path only: hardware_executed must remain false until a separate real-hardware gate produces hardware evidence. Compile-only evidence cannot close TEST-002A. The capped execution acceptance therefore builds and runs an instrumented compiler, exercises the software GPU emitter, then runs the instrumented Seen fixture before deriving the five-component evidence document.
TEST-001A defines the seen-test-discovery-v1 manifest consumed by the test runner. Discovery is byte-ordered by canonical repository-relative path. It rejects traversal, symlinks, duplicate or unordered entries, unsupported file shapes, unknown fields, and caller-supplied categories that disagree with the path policy.
The maintained roots and primary categories are:
| Root | Runnable shape | Primary category |
|---|---|---|
compiler_seen/tests/ | .seen | unit |
seen_std/tests/ | .seen | unit |
tests/fixtures/external_package/tests/ | .seen | integration |
tests/misc_root_tests/ | .sh, _unit.py, _test.seen | integration |
Names may add orthogonal ignored, slow, or privileged markers using a matching directory or underscore-delimited filename segment. Platform markers are linux, linux_x86_64, linux_arm64, macos, and windows; unmarked tests use all. These markers describe selection policy and do not grant privileges or silently skip an unsupported platform.
Legacy sources that cannot be linked as standalone programs carry the explicit ignored marker. They remain discoverable and reported, but run only after an operator supplies --include-ignored; the runner never keeps a hidden skip list.
Native Seen exposes TestPrimaryCategory and TestPlatformCategory; ignored, slow, and privileged remain independent flags so categories compose without stringly typed combinations.
scripts/discover_seen_tests.py --discover <absolute-repository-root> is the strict host oracle for the current compiler's canonical seen test execution command. It emits canonical JSON to standard output. Validation is available with --validate <manifest> and never repairs input.
The canonical seen test <path> command consumes that native inventory through the seen-test-run-v1 policy. Selection is deterministic and supports a filter, the default and ci profiles, explicit ignored/slow/privileged opt-ins, one bounded worker, and a timeout of at most one hour. Its contract executes Seen, shell, and Python test shapes through a narrow process boundary while retaining per-test logs below .seen_cache/test/run/.
Exit codes are stable: 0 means all selected tests passed, 1 means at least one selected test failed, 2 is invalid CLI or selection input, 3 is an infrastructure or report-write failure, and 130 is cancellation. Timeouts are test failures and never silently retry. JSON and JUnit reports are written transactionally with repeatable --report json:<path> and --report junit:<path> options; paths must remain below .seen_cache/test/ or .seen/ and may not traverse symlinks or parent segments.
The legacy scripts/run_all_tests.sh continues to record and validate the same discovery inventory while migration to the shipped runner proceeds.
TEST-001F defines seen-test-migration-v1, the fail-closed ownership map for compiler, standard-library, and external-package tests. Each source has an explicit package root, manifest, tests root, category, platform, and bounded test count. The host oracle rejects missing or symlinked topology, unsafe paths, unknown or duplicate fields, reordered sources, inconsistent totals, and input above the configured limits; native Seen exposes the equivalent typed plan and stable test.001f.* errors.
The package-owned migration contract includes the external-package fixture alongside compiler and standard-library tests. Every migrated .seen test runs from its own manifest directory, preserving package dependency resolution rather than assuming the repository root. CPU-only discovery does not locate or link accelerator SDKs. Platform execution evidence belongs to the individual acceptance harnesses; a portable schema is not a native execution claim.
TEST-001C provides native seen-test-assertion-v1 assertion evaluation and seen-test-snapshot-v1 exact-text snapshots. Assertions return structured results: a comparison failure is test data (passed: false), while invalid, cancelled, or over-limit requests return stable test.001c.* errors through a fallible API. Names are bounded portable identifiers and values are limited to 1 MiB before allocation or any filesystem side effect.
Snapshots compare bytes exactly; newline, spacing, and Unicode differences are significant. The typed update policy is ReadOnly, CreateMissing, or Replace. Updates are never implicit: comparison returns writeRequired, and the caller may then transactionally write the returned content to .seen_snapshots/<name>.snap. CPU-only use has no accelerator dependency. The contract is implemented for Linux x86-64 and is platform-neutral source on Windows, macOS, and other declared Seen targets; filesystem persistence remains the caller's bounded platform adapter.
TEST-001D defines the native seen-test-fixture-v1 plan. A fixture has a portable name, explicit deterministic seed and target, a byte-ordered unique file set, and a byte-ordered environment allowlist limited to LANG, LC_ALL, TZ, and SEEN_TEST_*. Ambient home, path, locale, timezone, randomness, network, and clock state are not silently captured.
Plans validate every bound before materialization and use the owned root .seen_cache/test/fixtures/<name>. A pre-existing root, traversal, symlinked base, duplicate/unordered path, unsupported target, or unapproved environment key or secret-bearing environment name fails closed. The materialization oracle creates files exclusively below a new physical root and returns an ownership token required for idempotence-safe cleanup; tests cover success, failure, cancellation, limits, and zero retained fixture roots. Linux x86-64 is verified now; Linux ARM64, macOS, and Windows share the declared source contract and require their target runner adapter.
TEST-001E defines seen-test-report-v1, one native validated rendering path for deterministic human text, canonical seen-test-run-v1 JSON, and escaped JUnit XML. Report counters, status/exit-code pairs, byte-ordered paths, suite identity, cancellation, and output bounds are validated before rendering. Human output is the canonical CLI display; JSON and JUnit are written only to the safe transactional report paths documented above. XML declarations and entities are rejected by the host oracle, and all formats end deterministically.