The top-level result contains version, status, exitCode, cases, errors, and manifest.

Each case contains its ID, status, assertion failures, execution errors, and the available snapshots:

  • before: the initial lint run.
  • fixed: the first autofix run.
  • after: the changed document, or the separate lint check of autofix output.
  • secondFix: the repeated autofix run used to check idempotence.

Snapshots contain normalized diagnostics and execution evidence. The effective rules are stored once in cases[].rules, shared by the case’s execution phases.

Assertion failures

{
  "assertion": "expect.execution.operation-id-required.invocations",
  "code": "count_mismatch",
  "expected": {"min": 1},
  "actual": 0
}

The rule did not call its function. Check the selector and fixture first.

Failure codes are:

  • missing_diagnostic, unexpected_diagnostic
  • execution_status, count_mismatch, missing_evidence
  • new_diagnostics, diagnostics_changed
  • document_mismatch, non_idempotent_fix

Execution and input errors live in errors, not in the assertion-failure list. They produce exit code 2. The result schema is also available from vacuum test --schema result.

Use code, phase and the identity fields to handle errors. message explains the failure.

{
  "code": "autofix_error",
  "message": "rule \"operation-id-required\" function \"addOperationId\": repair failed",
  "phase": "fixed",
  "caseId": "verify-repair",
  "ruleId": "operation-id-required",
  "function": "addOperationId",
  "source": "missing-id.yaml"
}

code, phase, and message are always present. caseId, ruleId, function, and source are included when known. Sources within a suite are relative to its directory. Load and command errors can occur before a suite location is known. A known failing dependency takes precedence over the entry document’s path.

Codes What failed
invalid_suite, invalid_selection, invalid_arguments The suite, selected cases, or command arguments.
input_error, invalid_document, dependency_error, ignore_file_error An input file, document, dependency, or ignore file.
ruleset_error, rule_configuration, unknown_function Ruleset loading or rule configuration.
function_load_error, function_error, autofix_error, rule_panic Function loading, execution, or a repair.
rule_timeout, lookup_error, cancelled, deadline_exceeded A bounded operation failed or execution stopped.
missing_evidence, incomplete_paths The evidence required to verify the case is unavailable.
bundle_error, worker_error, internal_error Bundle creation, the command worker, or an unexpected harness failure.
execution_error An engine failure without a more specific typed cause.

Phases are load, before, fixed, after, secondFix, change, fixOutput, bundle, and command. fixOutput identifies reading or comparing the expected repair fixture. Multiple independent execution errors remain separate records. A timeout during function loading has phase load; it is not a failed assertion.

Manifest

The manifest records the vacuum version, effective execution options, suite contract hash, selected case IDs, sample limit and SHA-256 hashes of tracked local files. An empty caseIds array means all cases in the suite were selected.

reproducible: false means some inputs cannot be captured. limitations lists the reasons, such as absolute paths, remote dependencies or custom functions that can read files and external state. Caller-registered functions also make a run non-reproducible.

Write a bundle

vacuum test suite.yaml --bundle ./replay

vacuum checks the tracked hashes again, copies local inputs while preserving their relative paths, and writes the result to .vacuum-test/result.json. It refuses existing destinations, dependencies outside the suite directory, reserved metadata paths, changed files, and incomplete manifests.

From the new directory, run the suite using the same vacuum version:

cd replay vacuum test suite.yaml

If the original run selected cases, repeat the manifest’s caseIds through --case. To reproduce detailed samples, use --evidence detailed with the recorded sampleLimit.

Use relative paths within the suite directory for fixtures, $ref files and ruleset extensions. Paths that escape the directory, including ../ dependencies, make the run non-reproducible. Absolute paths still point at the original checkout and prevent bundling. Bundles do not include remote responses.

Supply explicit fixtures and before/after states. The harness does not generate mutations or shrink failing fixtures.