A suite is a versioned YAML or JSON document. Version 1 requires ruleset and at least one case.
Unknown fields are errors. Each input and expected-output fixture must contain exactly one YAML or JSON document.
The example uses defined, which accepts an empty string. If your policy requires a nonempty ID,
add that check and include an empty-string fixture.
The example suite:
version: 1
ruleset: ./rules.yaml
cases:
- id: valid-operation
input: ./valid.yaml
rules: [operation-id-required]
expect:
diagnostics: []
execution:
operation-id-required:
status: completed
matches: {min: 1, max: 1}
invocations: {min: 1, max: 1}
- id: missing-operation-id
input: ./missing-id.yaml
rules: [operation-id-required]
expect:
diagnostics:
- rule: operation-id-required
severity: error
path: $.paths['/pets'].get
execution:
operation-id-required:
status: completed
invocations: {min: 1}
- id: verify-repair
input: ./missing-id.yaml
after: ./valid.yaml
rules: [operation-id-required]
expect:
diagnostics:
- rule: operation-id-required
afterExpect:
diagnostics: []
execution:
operation-id-required:
status: completed
invocations: {min: 1}
change:
noNew: true
removed:
- rule: operation-id-required
Suite settings
ruleset is a local ruleset file. Use the normal extends syntax to select built-in or shared rules.
Each case can select rule IDs with rules; omit it to execute the loaded ruleset. Unknown or disabled IDs
are errors. Rules need at least one nonempty selector and one action.
Document and reference diagnostics produced by the engine remain part of the results.
functions points to a directory of existing vacuum custom functions. ignoreFile points to the usual
ignore-file format. Both paths resolve relative to the suite.
Write functions with the JavaScript function guide
or Go function guide.
For JavaScript, getSchema().name is the name used by the rule’s then.function; runRule(input) returns
an array of findings. Set functions: ./functions in the suite to load the directory.
options:
timeoutMs: 5000
lookupTimeoutMs: 500
maxRuleConcurrency: 32
runTimeoutMs: 30000
remote: false
resolveAllRefs: false
nestedRefsDocContext: false
Zero or omitted per-rule, lookup, and concurrency limits use the defaults shown above. Omitted
runTimeoutMs means no overall motor deadline. The deadline applies to each engine invocation;
before/after and autofix cases perform more than one invocation. Preparation checks cancellation between
phases; it is not a hard interruption of an active parser or arbitrary Go callback.
JavaScript script loading and getSchema() evaluation each use timeoutMs. A timeout, thrown error,
or invalid schema fails the load, including when other functions in the directory loaded successfully.
remote controls document-reference lookup. Existing ruleset extensions use the ruleset loader;
remote extensions are recorded as a reproducibility limitation. JavaScript fetch retains vacuum’s existing
security defaults. The harness does not enable HTTP or private-network access.
Use format: auto (the default) for OpenAPI, AsyncAPI, and Arazzo. Use json-schema for JSON Schema,
or fragment to bypass the document-type check for a fragment.
Exact diagnostics
diagnostics: [] requires no findings. Otherwise every expected entry consumes exactly one diagnostic.
Extra findings fail the test. Ordering does not matter, duplicate counts do.
expect:
diagnostics:
- rule: operation-id-required
severity: error
path: $.paths['/pets'].get
source: missing-id.yaml
rule is required. Optional fields are severity, path, paths, source, line, column, and message.
Source coordinates are one-based. Sources are suite-relative. path checks the primary path; paths
checks the complete alternate-path set, without depending on order. An incomplete path set cannot satisfy
an exact paths assertion. paths: [] requires an empty alternate-path set; omit paths to skip that check.
Omit message if wording changes should not fail the test.
Omit diagnostics when a case only asserts execution evidence. Every expectation must contain at least
one diagnostic or execution assertion. An omitted diagnostic assertion does not mean “expect no findings”.
Prove that the rule ran
expect:
diagnostics: []
execution:
operation-id-required:
status: completed
matches: {min: 1}
invocations: {min: 1}
ignored: {max: 0}
Counts accept inclusive min and max; set both for an exact count. status accepts completed or
excluded. You cannot assert a timeout as a successful test. See execution evidence.
Verify a repair
Supply after and either afterExpect, change, or both. Both documents use the same rules and options.
change.added and change.removed are exact diagnostic lists. noNew: true rejects new findings.
unchanged: true requires no added or removed findings. Comparison accounts for source locations, reference
aliases and duplicate counts. Message or severity changes count as a removed finding and an added finding.
Verify an autofix
fix:
output: ./expected-fixed.yaml
expect:
diagnostics: []
The harness applies registered fixes in memory, compares the output semantically with the expected document,
lints it again, and checks that another fix makes no further semantic change. Input files are never overwritten.
fix.expect.diagnostics checks the separate lint run after the repair. fix.expect.execution checks the
repair pass, including its fixed, matches and invocations counts.
A case cannot combine fix and after.
Go callers register AutoFixFunctions in RunOptions. Go plugins can register repairs through
Manager.RegisterAutoFixFunction in their existing Boot function. Those repairs also work with
vacuum lint --fix --functions ./plugins. An unregistered repair is an execution error.
A repair callback error or panic fails the run, even if its unchanged output matches the expected fixture.
The harness does not infer repairs from a JavaScript lint function.
The suite JSON Schema describes the fields. Semantic checks also reject contradictory count bounds, duplicate case IDs, and incomplete before/after declarations. Case IDs start with a letter or digit and contain only letters, digits, underscores, dots, and hyphens.