Test rules and repairs with vacuum test. Give it a suite, fixtures and expected results. It uses vacuum’s lint engine and returns JSON with findings, execution evidence and assertion failures.

No findings doesn’t mean the rule ran. Add execution assertions to check selector matches and function calls.

Let’s test it

Download these four files into the same directory:

vacuum test suite.yaml

The suite checks that:

  1. A valid operation passes and the rule evaluated it.
  2. A missing operationId produces the expected finding.
  3. Restoring operationId removes the finding without adding another.

All three tests pass. The invalid fixture passes its test because it produces the expected finding.

Use it from an agent

Read the suite schema. Get it from the installed binary with:

vacuum test --schema suite

Write the cases. Include valid, invalid and boundary fixtures. Add execution assertions to check that functions ran.

Run the suite. Read exitCode, cases[].failures, and cases[].errors. A mismatch includes its assertion, expected value and actual value.

Fix the failure. Run just the affected case while working:

vacuum test suite.yaml --case missing-operation-id --evidence detailed

Run the full suite. Commit the suite and fixtures alongside the rules.

Same vacuum, same rules

OpenAPI, AsyncAPI, Arazzo, and JSON Schema use their existing execution paths. Custom functions, aliases, reference handling, and inline ignores keep their existing behavior. Select format: json-schema for a schema case, or format: fragment for a document fragment.

The suite supplies its settings explicitly. vacuum.conf.yaml and VACUUM_* configuration overrides do not change a test run. Paths are relative to the suite, so the agent can invoke it from another directory.

Test every operation your policy covers. A GET-only fixture does not test the other methods.

Reference