A rule with no findings may have passed, or its selector may have matched nothing. Execution evidence tells you which.

Every test snapshot includes an evidence array:

{
  "ruleId": "operation-id-required",
  "status": "completed",
  "resolved": true,
  "selectors": [{"given": "$.paths[*].get", "matches": 1}],
  "matches": 1,
  "invocations": 1,
  "findings": 0,
  "ignored": 0,
  "fixed": 0
}

This excerpt shows one rule’s summary.

Counters

Field What it means
matches Total nodes returned by the expanded selectors. Overlapping selectors can count a node more than once.
invocations Calls made to rule functions. A batch call counts once, even if it receives many nodes.
findings Findings retained for the rule after suppression and fixing.
ignored Inline ignore events plus findings removed by the suite’s ignore file.
fixed Findings reported as fixed by the engine.

Counters do not measure document coverage. A function receiving $ may inspect the entire model in one call.

Execution states

  • completed: execution finished. It may still have matched no nodes or produced findings.
  • excluded: the rule was excluded by format; reason explains the exclusion.
  • error: the rule returned an execution error.
  • timed_out: the rule did not finish within its limit.
  • cancelled: the invocation stopped before the rule completed.

Timed-out or cancelled workers cannot update counters after the result returns. A preparation failure may leave no evidence, but still fails the case with an execution error.

Inspect selector samples

vacuum test suite.yaml --evidence detailed --sample-limit 5

Each expanded selector includes up to five matched node locations. truncated: true means more nodes matched than were sampled. A sample limit of zero collects counters only.

Sample line and column values belong to the execution tree. Resolved trees can contain nodes from external files. Use diagnostic source, line, column, path, and paths for canonical finding locations.

To check a clean result, combine diagnostics: [] with a minimum invocation count. Add an ignored assertion to check suppression.