Agent test harness API

Use harness to run the same tests as the CLI.

package main

import (
    "context"
    "encoding/json"
    "os"

    "github.com/daveshanley/vacuum/harness"
)

func main() {
    result := harness.RunFile("suite.yaml", harness.RunOptions{
        Context: context.Background(),
        Version: "your-build-version",
        SampleLimit: 5,
    })
    if err := json.NewEncoder(os.Stdout).Encode(result); err != nil {
        panic(err)
    }
    os.Exit(result.ExitCode)
}

Load reads a suite file. Parse reads suite bytes. Suite.Validate checks a suite built in Go. Run executes it using a suite path for relative file resolution. RunFile loads and executes in one call. Schema("suite") and Schema("result") return the same schemas exposed by the command. Report.Errors and CaseResult.Errors contain harness.ExecutionError values. Use the typed Code and Phase constants and identity fields to handle failures. Message explains the error. Motor callers can inspect motor.RuleExecutionError with errors.As and retain its cause through errors.Is.

Custom functions and repairs

Use the JavaScript function guide or Go function guide to write custom functions.

Supply model.RuleFunction implementations through RunOptions.CustomFunctions and model.AutoFixFunction callbacks through RunOptions.AutoFixFunctions.

The harness treats JavaScript runtime failures, core-function bridge panics, and repair callback failures as execution errors. plugin.LoadFunctionsStrict also rejects script and schema loading failures and bounds JavaScript metadata evaluation. For custom Go functions, call RuleFunctionContext.ReportError(err) when it is non-nil to report an execution failure separately from findings. Call it synchronously during RunRule, never after the function returns. Direct motor callers can enable this callback with ExecutionOptions.ReportFunctionErrors.

Each case gets its own document state. Cases execute serially; rules within a case use the configured worker limit. Registered functions must be safe for concurrent rule execution. Cancellation does not stop Go callbacks. A timed-out callback may still be running, though its incomplete output is discarded.

The runner copies diagnostics and releases its motor resources. Callers own their registered functions. Read the report after Run completes.

The CLI isolates execution in a child process to keep plugin output off its JSON stream. The Go API does not redirect process-wide stdout or stderr; callers control how their own functions log.

Collect engine evidence directly

evidence := &motor.ExecutionEvidence{SampleLimit: 5}
result := motor.ApplyRulesToRuleSetWithOptions(execution, &motor.ExecutionOptions{
    Context: ctx,
    Evidence: evidence,
})
defer result.ReleaseOwnedResources()

// Inspect evidence.Rules after execution has returned.

Use a fresh evidence value for each invocation. Do not share it between concurrent runs. Leave Evidence nil when normal linting does not need these records. motor.ValidateRuleSet provides the harness’s strict function-name and option preflight without executing a fixture.