Skip to main content
An evaluation result needs the exact input, source identity, runner conditions, and independent acceptance rule. The evaluation guide gives setup and report commands. No fixture or offline control establishes a fresh model outcome.

Decision responses examine supplied guidance

source/evals/promptfoo/cases.mts owns active decision cases and expected answers. The suite adapter reads canonical skill text and case-specific routed references. It records their hashes and pairs skill inputs with no-skill controls. The decision runner disables tool execution, automatic host skill discovery, apps, plugins, network access, and web search. The response therefore shows a decision under supplied guidance. It does not prove automatic routing, file changes, native tool use, or host installation.
The first command runs types and offline controls. The second gets fresh model responses through the pinned Codex SDK. Normal CI runs the offline command without model calls or credentials. Raw reports stay under ignored .promptfoo/.

Execution scenarios need real pinned fixtures

Five plugin benchmark definitions declare 45 execution scenarios. The runtime needs an explicit fixture registry through SLOPSENTRAL_EXECUTION_FIXTURES. Each fixture binds the target, scenario, absolute repository path, full source revision, context, and independent verifier file and digest. Missing fixtures produce nonpassing BLOCKED_SETUP without a model call. A ready case uses a disposable private checkout and worktree. Its isolated HOME and CODEX_HOME preserve operator configuration. The runner projects fresh packages, installs through the native host, and records identities before the model turn. It preserves the benchmark’s model, sandbox, and approval settings. Network access, web search, retries, and caching stay disabled.
The runner keeps streamed JSONL events, including failed partial turns, initial and final Git status, diffs, fixture identities, and verifier outputs. The independent host verifier supplies evidence for every exact checklist item plus route and scope. Missing or unassessed evidence cannot pass. The model cannot grade itself.

Preserve historical scope

Routing fixtures describe expected routes. Archived assessments and field records describe their recorded inputs, revisions, and conditions. A retired route needs a fresh replay before it supports its replacement. Static package scores do not establish activation, useful outcomes, or measured improvement. The offline execution tests exercise adapters and grading contracts. They do not turn missing live fixtures into synthetic passes. A measured comparison needs matched conditions and actual fresh results. Use validation layers to state the mechanism that each result supports.