The current decision catalog contains 39 cases across 13 skills.
Each case runs with supplied skill text and a control that omits it.
The five execution definitions contain 45 scenarios.
A missing fixture produces nonpassing
BLOCKED_SETUP without a model call.
No tracked fixture registry establishes live execution results.
Start a decision run
Use Node.js 22.22.0 or later and the locked dependencies. Prepare the checkout before a live run:codex login before the live command.
The SDK uses the existing login when OPENAI_API_KEY and CODEX_API_KEY are absent.
Those variables select API authentication when present.
Keep credentials outside tracked files and reports.
Run the live suite and view its result:
gpt-6-sol, low reasoning effort, and concurrency of two.
The complete paired catalog schedules 78 calls.
Each case starts a fresh thread in an empty, read-only workspace.
Shell tools, web search, network access, host discovery, plugins, and apps stay off.
The prompt supplies the skill and case-selected references.
The control receives neither. The model must use the supplied evidence without tools.
The runtime adapter creates a temporary Codex home and links the existing login file.
It excludes personal configuration and removes its temporary directory on normal process exit.
The command disables telemetry, update notices, sharing, and response caching.
The JSON report uses .promptfoo/results/latest.json under the ignored report tree.
The local viewer keeps earlier runs. Save a named report for a comparison:
Read the decision result
The case catalog owns each scenario, choices, expected answer, rationale, and routed references. Cases include core engineering decisions, session regressions, CI evidence scenarios, and model instruction design. Keep additions in this owner instead of a parallel campaign runner. The grader compares the exact selected IDs with the independently authored expected set. It rejects missing, extra, invented, duplicate, and malformed choices. The explanation needs at least 20 characters. The grader does not assess its meaning. Read explanations before you accept a baseline. Each row records the skill path and content digest, supplied task digests, case-set digest, and grader digest.referenceSources records every explicitly supplied reference and its digest.
The sha256: prefix preserves public content hashes through promptfoo’s secret filter.
Provider settings, outputs, and token usage also belong to the result.
Compare runs only after their sources, cases, settings, and evidence match.
An equal paired score shows no measured decision benefit from the skill.
A decision score does not prove automatic routing, successful edits, compiler results, or native host consumption.
A single run does not establish a stable improvement.
The historical campaign and fixed execution cases preserve earlier runs and unassessed conditions.
Decision adaptations do not replay those tasks or transform historical ties into improvement claims.
Add a decision regression
Editsource/evals/promptfoo/cases.mts.
Use a stable ID for a concrete failure pattern.
When the case changes meaning, change its ID.
Keep the expected answer and rationale outside the rendered prompt.
Include a legal choice and its nearest plausible error.
Distribute correct choices across positions.
If the decision needs a bundled procedure, declare its skill-relative paths in references.
A missing reference fails the harness. The report records the supplied bytes.
Run the offline controls before the focused live case:
Run execution scenarios
The execution loader reads each validated benchmark definition and its scenarios. The runtime coordinates fixture setup, native installation, streamed turns, evidence, and independent assessment. Historicalplugin-eval analyze commands remain provenance, not current run instructions.
Supply an explicit, ignored fixture registry through SLOPSENTRAL_EXECUTION_FIXTURES.
Each entry binds a case ID to an absolute repository path, a full commit SHA, and task context.
The case ID selects its plugin target and scenario.
The entry also pins a host verifier by absolute path and content digest:
.js, .mjs, or .mts.
Hashes use sha256: plus 64 lowercase hexadecimal characters.
The revision uses the exact 40-character Git SHA.
Task context must not contain the expected answers.
Unknown and duplicate case IDs fail fixture admission.
An optional pinned prepare script provisions dependencies or seeds unrelated work before the initial status snapshot.
It receives a JSON request-file path with fixture, workspace, and artifacts.
A nonzero result blocks inference. The harness infers no dependency setup from a rubric.
origin remote. Agent staging cannot change the operator’s index or refs.
HOME and CODEX_HOME are isolated. Agent scratch stays inside its worktree.
Setup creates a fresh Codex projection and installs its native plugin.
Pkl 0.32.0 must exist on PATH or at PKL_BIN.
The current definitions select gpt-6-astra, workspace-write, and approval policy never.
The runtime preserves the definition’s settings and disables web search, sandbox network, replay retries, and response caching.
Setup and execution share a ten-minute budget. Post-run assessment has separate bounded timeouts.
These trusted local fixtures do not form a hostile-code security mechanism.
Review preparation and verifier scripts before a run.
Their dependency closure must be self-contained or independently pinned.
They run as host code and must stay within the disposable fixture.
They receive no authority for remote writes.
Assess execution evidence
The host verifier receives a request-file path withcase, fixture, workspace, artifacts, traceSha256, and completed.
It emits JSON to stdout and writes diagnostic artifacts beneath artifacts:
successChecklist string once.
Assess route and scope separately.
Every passing finding needs a regular evidence file relative to the artifact directory.
The trace digest must match the request.
Missing files, escaping paths, stale traces, incomplete turns, and unassessed findings cannot pass.
The harness also runs each benchmark’s original verifier commands and keeps their outputs and exit codes.
A failed command prevents a pass.
Examine its cause before you attribute failure to the target skill.
The host verifier must inspect actual effects, successful tool results, delivered skill bytes, forbidden actions, and unrelated-work preservation.
An echoed skill path or final answer cannot establish those facts.
The harness establishes evidence identity and completeness. It cannot establish the truth of an arbitrary verifier.
Never replace independent assessment with a script that copies checklist items into passing findings.
Leave unavailable native tools, companion routes, recovery, and performance evidence unassessed.
Keep the original SDK events, including failed partial turns, under ignored .promptfoo/execution/.
The same directory keeps fixture identities, projected bytes, installation records, initial and final status, diffs, verifier requests, and evidence.
The report uses .promptfoo/results/execution.json.
Read it with those artifacts before a result claim.
A measured comparison needs matching fixture revisions and settings, fresh turns, and independent effects evidence.
Successful installation, static controls, golden answers, and historical outcomes cannot substitute for that evidence.
Use the documentation practice when source changes make this guide or its linked explanation stale.