Skip to main content
Schemas define accepted external records. Evaluations examine decisions or task execution under explicit conditions. Each proves a different claim.

Schemas

source/schemas/ owns JSON contracts for primitives, plugins, profiles, hooks, Pi packages, and evaluation records. A schema describes accepted fields and finite variants. A parser or graph resolver adds filesystem, identity, and lifecycle rules. For example, resource references have an external schema. The graph resolver also examines the target’s file kind and dependency closure. A schema alone cannot establish that a target exists. Change the owning schema, producer, consumer, and accepted or rejected fixtures together. Keep compatibility decisions explicit. JSON Schema Management supplies the authoring procedure.

Evaluation layers

source/evals/ owns evaluation definitions and fixtures. tools/promptfoo/ owns the runtime adapters. The evaluation guide defines commands and report locations. source/evals/promptfoo/cases.mts owns active decision cases and expected answers. Extend that owner instead of a parallel catalog. Record the supplied reference hashes. The execution definitions under source/evals/promptfoo/execution/ need an explicit ignored fixture registry. Each fixture binds the scenario, repository, full commit SHA, context, and independent verifier identity. Missing fixtures produce nonpassing BLOCKED_SETUP without a model call. The model does not grade its own result.

Commands

After evaluation changes, run the offline gate:
For fresh decision evidence, run:
The latter command needs a configured Codex runtime and model access. Normal CI uses offline controls. An offline pass does not establish a live model pass or measured improvement.