Use a stable task set and assert outcome, files, tools, latency and usage—not subjective vibe.
The decision behind the technique
A workflow can look better on one demo while regressing other repositories or failure modes. Repeatable evaluations make tradeoffs visible.
Use a stable task set and assert outcome, files, tools, latency and usage—not subjective vibe.
A practical step-by-step workflow
Use this sequence as a default, then tighten or expand it only when the evidence changes.
- Select representative tasks with known acceptance checks
- Record the current workflow as a baseline
- Change one major variable at a time
- Compare correctness, side effects, latency and usage
Failure boundary and finish line
Do not optimize only for pass rate if the new workflow becomes unsafe, expensive or destructive.
Finish when another person can reproduce the check against the exact final state.
Frequently asked questions
What should I do first?+
Select representative tasks with known acceptance checks
What is the main failure mode?+
Do not optimize only for pass rate if the new workflow becomes unsafe, expensive or destructive.
OFFICIAL SOURCES
Verify against the source
Promptfoo coding-agent evaluationsThis guide provides general information, not an OpenAI allowance promise or support decision. Your account and official pages govern actual usage, benefits and recovery times.