01 / WHY
Why evaluation is product workânot only engineering work
AI outputs are probabilistic, and many tasks have no single reference answer. If âqualityâ remains a feeling, teams are left reviewing a few demos, chasing an aggregate score, or learning from failures in production. The hard part of evaluation is translating user value, product boundaries, and risk into operational criteria. That is product judgment.
Define behavior
What must happen, what must never happen, and what is merely a difference in expression.
Design evidence
Which samples represent use, which metrics diagnose, and which regressions must block release.
Make the call
A judge supplies signals; the product owner still owns trade-offs and the release decision.
02 / LOCATE
Locate the problem before changing it
I fix the optimization order as Contract â Workflow â Prompt. The further left a concern sits, the more deterministic it isâand the less it should rely on a model âgetting the idea.â
Contract
Freeze inputs, outputs, schema, error semantics, and primary/fallback parity. Elegant prose is irrelevant if the product cannot consume it.
Workflow
Put routing, validation, retries, aggregation, and fallback into observable nodes. Express deterministic rules structurally.
Prompt
Let prompts handle semantic judgment: what counts as an action, what matters, and which uncertainties must remain.
Anti-pattern: append a sentence to the prompt for every bad output until no one can attributeâor safely removeâanything.
03 / MEASURE
An evaluation system must answer three different questions
Evaluation is not one score. It is a set of cross-checking evidence. Hard failures cannot disappear inside an average, and semantic gains cannot justify unreliable execution.
Contract & hard rules
JSON / schema / evidence / empty results / routing
Deterministic pass or failSemantic quality
Faithfulness, coverage, structure, specificity, readability
Rubric + model judge + human calibrationExecution & traces
Success, fallback, latency, tokens, repeatability
Tracked separately from semantic scoreTwo dataset tiers keep optimization from contaminating acceptance
Tierâ1 ¡ Core regression
Small, slow-changing, and strictly frozen. It asks only: did this change break a floor we already protected?
Tierâ2 ¡ Campaign set
Created around one weakness. It shapes a hypothesis, supports slices, and permits targeted iteration within the campaign.
Turn one optimization into an attributable campaign
- 01Freeze contract and baseline
- 02Freeze both dataset tiers
- 03State one primary hypothesis
- 04Make the smallest candidate change
- 05Repeat runs and three-layer evaluation
- 06Human-sign the verdict
04 / BUILD
The work underneath the method
The Playbook was not written first and illustrated later. It was distilled backward from auditing, curating data, iterating candidates, and blind-reviewing a real set of meeting AI workflows.
execution SOPs
Contract, dataset, candidate, evaluation, reporting, and blind review
LLM nodes
Evidence guards and root status constraints applied
nodes / edges
Graph structure held constant to isolate the variable
frozen cases Ă runs
Candidate cleared the Contract P0 gate twice
05 / DECIDE
A real decisionânot just an attractive win rate
In a blind A/B on 20 frozen cases, the candidate won 17 of 19 decisive cases. More importantly, the same review surfaced two long-context cases where the baseline was better.
decisive candidate win rate
17 wins / 2 losses / 1 tie
candidate P0 defects
baseline: 16 / 20
candidate faithfulness & evidence
baseline: 2.60 / 5
06 / TOOL
Why Dify Workflow
Dify was not chosen because a visual canvas automatically improves quality. Nodes, routes, variables, fallbacks, and an exportable DSL give product hypotheses an inspectable carrier. The team can discuss which guarantees belong in code, which judgments belong to models, and where information disappears. It is a useful test bench; evaluation discipline still has to be built around it.
07 / REUSE
Take the method with you
The public edition removes internal business, data, and environment details while keeping a reusable operating skeleton: two dataset tiers, three evaluation layers, single-variable campaigns, blind review, and explicit claim boundaries.
- Five-step flow and six-SOP index
- Dataset and campaign folder structures
- Release gates and verdict template
- Checklist of domain contracts to replace