/taste-evalsNEW
An eval plan or results report with a calibrated judge, blind pairwise results and a pre-registered decision.
Then type /taste-evals in Claude Code, or describe the task and let it pick the skill.
What you get from the skill's Output section
Return one artifact:
- eval plan: change and decision, held-out set, arms, judge and calibration plan, tripwires, pre-registered rule and n;
- results report: manifest, judge calibration with intervals, tripwires, wins, losses and ties with an interval, human review and the decision;
- audit: violations ranked by impact, the direction of each bias and whether the verdict stands.
No sample run on this page yet.
When to use it
Use when a prompt, model or pipeline change has to prove its posts, captions or copy got better.
Skip it for live traffic tests (product-experiments) or correctness and cost evals (ai-product-development).
The thinking behind itA judge that was never checked against real hits and flops can be confidently backwards. Calibrate it first, blind every comparison, write the rule before the run, and say so when the sample can't decide.
How it checks its own work from the instructions
- Held-out items are excluded in code from prompts, retrieval and anchors
- The judge is calibrated against a bar set before the run
- Both orders ran and the flip rate is reported
- Language matches the interval
Requirements what it needs
- Works in
- Codex, Claude Code, Cursor
- Needs
- project files, terminal commands, files you provide
The instructions SKILL.md · 129 lines
Instructions & references
Back to overview ↑On this page 12 sections
Taste evals
Prove that a change makes subjective output better before it ships. Posts, captions, copy, summaries and designs described in text have no answer key. A judge's score means nothing until it has been checked against real outcomes. A win means nothing if the comparison could tell which arm was which.
Choose a mode
Infer the mode from what the user brings. Ask only if two modes fit equally.
- Design: a planned change, no results yet. Return an eval plan with the decision rule written before any run.
- Run and analyze: arms to compare, or outputs and scores already in hand. Return a results report.
- Audit: an existing eval, judge, scorecard or claim such as "the new prompt wins 70%". Return findings ranked by how much each could change the decision, and say whether the existing verdict survives.
Scope boundaries:
- Live traffic tests with assignment, exposure and conversion belong to the
product-experimentsskill. This skill decides which candidate is worth exposing to users. - Correctness, format adherence, refusal, latency and cost evals belong to the
ai-product-developmentskill. This skill covers the outputs where a person's taste decides what is good. - Factual accuracy of a generated claim is checked against sources, not judged for taste. A tripwire can flag a claim; it cannot verify one.
Inspect before asking. Read the generator, its prompts, retrieval and memory, any existing eval scripts and results, logs of user picks (post, copy, edit, star, keep, kill), and published outcomes with engagement. Ask only about what changes the design:
- what counts as a real hit, when no recorded outcome exists in the data;
- who the human tiebreaker is;
- the spend ceiling for judge calls, when a full run would be expensive.
Rules
Each rule exists because breaking it produces a confident wrong answer.
- Freeze the set and the context. Write a manifest before generating: the inputs, each arm's exact system prompt and user message per slot, retrieval results, few-shot examples, model and settings, seeds, and the judge's rubric and anchors in a fixed, seeded order. Retries replay the frozen prompts. Store the judging context with the scores. If context differs between arms or between runs, the comparison measures the drift.
- Hold out. Eval items never appear as few-shot examples, retrieval candidates, judge anchors or memory, for the generator or the judge. Enforce it with ID exclusion in code, not intention. When an arm must reproduce a real held-out item, brief it with a neutral one-line topic that carries none of the original's wording, structure or punchline.
- Keep failures in the denominator. A failed, empty or truncated generation scores as the worst outcome for its arm. Never drop it. Keep every failed candidate, every rejected draft and the judge's stated reasons; they are the analysis material.
- Blind and randomize position. Opaque labels. Position shuffled per input with a stable seed, so reruns match but no arm owns a column. Arm names never appear in the judge prompt or on a review page. Run every pairwise judgment in both orders. A verdict that flips with order is a tie. Report the flip rate.
- Calibrate the judge before its score is used. Test it on labelled real outcomes: known hits and flops, or a person's real picks among candidates. Report AUC with an interval for scores, agreement against chance with a Wilson interval for picks, and kill recall with kappa for keep or kill. Calibrate in the form the eval uses: a pairwise judge on hit-versus-flop pairs, a pick judge on real pick sessions. Compare to a bar written before the calibration run. Below the bar, the judge is not used. AUC well below 0.5 means the judge is inverted. Report that as a finding, change the design (often absolute to pairwise) and recalibrate. Do not flip the sign and ship it. Every rubric revision scored on the same calibration set spends it: iterate on a development split and confirm the final judge once on a split it has never seen.
- Prefer pairwise against a real reference. "Which of these two did the author write" or "which would you post", with the nearest real example as the rival, beats "rate this 1 to 10". A pairwise discrimination score has a fixed meaning: 0.5 is indistinguishable from real. Absolute scores drift with the rubric and reward polish. When the held-out set is real inputs with no real output to compare against, judge arm against arm, pairwise.
- Use an independent judge. Pick a model family different from the generator and from any critic inside the generation loop. Models favor their own outputs. If the loop's critic is also the eval judge, the eval measures how well the loop learned to please its critic.
- Tripwires for hard rules. Deterministic code, per arm, no model: length, banned phrases, em dashes, hashtags, repeated openers within a batch, named templates, invented numbers, and invented first-person claims (any claim about the author's life or work not supported by the input or the author's real record). Add distribution fit: the rate of each feature in the outputs against the base rate in the real corpus. Tripwires catch what judges forgive and explain why a judge prefers what it prefers.
- Pre-register the decision rule and the sample size. Before any generation or judge call: the primary metric, the arms, n, the judge's calibration bar, the win bar, and the action for a win, a loss and a null. Commit it, or record its hash and timestamp, before the first call.
- No re-scoring after seeing results. Do not swap the judge, rubric, anchors, exclusions or thresholds because of which arm they favor. A judge change is legitimate only when calibration motivated it and it is applied to every arm's saved samples. Report the old and new scores side by side.
- Human blind review is the tiebreak. Use it when the judge fails calibration, when the result sits inside its interval, or when judge and tripwires disagree. The decider sees one shuffled deck of single cards with arms mixed and the key kept outside the page. Any prefix of a shuffled deck is a fair sample, so a partial review still counts. Blind human picks outrank the judge.
- Report the effect with its uncertainty. The input is the unit: when an arm writes several drafts per input, reduce them to one verdict per input before counting. Wins, losses and ties per input, win rate on decisive pairs with a Wilson 95% interval, an exact sign test, and what this n can resolve. When the interval spans no difference, write "the eval cannot resolve this", give the n that would, and label any decision made on other grounds as unmeasured.
Restraint
- Never fabricate hit or flop labels, and never derive them from a model's opinion. Labels come from recorded outcomes: engagement in a fixed window, a person's posts, copies, edits, keeps or kills, an editor's accepts. If none exist, say the judge cannot be calibrated, run blind human review, and mark any judge result "uncalibrated, directional only".
- Check labels by hand before calibrating. Trace three to five labelled items to the real published artifact. Logging bugs, such as recording a card's screen position while storing drafts by planned slot, mislabel silently.
- Small samples get directional language: "leans toward", "no evidence of a difference". Never "the same", "proven" or "significantly better" from an underpowered test.
- Do not optimize to the judge. A generator tuned against the eval judge will win the eval. Keep the eval judge out of the loop, and watch tripwires and human picks for divergence from the judge.
- A change may ship on mechanism (cheaper, simpler, more principled) after a null result. Record it as "not a measured win".
- A run that dies partway, for example on an exhausted API balance, is not interpretable unless the call order was random by design. Log spend per run and stop at the ceiling.
- A judge that agrees with users in general has not been shown to agree with the specific decider. Calibrate on the population whose taste the decision serves, and report other populations as secondary.
Workflow
Design
- Name the change, the decision it serves, and who decides. One change per arm where possible. Hold model and retrieval constant when testing a prompt.
- Inventory evidence: recorded outcomes usable as labels, real items usable as held-out references, and any existing harness. Decide the held-out shape: real input-output pairs allow pairwise against the real output; real inputs alone allow arm against arm. Extend an existing harness before writing a new one.
- Size the held-out set from the smallest effect worth acting on. Read calibration and statistics for the power table. State the smallest win rate this n can detect.
- Build the calibration set from real outcomes: clear hits and clear flops, the middle band dropped, held-out items and anything in the judge's context excluded. Leave one out when the judge uses few-shot examples from the same pool.
- Choose arms: the baseline is what production runs today. Add challengers. Include a real-versus-real control when judging discrimination, to see the judge's ceiling.
- Choose the judge: independent family, pairwise, both orders, rubric and anchors frozen.
- Write the tripwires and the distribution-fit checks.
- Write the decision rule, the calibration bar and n. Pre-register them.
Run and analyze
- Confirm a committed plan exists. If not, write it now, before looking at any output, and label results already seen as exploratory.
- Freeze the manifest. Generate every arm. Keep failures.
- Run tripwires on every arm.
- Calibrate the judge for this exact rubric, anchors and model. If it misses the bar, stop the judge path and go to blind human review. Do not re-score earlier runs with it.
- Judge blind, both orders. Record the flip rate and the stated reasons.
- Analyze with calibration and statistics. Break results down by input category, with each category's n.
- Send ties, in-interval results and judge-tripwire conflicts to blind human review.
- Apply the pre-registered rule. Write the decision and what would change it.
Audit
Check the eval against rules 1 to 12. For each violation, state what it could bias, in which direction, and whether the verdict survives. Common findings:
- the judge was never calibrated, or was calibrated on examples sitting in its own prompt;
- the judge shares a model family with the generator or the in-loop critic;
- held-out items leak into few-shot examples or retrieval;
- failed generations were dropped from the denominator;
- judging ran in one order only;
- anchors were fetched live and unordered, so the rubric drifted between runs;
- a significant verdict from a judge with no demonstrated agreement with the decider;
- a non-significant small effect read as "no difference".
References
- Calibration and statistics: read when building a calibration set, computing AUC, agreement, kappa, the sign test, intervals or power, or setting a bar.
- Plan and results templates: read when writing the eval plan or the results report.
Output
Return one artifact, using the templates:
- Eval plan (design mode): change and decision, held-out set, arms, judge and its calibration plan, tripwires, pre-registered rule and n, the smallest detectable effect, spend estimate.
- Results report (run mode): set and manifest location, arms, judge calibration with intervals, tripwire table, results with W/L/T, win rate and interval, human review, decision, and what would change it.
- Audit (audit mode): violations ranked by impact, the direction of each bias, whether the verdict stands, and the smallest fix that would make it trustworthy.
Separate what was measured from what was inferred. Every number carries its n.
Verify
- The held-out IDs are excluded in code from every prompt, retrieval call and judge anchor. Grep for the exclusion; do not assume it.
- The manifest replays: rerunning one slot from it reproduces the same prompt bytes.
- Failed generations appear in the denominator.
- Position is shuffled, both orders ran, and the flip rate is reported.
- The judge's calibration result, its bar and the bar's timestamp are in the report, and the bar predates the calibration run.
- The judge's model family differs from the generator's and the critic's.
- The decision follows the pre-registered rule, or the report says why not.
- Language matches the interval: no "better" when the interval spans no difference.
Sources checked 2026-11-04
Research findings, not vendor defaults. Re-check if a newer study is cited against them.
- Position bias in LLM judges, and balancing by swapping order: Wang et al., "Large Language Models are not Fair Evaluators", https://arxiv.org/abs/2305.17926
- Position, verbosity and self-enhancement biases in LLM judges: Zheng et al., "Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena", https://arxiv.org/abs/2306.05685
- Self-preference linked to self-recognition: Panickssery et al., "LLM Evaluators Recognize and Favor Their Own Generations", https://arxiv.org/abs/2404.13076
- Pairwise preference outperforming direct scoring on the tasks studied: Liu et al., "Aligning with Human Judgement: The Role of Pairwise Preference in Large Language Model Evaluators", https://arxiv.org/abs/2403.16950
Calibration and statistics
The math behind rules 5, 6 and 12. Every formula here is standard; the defaults for bars and set sizes are heuristics to adapt, and the user's pre-registered numbers override them.
Contents
- Build the calibration set
- Score judges: AUC
- Pick judges: agreement against chance
- Keep or kill judges: the base-rate trap
- Set the bar before the run
- Gates inside the generation loop
- Pairwise discrimination against a real reference
- Compare arms
- What this n can resolve
- Snippets
- Sources checked 2026-11-04
Build the calibration set
- Choose the outcome. Use the outcome the decision serves. Engagement predicts reach. A person's posts, copies, edits and kills predict that person's taste. They are different judges, and calibrating on one says little about the other.
- Fix the window and the population. Count engagement over the same window for every item. Use one author, one channel and one period where audience size was roughly stable. Exclude items whose outcome was driven by something the judge cannot see: attached media for a text-only judge, a paid boost, a reply thread.
- Drop the middle. Take clear hits (for example the top band by engagement) and clear flops (the bottom band), and leave the middle out. A judge that cannot separate the extremes will not separate the middle. Keep a middle band only for a secondary check, such as mids against flops.
- Exclude what the judge has seen. Remove every item that sits in the judge's prompt as an anchor or few-shot example, every item that copies or paraphrases another item, and every held-out eval item. Leave one out when the judge's examples come from the same pool: when scoring item i, its examples exclude i and anything sharing i's idea.
- Check the labels by hand. Trace three to five items from each class to the real artifact. Confirm the stored text is what was actually published or picked. Record the check in the report.
- Balance and size. Sample the same number from each class with a fixed seed. Fewer than about 20 per class gives an interval too wide to act on (see the table below). Report n per class.
Score judges: AUC
AUC is the probability that the judge scores a random hit above a random flop, ties counted as half. It needs no threshold and no assumption about the score scale.
- 0.5: the judge cannot tell hits from flops.
- Above 0.5: it ranks hits higher. Read the lower bound of the interval, not the point.
- Well below 0.5: the judge is inverted. It ranks flops higher. An absolute "does this sound like the author" score can do this when it rewards polish the author does not have. Report it, change the design and recalibrate. Do not negate the score and ship: an inverted judge has shown it is measuring something else.
Interval: bootstrap over items (resample hits and flops separately, 2,000 draws, 2.5th and 97.5th percentiles), or the Hanley and McNeil standard error:
Q1 = A / (2 - A)
Q2 = 2A^2 / (1 + A)
SE = sqrt( (A(1-A) + (n1-1)(Q1 - A^2) + (n2-1)(Q2 - A^2)) / (n1 n2) )
95% interval = A ± 1.96 SE
Approximate 95% intervals from that formula, equal n per class (computed 2026-11-04):
| per class | AUC 0.60 | AUC 0.67 | AUC 0.70 | AUC 0.78 |
|---|---|---|---|---|
| 20 | 0.42 to 0.78 | 0.50 to 0.84 | 0.54 to 0.86 | 0.64 to 0.92 |
| 30 | 0.46 to 0.74 | 0.53 to 0.81 | 0.57 to 0.83 | 0.66 to 0.90 |
| 50 | 0.49 to 0.71 | 0.56 to 0.78 | 0.60 to 0.80 | 0.69 to 0.87 |
| 100 | 0.52 to 0.68 | 0.60 to 0.74 | 0.63 to 0.77 | 0.72 to 0.84 |
So at 50 per class, a signal scoring 0.60 has not shown it beats chance, while 0.67 has. Drop a signal whose lower bound sits at or below 0.5, even if its point estimate looks useful.
Calibrate in the form the eval uses. A judge that will compare two candidates gets hit-versus-flop pairs on similar topics, each item used once, both orders. Its pair accuracy is the AUC equivalent; put a Wilson interval on it. A judge that will score items one at a time gets AUC.
When a judge returns several signals, compute AUC for each and for any combination you plan to use. Pick the combination before looking at AUCs, or treat the choice as exploratory and confirm it on a fresh sample.
Pick judges: agreement against chance
When the label is "which of k candidates the person chose", score the judge by top-1 agreement: the judge's top candidate is one the person chose.
- Chance per session = chosen / k. A session where the person kept 3 of 4 has chance 0.75. Sessions where every candidate was chosen carry no information; drop them.
- Expected hits = the sum of per-session chance. Mean chance = expected hits / sessions.
- Interval: Wilson 95% on agreement. The judge passes when the Wilson lower bound sits above mean chance.
- p-value: the exact Poisson-binomial tail P(hits ≥ observed), since chance differs per session. A binomial against 1/k flatters the judge when some sessions have several chosen candidates.
- Average out position: show each session under at least three shuffles and average each candidate's position-free rank before taking the top pick.
- Compare two judges on the same sessions with an exact McNemar test on the discordant sessions (one right, the other wrong). Similar agreement with McNemar p near 1 means no evidence of a difference between them.
Worked example: 9 hits in 19 sessions is 47%, Wilson 95% 27% to 68%. If mean chance is 34%, the lower bound is below it, so the judge has not shown agreement, even though 47% looks better than 34%.
Keep or kill judges: the base-rate trap
When the person keeps most candidates, raw agreement is meaningless. If they keep 78%, a judge that keeps everything agrees 78% of the time and catches nothing. Report:
- Kill recall: of the person's kills, how many the judge also killed. This is the number that matters when the judge screens a library.
- Kill precision: of the judge's kills, how many the person also killed.
- Cohen's kappa: (observed agreement − chance agreement) / (1 − chance agreement), where chance agreement comes from each rater's own keep rate. Kappa near 0 is chance.
A judge that catches 1 of 9 kills is at chance whatever its raw agreement.
Set the bar before the run
Write the bar in the plan and commit it before the first calibration call. Heuristic defaults, to adjust to the stakes:
- Score judge: AUC lower 95% bound above 0.5, on at least 30 per class.
- Pick judge: Wilson lower bound above mean chance.
- Keep or kill judge: kill recall at least one half and kappa lower bound above 0.
Split the labelled outcomes before iterating on the judge. Revise the rubric against a development split as often as needed; score the final judge once on a confirmation split. A judge tuned until it passes on the set it is graded on has passed nothing.
State what the bar costs in power. At 19 sessions with mean chance 34%, passing takes 11 hits. A judge whose true agreement is 50% clears that about a third of the time. Say so in the plan, so a miss is read as "not shown", not "shown to fail".
Gates inside the generation loop
A critic that decides whether a draft is shown needs a threshold. Set it from where real items sit, not from a round number: score the real held-out items and the first drafts, then choose the cut that most real items clear and fewer drafts do. Record both rates next to the threshold, for example "60% of real items clear it, 27% of first drafts do". A second gate (say, predicted performance) gets its own cut from hits against flops.
The in-loop critic and the eval judge must differ. The eval exists to check the loop.
Pairwise discrimination against a real reference
For voice or style: pair each generated item with the real item on the same topic, or the nearest real item by embedding. Ask the judge which one the author wrote and how confident it is (0.5 to 1.0). Run both orders.
p_order1 = confidence if judge picked the generated item as real else 1 - confidence
p_order2 = the same with positions swapped
p_item = (p_order1 + p_order2) / 2
score = mean of p_item over items
0.5 means the judge cannot tell generated from real. 0.0 means it always spots the imitation. Also report the share of items with p_item above a pre-set floor, and the judge's stated tells, which feed the next prompt change.
Control: run the same judge on real-versus-real pairs (two real items on similar topics, one labelled "imitation"). The judge should land near 0.5 there. If it does not, it is keying on something other than authorship.
When the judge changes, re-run it on every saved sample from every arm, including old baselines, so all versions are read by the same reader. Report old and new judge numbers side by side.
Compare arms
The unit is the input. Per input, compare the challenger to the baseline: win, loss or tie (a tie includes an order-flipped verdict).
- Primary: win rate on decisive pairs = wins / (wins + losses), with a Wilson 95% interval.
- Test: exact two-sided sign test on wins against losses.
- Secondary: mean rating difference, per-category W/L/T with each category's n, tripwire rates per arm.
- Several drafts per input: reduce to one verdict per input (judge the batch as a whole, or compare each arm's best draft) before counting. Drafts from one input are not independent pairs; counting them inflates n.
- Several challengers against one baseline: name the primary comparison in the plan. Treat the rest as secondary, or correct for multiple comparisons.
Worked example: 16 wins, 9 losses, 5 ties on 30 inputs. Win rate 16/25 = 64%, Wilson 95% 45% to 80%, sign test p = 0.23. The interval spans 50%, so the eval cannot resolve this.
What this n can resolve
Decisive pairs needed for 80% power, exact two-sided sign test at alpha 0.05 (computed 2026-11-04, rounded):
| true win rate | decisive pairs |
|---|---|
| 0.80 | about 25 |
| 0.75 | about 35 |
| 0.70 | about 55 |
| 0.65 | about 90 |
| 0.60 | about 210 |
| 0.55 | about 800 |
Add the expected tie share to get inputs. At 23 decisive pairs, a true win rate of 0.61 is detected about 15% of the time. Print this next to every result, so a null at small n reads as "this harness cannot see it", not "there is no difference".
Snippets
Standard library only. Adapt to the project's language.
import math
from math import comb
def auc(pos, neg):
wins = sum((p > n) + 0.5 * (p == n) for p in pos for n in neg)
return wins / (len(pos) * len(neg))
def wilson(k, n, z=1.96):
p, d = k / n, 1 + z * z / n
c = (p + z * z / (2 * n)) / d
h = z * math.sqrt(p * (1 - p) / n + z * z / (4 * n * n)) / d
return c - h, c + h
def sign_test(wins, losses):
n, k = wins + losses, max(wins, losses)
return min(1.0, 2 * sum(comb(n, i) for i in range(k, n + 1)) / 2 ** n)
def kappa(a, b): # two lists of booleans (keep = True)
n = len(a)
po = sum(x == y for x, y in zip(a, b)) / n
pa, pb = sum(a) / n, sum(b) / n
pe = pa * pb + (1 - pa) * (1 - pb)
return (po - pe) / (1 - pe) if pe < 1 else float("nan")
Sources checked 2026-11-04
- AUC as the probability a random positive outranks a random negative, and its standard error: Hanley and McNeil, "The meaning and use of the area under a receiver operating characteristic (ROC) curve", Radiology 143(1):29-36, 1982, https://pubmed.ncbi.nlm.nih.gov/7063747/
- Score interval for a proportion: Wilson, "Probable Inference, the Law of Succession, and Statistical Inference", Journal of the American Statistical Association 22:209-212, 1927, https://doi.org/10.1080/01621459.1927.10502953
- Position bias and balancing by swapping order: https://arxiv.org/abs/2305.17926
- Tables in this file were computed from the formulas above, not taken from a source.
Plan and results templates
Fill the plan before any generation or judge call, commit it, then fill the results from the same headings. Leave a heading in with "not run" rather than deleting it. Every number carries its n.
Contents
Eval plan
# Eval plan: <change in one line>
Committed: <date, commit hash or file hash> (before any generation or judge call)
Decision this serves: <ship B over A as the default for X>
Decider and tiebreaker: <who reviews blind when the judge cannot settle it>
## Held-out set
- Source and selection rule: <e.g. 30 real items from <period>, fixed seed, filters>
- Exclusion enforced in: <file and function that removes held-out IDs from prompts, retrieval, anchors>
- Brief per item: <neutral one-line topic; how it was written so it carries no wording from the original>
- Manifest: <path; contains prompts per slot, retrieval, examples, model, settings, seeds, judge rubric and anchors>
## Arms
| arm | what changes | held constant |
|---|---|---|
| baseline | what production runs today | |
| challenger | | model, retrieval, examples |
## Judge
- Model family: <differs from generator family and from in-loop critic family>
- Form: <pairwise vs nearest real item | pick best of k | keep or kill>, both orders, position seeded per input
- Calibration set: <outcome used as label, window, hits n, flops n, middle band dropped, exclusions, leave-one-out rule>
- Label check: <which items were traced to the real artifact>
- Bar: <e.g. AUC lower 95% bound above 0.5>. If missed: judge not used; blind human review decides.
## Tripwires (deterministic, per arm)
- <length range, banned phrases, em dashes, hashtags, repeated openers, named templates, invented numbers, invented first-person claims>
- Distribution fit against the real corpus: <features and their base rates>
## Decision rule
- Primary metric: <win rate on decisive pairs, challenger vs baseline>
- n: <inputs>, expected decisive pairs <n>, smallest detectable win rate at 80% power: <x>
- Ship challenger if: <Wilson lower bound above 0.5 and no tripwire rate worse by more than <y> points>
- Keep baseline if: <upper bound below 0.5, or a tripwire regression>
- Otherwise: the eval cannot resolve this. Then: <ship on mechanism and label unmeasured | blind human review | grow n to <z>>
- No re-scoring: judge, rubric, anchors, exclusions and thresholds are fixed by this file.
## Spend
- Estimated calls and cost: <n calls, ceiling>. A run that stops at the ceiling is not interpreted.
Results
# Eval results: <change in one line>
Plan: <path and commit> Run: <id, date> Manifest: <path>
Status: <ship | keep baseline | cannot resolve | invalid run>
## Set and arms
<n inputs, n per arm generated, n failed per arm (kept as worst outcome)>
## Judge calibration
| signal | n hits | n flops | AUC or agreement | 95% interval | bar | passed |
|---|---|---|---|---|---|---|
<for pick judges: hits, sessions, mean chance, Wilson interval, Poisson-binomial p>
<for keep or kill: kill recall, kill precision, kappa>
Position flip rate: <x% of pairs changed verdict with order>
Finding: <e.g. absolute critic inverted; pairwise variant passed>
## Tripwires
| arm | template match | banned phrase | repeated opener | invented claim | median length | other |
|---|---|---|---|---|---|---|
## Results
| comparison | W | L | T | win rate (decisive) | Wilson 95% | sign test p |
|---|---|---|---|---|---|---|
Per category: <category, n, W/L/T>
What this n can resolve: <decisive pairs, power at the observed effect, n needed for 80% power>
## Human blind review
<deck size, cards reviewed, picks by arm, notes; or "not needed" with the reason>
## Decision
<the pre-registered rule applied, in one paragraph. If decided on other grounds, say so and label it unmeasured.>
## What would change it
<the result, n or calibration outcome that would reverse the decision>
## Kept for analysis
<paths to failed candidates, rejected drafts, judge reasons, tells>
Audit
# Eval audit: <eval or claim audited>
Verdict under audit: <the claim as stated>
Survives: <yes | no | not shown>
| # | rule broken | evidence (file:line or run id) | bias it introduces | direction | fix |
|---|---|---|---|---|---|
Smallest change that makes the verdict trustworthy: <one paragraph>
Claude app, ChatGPT and ZIP download
local coding agent required
This skill requires project files, terminal commands, and files you provide. Uploading it to a chat app does not provide equivalent execution.
ChatGPT Skills
This workflow needs a local coding environment or capabilities that a chat-only Skills upload does not provide.
Package details
- Package
- taste-evals/SKILL.md
- Author
- @tushaarmehtaa
- Added
- 4 Nov 2026
- Tags