00 · Experiment, score, pick winners
AI Evals
Run experiments on live traffic, keep cohorts stable, and compare variants against real outcome signals. The substrate for evaluating models, prompts, and rewrites in production.
eventuser.signed_up · 4 listeners
Patterns
01Run experiments in productionUse group.experiment() to split traffic, keep cohorts stable, compare variants, and roll changes forward safely.02Score agents on real outcomesAttach scores to runs and steps, and defer scoring until the real-world signal arrives, so you measure agents on what actually happened.03Score agent accuracyGrade an agent against a known answer (exact match, set overlap, or numeric tolerance) and record the score on the run that produced it.