Your AI Eval Is Lying To You

When you set temperature=0 and run your AI eval, you expect the same input to give the same output. It doesn't. Recent measurements on Qwen3-235B at temperature=0 produced 80 unique completions on a single prompt. So when your eval reports "92% pass rate," what does that actually mean? Is it 92% capable, 92% reliable, or 92% lucky on a small sample? This talk is about the gap between how the AI eval ecosystem talks about scores and what those scores can actually support. We walk through five specific tools that fix the gap, all anchored to published methods: 1. Pass@k versus pass^k: capability versus reliability, two different questions that one number obscures (Chen et al. 2021, OpenAI Codex paper). 2. Wilson confidence intervals with proper boundary handling, so your "92%" comes with an honest range (Brown et al. 2001). 3. Bayesian pass@k with Beta-Binomial conjugacy, when you want a posterior rather than a point estimate (Hariri et al., ICLR 2026). 4. Sequential drift detection with EWMA, CUSUM, and OLS, to catch eval regression while it's small instead of after a customer reports it (Lucas-Saccucci 1990, Page 1954, Montgomery 2012). 5. Family-wise error control via Benjamini-Hochberg, Benjamini-Yekutieli, and e-BH FDR procedures, for when you're running multiple correlated drift checks in parallel and don't want false alarms (Benjamini-Hochberg 1995, Wang-Ramdas 2022). Each method gets a short demo in pure Python with no framework dependency. The audience leaves with reference implementations they can paste into an existing pytest setup tonight. The talk also previews an open-core pytest plugin shipping in July 2026 that packages these methods into a single eval pipeline with SARIF reporting and a baseline-regression workflow. The talk shows the open primitives and the methodology that drives them. The production AI-eval ecosystem (LangSmith, Arize Phoenix, Evidently, DeepEval, Promptfoo, and others) mostly uses absolute thresholds and simple averages. None of the ten platforms I surveyed combine sequential testing with FDR control on bounded scoring scales. The framing here isn't competitive; it's a methodological gap every team shipping production AI evals will hit eventually.