SafetyPage

Government · Government · NIST

Expanding the AI Evaluation Toolbox with Statistical Models

13 Feb 2026

ReadSource at nvlpubs.nist.gov ↗Read herePublic domain (US government work)

Outline · 64 pages

  1. Introductionp. 9
  2. Motivationp. 10
  3. Why LLM Evaluations Need a Statistical Modelp. 10
  4. A Statistical Model for LLM Evaluationsp. 13
  5. Expanding the Evaluation Toolboxp. 15
  6. Choosing an Accuracy Estimandp. 17
  7. Variance Statistics for Benchmark Evaluationp. 17
  8. Regression-Free Estimatorsp. 18
  9. The Importance of Accounting for Multiple Trialsp. 19
  10. Limitations of Regression-Free Estimatorsp. 19
  11. Leveraging Generalized Linear Mixed Models for Evaluation Datap. 20
  12. Computing Evaluation Statistics with a GLMMp. 21
  13. Validating the Data-Efficient Properties of GLMMsp. 24
  14. Evaluating GLMM Estimates in a Toy Settingp. 25
  15. Evaluating GLMM Estimates on GPQA-Diamondp. 27
  16. Comparing GLMM and Regression-Free Generalized Accuracy Estimates.p. 28
  17. Interpreting Random Effect Estimatesp. 29
  18. Considering Individual Random Effectsp. 29
  19. The Role of Variance in Interpreting Evaluationsp. 32
  20. Variations in Variance within Benchmarksp. 33
  21. Discussionp. 36
  22. GLMM Extensionsp. 36
  23. Assumptions and Limitations of the Statistical Modelsp. 37
  24. Conclusionp. 40
  25. Variable Glossaryp. 48
  26. Evaluation Setting and Estimand Definitionsp. 48
  27. Regression-Free Estimatorsp. 50
  28. An Incorrect Standard Error Calculationp. 53
  29. Central Limit Theorem Confidence Intervalsp. 54
  30. Generalized Linear Mixed Modelsp. 55
  31. GLMM Estimationp. 56
  32. Toy Experiment Detailsp. 57
  33. Benchmark Evaluation Detailsp. 60