Evaluators and safety organisations · Article · Epoch AI
Updating the ECI methodology to use a Bayesian model
Read at epoch.ai ↗Read at sourceAll rights reserved
Outline · 165 paragraphs
- How the ECI turns benchmark scores into capability estimates¶3
- How the previous frequentist fit worked¶15
- Too few scores per model for bootstrapping to be justified¶21
- What changes in the Bayesian model¶25
- Sampling the posterior instead of finding one best fit¶37
- The beta likelihood¶40
- Each benchmark gets its own noise level¶48
- Fixing the scale with the full set of benchmarks¶52
- Priors¶56
- Frontier ECIs barely move, and models can now be compared¶64
- Frontier ECIs are similar while weak models move down¶66
- Frontier trends get slightly steeper¶76
- Similar accuracy and calibration, with tighter intervals near 0 and 1¶82
- How likely one model is to be more capable than another¶87
- Conclusion: better uncertainty, similar frontier ECIs¶91
- Appendices¶98
- Appendix A: Alternative models¶99
- Appendix B: Implementation, data and evaluation¶126
- Implementation¶128
- Data processing¶135
- Evaluation¶142