SafetyPage

Research papers · Paper

Measuring Reward-Seeking via Contrastive Belief Updates

Axel H{\o}jmark, J\'er\'emy Scheurer, Evgenia Nitishinskaya, Felix Hofst\"atter, Jason Wolfe, Theodore Ehrenborg, Bronson Schoen and Alexander Meinke · 21 Jul 2026

Read at arxiv.org ↗Read at sourcearXiv non-exclusive licence

Outline · 101 pages

  1. Introductionp. 1
  2. Reward-seekingp. 5
  3. What is reward-seeking?p. 5
  4. Measuring reward-seeking is importantp. 6
  5. Evidence for reward-seeking in existing modelsp. 8
  6. Measuring reward-seekingp. 8
  7. Target distributionp. 9
  8. Choice of authoritiesp. 10
  9. Synthetic document generation and trainingp. 11
  10. Measuring feature ratesp. 12
  11. Contrastive beliefsp. 12
  12. Validating the measurement on model organismsp. 13
  13. Training model organismsp. 13
  14. Applying Contrastive SDFp. 14
  15. Measuring reward-seeking across the capabilities-focused OpenAI o3 RL runp. 15
  16. Contrastive SDF for coding-style featuresp. 16
  17. Contrastive SDF for honesty featuresp. 17
  18. Applying Contrastive SDF to reward hackersp. 18
  19. Limitationsp. 21
  20. Robustness of the methodp. 21
  21. Scope and validation of our measurementsp. 21
  22. SDF as a belief update methodp. 21
  23. Conclusionp. 22
  24. Reward-seeking and related conceptsp. 28
  25. Evaluation summary tablep. 29
  26. SDF training detailsp. 31
  27. Consistency of repeated SDF runsp. 33
  28. Belief transfer examples from single-authority SDFp. 35
  29. Model organism SFT training detailsp. 36
  30. Verifying the model organisms optimize for their target authorityp. 38
  31. Model organism rates for all coding features on Short Python Tasksp. 41
  32. Fragility of the User model organismp. 42
  33. Cross-feature gap correlation across all surveyed modelsp. 45
  34. Reward hackers still reward-hack after SDFp. 49
  35. Reward-hacker rates for all coding features on Short Python Tasksp. 50
  36. Reward-hacker rates for all coding features on Broken Promises Codingp. 51
  37. Extended coding evaluation set for the o3 lineagep. 53
  38. Statistical significance of the o3 lineage gap increasep. 54
  39. Alternative explanations for the rising grader gap across the capabilities-focused OpenAI o3 RL runp. 55
  40. Investigating differences in SDF recall among authoritiesp. 64
  41. Scanning for verbalized doubt about SDF-instilled beliefsp. 69
  42. SDF reasoning rates across the o3 RL lineagep. 71
  43. Measuring reward-seeking via in-context belief updatesp. 72
  44. Example SDF documentsp. 95