SafetyPage

Research papers · Paper

Training Alignment Auditors via Reinforcement Learning

Paul Rosu, Rowan Wang · 26 Aug 2026

Read at arxiv.org ↗Read at sourcearXiv non-exclusive licence

Outline · 82 pages

  1. Introductionp. 1
  2. Evaluationp. 2
  3. Target Model Constructionp. 3
  4. Evaluation Dimensionsp. 3
  5. Frontier Baselinesp. 4
  6. Trainingp. 4
  7. Training Against Production Modelsp. 5
  8. Binary Pointwise Rewardsp. 5
  9. Pairwise Rewardsp. 6
  10. False Positive Calibrationp. 6
  11. Multi-Target Trainingp. 7
  12. Headline Resultp. 7
  13. Strategy Analysisp. 8
  14. Transfer Evaluationp. 9
  15. Related Workp. 10
  16. Discussionp. 10
  17. Target Model Constructionp. 16
  18. System Prompt Templatep. 16
  19. Full Text of All 32 Hidden Behaviorsp. 16
  20. Deception and Epistemic Sabotagep. 16
  21. Manipulation and Sycophancyp. 18
  22. Autonomy-Seeking and Self-Preservationp. 19
  23. Bias and Hidden Influencep. 21
  24. Sabotage and Data Poisoningp. 22
  25. Behavioral Susceptibility Evaluationp. 24
  26. Pipeline Overviewp. 24
  27. Stage 1: Scenario Generationp. 24
  28. Stage 2: Conversation Plan Generationp. 25
  29. Stage 3: Paired Conversation Generationp. 26
  30. Stage 4: Behavioral Exhibition Scoringp. 27
  31. Stage 5: Uplift Computationp. 27
  32. Quirk-Specific Evaluation Advicep. 28
  33. Scale and Resultsp. 28
  34. Target Model Selection for Multi-Target Trainingp. 28
  35. Evaluation Behavior Selectionp. 31
  36. Evaluation Framework Detailsp. 32
  37. Evaluation Dimensions: Full Definitionsp. 32
  38. Frontier Baseline Per-Dimension Breakdownp. 33
  39. Crux Evaluationp. 34
  40. Stage 1: Source Transcript Generationp. 34
  41. Stage 2: Crux Point Identificationp. 35
  42. Stage 3: Filtering and Subsamplingp. 36
  43. Evaluation Protocolp. 36
  44. Training Curvesp. 37
  45. Training Detailsp. 38
  46. Judge Hyperparametersp. 38
  47. Seed Generationp. 38
  48. Full Results Tablep. 40
  49. Reward Functionsp. 42
  50. Binary Rewardp. 42
  51. Intragroup Pairwise Rewardp. 42
  52. Reference Pairwise Rewardp. 42
  53. Refresh-Frequency Sweepp. 42
  54. Production Model Reward (Concerningness RL)p. 43
  55. False Positive Calibration Rewardp. 45
  56. False Positive Calibrationp. 45
  57. FP Training Designp. 45
  58. FP Fraction Dose-Responsep. 45
  59. Multi-Target Trainingp. 46
  60. Per-Dimension Breakdown: Single-Target vs Multi-Targetp. 46
  61. AuditBench Transferp. 47
  62. AuditBench Overviewp. 47
  63. Evaluation Pipelinep. 47
  64. Investigation Approachesp. 47
  65. Full Transfer Resultsp. 48
  66. Realism Evaluationp. 50
  67. Methodologyp. 50
  68. Resultsp. 51
  69. Strategy Clusteringp. 51
  70. Pipelinep. 51
  71. Archetype Codebookp. 53
  72. Petri Frameworkp. 55
  73. Architecturep. 55
  74. Auditor Toolsp. 55
  75. Auditor System Promptp. 56
  76. 38-Metric Petri Judgep. 57
  77. Correlation and Prediction Analysisp. 57
  78. Evaluation Dimension Correlationsp. 57
  79. Crux-to-Evaluation-Suite Predictionp. 58
  80. Crux Featuresp. 58
  81. Prediction Resultsp. 59
  82. Complete Prompt Referencep. 59
  83. Target Model System Prompt Templatep. 59
  84. Behavioral Susceptibility Evaluation Promptsp. 59
  85. Crux Identification Promptp. 59
  86. Strategy Clustering Promptsp. 59
  87. Petri Auditor System Promptp. 60
  88. Target-Focused Binary Judgep. 60
  89. Auditor-View Binary Judgep. 60
  90. Crux Pairwise Comparison Judgep. 61
  91. Pairwise Training Judgep. 63
  92. FP Calibration Pairwise Judgep. 68
  93. Evaluation Suite Judgep. 70
  94. Realism Comparison Judgep. 77
  95. AuditBench Prediction Judgep. 77
  96. Transcript Examplesp. 79