SafetyPage

Alignment Forum and LessWrong · Forum post · Alignment Forum

Deconfusing Corrigibility

berns · 17 Oct 2026

Read at alignmentforum.org ↗Read at sourceNo licence statement found

Outline · 220 paragraphs

  1. Introduction¶2
  2. From normative learning to corrigibility¶6
  3. The complete normative learning problem¶7
  4. Three kinds of uncertainty¶13
  5. Dangers of a complete normative goal¶24
  6. An analogy from human relationships¶30
  7. The two component goals of corrigibility¶37
  8. Confusions in existing pictures of corrigibility¶43
  9. Over-epistemicized corrigibility¶44
  10. Fully updated deference¶48
  11. Inputs to the utility function¶56
  12. Emergent corrigibility is not enough¶62
  13. Why not a singular target¶67
  14. Evaluatively-thickened corrigibility¶71
  15. Unnatural reasoning in singular-target agents¶77
  16. A new picture of corrigibility¶89
  17. Allocating authority¶90
  18. Effective exercise of reserved powers¶101
  19. Latency and the cost of non-delegation¶106
  20. Operationalizing Goal 2¶110
  21. Legitimacy¶118
  22. A formalism for corrigibility¶125
  23. Definitions¶127
  24. Prioritization under certainty¶136
  25. Prioritization under uncertainty¶144
  26. Learning with logical induction¶166
  27. Limitations and extensions¶188
  28. The state of the corrigibility model¶189
  29. Who is the human principal?¶191
  30. Instilling goals in practical training¶199
  31. A thicker concept of legitimacy¶202