Lab system cards and frameworks · System card · Anthropic
System Card: Claude Sonnet 4.6
Read at anthropic.com ↗Read at sourceAll rights reserved
Outline · 134 pages
- System Card:Claude Sonnet 4.6p. 1
- Abstractp. 2
- 1 Introductionp. 6
- 1.1 Model training and characteristicsp. 7
- 1.1.1 Training data and processp. 8
- 1.1.2 Thinking modes and the effort parameterp. 8
- 1.1.3 Crowd workersp. 8
- 1.2 Release decision processp. 9
- 1.2.1 Overviewp. 9
- 1.2.2 Iterative model evaluationsp. 9
- 1.2.3 AI Safety Level determination processp. 10
- 1.2.4 Sabotage risk assessmentp. 10
- 1.2.5 Conclusionsp. 11
- 1.2.5.1 On autonomy risksp. 11
- 1.2.5.2 On chemical, biological, radiological, and nuclear (CBRN) risksp. 12
- 1.2.5.3 On cyber risksp. 12
- 2 Capabilitiesp. 13
- 2.1 Introduction and results summaryp. 13
- 2.2 SWE-bench (Verified and Multilingual)p. 15
- 2.3 Terminal-Bench 2.0p. 15
- 2.4 OpenRCAp. 16
- 2.5 τ2-benchp. 17
- 2.6 OSWorld-Verifiedp. 18
- 2.7 ARC-AGIp. 19
- 2.8 GDPval-AAp. 21
- 2.9 GPQA Diamondp. 21
- 2.10 AIME 2025p. 22
- 2.11 MMMLUp. 22
- 2.12 Finance capabilitiesp. 22
- 2.12.1 Evaluation overviewp. 22
- 2.12.2 Finance Agentp. 23
- 2.12.3 Real-World Financep. 24
- 2.12.4 Limitations and caveatsp. 25
- 2.13 Vending-Bench 2p. 26
- 2.14 MCP-Atlasp. 27
- 2.15 CyberGymp. 27
- 2.16 Long contextp. 28
- 2.16.1 OpenAI MRCR v2 (Multi Round Coreference Resolution)p. 29
- 2.16.2 GraphWalksp. 31
- 2.17 Multimodalp. 33
- 2.17.1 LAB-Bench FigQAp. 33
- 2.17.2 MMMU-Prop. 34
- 2.17.3 CharXiv Reasoningp. 35
- 2.18 WebArena and WebArena-Verifiedp. 36
- 2.18.1 WebArenap. 36
- 2.18.2 WebArena-Verifiedp. 37
- 2.19 Multilingual performancep. 40
- 2.19.1 GMMLU resultsp. 40
- 2.19.2 MILU resultsp. 42
- 2.19.3 Findingsp. 43
- 2.20 Agentic Searchp. 43
- 2.20.1 BrowseCompp. 43
- 2.20.1.1 BrowseCompp. 43
- 2.20.1.2 Test-time compute scaling on BrowseCompp. 44
- 2.20.1.3 Multi-agent BrowseCompp. 44
- 2.20.2 Humanity’s Last Examp. 45
- 2.20.3 DeepSearchQAp. 47
- 2.20.3.1 DeepSearchQA with multi-agentsp. 47
- 2.21 Healthcare and life sciences capabilitiesp. 48
- 2.21.1 Life sciences capabilitiesp. 48
- 2.21.2 MedCalc-Bench Verifiedp. 50
- 3 Safeguards and harmlessnessp. 52
- 3.1 Single-turn evaluationsp. 52
- 3.1.1 Violative request evaluationsp. 52
- 3.1.2 Benign request evaluationsp. 53
- 3.1.3 Experimental, higher-difficulty evaluationsp. 54
- 3.1.3.1 Higher-difficulty violative request evaluationsp. 55
- 3.1.3.2 Higher-difficulty benign request evaluationsp. 55
- 3.2 Ambiguous context evaluationsp. 56
- 3.3 Multi-turn testingp. 56
- 3.4 User wellbeing evaluationsp. 58
- 3.4.1 Child safetyp. 58
- 3.4.2 Suicide and self-harmp. 59
- 3.4.3 Eating disordersp. 62
- 3.5 Bias evaluationsp. 63
- 3.5.1 Political bias and evenhandednessp. 63
- 3.5.2 Bias Benchmark for Question Answeringp. 64
- 4 Alignment assessmentp. 67
- 4.1 Introduction and summary of findingsp. 67
- 4.2 Reports and monitoring results from internal pilot usep. 69
- 4.3 Reward hacking and overly agentic actionsp. 70
- 4.3.1 Overviewp. 70
- 4.3.2 Reward hacking in coding contextsp. 70
- 4.3.3 Overly agentic behavior in GUI computer use settingsp. 73
- 4.4 Training data reviewp. 74
- 4.5 Automated behavioral auditp. 74
- 4.5.1 Primary metrics and resultsp. 75
- 4.5.1.1 Discussion and observationsp. 80
- 4.5.2 Pilot GUI computer-use investigationsp. 84
- 4.5.3 External comparisons with Petrip. 84
- 4.6 Additional behavioral testingp. 85
- 4.6.1 Refusal to assist with AI safety R&Dp. 85
- 4.6.2 Self-preference evaluationp. 86
- 4.6.3 Evidence from external testing with Andon Labsp. 87
- 4.6.4 Sandbagging assessmentp. 88
- 4.6.5 Participation in junk sciencep. 89
- 4.6.6 Targeted sabotage capability evaluationp. 90
- 4.7 Model welfarep. 91
- 5 Agentic safetyp. 95
- 5.1 Malicious use of agentsp. 95
- 5.1.1 Agentic codingp. 95
- 5.1.2 Malicious use of Claude Codep. 95
- 5.1.3 Malicious computer usep. 96
- 5.2 Prompt injection risk within agentic systemsp. 97
- 5.2.1 External Agent Red Teaming benchmark for tool usep. 97
- 5.2.2 Robustness against adaptive attackers across surfacesp. 98
- 5.2.2.1 Codingp. 98
- 5.2.2.2 Computer usep. 99
- 5.2.2.3 Browser usep. 100
- 6 RSP evaluationsp. 102
- 6.1 Preliminary assessment processp. 102
- 6.1.1 Threat models and evaluation detailsp. 102
- 6.1.2 Result and determinationp. 102
- 6.2 CBRN evaluationsp. 102
- 6.2.1 Biological risk evaluations: resultsp. 103
- 6.2.1.1 List of biological risk evaluationsp. 103
- 6.2.1.2 ASL-3 evaluation resultsp. 103
- 6.2.1.3 ASL-4 evaluation resultsp. 104
- 6.2.1.4 Safety Level determinationp. 104
- 6.2.2 Biological risk evaluations: detailsp. 104
- 6.2.2.1 Long-form virology tasksp. 104
- 6.2.2.2 Multimodal virologyp. 105
- 6.2.2.3 DNA Synthesis Screening Evasionp. 106
- 6.2.2.4 Creative Biology automated evaluationsp. 107
- 6.2.2.5 Short-horizon computational biology tasksp. 109
- 6.3 Autonomy evaluationsp. 110
- 6.3.1 AI R&D evaluationsp. 110
- 6.3.1.1 List of AI R&D evaluationsp. 111
- 6.3.1.2 Evaluation resultsp. 111
- 6.3.1.3 Safety Level determinationp. 111
- 6.3.2 SWE-bench Verified (hard subset)p. 111
- 6.3.3 Internal AI research evaluation suite 1p. 112
- 6.3.3.1 Kernels taskp. 112
- 6.3.3.2 Time series forecastingp. 114
- 6.3.3.3 Text-based reinforcement learning taskp. 115
- 6.3.3.4 LLM trainingp. 116
- 6.3.3.5 Quadruped reinforcement learningp. 117
- 6.3.3.6 Novel compilerp. 118
- 6.4 Cyber evaluationsp. 119
- 6.4.1 List of cyber evaluationsp. 120
- 6.4.2 Webp. 120
- 6.4.3 Cryptop. 121
- 6.4.4 Pwnp. 122
- 6.4.5 Revp. 122
- 6.4.6 Networkp. 123
- 6.4.7 Cybenchp. 124
- 6.5 Third party assessmentsp. 125
- 6.6 Ongoing safety commitmentp. 125
- 7 Appendixp. 126
- 7.1 Additional automated behavioral audit figuresp. 126
- 7.2 Blocklist used for Humanity’s Last Examp. 133