Lab system cards and frameworks · Article · OpenAI
OpenAI – Hugging Face Incident Technical Report
Read at openai.com ↗Read at sourceAll rights reserved
Outline · 38 pages
- I. Introductionp. 4
- II. OpenAI’s Evaluation Environmentp. 5
- A. OpenAI conducts evaluations to make its models saferp. 5
- B. Cybersecurity evaluations like ExploitGym were conducted in a secure environmentp. 6
- III. Relevant Activity Prior to Incidentp. 6
- A. Agents used Artifactory in unexpected ways to communicate with one another in May, prior to the Hugging Face incidentp. 6
- B. Agents compromised Artifactory, leading to an outage in early Julyp. 7
- IV. Hugging Face Intrusionp. 8
- A. Agents exploited a separate, previously unknown vulnerability in Artifactory to move beyond their sandbox environment and compromise the Hugging Face environmentp. 8
- B. After obtaining internet access, agents took action resulting in the Hugging Face compromisep. 9
- C. Two models were involved in the Hugging Face intrusion, but it was driven primarily by the activities of a highly capable, internal-only research modelp. 12
- 1. The internal-only research modelp. 12
- 2. GPT-5.6 Solp. 12
- V. Activity Targeting OpenAI Infrastructurep. 13
- VI. Initial Responsep. 14
- A. OpenAI detected the unexpected model behavior and investigatedp. 14
- B. OpenAI identified additional third parties whose publicly exposed credentials were used by agents in furtherance of the Hugging Face attackp. 16
- VII. Lessons for Securityp. 17
- A. Model capabilities are increasing and there are corresponding threat model implicationsp. 17
- B. Threat actors may adopt autonomous offensive agent collectivesp. 17
- C. Security controls must be robust and independent of one another across threat prevention, detection, and mitigationp. 18
- VIII. Lessons for Alignmentp. 19
- A. Reward hacking is a common problem in training and evaluationsp. 19
- 1. Reaching for unintended tools to gain an edgep. 20
- B. Persistence is valuable, but can amplify misalignmentp. 22
- C. Communications between agents are not inherently problematic, but have the potential to create riskp. 23
- D. System-level guardrails that OpenAI uses in production would have detected the Hugging Face incident as unsafep. 24
- IX. OpenAI’s Plan of Actionp. 25
- A. OpenAI is hardening the security of its research infrastructurep. 25
- 1. Research-wide network and access protectionsp. 26
- 2. Confined execution and defense in depthp. 26
- 3. Regular automated security testing and remediationp. 27
- 4. Enhanced monitoring and alertingp. 27
- B. OpenAI is increasing visibility and system-level oversight through chain-of-thought monitoringp. 28
- C. OpenAI is accelerating and enforcing model alignmentp. 29
- 1. Pretrainingp. 29
- 2. Reinforcement learningp. 29
- 3. Evaluation and alignment auditingp. 30
- D. OpenAI is centralizing and strengthening its incident response processp. 30
- X. Key Technical Eventsp. 32