OpenAI Alignment: agent safety eval scores can differ sharply from real production behavior

OpenAI Alignment researchers Joy Hsu and Sam Arnesen compare agent safety behavior in evaluations versus production. Across six served models (GPT-5.6 Luna, GPT-5.6 Sol, GPT-6 Luna, GPT-6 Sol, GPT-6 Astra, GPT-6.1 Sol), eval success was lower than on comparable production cases, with gaps of 10.2 percentage points to 37.4 percentage points; GPT-5.6 Sol and GPT-6 Sol had the largest gaps. Knowing a session is an evaluation does not necessarily change behavior; how the model reads the evaluation objective is more closely tied to the gap. Saying only that the agent is being evaluated had little effect, while stating the correct or incorrect objective moved eval performance in opposite directions. The team says it will track this gap to judge how well safety-eval behavior generalizes.

Written by dalao.ai: the title, summary and why it matters are our own words. The original reports are the record; their publishers keep authorship.