Analyzing GPT-5.5 & Opus 4.7 with ARC-AGI-3
AI benchmarks are powerful tools for evaluating the capabilities of different models, but they often only provide a pass or fail outcome. The ARC-AGI-3 framework, however, offers a deeper insight into the reasoning processes behind these scores. This article explores the findings from a recent analysis of OpenAI’s GPT-5.5 and Anthropic’s Opus 4.7, focusing on their performance in novel, long-horizon environments.
Overview of ARC-AGI-3
ARC-AGI-3 consists of 135 unique environments specifically designed to test AI models’ adaptability to new and unfamiliar situations. Unlike traditional benchmarks, participants—whether human or AI—are not provided with instructions on how to navigate these environments. To succeed, they must:
- Explore unfamiliar interfaces
- Infer rules from sparse feedback (known as world modeling)
- Form and test hypotheses
- Recover from incorrect assumptions
- Transfer knowledge from one level to the next (also known as continual learning)
Each environment is crafted to isolate abstract reasoning, making ARC-AGI-3 a valuable tool for assessing the cognitive demands that real-world tasks place on agents.
Performance Analysis
In our analysis, we evaluated 160 replays and reasoning traces from both models. The results were revealing:
- GPT-5.5 Score: 0.43%
- Opus 4.7 Score: 0.18%
These scores were derived from a semi-private dataset, and while they provide a quantitative measure, the qualitative insights from the reasoning processes are equally important.
Failure Modes Identified
Through our analysis, we identified three primary failure modes that both models exhibited:
1. True Local Effect, False World Model
The most prevalent failure mode involved the models recognizing local effects without integrating them into a broader world model. For example, while Opus 4.7 understood that a specific action (ACTION3) rotated an object, it failed to connect this action to the necessary subsequent steps to achieve the desired outcome. This disconnect led to ineffective strategies and poor performance in tasks.
2. Wrong Level of Abstraction from Training Data
The second failure mode stemmed from the models applying incorrect abstractions derived from their training data. Throughout the runs, both models frequently misinterpreted unfamiliar mechanics by relating them to known games, such as Tetris or Frogger. This reliance on familiar game mechanics often resulted in misguided actions and wasted attempts to engage with the new environment.
3. Solved the Level, Didn’t Learn the Game
The final failure mode highlighted that achieving success in a level did not equate to a comprehensive understanding of the game mechanics. For instance, Opus 4.7 might solve a level by chance, but when faced with new challenges, it would revert to incorrect strategies based on its previous misconceptions. This pattern demonstrated that early successes could mask deeper misunderstandings, leading to repeated failures in subsequent levels.
Insights from the Analysis
By examining the runs of GPT-5.5 and Opus 4.7, we noticed distinct differences in their failure patterns:
- Opus 4.7: Tended to have the wrong compression of information, often misapplying learned mechanics aggressively.
- GPT-5.5: Struggled with compressing information effectively, leading to a lack of adaptability in new situations.
For example, Opus 4.7 quickly identified and executed short-horizon mechanics in some levels but often latched onto incorrect theories of gameplay. In contrast, GPT-5.5’s inability to compress information resulted in a failure to adapt to novel challenges, even when it had previously succeeded in similar tasks.
Conclusion
The analysis of GPT-5.5 and Opus 4.7 using the ARC-AGI-3 framework has provided valuable insights into the reasoning processes of AI models. By understanding their failure modes, developers can work towards creating more robust AI systems capable of navigating complex environments. The findings emphasize the importance of not only achieving high scores but also ensuring that models genuinely understand the mechanics of the tasks they are performing.
Note: This article is based on the analysis conducted in May 2026 and reflects the performance of the AI models at that time.
