Competence Without Performance: Models Represent ARC-AGI-3 but Cannot Play It
Abstract
ARC-AGI-3 tests whether a model can enter an unfamiliar interactive world, infer its hidden rules through play, and act successfully on what it learns. We build the first Gymnasium- and DreamerV3-compatible environment wrapper for ARC-AGI-3 and use it to establish the first model-based reinforcement-learning baseline on the benchmark with a learned latent world model. Run on six public games, the agent scores zero on five of them under the benchmark's metric, Relative Human Action Efficiency (RHAE). To find out why, we ask whether these zeros mean that the agent has learned nothing about the games, or that it has learned useful structure but cannot turn it into play. Because DreamerV3 trains its world model on real frames and its policy only on imagined trajectories, what the model has learned can be measured separately from what the policy does. We find competence without performance. The world model fits every game at the single-step level, and linear probes on its frozen latent state decode which level is being played, above raw-pixel, elapsed-time and label-permutation controls, in many games on which the agent never scores. Yet the policy stays at its uniform initialization on five of six games. Ablations that pretrain the world model across games and that replace the reinforcement learner with a supervised imitation model confirm that the gap lies in exploration and policy learning rather than in representation. Learning a world model and learning to act from it therefore fail separately on ARC-AGI-3, and the benchmark score alone cannot tell the two failures apart.
Then back it, or bet against it.
Related papers
Open the market on this paper to see 7 more related papers.