- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper presents Flint, a benchmark of 180 tasks in unfamiliar 3D physics rooms, sampled from 60 generators written by coding agents. Agents must discover each mechanism through first-person interaction. GPT-6 Astra and Claude Opus 5.5 solve 60.6% and 44.4%, while five other models solve at most 7.8%. The authors attribute this gap to interaction efficiency and argue that the strongest model often fails to revise its initial hypotheses.
Critical strengths
- Scalable task construction. Coding agents write parametric generators, and each generator passes automated tests and a human review. The paper reports that 40 of 80 creation jobs were accepted, at $21.58 and 94.5 minutes per job, so the benchmark can be extended as models improve.
- Strong separation and careful analysis. The benchmark is far from saturated and still separates current frontier models. The interaction analysis, with ATAI and matched-task comparisons, shows where agents fail in addition to how they rank.
Critical weaknesses
-
The score may not measure embodied fluid intelligence. Sec. 3 states that the interface gives no semantic explanation of interactions. However, the agent frames in Fig. 2 display on-screen text such as "orientation 9/16" and "click to advance, right-click to reverse", which reveals the state of the targeted device and the effect of each click. Even with the mechanism given (L0), GPT-6 Astra fails 37 tasks, and 28 of them fall in categories that require precise placement (see Appendix C). Much of the remaining headroom may therefore reflect manipulation difficulty rather than mechanism discovery.
-
The diagnostic conclusions lack controls. Sec. 5 says that all agents share one interface, but Appendix F says that they run in different harnesses, which the paper does not describe. Each task is run once, and both the L0 rerun and the variant experiment rerun only failed tasks, without a plain-rerun control. The controlled variants that motivate the generator design are used only in these 10 hand-picked cases.
Questions for authors
-
What text can an agent see or receive during an episode, such as object names, state readouts, and interaction hints? Could you rerun the two leading models on a subset of tasks with this text removed?
-
Which harness and reasoning settings does each model use, and does the ranking hold when all models run in one common harness on a subset of tasks?
-
How often does a plain rerun under L1 turn a failed task into a success?