acceptodds
Under review as a conference paper at ICLR 2027

Benchmarking Embodied Fluid Intelligence

12 views

Abstract

We study fluid intelligence, the capacity to efficiently infer and exploit the actionable structure of an unfamiliar environment from limited action–perception experience, in interactive 3D worlds. We present Flint, a benchmark of 180 tasks, validated for solvability, unfamiliarity, and meaningful controlled variation. Each task is an unfamiliar puzzle in a physics-enabled 3D room. Instead of manually creating this benchmark, we design a framework that uses coding agents to automatically create such tasks. In particular, we ask coding agents to author task generators that allow us to generate not only individual tasks but their controlled variants. Evaluating frontier multimodal LLM agents on Flint, we find that GPT-6 Astra and Claude Opus 5.5 have increased the solve rate by > 35 percentage points over prior models. Yet, while they saturate many other benchmarks, they leave more than a third of the tasks unsolved on Flint. Our analysis sheds light on the challenges and opportunities for advancing embodied fluid intelligence.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Spatial Intelligence, Embodied Intelligence

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_7BBXOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents Flint, a benchmark of 180 tasks in unfamiliar 3D physics rooms, sampled from 60 generators written by coding agents. Agents must discover each mechanism through first-person interaction. GPT-6 Astra and Claude Opus 5.5 solve 60.6% and 44.4%, while five other models solve at most 7.8%. The authors attribute this gap to interaction efficiency and argue that the strongest model often fails to revise its initial hypotheses.

Critical strengths
  1. Scalable task construction. Coding agents write parametric generators, and each generator passes automated tests and a human review. The paper reports that 40 of 80 creation jobs were accepted, at $21.58 and 94.5 minutes per job, so the benchmark can be extended as models improve.
  2. Strong separation and careful analysis. The benchmark is far from saturated and still separates current frontier models. The interaction analysis, with ATAI and matched-task comparisons, shows where agents fail in addition to how they rank.
Critical weaknesses
  1. The score may not measure embodied fluid intelligence. Sec. 3 states that the interface gives no semantic explanation of interactions. However, the agent frames in Fig. 2 display on-screen text such as "orientation 9/16" and "click to advance, right-click to reverse", which reveals the state of the targeted device and the effect of each click. Even with the mechanism given (L0), GPT-6 Astra fails 37 tasks, and 28 of them fall in categories that require precise placement (see Appendix C). Much of the remaining headroom may therefore reflect manipulation difficulty rather than mechanism discovery.

  2. The diagnostic conclusions lack controls. Sec. 5 says that all agents share one interface, but Appendix F says that they run in different harnesses, which the paper does not describe. Each task is run once, and both the L0 rerun and the variant experiment rerun only failed tasks, without a plain-rerun control. The controlled variants that motivate the generator design are used only in these 10 hand-picked cases.

Questions for authors
  1. What text can an agent see or receive during an episode, such as object names, state readouts, and interaction hints? Could you rerun the two leading models on a subset of tasks with this text removed?

  2. Which harness and reasoning settings does each model use, and does the ranking hold when all models run in one common harness on a subset of tasks?

  3. How often does a plain rerun under L1 turn a failed task into a success?

Reviewer_y3qoOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The work outlines the (somewhat complicated) process of generating a benchmark of puzzles. It's an agents on agents on agents setup to create the generators that create the environments that produce outputs that agents (and humans) can verify. The notion is that you need to formulate a hypothesis about the space and then take actions within it. Models do pretty well in some cases but I'll hold on that for the other boxes.

Critical strengths

The paper is clear, the idea of a hard benchmark that requires action and physical reasoning is very appealing, and some of the insights (e.g. that early hypothesis generation without revision leads to failure) are very interesting.

Critical weaknesses

No human baseline. I wanted to repeat this over and over while reading the paper. Validation is good but insufficient here, particularly when all of the results are capped (e.g. cost capped, length capped, etc) and no additional strategies are investigated for improving model performance.

  1. If it (for example) turns out that humans also fall for "local" optima, don't explore enough options, click around too often on elements that do not have an effect, etc. then we have a very different story then if they solve everything quickly and trivially.
  2. If models with a very slightly different prompt, which are given the ability to explore with a few agents themselves, or some other minor agentic change are able to saturate the dataset then that's also a very useful datapoint. In particular, if I had a prompt to outline a few hypotheses before charging in, would that already eat half the headroom?
Questions for authors

A. See two questions in weaknesses. B. Which model for the coding agents was used and does that affect downstream performance? If I coded it am I better at guessing what aspects of the scene are interactable? C. Did the largest models hit their reasoning or compute limits at the $20 cap? Can I throw more money at the problem and see the top left corner of the plot continue to climb? D. Are solutions unique? E. In the case of action failures, how much of the degradation in performance is really just that the context window fills with no-ops which could also just be dropped automatically to keep the context clean?

Reviewer_jJvkOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces Flint, a benchmark for embodied fluid intelligence consisting of 180 interactive 3D tasks sampled from 60 generators. A coding-agent workflow composes shared, verified assets into parameterized task families, followed by automated checks and human validation. Agents must infer unfamiliar mechanisms from egocentric observations and interactions to achieve a specified goal. The evaluation covers seven multimodal agents and reports success rates, inference costs, and interaction efficiency. The paper also analyzes interaction behavior, reruns failed tasks with mechanism descriptions, and uses targeted task variants to investigate remaining errors.

Critical strengths
  1. The generator-based design is the strongest contribution. It produces related tasks with controlled changes to properties such as causal depth and action ambiguity, giving the benchmark practical value for analyzing failures as well as ranking agents. The automated checks, oracle solutions, and human inspection provide useful quality control for this generated task collection.

  2. The evaluation covers varied interactive mechanisms and shows clear differences among agents. At the $20 cap, GPT-6 Astra and Claude Opus 5.5 solve 60.6% and 44.4% of the tasks, respectively, while the other agents remain below 10%. The category-level results and interaction analyses, including matched-task comparisons in App. F, make the benchmark informative beyond its aggregate solve rate.

Critical weaknesses
  1. The failure analysis does not adequately separate the effect of the interventions from an additional attempt. Sec. 5 reruns only the 71 initial L1 failures under L0 and recovers 34, without reporting fresh L1 reruns on the same tasks. This leaves the improvement attributable to the mechanism description unclear. The 7/10 recovery under manually selected variants is also suggestive, but changes such as starting blocks on their target pedestals simplify the task and provide solution-relevant information (Fig. 7). These experiments motivate further investigation of mechanism acquisition and hypothesis revision, but do not isolate their contribution to the observed failures. Repeated, matched comparisons with unchanged-task controls would strengthen these central conclusions.

  2. The exploration budget needs stronger validation. The limit of 10 times the oracle interaction length is based on executing a known solution, which need not reflect the exploration required to discover it. The human validation in Sec. 4 shows oracle solution replays before inspection, so it does not establish independent goal-only solving within this limit. This matters because 28 of the 37 remaining L0 failures terminate at the interaction limit (App. C). Human solve rates and click counts on a representative subset, or sensitivity to the click-budget multiplier, would help determine how much of the reported difficulty depends on the chosen budget. The existing sweep over inference cost does not answer this question.

Questions for authors
  1. Can you provide repeated, matched L1/L0 results on a representative subset of the 71 initial failures, using fresh contexts and the same interaction and cost limits? For the targeted variants, how does recovery compare with fresh runs of the original tasks?

  2. Do you have solve rates and click counts from participants who have not seen the mechanisms or solutions, or results under several interaction limits? How stable are the reported failure patterns across these limits?

  3. Please provide representative L0 prompts alongside the corresponding L1 prompts. How do you distinguish mechanism descriptions from task-specific solution hints?

  4. Which harness and inference settings were used for each agent, particularly for action batching, image access, and context management? Was the three-second observation-timing rule in App. F checked for agents other than GPT-6 Astra?

  5. Which models and versions authored the engine, assets, and task generators? Were any evaluated agents’ solve outcomes used to revise or select generators?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.