acceptodds
Under review as a conference paper at ICLR 2027

Game2Real-Bench: Do Toy Games Reveal and Transfer to Real-World Visual Reasoning?

Abstract

Visual reasoning is an important capability for vision-language models (VLMs) to understand visual evidence, spatial structure, constraints, and state changes. Existing evaluations mainly rely on either real-world data or procedurally generated tasks. Real-world data preserve the complexity of natural scenes, while procedural games provide controllable generation and verifiable solutions. However, it remains unclear whether the abilities measured in toy games correspond to those required in real-world visual reasoning, and whether game examples can provide useful signals for solving real-world tasks. We introduce Game2Real-Bench, a benchmark for studying these connections across four capability groups, including fine-grained perception, spatial reasoning, constraint reasoning, and sequential planning. On the game side, we construct 1,000 independently generated samples from executable task specifications and verify their answers programmatically. On the real-world side, we curate and construct 400 samples from diverse real-world sources. We connect the two settings according to shared reasoning requirements, including visual evidence, spatial operations, constraints, and state transitions, rather than visual similarity. We evaluate 25 representative VLMs across both settings. Performance on game and real-world tasks shows consistent cross-setting trends, with a Pearson correlation of 0.935 and a Spearman correlation of 0.935. We further examine game-to-real transfer through one-shot in-context learning on four models. The effect varies across capabilities and models. Constraint reasoning shows positive gains for several models, while fine-grained perception consistently declines with game demonstrations. These results show that toy games can reflect some reasoning patterns observed in real-world tasks and can provide useful demonstrations in selected settings. To further support model training, we construct Game2Real-Dataset, containing nearly 24K verified game samples across 18 tasks and 65 QA families. The samples are generated from executable task specifications and verified programmatically, providing scalable training data for visual reasoning. We will publicly release the benchmark, dataset, and code.

Primary area:
datasets and benchmarks
Keywords:
Benchmark, Visual Reasoning, Vision-Language Models

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_BZ9MOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper builds a benchmark that pairs procedurally generated toy-game tasks with real-world visual reasoning items under four capability groups: fine-grained perception (FP), spatial reasoning (SR), constraint reasoning (CR) and sequential planning (SP).

  • Game side: 18 tasks and 65 QA families. Generators come with solvers and verifiers that check uniqueness and optimality where required. The pipeline yields a 23,893-sample training set and a separately seeded 1,000-sample evaluation set.
  • Real-world side: 400 items taken mostly as-is or lightly reformulated from 11 existing benchmarks and datasets, each tagged with a "primary" game task.

Main results:

  • Across 25 VLMs, overall game and real-world scores correlate at Pearson/Spearman 0.935.
  • Game tasks are harder and spread models out more. And thinking mode helps more for larger models.
  • One-shot in-context learning with random game demonstrations (four Qwen models, three seeds) helps CR on average (+4.9), hurts FP for all four models (−4.9), and lowers overall accuracy by 1.34.
Critical strengths
  1. The game side is well engineered and reusable. The paper has executable generators, task-specific solvers that check uniqueness and optimality, disjoint seeds and layouts between the 24K training set and the evaluation set, and de-duplication by semantic fingerprints and image hashes. Verifiable multi-step visual tasks like Rush Exit, Mancala and Pipe are useful for RL-style training regardless of the game-to-real question.
  2. The correlation analysis is checked carefully, and negative results are reported. The paper includes leave-one-source-out, answer-format subsets and model bootstrap (App. D.2), and it states plainly that one-shot game demonstrations lower overall accuracy.
Critical weaknesses
  1. The 0.935 correlation mostly reflects overall model strength, not capability-specific alignment. With 25 models spanning 2B to frontier, almost any two VLM benchmarks correlate this strongly. The test that matters is whether game capability c predicts real capability c better than it predicts the other real capabilities. Recomputing from Table 1 (my numbers reproduce your overall r = 0.934):

    • Game FP correlates 0.79 with real FP, but 0.84–0.86 with each of the other three.
    • Game SR correlates 0.85 with real SR, but 0.87 with real CR.
    • Partial correlations for matched pairs, controlling for the other three real capabilities, are 0.05 (FP), 0.22 (SR), 0.75 (CR) and 0.38 (SP).

    Only CR shows a specific correspondence. Coverage is also thin: four game tasks have no real-world counterpart, and Cube Net ↔ IKEA has only 5 items.

  2. The game-example prompting experiment doesn't support a transfer claim. The net effect is negative (−1.34). The CR gain comes mostly from Qwen3.5-9B (+15.6), whose zero-shot CR (24.3) is oddly below the 4B model's (28.9). Half of the real CR items are SpatiaLQA, which require structured outputs, so the solved example may simply be showing the model the answer format. A zero-shot run that states the format requirements explicitly in the system prompt would check this: if it recovers the gain, the effect is format, not transfer. More fundamentally, the paper releases a 24K training set but runs no training. A small SFT or RL run on Game2Real-Dataset, evaluated on the 400 real items by capability, would answer the title question directly.

Questions for authors
  1. Please report the capability-level cross-correlation matrix and partial correlations. Do you agree that only CR shows specific correspondence? If not, what explains game FP predicting real CR better than real FP?
  2. What is the correlation between your real-world set and a benchmark not aligned with any game task, over the same models?
  3. For Qwen3.5-9B on real CR, what fraction of zero-shot failures are format errors, truncation, or unparseable outputs, and how does that fraction change once a game example is added to the prompt?
Reviewer_WkHiOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces Game2Real-Bench, a benchmark for studying whether visual reasoning performance on procedurally generated toy games corresponds to performance on real-world visual reasoning tasks, and whether game demonstrations can transfer to real-world problems. The benchmark covers four capability groups—fine-grained perception, spatial reasoning, constraint reasoning, and sequential planning—and contains 1,000 independently generated toy-game samples and 400 curated real-world samples. The authors additionally construct Game2Real-Dataset, containing 23,893 programmatically verified game samples across 18 tasks and 65 QA families. The paper evaluates 25 VLMs and reports a strong model-level correlation between aggregate toy-game and real-world performance (Pearson r = 0.935, Spearman ρ = 0.935). It further studies one-shot game-to-real in-context learning on four Qwen models using randomly assigned game demonstrations. The transfer results are mixed: constraint reasoning improves on average, fine-grained perception consistently declines, and overall performance slightly decreases on average. The paper also analyzes thinking-mode scaling and representative reasoning failures.

Critical strengths
  1. The benchmark construction is extensive and verifiable. The paper combines 18 procedurally generated game tasks, 65 QA families, executable generators, and task-specific verification with a separately generated 1,000-sample game evaluation set. The real-world side draws from multiple existing sources and undergoes automated checks and manual review. This creates a useful resource for controlled evaluation of visual reasoning and makes the benchmark substantially more reproducible than benchmarks relying predominantly on manually authored questions.

  2. The empirical evaluation is broad and the reported model-level correlation is tested carefully. The authors evaluate 25 VLMs and do more than report the headline r = 0.935: the appendix examines alternative aggregation, excludes proprietary models, removes individual real-world sources, separates answer formats, and reports uncertainty estimates. The correlation remains high under most of these variants. This makes the empirical observation that strong models on the game suite tend also to be strong on the selected real-world suite convincing.

Critical weaknesses

The main correlation shows strong overall correspondence between toy-game and real-world performance, but does not validate the specific game–real task alignment. A model that is generally stronger could score well in both settings even if the matched toy and real-world tasks are only weakly related. A stronger test would ask whether performance on a game task or capability predicts its matched real-world counterpart better than mismatched or randomly permuted counterparts, ideally controlling for overall model strength.

A similar issue appears in the one-shot transfer experiment. Game demonstrations are randomly assigned without task- or capability-level matching, so any gains cannot be attributed specifically to transfer from a related game task or capability. They could instead come from the generic effect of adding a solved example. A stronger test would compare matched game demonstrations against mismatched or random demonstrations and a non-game one-shot control.

Questions for authors
  1. Does performance on a toy-game task or capability predict performance on its matched real-world counterpart better than on mismatched or randomly permuted counterparts? An analysis of this kind would materially affect my assessment because it would directly test whether the proposed game–real alignment itself is informative.

  2. In the one-shot experiment, have you compared matched game demonstrations with mismatched/random game demonstrations or with a non-game one-shot control? This would help determine whether the observed gains come from transfer between related reasoning tasks or from the generic effect of adding a solved example.

Reviewer_9Jo8OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies whether toy-game performance reflects real-world visual reasoning and whether game examples help on real-world tasks. It evaluates 25 VLMs and finds strong score correlations.

Critical strengths
  1. The benchmark construction is systematic, with procedural generation, answer verification, and separate evaluation instances.
  2. The game–real correlation is strong. The appendix includes useful robustness checks across model subsets, data sources, and weighting schemes.
  3. The paper reports both positive and negative one-shot results and acknowledges their limitations.
Critical weaknesses
  1. The alignment is not directly validated. Stronger models may simply perform better in both domains. The correlation analysis does not show whether matched capabilities are more predictive than mismatched ones.
  2. The one-shot experiment lacks key controls. Random game demonstrations are compared only against zero-shot evaluation. Without matched, mismatched, and non-game demonstrations, the gains cannot be attributed specifically to game reasoning. The overall average decline also limits the evidence for transfer.
  3. The constraint-reasoning gains need closer examination. This category combines SSI-Bench and SpatiaLQA. The latter uses a modified rule-based evaluator, but its agreement with human or official judgments is not reported. Source-level results and evaluator validation are needed.

The insight and the question are worthwhile. My main concerns are about experimental rigor and the interpretation of the results. These concerns affect how much we can conclude from the experiments. I would consider raising my score if the authors address them with additional evidence.

Questions for authors
  1. Do matched game–real capabilities show stronger associations than mismatched ones, after accounting for general model strength?
  2. Do the one-shot gains persist against non-game demonstrations? Does capability matching improve the results?
  3. How much of the constraint-reasoning gain comes from each source? Does the modified SpatiaLQA evaluator agree with human or official judgments?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.