- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper builds a benchmark that pairs procedurally generated toy-game tasks with real-world visual reasoning items under four capability groups: fine-grained perception (FP), spatial reasoning (SR), constraint reasoning (CR) and sequential planning (SP).
- Game side: 18 tasks and 65 QA families. Generators come with solvers and verifiers that check uniqueness and optimality where required. The pipeline yields a 23,893-sample training set and a separately seeded 1,000-sample evaluation set.
- Real-world side: 400 items taken mostly as-is or lightly reformulated from 11 existing benchmarks and datasets, each tagged with a "primary" game task.
Main results:
- Across 25 VLMs, overall game and real-world scores correlate at Pearson/Spearman 0.935.
- Game tasks are harder and spread models out more. And thinking mode helps more for larger models.
- One-shot in-context learning with random game demonstrations (four Qwen models, three seeds) helps CR on average (+4.9), hurts FP for all four models (−4.9), and lowers overall accuracy by 1.34.
Critical strengths
- The game side is well engineered and reusable. The paper has executable generators, task-specific solvers that check uniqueness and optimality, disjoint seeds and layouts between the 24K training set and the evaluation set, and de-duplication by semantic fingerprints and image hashes. Verifiable multi-step visual tasks like Rush Exit, Mancala and Pipe are useful for RL-style training regardless of the game-to-real question.
- The correlation analysis is checked carefully, and negative results are reported. The paper includes leave-one-source-out, answer-format subsets and model bootstrap (App. D.2), and it states plainly that one-shot game demonstrations lower overall accuracy.
Critical weaknesses
-
The 0.935 correlation mostly reflects overall model strength, not capability-specific alignment. With 25 models spanning 2B to frontier, almost any two VLM benchmarks correlate this strongly. The test that matters is whether game capability c predicts real capability c better than it predicts the other real capabilities. Recomputing from Table 1 (my numbers reproduce your overall r = 0.934):
- Game FP correlates 0.79 with real FP, but 0.84–0.86 with each of the other three.
- Game SR correlates 0.85 with real SR, but 0.87 with real CR.
- Partial correlations for matched pairs, controlling for the other three real capabilities, are 0.05 (FP), 0.22 (SR), 0.75 (CR) and 0.38 (SP).
Only CR shows a specific correspondence. Coverage is also thin: four game tasks have no real-world counterpart, and Cube Net ↔ IKEA has only 5 items.
-
The game-example prompting experiment doesn't support a transfer claim. The net effect is negative (−1.34). The CR gain comes mostly from Qwen3.5-9B (+15.6), whose zero-shot CR (24.3) is oddly below the 4B model's (28.9). Half of the real CR items are SpatiaLQA, which require structured outputs, so the solved example may simply be showing the model the answer format. A zero-shot run that states the format requirements explicitly in the system prompt would check this: if it recovers the gain, the effect is format, not transfer. More fundamentally, the paper releases a 24K training set but runs no training. A small SFT or RL run on Game2Real-Dataset, evaluated on the 400 real items by capability, would answer the title question directly.
Questions for authors
- Please report the capability-level cross-correlation matrix and partial correlations. Do you agree that only CR shows specific correspondence? If not, what explains game FP predicting real CR better than real FP?
- What is the correlation between your real-world set and a benchmark not aligned with any game task, over the same models?
- For Qwen3.5-9B on real CR, what fraction of zero-shot failures are format errors, truncation, or unparseable outputs, and how does that fraction change once a game example is added to the prompt?