- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper introduces ComboBench, a text-only benchmark for translating VR-game goals and controller conventions into ordered device-manipulation instructions. It reports 262 scenarios across four games and evaluates twelve models with different numbers of demonstrations. The study includes a human text baseline, multiple reference paths, and analyses of command-attribute sensitivity and score normalization. Its main contribution is a domain-specific planning dataset and an evaluation diagnostic study, rather than a new planning method.
Critical strengths
- Specifying controller side, operation parameters, and action order provides a concrete refinement of high-level action descriptions. Separating this written-planning task from perception and online control is an interesting research choice.
- The evaluator diagnostics are informative. The attribute perturbations and empty-output analysis expose specific ways that semantic matching and normalization can produce misleading scores.
Critical weaknesses
-
The distinct benchmark contribution is insufficiently established. Goal-to-action generation and the gap between plausible descriptions and actionable plans have already been studied in previously, such as Language Models as Zero-Shot Planners[1]. Embodied Agent Interface [2] also evaluates action sequencing and provides fine-grained planning-error diagnostics. The clearest distinction is its VR controller-level granularity, but the experiments do not establish what additional model failure this reveals beyond game knowledge, procedural recall, or reference-format matching. In particular, the paired analysis identifying semantically adequate outputs with incorrect device commands is explicitly left unfinished in Appendix E. A small benchmark can be valuable, but it needs a focused, validated insight. Here, the contribution remains closer to a domain-specific textual planning dataset with scoring diagnostics than an established measurement of a distinct planning capability.
-
The metrics are not sufficiently validated for the command distinctions that motivate the benchmark. The evaluation relies on embedding similarity and reference path matching rather than verifying actual controller actions. Consequently, valid alternative paths receive low scores, while substitutions indicate matcher insensitivity rather than true execution errors. Without independent validation, the scores fail to support the benchmark's main novelty.
Refs:
[1] Wenlong Huang, Pieter Abbeel, Deepak Pathak, and Igor Mordatch. Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agents. ICML 2022.
[2] Manling Li et al. Embodied Agent Interface: Benchmarking LLMs for Embodied Decision Making. NeurIPS 2024, Datasets and Benchmarks Track.
Questions for authors
- What does ComboBench reveal beyond existing action-planning benchmarks? Do models still make device-command errors when given a correct high-level plan and complete controller conventions?
- Have authors checked whether the metric agrees with human judgments? For example, would a correct alternative solution score lower than an answer that follows the reference but presses the wrong button?
- Please show the actual zero-shot and few-shot prompts. Does the zero-shot prompt already explain all the controls? I’m concerned that the examples may mainly teach the controls and expected answer format.
- Which exact scenario set and scorer version produced Table 1 and 2? This is unclear.
- The human SSM is only 1.2%, below several models. Are the human answers actually wrong, or just different from the references? And I noticed that in most results, 'Human' achieved a lower score.