acceptodds
Under review as a conference paper at ICLR 2027

ComboBench: From Semantic Goals to Device-Command Plans

Abstract

Translating a goal into device commands requires more than naming relevant actions: a plan must specify the right controller, operation, parameters, and sequence. We introduce ComboBench, a text-only benchmark of device-command planning drawn from four virtual-reality games. Goals and controller conventions are mapped to ordered manipulation steps, allowing command specification to be studied separately from visual perception and online control. The twelve-model evaluation reveals complementary strengths rather than a single best system. Five demonstrations increase reference-prefix recovery for every model in the reported aggregate, yet strict full-sequence matching remains at most 11.7%. A 50-scenario study with additional valid reference paths examines the ambiguity of single-path scoring, alongside a human text baseline and game-level analyses. We further investigate what these scores measure: a semantic matcher accepts 263 of 374 constructed command-attribute mismatches, and snapshot-wide normalization can give an empty plan a score of 0.867. A 68-scenario reanalysis supplies paired uncertainty estimates. Together, the benchmark and diagnostics make device-command specificity measurable while distinguishing reference agreement, format validity, and execution success. The artifact is available at [https://anonymous.4open.science/r/ComboBench-623B/](https://anonymous.4open.science/r/ComboBench-623B/).

Primary area:
datasets and benchmarks
Keywords:
large language models, language-to-action planning, virtual reality, benchmark, evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_zE9VOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces ComboBench, a text-only benchmark for translating VR-game goals and controller conventions into ordered device-manipulation instructions. It reports 262 scenarios across four games and evaluates twelve models with different numbers of demonstrations. The study includes a human text baseline, multiple reference paths, and analyses of command-attribute sensitivity and score normalization. Its main contribution is a domain-specific planning dataset and an evaluation diagnostic study, rather than a new planning method.

Critical strengths
  1. Specifying controller side, operation parameters, and action order provides a concrete refinement of high-level action descriptions. Separating this written-planning task from perception and online control is an interesting research choice.
  2. The evaluator diagnostics are informative. The attribute perturbations and empty-output analysis expose specific ways that semantic matching and normalization can produce misleading scores.
Critical weaknesses
  1. The distinct benchmark contribution is insufficiently established. Goal-to-action generation and the gap between plausible descriptions and actionable plans have already been studied in previously, such as Language Models as Zero-Shot Planners[1]. Embodied Agent Interface [2] also evaluates action sequencing and provides fine-grained planning-error diagnostics. The clearest distinction is its VR controller-level granularity, but the experiments do not establish what additional model failure this reveals beyond game knowledge, procedural recall, or reference-format matching. In particular, the paired analysis identifying semantically adequate outputs with incorrect device commands is explicitly left unfinished in Appendix E. A small benchmark can be valuable, but it needs a focused, validated insight. Here, the contribution remains closer to a domain-specific textual planning dataset with scoring diagnostics than an established measurement of a distinct planning capability.

  2. The metrics are not sufficiently validated for the command distinctions that motivate the benchmark. The evaluation relies on embedding similarity and reference path matching rather than verifying actual controller actions. Consequently, valid alternative paths receive low scores, while substitutions indicate matcher insensitivity rather than true execution errors. Without independent validation, the scores fail to support the benchmark's main novelty.

Refs:

[1] Wenlong Huang, Pieter Abbeel, Deepak Pathak, and Igor Mordatch. Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agents. ICML 2022.

[2] Manling Li et al. Embodied Agent Interface: Benchmarking LLMs for Embodied Decision Making. NeurIPS 2024, Datasets and Benchmarks Track.

Questions for authors
  1. What does ComboBench reveal beyond existing action-planning benchmarks? Do models still make device-command errors when given a correct high-level plan and complete controller conventions?
  2. Have authors checked whether the metric agrees with human judgments? For example, would a correct alternative solution score lower than an answer that follows the reference but presses the wrong button?
  3. Please show the actual zero-shot and few-shot prompts. Does the zero-shot prompt already explain all the controls? I’m concerned that the examples may mainly teach the controls and expected answer format.
  4. Which exact scenario set and scorer version produced Table 1 and 2? This is unclear.
  5. The human SSM is only 1.2%, below several models. Are the human answers actually wrong, or just different from the references? And I noticed that in most results, 'Human' achieved a lower score.
Reviewer_pC4JOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies the translation of high-level goals into ordered, device-specific command sequences using tasks drawn from four VR games. The benchmark isolates textual command planning from visual perception and online execution. It evaluates 12 language models under several demonstration settings using 4 reference-agreement metrics, supplemented by human responses, a multi-reference study, and a statistical reanalysis of an Alyx subset. It reports improved reference-prefix recovery with demonstrations, while full-sequence agreement remains uncommon. It also identifies weaknesses in semantic matching and score normalization, advocating evaluation that distinguishes semantic similarity, command specificity, and execution success.

Critical strengths
  1. The attribute-substitution experiment and empty-plan normalization example expose specific measurement failures. These are more informative than simply observing disagreement between aggregate metrics.
  2. Careful interpretation of several results. The authors explicitly distinguish reference agreement from execution success, acknowledge valid alternative procedures, and avoid treating different metric scales as directly comparable measures of capability.
Critical weaknesses
  1. The reported benchmark lacks a sufficiently established connection between data, implementation, and results.
  2. The evaluator does not reliably preserve the distinctions that motivate the benchmark. All four metrics rely on a similarity threshold calibrated using equivalent paraphrases. The stress test subsequently shows acceptance of 263/374 constructed attribute changes, including all 91 handedness substitutions. These are not adjudicated execution failures, but they demonstrate that the matching rule can overlook explicitly changed command fields. Even SSM therefore need not establish exact command agreement. A held-out validation set containing valid paraphrases, operationally different commands, and naturally occurring model outputs is needed. Independent expert judgments should establish both false acceptance and false rejection rates.
  3. Identifying the normalization problem does not resolve its effect on the main evaluation. The paper retains NSAS as a principal metric without providing a complete, consistently rescored comparison. Its dependence on snapshot extrema also makes absolute scores unstable across evaluation populations unless the transformation is fixed. The controlled empty-output adjustment is useful diagnostically, but it is not a validated replacement. The main results should report raw alignment or a justified fixed-scale metric, explicit empty-output handling, and corresponding uncertainty.
Questions for authors
  1. CAfter rescoring all systems with one frozen, independently validated evaluator and explicit empty-output handling, which model comparisons and prompting conclusions remain?
  2. On the same 50 multi-reference scenarios, how do canonical-only and best-of-four scores differ under an identical scoring transformation? Do model rankings change?
Reviewer_xAGKOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces ComboBench, a benchmark containing VR gaming scenarios that map high-level goals to device-command sequences. It evaluates twelve LLMs and examines their performance using several sequence-matching metrics.

Critical strengths

(+) The paper addresses an interesting problem of translating high-level goals into low-level device commands. This is a potentially useful capability for embodied AI systems.

(+) The authors put effort into collecting and annotating scenarios from four VR games, including recording controller-level actions during gameplay.

(+) The evaluation includes twelve models, multiple prompting settings, and human baselines. The authors also identify important limitations in their own evaluation metrics.

Critical weaknesses

(-) It is unclear who would use this dataset and for what purpose beyond comparing LLM-generated command sequences. A concrete application or demonstration of how the benchmark helps develop better embodied agents would strengthen the contribution.

(-) Prior benchmarks such as PlanBench and Embodied Agent Interface already study action planning, decomposition, and sequencing. The main distinction here seems to be VR controller-specific commands, but the paper does not convincingly establish why this narrower representation requires a separate benchmark or what new insights it enables.

(-) The dataset is relatively small and contains only 262 scenarios from four games, with no convincing analysis of whether these adequately represent the diversity and complexity of VR interactions. More systematic coverage of action types, task difficulty, and controller configurations would make the dataset more valuable.

(-) Although the authors describe collecting gameplay actions and constructing reference sequences, there is little evidence of annotation consistency, independent validation, or ground-truth quality. Since dataset creation is the main contribution, these aspects deserve much stronger attention.

(-) The authors show that their semantic matcher accepts 263 of 374 incorrect command-attribute substitutions and that even an empty plan can receive a high score. Without a validated scoring method or execution-based checks, it is difficult to know whether the benchmark can reliably distinguish correct plans from incorrect ones. The unresolved dataset-count discrepancy further weakens reproducibility.

Questions for authors
  1. What specific research problems or practical applications would benefit from ComboBench that cannot be adequately studied using existing planning or embodied-agent benchmarks?

  2. How were the reference command sequences independently validated, and what measures were taken to ensure their correctness and consistency across annotators?

  3. How do the authors justify the coverage of 262 scenarios, and would the benchmark generalize to unseen VR games or controller configurations?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.