- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper explore if plan-level evaluators predict execution success. It evaluates plans from 15 planners through 24,615 rollouts on ALFRED-derived tasks. The main finding is that reference-plan overlap, particularly LCS, predicts success better than the proposed rubric-based LLM judge, EMBER. The paper also studies whether judges help identify failures, recognize alternative plans, or evaluate plans without a reference. Overall, I find the evaluation question worthwhile, but concerns about the executor and judge implementation affect the interpretation of the main result. Also, the limited execution setting weakens the paper.
Critical strengths
- I like the execution-based evaluation: Testing whether evaluator scores correspond to actual execution success is useful, and the comparison against simple overlap metrics provides an informative baseline.
- The paper makes a good effort toward transparent evaluation: The preregistered tests, clustered bootstraps, and replication analyses make the study better. I also appreciate that the authors report negative findings and discuss claims that their experiments did not support.
Critical weaknesses
-
The executor’s language restrictions complicate the main conclusion. Also contribution statement is too spread out The reported synonym-related failures raise a major concern about whether the evaluation measures plan quality or compatibility with the executor’s vocabulary. A judge may reasonably accept “refrigerator” instead of “fridge,” while execution rejects it. Removing these rollouts does not establish what would happen if the executor resolved synonyms correctly. So I find the headline comparison difficult to interpret without re-execution.
-
The judge implementation introduce an important source of uncertainty. Missing verdict fields are converted into zero scores, but raw responses were not retained. As a result, it is unclear how much of the judge’s poor performance reflects its assessment versus parsing failures. The frontier judge’s reasoning configuration also needs clarification before drawing a broad conclusion about the limitations of capable LLM judges.
-
The experiment result supports a more narrower conclusion than the recommendations suggest. Execution is validated only on ALFRED under an oracle controller, with relatively low-success planners. But, pooled AUROC mostly compares plans across different instructions, so task difficulty can contribute a lot to the result. The within-instruction analysis is useful, but it also needs uncertainty estimates and a task-type breakdown. These issues leave me unconvinced that overlap would retain its advantage for stronger planners producing diverse, valid alternatives.
-
Appendix C says every gpt-5.1 number comes from a judge without the object list, yet the main text reports the object-given arm. At many places there is difference in main text vs appendix.
-
Use of older model also raises concern and no justification is provided.
Questions for authors
-
Could you re-execute the alias-miss rollouts with synonym resolution? Does the overlap–judge gap persist when ordinary naming variations are accepted?
-
How many verdicts received parser-default values? Could you audit a sample with raw outputs retained to establish whether parsing materially affects the reported zero scores?
-
What reasoning configuration and output budget were used for the frontier judge? Does its deficit persist with reasoning explicitly enabled?