acceptodds
Under review as a conference paper at ICLR 2027

Sensitivity Is Not Validity: LLM Plan Judges Predict Execution Worse Than Overlap

Abstract

We ask whether plan evaluators predict what happens when the plan is executed – in AI2-THOR, under an oracle low-level controller that removes perception and actuation as causes of failure. Plan-level evaluation promises to separate an embodied agent's reasoning from its perception and control, but only if its scores predict something real. We meta-evaluate deterministic, symbolic, embedding-based and rubric-guided LLM-judge evaluators by how well each predicts execution of the same plans. Across 24,615 rollouts of 1,641 ALFRED-derived episodes, a plain overlap metric predicts execution better than EMBER – our own rubric judge, a contestant here rather than a proposal (primary pool, n=6,564: AUROC 0.890 vs. 0.786; preregistered difference -0.104, 95% CI [-0.125,-0.082]). The deficit holds for all fifteen planners we executed, eleven of them in four families scoring the same episodes, with a mean of -0.114 over those eleven. One family is architecturally disjoint from the one we developed on, and one planner is multimodal, its image moving plans but not execution. A frontier judge given the same inputs still trails (0.857). The judge's usual defence – that it credits valid alternatives overlap penalizes – fails on natural output and holds only weakly under control: on execution-verified rewrites of gold plans it beats LCS by a preregistered +0.102, but by +0.079 once no rewrite names a new object, it is near chance on precondition violations a symbolic checker catches, and an order-free overlap metric beats it throughout. A second preregistered test finds no localization gain over a single overlap scalar. The ordering holds for decisions: choosing one of eleven planners' plans per instruction, overlap picks an executable plan 50.2% of the time, the judge 36.0%. What the judge adds with a reference is small (exploratory): +0.016 AUROC over overlap, reordering plans below the top rather than changing which is chosen, and +0.005 over a fitted composite of cheap text features, the strongest scorer we have (0.947). Without a reference, a frontier judge is the best option we tested. An evaluator's validity has to be established for the decision it serves. We contribute the protocol, a validated executor, and these execution-anchored results.

Primary area:
datasets and benchmarks
Keywords:
LLM-as-a-judge, meta-evaluation, predictive validity, embodied planning, plan evaluation, reference-free evaluation, measurement validity, ALFRED, AI2-THOR

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_zNAcOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper explore if plan-level evaluators predict execution success. It evaluates plans from 15 planners through 24,615 rollouts on ALFRED-derived tasks. The main finding is that reference-plan overlap, particularly LCS, predicts success better than the proposed rubric-based LLM judge, EMBER. The paper also studies whether judges help identify failures, recognize alternative plans, or evaluate plans without a reference. Overall, I find the evaluation question worthwhile, but concerns about the executor and judge implementation affect the interpretation of the main result. Also, the limited execution setting weakens the paper.

Critical strengths
  • I like the execution-based evaluation: Testing whether evaluator scores correspond to actual execution success is useful, and the comparison against simple overlap metrics provides an informative baseline.
  • The paper makes a good effort toward transparent evaluation: The preregistered tests, clustered bootstraps, and replication analyses make the study better. I also appreciate that the authors report negative findings and discuss claims that their experiments did not support.
Critical weaknesses
  • The executor’s language restrictions complicate the main conclusion. Also contribution statement is too spread out The reported synonym-related failures raise a major concern about whether the evaluation measures plan quality or compatibility with the executor’s vocabulary. A judge may reasonably accept “refrigerator” instead of “fridge,” while execution rejects it. Removing these rollouts does not establish what would happen if the executor resolved synonyms correctly. So I find the headline comparison difficult to interpret without re-execution.

  • The judge implementation introduce an important source of uncertainty. Missing verdict fields are converted into zero scores, but raw responses were not retained. As a result, it is unclear how much of the judge’s poor performance reflects its assessment versus parsing failures. The frontier judge’s reasoning configuration also needs clarification before drawing a broad conclusion about the limitations of capable LLM judges.

  • The experiment result supports a more narrower conclusion than the recommendations suggest. Execution is validated only on ALFRED under an oracle controller, with relatively low-success planners. But, pooled AUROC mostly compares plans across different instructions, so task difficulty can contribute a lot to the result. The within-instruction analysis is useful, but it also needs uncertainty estimates and a task-type breakdown. These issues leave me unconvinced that overlap would retain its advantage for stronger planners producing diverse, valid alternatives.

  • Appendix C says every gpt-5.1 number comes from a judge without the object list, yet the main text reports the object-given arm. At many places there is difference in main text vs appendix.

  • Use of older model also raises concern and no justification is provided.

Questions for authors
  • Could you re-execute the alias-miss rollouts with synonym resolution? Does the overlap–judge gap persist when ordinary naming variations are accepted?

  • How many verdicts received parser-default values? Could you audit a sample with raw outputs retained to establish whether parsing materially affects the reported zero scores?

  • What reasoning configuration and output budget were used for the frontier judge? Does its deficit persist with reasoning explicitly enabled?

Reviewer_15GzOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates whether LLM judges can evaluate the quality of action plans by predicting their execution success. The paper compares different evaluation metricsand show that simple overlap metrics such as LCS outperform LLM judges in predicting execution success when there is a reference plan. They also investigate different settings, e.g., planner models, judge models.

Critical strengths
  1. Using execution outcomes rather than relying on human/LLM judgments is intuitive and is more objective to evaluate a plan quality.
  2. The paper evaluates multiple planner and judge models, including different model families and sizes.
  3. The statistical analyses and robustness checks are comprehensive.
Critical weaknesses
  1. The evaluation is limited to “ALFRED” tasks that are executed with an oracle. This excludeds many challenges of real-world execution, e.g., perception and low-level control. It is unclear how much the findings generalize to more natural scenarios where execution success depends on these factors.
  2. The oracle executor only achieves 88% success on gold plans, suggesting that the execution outcome is not a perfect ground truth.
  3. It is unclear how mucg the conclusion that simple overlap metrics outperform LLM judges can be applied to realistic agent evaluation. The conclusion relies on the presence of gold reference plans. However, in many practical applications, reference plans are not available. On the other hand, in the reference-free setting, the paper actually shows that stronger LLM judges outperform the symbolic baselines.
Questions for authors
  1. How would the results change if the evaluation us only done on episodes where the gold plans successfully execute, given that the oracle executor fails on some gold plans?
Reviewer_c6JvOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This is the last paper in my batch and the fifth one I've read that seems totally AI generated, at least the text itself. My summary of the abstract appears below in paper weaknesses and contains my attempt to understand what this paper is about. The authors didn't write the paper, so it does not seem even a little bit reasonable to me to ask reviewers to read it and try to understand its contributions.

Critical strengths

N/A. This submission is not a paper, it is LLM output in the shape of a paper.

Critical weaknesses

The abstract is thick with LLM-generated sentences that are overly long and difficult to parse, included the dreaded and extremely obvious case where there's an em-dash parenthetical in the first part of a complete sentence followed by a colon and a second totally independent sentence tied to the first. ("The judge’s usual defence – that it credits valid alternatives overlap penalizes – fails on natural output and holds only weakly under control: on execution-verified rewrites of gold plans it beats LCS by a preregistered +0.102, but by +0.079 once no rewrite names a new object, it is near chance on precondition violations a symbolic checker catches, and an order-free overlap metric beats it throughout.") The abstract LLM sentences include vague and undefined referents ("A frontier judge given the same inputs still trails (0.857)." still trails what??). The abstract also fails to amke it clear what's actually being measured or tested. At first it sounds like an evaluation of the efficacy of LLM-as-judge for determining plan outcomes ("We meta-evaluate deterministic, symbolic, embedding-based and rubric-guided LLM-judge evaluators by how well each predicts execution of the same plans"), but a few sentences later it seems to be that planners are being evalauted ("The deficit holds for all fifteen planners we executed, eleven of them in four families scoring the same episodes, with a mean of −0.114 over those eleven.").

The introduction similarly starts out weak with LLM-sentence after LLM-sentence, reading more like a LinkedIn influencer blog than a scientific paper. The language style is grating, condescending, and repeats in every paragraph.

Questions for authors

An honest question: why do the authors expect reviewers to read this paper if they did not themselves bother to write it?

How can the authors not feel embarassed by having submitted this to a top-tier scientific conference?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.