acceptodds
Under review as a conference paper at ICLR 2027

CAP-7: Binding-Aware and Commitment-Complete Verification for Task-Level Affordance Reasoning

Abstract

Task-level affordance reasoning requires vision-language models to turn images and manipulation instructions into plans that connect objects, actions, states, and motions. Yet evaluating these elements separately leaves two failures unresolved: plausible elements may be connected incorrectly, which we term , and required steps may be omitted without being penalized, which we term . To expose incorrect bindings, we introduce CAP-7, a typed program representation linking objects and roles to actions, states, dependencies, and motions. We construct CAP-7-PD for supervision and a extension for community research. Controlled changes to roles, action order, and motion paths test whether the resulting plans preserve the relations required by the task. To detect omissions, we develop commitment-complete verification that retains missing requirements in the score and penalizes unsupported additions. Combined with a check of whether the submitted action order executes legally, this verification provides a training reward. Rescoring unchanged model outputs reveals misleading gains under the original verifier. Under matched training conditions, replacing only the reward raises the mean share of sampled orders judged legal by an independent symbolic engine from 51.3% to 94.4%, compared with 85.2% for the shared supervised initialization; unmatched step pairs also decrease. These results show that complete task verification can improve the structure and symbolic executability of learned manipulation plans.

Primary area:
applications to robotics, autonomy, planning
Keywords:
affordance reasoning, task-level affordance, robotic manipulation, vision-language models, embodied AI, program representation, verifiable programs, binding, partial-order planning, dataset and benchmark

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_g45mOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies task-level affordance reasoning, where models must not only identify interaction regions but also correctly connect object roles, actions, states, dependencies, and motions. The authors highlight two failure modes: Binding Blindness, where these elements are connected incorrectly, and Omission Blindness, where missing requirements escape evaluation. They introduce CAP-7, a task-program representation with seven linked fields, along with CAP-7-PD, a dataset of 1,698 programs, and a separate 20k-program extension. The main contribution is commitment-complete verification, which accounts for omitted reference requirements and penalizes unsupported additions. Combined with state and execution-order checks, it provides a training reward. With the same supervised initialization and training budget, replacing the original reward increases average sampled-order legality under an independent symbolic engine from 51.3% to 94.4%, compared with 85.2% for the supervised starting model. However, goal achievement and some relational metrics do not show corresponding improvements.

Critical strengths

The paper identifies an important evaluation gap. Separating coverage from the correctness of expressed relations helps reveal misleading performance gains. The reward comparison is well controlled, using the same supervised initialization and training budget. The paper is well presented, with clear figures and tables.

Critical weaknesses

The main gain is in execution legality, not task completion. Goal achievement does not significantly improve (0.6320 vs. 0.6456), while step-deletion CBA drops significantly (0.654 to 0.589). Further analysis is needed to show whether actual omissions decrease, or whether the model simply produces more conservative plans. Reference completeness is not the same as task completeness. Valid plans may use different steps, tools, or action granularities. The verifier should show that it does not unfairly penalize these alternatives. The novelty over existing planning and verification methods needs clarification. Program representations, precondition checks, and plan validation are already established in systems such as ProgPrompt and VAL. The paper should explain what CAP-7 adds beyond these capabilities.

Questions for authors

The complete reward improves legality, but not goal achievement, while step-deletion CBA decreases. How do you explain this? Could you report key-step recall, omission types, and the joint rate of legal execution and goal achievement? Which reward components drive the gains? Could you compare execution gating alone, coverage and redundancy penalties alone, and the full reward, alongside a simple execution-legality reward baseline? How does the verifier handle valid alternatives to the reference program? Could you test human-verified equivalent plans with different tools, action granularities, and alternative steps, and report how often they are incorrectly penalized?

12 comments
AuthorsOpenReview ↗

Q1. The complete reward improves legality, but not goal achievement, while step-deletion CBA decreases. How do you explain this?

Thank you for taking the time to review our work.

Response

Thank you for raising this—it is the most valuable comment we received on whether the mechanism is clearly explained. Neither of the two phenomena you observed can be explained from the text of the original draft, and on rereading we believe the problem is that we discussed errors from two different sources together, so that readers could see the phenomena but not their causes. By putting the two side by side you led us to make the decomposition explicit—which is why §4.1 through §4.4 were rewritten for the revision.

The two phenomena you mention correspond to two distinct design defects:

Cause 1: the goal-attainment rate does not move, because the goal is not in the check set.

The seven check heads of the original draft cover objects, bindings, step matching, dependency order, state, trajectory, and geometry—all of them measure “is the plan written correctly”; “are the two sub-goals achieved” is not among them. A reward trained on this can therefore only push the “written correctly” side, and legality rises accordingly; the goal-attainment rate lies outside what the checks can see, and so cannot be changed by this reward. In the notation of Eq. (3) of the revision, this part of the error is εirr(Z)\varepsilon_{\mathrm{irr}}(Z): it is determined by the checks ZZ and is independent of the scoring RR:

∥U−R∥2=∥U−R⋆∥2⏟εirr(Z)+∥R⋆−R∥2⏟εred.\lVert U-R \rVert^2 = \underbrace{\lVert U-R^\star \rVert^2}_{\varepsilon_{\mathrm{irr}}(Z)} + \underbrace{\lVert R^\star-R \rVert^2}_{\varepsilon_{\mathrm{red}}}.

As long as no goal is in ZZ, no matter how much better RR becomes, εirr(Z)\varepsilon_{\mathrm{irr}}(Z) stays exactly where it is.

Cause 2: deletion-type CBA is distorted, because the denominator normalises along with whatever is expressed.

The original draft scored each head separately and took a weighted average (S=2M−1S = 2M - 1), with each head using “what has been expressed” as its denominator. When deletion-type corruption shrinks the reference obligation set to Uγ⋆⊂U⋆U^\star_\gamma \subset U^\star, there are fewer pairs to cover, so the denominator shrinks with it: numerator and denominator fall together and the score does not drop. The outcome of this family of contrasts is thus decided mostly by how far the denominator collapses, rather than by whether the prediction actually distinguishes a complete reference from a truncated one. In the notation of Eq. (3), this part is εred>0\varepsilon_{\mathrm{red}} > 0.

AuthorsOpenReview ↗

Q1. The complete reward improves legality, but not goal achievement, while step-deletion CBA decreases. How do you explain this?

Action

  1. Fixed the denominator by C1–C3 (Eq. (5) of the main text):

    D=∣U⋆∣+um+(u2),D = |U^\star| + um + \binom{u}{2},

    which keeps both the missing reference pairs and the unmatched predicted pairs in the denominator, so that neither side collapses under corruption or omission, giving εred=0\varepsilon_{\mathrm{red}} = 0.

  2. Extended the check set, folding the goal predicates into Z+=(Z,B)Z^+ = (Z,B) (Eq. (4) of the main text):

    εirr(Z)−εirr(Z+)=∥E[U∣Z+]−E[U∣Z]∥2≥0.\varepsilon_{\mathrm{irr}}(Z) - \varepsilon_{\mathrm{irr}}(Z^+) = \lVert\mathbb{E}[U\mid Z^+] - \mathbb{E}[U\mid Z]\rVert^2 \geq 0.

  3. Added the definition and implementation of the goal term (Eq. (9) of the main text). The reward is Rgoal=clip⁡(Rcc+γ(g−1),−1,1)R_{\mathrm{goal}}=\operatorname{clip}(R_{cc}+\gamma(g-1),-1,1) with γ=1.0\gamma=1.0. Here gg is the mean fraction of reference core-goal facts satisfied in the final state, up to an injective renaming of entities, over four random execution orders consistent with the declared dependencies. The core facts are the facts added by the reference plan whose predicates are stroke_count, on_top_of, contained_in, poured_into, or orientation_deg. An empty core-goal set gives g=1g=1. Programs rejected by the RccR_{cc} gates retain a reward of −1-1.

  4. Added execution-level validation, so that the results do not stop at the symbolic level (§5.4, Table 5; 19,074 real episodes):

    Upper-level plannerBoth sub-goals written ↑\uparrowWhole-task success ↑\uparrowSub-goal attainment ↑\uparrow
    A00.3780.2940.400
    Complete scoring0.3450.2700.368
    Complete scoring + goal term0.6360.4910.653
  5. Added the CBA rescoring experiment (§5.3, Table 4). With the programs held fixed and only the scoring rule replaced, CBA rises for all six conditions, while the spread among them narrows from 0.065 to 0.020:

    ConditionCBA original ↑\uparrowCBA complete ↑\uparrowGain ↑\uparrow
    A00.7070.828+0.121
    P60.7720.846+0.074
    L00.7650.848+0.083
    RFT-10.7370.835+0.098
    RFT-20.7410.842+0.101
    RFT-new0.7490.840+0.091

    The narrowing of the spread is exactly the effect of εred→0\varepsilon_{\mathrm{red}} \to 0: the differences among conditions originally came from how far each one fell into the gap, and once that term is removed only εirr\varepsilon_{\mathrm{irr}} remains; since all six conditions face the same ZZ, the differences vanish with it. P6 is the clearest example—under the original score it is 0.772, the highest in the table and 0.065 ahead of A0, whereas under complete scoring it leads by only 0.018; and it is precisely the condition with the lowest coverage (0.283) and conditional correctness (0.709). Its original advantage therefore came from the gap, not from binding quality.

  6. Added an explanation for why CBA and the order agreement move in opposite directions. Your observation prompted us to state explicitly in the caption of Table 4 that, relative to the supervised start, contrastive accuracy rises for all five reward variants while the order agreement (ORD3) and conditional correctness fall for all of them—exactly the direct evidence for “optimizing along the gap”, and the reason why CBA alone leads to the opposite conclusion.

  7. Added the necessity half of the proposition's proof to Appendix B (the original draft carried only the heading “sufficiency”). A new “Step 4: necessity of the conditions” shows explicitly that, in the absence of C1/C2, two plans land on the same atom and no scoring can distinguish them, while the necessity of C3 follows directly from the definition.

AuthorsOpenReview ↗

Q2.Could you report key-step recall, omission types, and the joint rate of legal execution and goal achievement? Which reward components drive the gains?

Response

Thank you for the suggestion. These three items are indeed things we should have put in the main text and did not—the joint value especially, which was already computed per task among the registered metrics of the supplementary material but never made it into the main text. That was an oversight in our presentation. Your comment led us to promote all three to formal content in the main text or appendix.

Action

  1. Recall is now reported in §5.2 of the main text (Table 3). Coverage is the relational counterpart of recall: it measures how many reference step pairs the plan expresses. The three supervision configurations give 0.357 (N-1, flat-sequence objective), 0.352 (B-0, Qwen3-VL-8B backbone), and 0.362 (A0, program-level supervision), all within the narrow band from 0.352 to 0.362—coverage is determined by the supervision data itself, and changing the supervision objective or the backbone barely moves it. The difference appears in accuracy at the same coverage: conditional correctness 0.869 / 0.893 / 0.898 and replay legality 0.186 / 0.874 / 0.890.

  2. Named the B-0 backbone (the original draft said only “the second backbone”). The revision states Qwen3-VL-8B-Instruct in three places—main text, table caption, and table row—so that readers can judge what scale “transferable” actually transferred to.

  3. Omission types are now priced separately by relation type (§4.2, Eq. (6)). In the denominator D=∣U⋆∣+um+(u2)D = |U^\star| + um + \binom{u}{2}, ∣U⋆∣|U^\star| retains every reference step pair whether expressed or not, umum counts the pairs between unmatched predicted steps and reference steps, and (u2)\binom{u}{2} counts the pairs among unmatched steps themselves. The two kinds of omission therefore enter the denominator separately: a whole sub-chain that never appears lowers coverage, while a step that appears but is bound to the wrong object lowers conditional correctness. Figure 4 of the revision shows these two forms qualitatively.

  4. The joint value is now in the main text and appendix. 360 test tasks, two seeds:

    ConditionLegal-execution rateGoal-attainment rateJoint (legal and achieved)
    A0 (supervised start)0.841 / 0.8490.608 / 0.6060.549 / 0.543
    Complete scoring0.933 / 0.9280.622 / 0.6280.581 / 0.575
    Complete scoring + goal term0.930 / 0.9330.644 / 0.6530.594 / 0.597

    Paired differences (both seeds pooled, 95% intervals): goal term −- complete scoring on the joint value is +0.018+0.018 [−0.004,+0.042][-0.004, +0.042]; goal term −- A0 is +0.050+0.050 [+0.019,+0.082][+0.019, +0.082], an interval that excludes zero. The joint value improves less than either component alone, which is what we expect: the joint requires both things to hold at once, failure in either one is not counted, and it is therefore sensitive to the shortcoming of each factor.

  5. The contribution of each reward component is now broken out item by item:

    ModificationQuantity it raisesReading
    Fix the denominator, count unmatched stepsLegal-execution rate0.841 → 0.933 (seed 42), 0.849 → 0.928 (seed 123)
    Extend the check set, include goal predicatesGoal-attainment rate, joint valuegoal 0.622 → 0.644, joint 0.581 → 0.594 (seed 42)
    The two togetherWhole-task success0.294 → 0.491 (frozen controller, 19,074 executions)
  6. Added Appendix D (Table 9), training each of the eight reward components in isolation and reporting execution results arm by arm for both seeds, so that “which component contributed what” can be read off directly instead of inferred backwards from the combined result.

  7. Added Appendix I (Table 14), disclosing the itemised composition of all 59 run directories and 19,074 executions, so that every reading above is traceable.

Reviewer_hh7YOpenReview ↗
Rating
3
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper addresses Binding Blindness and Omission Blindness, and introduced introduce CAP-7. Instead of predicting the elements for sub tasks, it predicts a plan’s relations by explicitly representing relations and define requirements. To address omission blindness, it make required relations remain countable after omission. The paper also builds CAP-7-PD dataset for evaluation.

Critical strengths
  1. Fig.2 is a good survey that supports the motivation and a good proof of the Omission Blindness problem.
  2. The design of seven question checks is good and intuitive.
  3. The proposed "missing steps must not erase the relations" is a critical design and novel.
  4. Experimental results proof that the proposed pipeline is working.
Critical weaknesses
  1. The language input is a critical factor that affecting the overall pipeline performance. However, it seems the intented task order described in the language input is mostly in correct order. There is no harder task description showing the VLM is actually understanding the language, e.g. instead of a regular "please do A and do B", but input "please do B after do A", the VLM should understand the first task is do A, and the action relation will be more challenging.
Questions for authors

See weakness

2 comments
AuthorsOpenReview ↗

Q:Language input is critical to pipeline performance, yet the evaluated instructions mostly specify the correct task order. More challenging formulations, such as “do A and B” versus “do A, then B,” are needed to assess the VLM’s understanding of action order and dependencies.

Thank you for taking the time to review our work.

Response

This is among the most helpful comments we received for improving the experimental design. It raises a concern that can be settled directly by experiment: if the task sentence itself already states the order, the model may merely be copying a sentence pattern rather than reasoning about the relation between an object and its destination. The contrast you give—“do A and B” against “do A, then do B”—is precisely the cleanest form of that test.

Let us answer first the point you care about most: the prompt used in this paper is set up in the “do A and B” form from the outset, not “do A, then do B”. The task sentence handed to the planner is two LIBERO task descriptions joined directly by and, with no ordering word anywhere in it—no then, no first, no after:

put the bowl on the plate and put the wine bottle on the rack.

What that sentence fixes is therefore “which two things must hold” and “which object belongs to which destination”; it fixes no execution order at all. A planner that matched on sentence pattern alone would have nothing to base an order on. The “harder formulation” you suggest is in fact the default condition of this paper—the original draft simply did not make that clear, and offered no evidence for it.

We agree with the deeper concern behind the comment: even when the sentence states no order, might the model still get the order right through some surface cue? That doubt cannot be removed by explanation; only experiment can answer it. Below we set out all the work we added for this.

AuthorsOpenReview ↗

Q:Language input is critical to pipeline performance, yet the evaluated instructions mostly specify the correct task order. More challenging formulations, such as “do A and B” versus “do A, then B,” are needed to assess the VLM’s understanding of action order and dependencies.

Action

  1. The “no ordering word” setting is now stated in the main text and made checkable. §5.1 states explicitly that the 320 prompts used in the execution experiments consist of 8 tasks × 20 initial placements × 2 instruction word orders; the two orders only swap which of the two sub-goals is mentioned first (the bowl or the wine bottle), and carry no information about execution order. Readers can therefore judge directly that surface order and correct execution order do not coincide, so a model that gets the order right cannot have done so by copying the sentence.

  2. Added a controlled experiment with three instruction rewrites (320 prompts each, frozen controller, same scenes). The scenes, tasks, initial placements, images, GR00T N1.7 weights, and all planners are held fixed; the only thing changed is that one instruction sentence:

    TemplateExampleWhat it removes
    p1I want the bowl on the plate and the wine bottle on the rack.keeps and, drops the imperative verb
    p2Move the bowl and the wine bottle: the first goes on the plate, the second on the rack.one verb shared by both objects; split at and, each half has objects but no destination
    p3The bowl should end up on the plate; the wine bottle belongs on the rack.neither and nor an imperative; the goal is written as a state rather than an action
  3. Result: the gain holds under all three formulations:

    Upper-level plannerp1p2p3Overall
    A0, supervised start0.2030.3090.2370.250
    Complete scoring + goal term (two seeds)0.3310.4940.4280.418
  4. Paired gains and intervals (95% scene-cluster bootstrap):

    Comparisonp1p2p3
    Both seeds pooled+0.128 [+0.083, +0.175]+0.184 [+0.134, +0.234]+0.191 [+0.131, +0.250]
    Seed 42+0.119 [+0.069, +0.169]+0.247 [+0.184, +0.306]+0.191 [+0.128, +0.250]
    Seed 123+0.138 [+0.088, +0.188]+0.122 [+0.072, +0.175]+0.191 [+0.128, +0.253]
    All three formulations——+0.168 [+0.128, +0.209]

    All three intervals exclude zero, and each formulation is individually significant: the gain does not depend on the conjunction, on imperative mood, or on both goals sharing the same grammatical form.

  5. The harder case you describe is exactly where our method gains most. The three templates are not equally difficult. In p1 and p3, each half of the sentence still gives both an object and a destination, so the binding can be read off literally. p2 cannot: the two objects appear together, the destinations sit in another clause, and only “the first” and “the second” index them, so deciding which object goes where requires linking the two clauses. This is precisely the situation you point to, and it is where our method gains the most (0.184 over the baseline, reaching 0.247 at seed 42). A planner that merely restates the sentence gains nothing here, because the sentence no longer contains that pairing.

  6. The same conclusion reproduces from a stronger supervised start. Starting from A5 (16 times the supervision data, CAP-7-PD-20K), adding complete scoring and the goal term gives a gain of +0.132 [+0.089, +0.175], in the same direction across all three formulations. This shows the result is not a peculiarity of one weaker start.

  7. The conclusion this yields. What a change of wording alters is how many pairings can be read off literally; the less the surface supplies, the larger the gain of our method. The mechanism the paper proposes—that scoring must include the omitted relations and the goal predicates—does not depend on how the sentence is written. We have added this point to the revision.

Thank you for taking the time to review our work!

Reviewer_7P6yOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper says that many robot-planning benchmarks are too easy to fool because they only check whether the right objects and actions appear in the answer. Instead, the authors check whether the robot used the right object for the right job, performed the steps in the correct order, and didn't skip important steps. Evaluation is performed on open source models and a RL post-trained model.

Critical strengths
  • The paper address an important problem.
Critical weaknesses
  • The key novelty is not clear. It appears that the paper proposes a CAP-7 verification system. However, it's use case it not clear. The proposed method is used to evaluate plans generated by LLMs. However, these plans are never executed.

  • The planning and verification are done completely in a open loop setting. This is not realistic. In the real-world the planner will be invoked every few timesteps to replan based on the state of the world at that time.

  • As baselines, only LLMs are used. True VLAs like GROOT N1.7 which can generate complex plans and execute them at the same time are not used in the evaluation. Thus, evaluation is highly limited.

  • Moreover, for evaluation older models such as Qwen3-VL and Llama-3.2 are used. Newer models state of the art models such as Qwen3.5/3/6, Cosmos 2 Reason and frontier models such as Gemini-3 or GPT-5 are considered in the evaluation.

  • The proposed CAP-7-PD dataset is really small in size, with ~1.7K examples. CAP-7-PD-20K is not used in the main experiments reported in the paper.

  • Large sections of the paper appear to be AI generated. This is especially evident in the figures, which are clearly generated by a version of Claude. In Figure 5 D, the graphs have no relation to anything else in the figure.

Questions for authors
  • The main contribution of the paper should be explain in more detail.
  • The realism of the evaluation in the paper should also be discussed in more detail.
  • The choice of baselines used for evaluation should also be discussed in more detail.
6 comments
AuthorsOpenReview ↗

Q0. On the openness of data and code.

Thank you for taking the time to review our work.

Response

Thank you for your valuable time. We thank the reviewer sincerely for raising this point. We fully agree with the judgement behind it: for a paper whose central claim is that a verifier should price task requirements differently, releasing only the conclusions while withholding the representation, the data, and the decision procedure would leave every reported number unverifiable, and the claim itself untestable. It is also because the reviewer singled out openness as a point in its own right that we brought the anonymous release of CAP-7-PD and CAP-7-PD-20K forward into this revision, so that both datasets can be inspected during review rather than only at publication.

On how the research object was chosen. We should be explicit that this work was not aimed at adding another decimal point to an established benchmark. When we began, we did not start from a leaderboard gap; we started from a failure we kept observing in real manipulation scenes—plans that score well and then fail to execute, because the scene's actual requirements (which object is the tool, which surface receives it, in what order the grips must occur) had never entered the score at all. Turning that observation into a scientific question, and deciding precisely which object to study, took roughly a third of the project's total time. Most of that time went into discarding formulations that were only descriptive: a taxonomy of failure modes is not a research question, and neither is a new metric that cannot state the condition under which it is provably correct. What we settled on is narrow on purpose—the reward as a projection onto the information the checks carry, the three conditions under which the reducible part of the error can be driven to zero, and the extension of the check set that the residual then demands. We believe this is the right way to contribute: identify a question that the scene itself poses, state it precisely enough that it can be wrong, and then hand over the instruments needed to test it.

Action

  1. Anonymous release of the code. The construction of the CAP-7 seven-tuple representation, the seven check heads, the commitment-complete score (including the denominator rule D=∣U⋆∣+um+(u2)D=|U^\star|+um+\binom{u}{2}), the execution gate, and the goal term are all released in an anonymous repository:

    https://anonymous.4open.science/r/CAP7-EBF5/

  2. Anonymous release of the data (two datasets).

    CAP-7-PD (1,698 task programs: 1,254 train / 84 calibration / 360 test; nine task families with image-disjoint splits):

     [https://huggingface.co/datasets/131kjhsqos0u02ouwu2/CAP7](https://huggingface.co/datasets/131kjhsqos0u02ouwu2/CAP7)
    

    CAP-7-PD-20K (an extension of 20,276 programs, 16 times the primary start):

     [https://huggingface.co/datasets/131kjhsqos0u02ouwu2/CAP-7-PD-20k](https://huggingface.co/datasets/131kjhsqos0u02ouwu2/CAP-7-PD-20k)
    
  3. Where this appears in the manuscript. A public-release statement newly added at the end of the abstract lists the three addresses; contribution (i) now points to the concrete dataset instead of promising a future release of CAP-7-PD-20K. The size of the dataset used in the paper is also deliberately kept at a level that can be audited by hand while still supporting a systematic analysis of the failure modes that matter: the primary set is 1,698 programs.

  4. Reproducibility support. Appendix I newly adds a per-directory audit of execution records that lists all 59 run directories and 19,074 executions grouped by experimental purpose (Table 14), so that any reader can check every execution readout in the main text. Appendix C adds the program and dataset specification (Tables 6 and 7), so that the data format and the check conventions can be compared directly against the code.

AuthorsOpenReview ↗

Q1.The key novelty is not clear. It appears that the paper proposes a CAP-7 verification system. However, it's use case it not clear. The proposed method is used to evaluate plans generated by LLMs. However, these plans are never executed.

Response

This is the comment that helped us most in repositioning the paper. After re-reading the manuscript we agree with it: the original draft placed the CAP-7 representation where the claim should have been, so a reader following the introduction naturally mistook the instrument for the conclusion. That was a failure of our exposition, not of the reviewer's reading.

What we actually intend to claim is narrower and far easier to test: when the controller, the data, and the policy weights are all held fixed and only the reward is replaced, the programs the planner learns to write change, and the success rates those programs achieve once executed change as well. CAP-7 and CAP-7-PD serve this statement by making such a verifier computable and reproducible, but they are not themselves the paper's conclusion. We now state this distinction in a sentence of its own in the rewritten §1, and the three contributions are presented separately.

On the application scenario, our position is that the verifier is a module inside the planning layer, not a new policy: it determines what the frozen policy is asked to do and produces no actions of its own.

On “the plans are never executed”—this is precisely right, and it was the largest gap in the original draft. That gap is now closed by an entirely new block of experiments: the official, unmodified GR00T N1.7 LIBERO-goal weights, run over 320 prompts for 19,074 real episodes, with the reward as the only manipulated variable. The details of this addition follow.

Action

  1. Rewrote the claim structure of §1. “Verifier as reward” is now stated as the claim on its own; CAP-7 and CAP-7-PD are demoted to supporting instruments; the three contributions are reordered as (i) representation and dataset, (ii) projection formulation and the three necessary-and-sufficient conditions, and (iii) training validation on a frozen controller, so that readers no longer have to assemble the argument themselves.

  2. Added an entire block of execution validation, which the original draft lacked completely. Checking the original submission confirms that it mentions GR00T zero times and reports zero real executions. The revision adds:

    • The official, unmodified GR00T N1.7 LIBERO-goal weights, with the controller receiving no gradients and undergoing no fine-tuning.
    • 320 prompts = 8 tasks × 20 initial placements × 2 instruction word orders.
    • 19,074 real episodes in total across 59 run directories.
    • The reward as the only manipulated variable, with no planner trained on LIBERO images.
    • Before the main runs we reproduced the official single-task score of these weights (98.5% on our platform against 97.5% officially), confirming that neither the environment nor the weights were at fault.
  3. Execution results (§5.4, Table 5):

    Upper-level plannerBoth sub-goals written correctly ↑\uparrowWhole-task success ↑\uparrowSub-goal attainment ↑\uparrow
    A0 (supervised start)0.3780.2940.400
    Complete scoring0.3450.2700.368
    Complete scoring + goal term0.6360.4910.653

    Complete scoring plus the goal term raises whole-task success by 0.197 (95% scene-cluster bootstrap interval [+0.133, +0.259]).

  4. Added two latency arguments so that “usable as an embedded module” now rests on measurement. One verifier judgement takes 0.80 μs (single core of an Intel Xeon Platinum 8470Q), whereas one planner forward pass takes 793 ms—six orders of magnitude apart. On this basis we argue that the judgement can be placed inside every planning call without entering the latency budget (§6).

  5. Added Appendix D (Table 9), reporting execution results for all eight reward components arm by arm, and Appendix I (Table 14), disclosing the composition of all 59 run directories.

  6. Stated the boundary of the claim. Complete scoring used alone falls below the supervised start under both seeds (0.256 and 0.284 against 0.294), so the gain has to be carried by the goal term. We retain this negative result in full in Table 5 and Table 9, because it is exactly the distinction the paper sets out to draw.

AuthorsOpenReview ↗

Q2.The planning and verification are done completely in a open loop setting. This is not realistic. In the real-world the planner will be invoked every few timesteps to replan based on the state of the world at that time.

Response

We agree entirely with the situation the reviewer describes, and this comment sent us back to our own formalisation, where we found a genuine ambiguity: the original draft left the impression that one plan corresponds to an entire episode. That is not the case. The verifier is defined for each planning call, not for an episode. A closed-loop planner is invoked every few steps, each call produces one program, and the verifier scores exactly that program. The formalisation therefore never assumed a single plan per episode.

We also agree, however, that defining the verifier per call is only a clarification and cannot substitute for actually running the system under scheduling. We therefore treated this comment as a request for an experiment rather than a request for a clarification.

Action

  1. Added a latency argument to §6, giving “verify per call” a quantitative basis. One verifier judgement takes 0.80 μs (single core of an Intel Xeon Platinum 8470Q) against 793 ms for one planner forward pass, six orders of magnitude apart. The judgement can therefore be placed inside every planning call while total latency remains set by the planner forward pass. This subsection is entirely new; the original draft had nothing of the kind.

  2. Added a dependency-dispatch experiment (Appendix F, Table 12). The situation the reviewer has in mind is that the system does not replay the submitted order but schedules steps as their preconditions become satisfied. We therefore sample uniformly among the execution orders permitted by the dependencies the program declares—exactly the set a precondition-driven scheduler can reach. The effect does not weaken; it becomes slightly stronger:

    ComparisonOwn orderDependency dispatch
    Complete scoring + goal term −- supervised start+0.197 [+0.133, +0.259]+0.205 [+0.144, +0.264]

    This shows that the dependencies declared by plans trained with our method are sufficient on their own to support legal scheduling, whereas plans trained with the original score degrade markedly under dependency dispatch because their declared dependencies cannot support arbitrary legal scheduling.

  3. Added a controlled order intervention (960 trials). This block involves no planner: tasks and initial placements are fixed and only the dispatch order of the six actions changes.

    Execution order categoryTrialsSuccess rate
    Legal: finish one object (grasp, place, release) before the next960.781 [0.698, 0.865]
    Placed but not released before the next grasp2880.542 [0.469, 0.608]
    Next grasp dispatched while still holding one5760.003 [0.000, 0.009]

    The frozen controller cannot recover from the third category: 574 of the 576 trials block until timeout and only 2 succeed. The intervals of the three categories do not overlap, and the only difference among them is whether the next grasp is dispatched before or after the release—nothing to do with what the plan says.

  4. Added Figure 8 to Appendix F, breaking down all twenty interleavings category by category so that readers can check the table above.

  5. These results in turn revised the design argument of §4.3. Precisely because they show that a verifier which does not price order legality will pass plans that any scheduler would actually turn into failures, we place the execution gate inside the reward rather than using it as a post-hoc filter. The original draft had only the formal argument; the revision has experimental support.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.