- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper studies task-level affordance reasoning, where models must not only identify interaction regions but also correctly connect object roles, actions, states, dependencies, and motions. The authors highlight two failure modes: Binding Blindness, where these elements are connected incorrectly, and Omission Blindness, where missing requirements escape evaluation. They introduce CAP-7, a task-program representation with seven linked fields, along with CAP-7-PD, a dataset of 1,698 programs, and a separate 20k-program extension. The main contribution is commitment-complete verification, which accounts for omitted reference requirements and penalizes unsupported additions. Combined with state and execution-order checks, it provides a training reward. With the same supervised initialization and training budget, replacing the original reward increases average sampled-order legality under an independent symbolic engine from 51.3% to 94.4%, compared with 85.2% for the supervised starting model. However, goal achievement and some relational metrics do not show corresponding improvements.
Critical strengths
The paper identifies an important evaluation gap. Separating coverage from the correctness of expressed relations helps reveal misleading performance gains. The reward comparison is well controlled, using the same supervised initialization and training budget. The paper is well presented, with clear figures and tables.
Critical weaknesses
The main gain is in execution legality, not task completion. Goal achievement does not significantly improve (0.6320 vs. 0.6456), while step-deletion CBA drops significantly (0.654 to 0.589). Further analysis is needed to show whether actual omissions decrease, or whether the model simply produces more conservative plans. Reference completeness is not the same as task completeness. Valid plans may use different steps, tools, or action granularities. The verifier should show that it does not unfairly penalize these alternatives. The novelty over existing planning and verification methods needs clarification. Program representations, precondition checks, and plan validation are already established in systems such as ProgPrompt and VAL. The paper should explain what CAP-7 adds beyond these capabilities.
Questions for authors
The complete reward improves legality, but not goal achievement, while step-deletion CBA decreases. How do you explain this? Could you report key-step recall, omission types, and the joint rate of legal execution and goal achievement? Which reward components drive the gains? Could you compare execution gating alone, coverage and redundancy penalties alone, and the full reward, alongside a simple execution-legality reward baseline? How does the verifier handle valid alternatives to the reference program? Could you test human-verified equivalent plans with different tools, action granularities, and alternative steps, and report how often they are incorrectly penalized?
12 comments
Q1. The complete reward improves legality, but not goal achievement, while step-deletion CBA decreases. How do you explain this?
Action
-
Fixed the denominator by C1–C3 (Eq. (5) of the main text):
which keeps both the missing reference pairs and the unmatched predicted pairs in the denominator, so that neither side collapses under corruption or omission, giving .
-
Extended the check set, folding the goal predicates into (Eq. (4) of the main text):
-
Added the definition and implementation of the goal term (Eq. (9) of the main text). The reward is with . Here is the mean fraction of reference core-goal facts satisfied in the final state, up to an injective renaming of entities, over four random execution orders consistent with the declared dependencies. The core facts are the facts added by the reference plan whose predicates are
stroke_count,on_top_of,contained_in,poured_into, ororientation_deg. An empty core-goal set gives . Programs rejected by the gates retain a reward of . -
Added execution-level validation, so that the results do not stop at the symbolic level (§5.4, Table 5; 19,074 real episodes):
Upper-level planner Both sub-goals written Whole-task success Sub-goal attainment A0 0.378 0.294 0.400 Complete scoring 0.345 0.270 0.368 Complete scoring + goal term 0.636 0.491 0.653 -
Added the CBA rescoring experiment (§5.3, Table 4). With the programs held fixed and only the scoring rule replaced, CBA rises for all six conditions, while the spread among them narrows from 0.065 to 0.020:
Condition CBA original CBA complete Gain A0 0.707 0.828 +0.121 P6 0.772 0.846 +0.074 L0 0.765 0.848 +0.083 RFT-1 0.737 0.835 +0.098 RFT-2 0.741 0.842 +0.101 RFT-new 0.749 0.840 +0.091 The narrowing of the spread is exactly the effect of : the differences among conditions originally came from how far each one fell into the gap, and once that term is removed only remains; since all six conditions face the same , the differences vanish with it. P6 is the clearest example—under the original score it is 0.772, the highest in the table and 0.065 ahead of A0, whereas under complete scoring it leads by only 0.018; and it is precisely the condition with the lowest coverage (0.283) and conditional correctness (0.709). Its original advantage therefore came from the gap, not from binding quality.
-
Added an explanation for why CBA and the order agreement move in opposite directions. Your observation prompted us to state explicitly in the caption of Table 4 that, relative to the supervised start, contrastive accuracy rises for all five reward variants while the order agreement (ORD3) and conditional correctness fall for all of them—exactly the direct evidence for “optimizing along the gap”, and the reason why CBA alone leads to the opposite conclusion.
-
Added the necessity half of the proposition's proof to Appendix B (the original draft carried only the heading “sufficiency”). A new “Step 4: necessity of the conditions” shows explicitly that, in the absence of C1/C2, two plans land on the same atom and no scoring can distinguish them, while the necessity of C3 follows directly from the definition.
Q2.Could you report key-step recall, omission types, and the joint rate of legal execution and goal achievement? Which reward components drive the gains?
Response
Thank you for the suggestion. These three items are indeed things we should have put in the main text and did not—the joint value especially, which was already computed per task among the registered metrics of the supplementary material but never made it into the main text. That was an oversight in our presentation. Your comment led us to promote all three to formal content in the main text or appendix.
Action
-
Recall is now reported in §5.2 of the main text (Table 3). Coverage is the relational counterpart of recall: it measures how many reference step pairs the plan expresses. The three supervision configurations give 0.357 (N-1, flat-sequence objective), 0.352 (B-0, Qwen3-VL-8B backbone), and 0.362 (A0, program-level supervision), all within the narrow band from 0.352 to 0.362—coverage is determined by the supervision data itself, and changing the supervision objective or the backbone barely moves it. The difference appears in accuracy at the same coverage: conditional correctness 0.869 / 0.893 / 0.898 and replay legality 0.186 / 0.874 / 0.890.
-
Named the B-0 backbone (the original draft said only “the second backbone”). The revision states Qwen3-VL-8B-Instruct in three places—main text, table caption, and table row—so that readers can judge what scale “transferable” actually transferred to.
-
Omission types are now priced separately by relation type (§4.2, Eq. (6)). In the denominator , retains every reference step pair whether expressed or not, counts the pairs between unmatched predicted steps and reference steps, and counts the pairs among unmatched steps themselves. The two kinds of omission therefore enter the denominator separately: a whole sub-chain that never appears lowers coverage, while a step that appears but is bound to the wrong object lowers conditional correctness. Figure 4 of the revision shows these two forms qualitatively.
-
The joint value is now in the main text and appendix. 360 test tasks, two seeds:
Condition Legal-execution rate Goal-attainment rate Joint (legal and achieved) A0 (supervised start) 0.841 / 0.849 0.608 / 0.606 0.549 / 0.543 Complete scoring 0.933 / 0.928 0.622 / 0.628 0.581 / 0.575 Complete scoring + goal term 0.930 / 0.933 0.644 / 0.653 0.594 / 0.597 Paired differences (both seeds pooled, 95% intervals): goal term complete scoring on the joint value is ; goal term A0 is , an interval that excludes zero. The joint value improves less than either component alone, which is what we expect: the joint requires both things to hold at once, failure in either one is not counted, and it is therefore sensitive to the shortcoming of each factor.
-
The contribution of each reward component is now broken out item by item:
Modification Quantity it raises Reading Fix the denominator, count unmatched steps Legal-execution rate 0.841 → 0.933 (seed 42), 0.849 → 0.928 (seed 123) Extend the check set, include goal predicates Goal-attainment rate, joint value goal 0.622 → 0.644, joint 0.581 → 0.594 (seed 42) The two together Whole-task success 0.294 → 0.491 (frozen controller, 19,074 executions) -
Added Appendix D (Table 9), training each of the eight reward components in isolation and reporting execution results arm by arm for both seeds, so that “which component contributed what” can be read off directly instead of inferred backwards from the combined result.
-
Added Appendix I (Table 14), disclosing the itemised composition of all 59 run directories and 19,074 executions, so that every reading above is traceable.
Q1. The complete reward improves legality, but not goal achievement, while step-deletion CBA decreases. How do you explain this?
Thank you for taking the time to review our work.
Response
Thank you for raising this—it is the most valuable comment we received on whether the mechanism is clearly explained. Neither of the two phenomena you observed can be explained from the text of the original draft, and on rereading we believe the problem is that we discussed errors from two different sources together, so that readers could see the phenomena but not their causes. By putting the two side by side you led us to make the decomposition explicit—which is why §4.1 through §4.4 were rewritten for the revision.
The two phenomena you mention correspond to two distinct design defects:
Cause 1: the goal-attainment rate does not move, because the goal is not in the check set.
The seven check heads of the original draft cover objects, bindings, step matching, dependency order, state, trajectory, and geometry—all of them measure “is the plan written correctly”; “are the two sub-goals achieved” is not among them. A reward trained on this can therefore only push the “written correctly” side, and legality rises accordingly; the goal-attainment rate lies outside what the checks can see, and so cannot be changed by this reward. In the notation of Eq. (3) of the revision, this part of the error is εirr(Z): it is determined by the checks Z and is independent of the scoring R:
∥U−R∥2=εirr(Z)∥U−R⋆∥2+εred∥R⋆−R∥2.
As long as no goal is in Z, no matter how much better R becomes, εirr(Z) stays exactly where it is.
Cause 2: deletion-type CBA is distorted, because the denominator normalises along with whatever is expressed.
The original draft scored each head separately and took a weighted average (S=2M−1), with each head using “what has been expressed” as its denominator. When deletion-type corruption shrinks the reference obligation set to Uγ⋆⊂U⋆, there are fewer pairs to cover, so the denominator shrinks with it: numerator and denominator fall together and the score does not drop. The outcome of this family of contrasts is thus decided mostly by how far the denominator collapses, rather than by whether the prediction actually distinguishes a complete reference from a truncated one. In the notation of Eq. (3), this part is εred>0.