acceptodds
Under review as a conference paper at ICLR 2027

A2-CAP: A Causal Attribute–Affordance Graph Harness for Frozen VLA Manipulation

Abstract

Long-horizon robotic manipulation requires reasoning about which object properties enable skills and whether their intended effects occur. We introduce **A2-CAP**, an inference-time harness around a frozen vision-language-action (VLA) policy. Its registry of 12 mechanism-level strategies, plus an LLM fallback, defines a causal attribute–affordance graph whose directed edges encode preconditions and effects. **PiCausal** grounds video evidence in this graph; **CAP-Plan** selects skills, verifies their outcomes, and handles failures without updating the policy. We introduce **A2-CausalHOI**, 1,305 hand–object interaction videos spanning 85 object categories and 256 tasks, with registry-derived target graphs. On a 100-clip graph-grounding sample, PiCausal raises edge F1 from to over a matched VLM-only baseline. On 50 scored EPIC-KITCHENS-100 segments, gains are to across three backbones; on 14 segments annotated before viewing model drafts, they are to . With pre-generated graphs on two dual-arm tasks, the frozen policy under the harness scores task SR over eight T1 rounds versus over ten rounds for each of two baselines, under a joint execution-and-planning rubric with partial credit. On T2, which reverses the arm roles, it scores task SR over 11 rounds without a baseline comparison.

Primary area:
applications to robotics, autonomy, planning
Keywords:
robotic manipulation, long-horizon task planning, object affordances, video-to-graph grounding, benchmark and dataset, frozen VLA policies

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_fiBdOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes A2-CAP, a causal attribute–affordance graph framework for long-horizon manipulation with frozen VLA policies. PiCausal constructs structured graphs using predefined strategies, while CAP-Plan uses these graphs for skill planning, outcome verification, and failure recovery. The authors introduce A2-CausalHOI, containing 1,305 interaction videos, and evaluate graph recovery across three datasets. Real-robot experiments on two dual-arm tasks demonstrate improvements over the reported baselines.

Critical strengths

The attribute–affordance graph provides an interpretable way to connect object properties, executable skills, and expected outcomes without updating the VLA policy during execution. The new dataset and evaluations across three VLM backbones demonstrate consistent improvements in graph recovery. The additional experiments on independent annotations help assess the robustness of these results.

Critical weaknesses

The ground-truth graphs are generated using the same strategy templates employed by PiCausal, while the VLM-only baseline does not have access to these templates. Although the authors provide additional annotation controls, the large F1 improvements may partly reflect alignment with the predefined graph representation rather than general causal understanding. All scored robot experiments use pre-generated graphs, and no valid trial exercises graph-guided rollback. Furthermore, the task success metric includes graph-dependent planning penalties, making it difficult to isolate improvements in physical execution. The small number of trials also limits the strength of the conclusions.

Questions for authors

Appendix E.6 indicates that strategy activation in the main evaluation relies on the object database rather than fused visual observations. How does PiCausal adapt to changes in an object's observed state when its category remains the same? Could the authors report physical task success separately from the graph-dependent planning score? This would help clarify whether the improvements reflect better physical execution or agreement with the predefined graph. Could the authors provide a controlled evaluation of the complete online graph-to-action pipeline, including a failure requiring graph-guided rollback rather than a local retry?

Reviewer_69rrOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper presents A2-CAP, an attribute-affordance graph interface for manipulation with a frozen VLA policy. Declared mechanism templates connect object properties to skill preconditions and effects. PiCausal grounds video interactions into this representation, while CAP-Plan uses the graph to order skills, verify outcomes, and select recovery actions. The authors also introduce A2-CausalHOI, with 1,305 videos covering 85 object categories and 256 tasks. The contribution is a shared symbolic interface for perception, planning, and execution monitoring, supported by graph-grounding studies and robot demonstrations of graph-conditioned execution.

Critical strengths
  1. The paper addresses a useful systems question: how to organize existing manipulation skills so that their prerequisites, intended effects, and failures can be inspected explicitly. The attribute-affordance graph gives perception, scheduling, and verification a common vocabulary. Its value is the connection between an object's physical state, an executable skill, and the state that should result. This provides a concrete design for composing frozen policies and localizing failures without changing policy weights, which is relevant to researchers building reusable manipulation systems.

  2. The combination of a mechanism vocabulary, a video-grounding module, and a corresponding dataset provides a useful research resource. A2-CausalHOI makes object attributes and action effects explicit in real hand-object interactions, allowing subsequent work to study grounding and execution against a common representation. The broader insight is that a structured interface can make pretrained model outputs operational: the same graph specifies what to execute, what to check, and which dependencies matter after a failure. This is an informative system-level contribution even though its individual ingredients have precedents.

Critical weaknesses
  1. The interface is useful, but the account of its added contribution beyond existing precondition-effect skill planning is not yet sufficiently clear. As the authors explain, the causal edges express declared mechanisms rather than mechanisms discovered from video. Under the main object-lab contract, the fused visual attributes do not reach the mechanism templates (Appendix E.6), so the distinction between dynamic attribute reasoning and category-prior-based template instantiation matters. The paper's strongest contribution is therefore an operational interface, but its account does not yet clearly establish what additional state-sensitive reasoning that interface enables. This limits the new methodological knowledge supplied by the work, despite its practical organization of existing components.

  2. The proposed topology-based recovery principle has a substantive semantic gap. Out-degree greater than one establishes that several actions share an attribute, but does not establish that the attribute remains physically valid after failure. A tool can be dropped even when multiple later actions depend on it being held. Treating such a node as stable and stopping the backward search can therefore select a local retry without restoring the missing prerequisite. Since graph-guided recovery is a central contribution, the method needs a connection between dependency structure and verified state persistence before this rule can serve as a general recovery principle.

Questions for authors
  1. What state-sensitive decisions does the attribute-affordance interface enable beyond an ordinary skill planner with preconditions and effects? Please clarify which decisions depend on observed dynamic attributes, given the object-lab template contract in Appendix E.6.

  2. What is the intended procedure for adding a new mechanism or accepting an LLM-generated fallback? Which consistency checks ensure that new preconditions and effects remain meaningful to CAP-Plan?

  3. Under what assumptions can shared attributes be treated as stable during recovery? If execution invalidates a shared state, such as a tool being held, how should state verification interact with the topology-based retry or rollback decision?

Reviewer_URTFOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper adds a symbolic planning and monitoring layer around a frozen VLA robot policy, using an attribute–affordance graph to sequence skills, verify outcomes, and guide failure recovery.

Critical strengths

The paper addresses an important problem for real-world robot deployment. The symbolic graph makes the intended action sequence, verification targets, and failure responses explicit and inspectable, providing a useful interface for understanding and debugging policy execution.

Critical weaknesses
  1. Several framing choices suggest stronger contributions than are demonstrated. Although "harness" can describe an external execution layer, the paper does not clearly establish what this framing adds beyond a symbolic task executive with monitoring and recovery rules. Similarly, the "causal" graph consists of manually declared precondition-effect relations, and the paper does not perform causal discovery or experimentally validate these relations as causal mechanisms.
  2. The experimental comparisons do not convincingly establish the method’s advantage. Complete T1 task success is too low for both the method and baseline, leaving substantial uncertainty about whether the difference reflects a reliable improvement (it could be just noise). Moreover, the joint execution-and-planning rubric penalizes deviations from the prescribed graph even when actions succeed. The graph-recovery comparison also removes the registry and associated knowledge from the baseline, making it difficult to separate methodological gains from the benefit of supplying additional structure. Also, the paper merely uses 12 predefined interaction strategies.
  3. The central integrated claims remain untested. The scored robot experiments use pre-generated graphs, and hence, there is no evaluated end-to-end pipeline from visual grounding to action execution. No valid robot trial exercises graph-guided rollback, and the second task has no baseline comparison. Consequently, the evidence primarily supports execution monitoring from supplied graphs, while the full architecture and its distinctive recovery mechanism remain insufficiently validated.
Questions for authors

See above weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.