acceptodds
Withdrawn from ICLR 2027

TriRelVLA: Triadic Relational Structure for Generalizable Embodied Manipulation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Vision-language-action (VLA) models perform well on training-seen robotic tasks but struggle to generalize to unseen scenes and objects. A key limitation is their implicit visual representations, which entangle object appearance, background, and scene layout, making policies sensitive to visual variations. Prior work improves transferability through structured intermediate representations, but these mainly capture scene semantics rather than action-relevant relations. We argue that manipulation actions are fundamentally governed by the object–hand–task relational structure, which defines interactions among task requirements, robot states, and object properties. Based on this observation, we propose TriRelVLA, a triadic relational VLA framework for generalizable embodied manipulation. Our approach consists of three components: 1) We construct explicit object–hand–task triadic representations from multimodal inputs as relational primitives. 2) We build a task-grounded relational graph. Task-guided cross-attention forms nodes, and a relation-aware graph transformer models interactions among them. 3) We perform relation-conditioned action generation. The relational structure is compressed into a bottleneck space and projected into the LLM for action prediction. This triadic relational bottleneck reduces reliance on appearance statistics and enables transfer across scenes, objects, and task compositions. We further introduce a real-world robotic dataset for fine-tuning. Experiments show strong performance on fine-tuned tasks and clear gains in cross-scene, cross-object, and cross-task generalization. Our code will be released upon acceptance.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Embodied AI, Robotic Manipulation, VLA, Structured Representation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_WcF7OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

TriRelVLA replaces dense visual tokens in a VLA with an explicit object-hand-task relational representation: object tokens are grounded from fused SigLIP/VGGT features, a hand token is anchored on proprioception, and the instruction is decomposed into action, role, constraint and stage tokens. A task-guided graph with typed edges (task-object, task-hand, object-hand, object-object) is updated by a relation-aware graph transformer, and a bottleneck projects it into a Qwen3-4B LLM for action prediction. Training is three-stage (OXE, DROID, LIBERO + CSOT-Bench) with object/hand mask auxiliary losses. The authors also introduce CSOT-Bench, a real-robot benchmark with cross-scene, cross-object and cross-task suites. Against eight VLAs ranging from OpenVLA to SemanticVLA, the method is at parity on fine-tuned LIBERO/CSOT (97.6/84.9) and reports large gains on the CSOT generalization suites (e.g., cross-object 80 vs. 47).

Critical strengths
  1. The triadic object-hand-task representation is a well-motivated architectural idea. It moves beyond scene-level structured representations to action-centric relations, and the implementation - task-guided nodes, a relation-aware graph transformer, and a relational bottleneck - is coherent and clearly specified. The ablation trends (triadic representation, graph, proprioception, task grounding) all point in the same direction, although, as noted below, their magnitudes are not yet reconcilable. Fine-tuned performance is competitive with strong VLA baselines (97.6 LIBERO, 84.9 CSOT). This is consequential because it offers a concrete alternative to dense visual tokens for generalization.
  2. The paper reports large cross-scene/object/task gains on a real-world benchmark. On CSOT-Bench, TriRelVLA improves over SemanticVLA by 21, 33, and 25 points on the first task of each suite (78 vs. 57 cross-scene, 80 vs. 47 cross-object, 71 vs. 46 cross-task), while remaining comparable on fine-tuned tasks. If these gains hold under a controlled evaluation, they would be significant for real-world VLA deployment.
Critical weaknesses
  1. The ablation is not interpretable as presented. The "Generalized task" column in Tables 2-6 is never defined, and 79.5 for the full model cannot be reconciled with Fig. 7 (79/81/73 cross-scene/object/task), so it is unclear what the column aggregates. The jump from 43.0 to 79.5 when adding the relational graph (+36.5) is unexplained, while the gain over the best baseline is only +0.7; no per-suite results, parameter counts, or training controls are given, so the graph may be confounded with capacity or training. The grid is also incomplete: Table 3 has no pairwise combinations (object+hand, object+task, hand+task), Table 4 does not isolate L_obj and L_hand, Table 6 omits the cell with object grounding off and hand grounding on, and no table reports error bars, seeds, or the number of rollouts. These gaps are consequential because the ablation is the main evidence for the architectural claims.
  2. The comparison is selective and not controlled for model size, training data, or compute. Table 1 compares eight baselines, but Figures 6-7 compare only four (OpenVLA, Octo, CogACT, SemanticVLA), omitting SpatialVLA (CSOT 80.8) and CogVLA (82.5), the strongest structured baselines in Table 1, while keeping Octo (72.0). The paper reports no parameter counts, pretraining data, or compute for the baselines, though TriRelVLA uses a 4B Qwen3 LLM with SigLIP/VGGT and a three-stage OXE/DROID pipeline, whereas Octo is ~0.1B and DiffusionPolicy is trained from scratch. The large gains are on CSOT-Bench, a self-constructed benchmark that is also used for Stage-3 training and that is described without task, demonstration, object, or split counts, and for which only code release is promised. This is consequential because the paper does not isolate the claim that the triadic structure, rather than extra data or capacity, drives generalization.
Questions for authors
  1. The relational graph adds +36.5 generalization (43.0→79.5) but only +0.7 fine-tuned (84.2→84.9). What explains this asymmetry? Please report a controlled ablation with the same data, training stages, parameter count, and compute. The basic architecture’s generalized score (32.1) is close to OpenVLA’s generalized average (~34.5 in Figure 7), so the surprising result is the full model’s 79.5, which is far above every baseline (SemanticVLA ~49.7). What causes this jump?
  2. The comparison lacks several recent related works, such as GraphCoT-VLA (Huang et al., 2026) and OpenVLA-OFT (Kim et al., 2025). Can you compare against them, or explain why they are not applicable?
Reviewer_cz26OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes TriRelVLA, which models relationships among objects, the robot hand, and task requirements to improve VLA performance on unseen scenes, objects, and tasks. The method extracts three types of representations from images, language instructions, and robot states. It then uses task-guided attention and a relational graph to model their interactions, before compressing the features and feeding them into a language model for action prediction.

The authors evaluate the method on LIBERO and their own real-world benchmark, CSOT-Bench. The method performs similarly to strong baselines on fine-tuned tasks and shows improvements in cross-scene, cross-object, and cross-task tests.

Critical strengths

Modeling objects, the robot hand, and task requirements together makes sense for robotic manipulation. Using proprioception to extract hand features and task information to refine object representations allows action prediction to account for both the robot’s current state and the goal.

The paper includes both simulation and real-world experiments, with strong results in generalization tests. In Table 2, adding the relational graph increases the generalization success rate from 43.0% to 79.5%. The ablations also show the benefits of proprioception and auxiliary supervision. In addition, the relational bottleneck reduces computation and memory usage while largely maintaining the success rate.

Critical weaknesses

Object-centric representations, task-guided attention, and graph-based relational modeling have all been explored in prior work. The main differences between this method and those approaches are not clear enough. The relational graph also introduces information such as relative positions, poses, distances, and contact or reachability scores, so the large gain in Table 2 cannot be attributed directly to relational reasoning. The paper does not test a model that uses the same geometric information without the relational graph, making it hard to tell how much each contributes.

The generalization claims rely heavily on CSOT-Bench, but the dataset size, train/test splits, number of evaluation trials, and variation in results are not clearly reported. In particular, the cross-task tests do not clearly distinguish new combinations of known skills from entirely unseen manipulation skills. TriRelVLA also uses multiple pretrained encoders, object and hand mask supervision, and multi-stage training, but it is unclear whether the baselines use comparable data, supervision, and training budgets. These differences affect how the results should be interpreted.

Questions for authors

Q1. In Table 2, adding the relational graph increases the generalization success rate from 43.0% to 79.5%. What happens if the graph is replaced with a standard Transformer while keeping the same geometric information? This comparison would help separate the effects of geometric information and relational modeling.

Q2. How are the relative positions, poses, and contact or reachability scores in the graph computed? Which come from visual predictions, and which come from robot states, additional annotations, or other modules?

Q3. Please provide the train/test splits for CSOT-Bench, explain what counts as an “unseen task” in the cross-task tests, and report the number of evaluation trials and variation in results. Please also clarify whether the main baselines use comparable training data, auxiliary supervision, and training budgets.

Reviewer_vCB5OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes TriRelVLA, a VLA framework that learns an explicit object–hand–task triadic relational representation for generalizable robotic manipulation. The framework has three stages: (1) construct object, hand, and task tokens from multimodal inputs as relational primitives; (2) build a task-grounded relational graph with task-guided cross-attention and a relation-aware graph transformer; (3) compress the relational structure into a bottleneck and project it into the LLM for relation-conditioned action generation. The authors also introduce CSOT-Bench, a self-constructed real-world manipulation dataset with scene/object/task generalization suites. Experiments compare TriRelVLA with implicit and structured VLA baselines on LIBERO and CSOT-Bench, reporting fine-tuning and cross-scene/object/task generalization results, plus ablations on the triadic architecture, representation components, losses, proprioception, task grounding, and bottleneck.

Critical strengths
  1. The triadic relational representation is well motivated and conceptually clean. The paper argues that action-relevant visual cues center on the target object and the robot hand, while language specifies task goals and constraints, and that modeling their relations should reduce reliance on appearance statistics. This is a reasonable and interpretable inductive bias, and the graph-transformer + bottleneck design is a sensible way to inject it into a VLA.

  2. The paper attempts a systematic generalization evaluation. Reporting cross-scene, cross-object, and cross-task generalization on both a simulation benchmark and a self-built real-world dataset is the right direction, and the ablation tables cover several design choices.

Critical weaknesses

1. The generalization comparison is likely unfair because baseline training protocols are unclear.
CSOT-Bench is the authors' own dataset, and TriRelVLA is fine-tuned on it in Stage 3. It is not stated whether OpenVLA, Octo, CogACT, DiffusionPolicy, SpatialVLA, CogVLA, and SemanticVLA are also fine-tuned on the same CSOT-Bench training split before the generalization evaluation. If TriRelVLA is trained on CSOT-Bench while baselines are only evaluated with their pretrained weights, the large cross-scene/object/task gaps in Fig. 7 are not evidence of better generalization. This is the single most consequential concern.

2. Numbers are inconsistent across Fig. 1(d), Table 1, Fig. 6, and Fig. 7.
Fig. 1(d) shows fine-tuning bars around 83/85/85 and cross-scene/object/task bars of 41/55/79, 32/48/81, and 27/45/73. Table 1 reports LIBERO averages of 76.5 (OpenVLA), 78.1 (CogACT), 97.7 (SemanticVLA), and 97.6 (TriRelVLA), which do not match the 83/85/85 impression. Fig. 7 shows SemanticVLA at roughly 57/55/47/50/46/43 across six tasks, which averages near 50 rather than 55. These discrepancies need to be reconciled; as written, the figures appear selectively summarized.

3. Ablations have confounds and unclear settings.
Table 2 reports only single numbers for "fine-tuned task" and "generalized task" without specifying which generalization dimension is averaged, how many trials, or whether the "basic arch." baseline shares the same backbone and training budget. Table 3 removes one token type at a time, but it is unclear whether the graph structure, edge set, and node count change accordingly. Table 5's "w/o Proprio" variant does not state what replaces the proprio-anchored hand query, so the 7.9 pp gap may reflect a much weaker hand-localization mechanism rather than the value of proprioception per se. Table 7's "w/o Bottleneck" reports 79.3 vs. 79.5, but the column label says "Succ" while Table 2's generalized task is 79.5, so it is unclear whether this is fine-tuned or generalized success.

4. The comparison set omits strong modern VLA baselines.
The baselines are OpenVLA, Octo, CogACT, DiffusionPolicy, SpatialVLA, CoA-VLA, CogVLA, and SemanticVLA. Flow-matching VLAs such as π0\pi_0 and π0.5\pi_{0.5}, which are now standard strong baselines, are not compared. The LLM backbone (Qwen3-4B) also differs from several baselines, and parameter counts/inference cost are not reported, so the comparison is not controlled for capacity.

5. The "structured representation" framing and citation support are partly inaccurate.
The paper cites Liu et al. (2024a) (MOKA, mark-based visual prompting) as constructing intermediate representations, which is not quite the same category. More importantly, the paper claims prior structured methods "mainly capture scene semantics rather than action-relevant relations," but SemanticVLA and CogVLA are strong structured baselines that already perform comparably on fine-tuned tasks. The paper needs a sharper argument for why the triadic relation is the missing ingredient, rather than a small margin over existing structured VLAs.

Questions for authors
  1. Are all baselines in Table 1 and Fig. 7 fine-tuned on the same CSOT-Bench training split as TriRelVLA? If not, please report both pretrained-only and fine-tuned baseline results, and clarify which comparison supports the generalization claim.

  2. Please provide CSOT-Bench statistics: number of trajectories, tasks, object categories, scene variants, viewpoints, demonstrations per task, and the train/validation/generalization split protocol. Also report the DROID and LIBERO mask annotation method and quality.

  3. Please reconcile the numbers in Fig. 1(d), Table 1, Fig. 6, and Fig. 7. In particular, how are the summary bars in Fig. 1(d) computed from the per-task results, and why do they differ from Table 1?

  4. On LIBERO, TriRelVLA (97.6) is essentially tied with SemanticVLA (97.7). What is the evidence for "state-of-the-art" performance? Please report confidence intervals and trial counts for the main comparisons.

  5. How are object queries assigned to multiple objects, and how is LobjL_{obj} defined in multi-object scenes? How are Δp\Delta p, ΔR\Delta R, dd, and ss computed from the visual latent?

  6. What supervision ensures that the four task tokens correspond to action, role, constraint, and stage? If there is none, please soften the "structured task primitives" claim.

  7. In Table 5, what replaces the proprio-anchored hand query in the "w/o Proprio" variant? In Table 7, is the reported 79.3/79.5 fine-tuned or generalized success?

  8. Please add π0\pi_0/π0.5\pi_{0.5} or another flow-matching VLA baseline, and report parameter count and inference cost for all methods.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.