acceptodds
Withdrawn from ICLR 2027

TAGER: Temporal Action Geometry for Robust Robot Manipulation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Vision-language-action (VLA) models inherit rich visual-semantic knowledge from pretrained vision-language models (VLMs), yet remain vulnerable to distribution shifts in visual observations, language instructions, and manipulation conditions. A key limitation is that pretrained VLMs lack explicit geometric awareness and spatial relationships required for manipulation. Recent methods mitigate this limitation by aligning VLA representations with geometric teacher features. However, directly injecting such supervision into shared visual representations may interfere with the visual-semantic priors inherited from pretraining. Moreover, static spatial representations fail to capture the action-conditioned evolution of visual observations and robot states, thereby restricting the policy's capacity to adapt subsequent actions under perturbations. To address these challenges, we introduce TAGER, a Temporal Action Geometry framework for robust Embodied Robot manipulation that couples Sparse Spatial Query Distillation with an Action Response Residual Flow. To enhance spatial awareness while while mitigating interference with pretrained visual-semantic priors, sparse spatial query distillation uses dual independent query transformers to distill dense geometric and semantic knowledge from frozen VGGT and SAM3 teachers into compact tokens. These tokens provide complementary world-grounded context to the action expert without directly imposing teacher-feature matching constraints on shared VLA representations. To support adaptive action generation under perturbations, the action response residual flow relates recent executed actions to subsequent changes in robot states and spatial tokens representations, providing evidence about the current visual-motor relationship. The resulting execution context drives a residual flow that adaptively refines subsequent action generation. Experiments on LIBERO and VLABench, together with zero-shot evaluations on LIBERO-Plus and LIBERO-PRO, demonstrate that TAGER achieves outstanding manipulation performance while generalizing robustly across diverse perturbations.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Vision-Language-Action Models(VLA), Spatial Representation Learning, Temporal Action-Response Modeling

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_aV3qOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

TAGER augments a π₀.₅ vision-language-action policy with two components. Sparse Spatial Query Distillation uses independently parameterized query transformers to reconstruct features from frozen SAM3 and VGGT teachers, providing compact semantic and geometric context to the action expert. Action Response Residual Flow encodes recent executed actions, proprioceptive changes, and SAM3-query differences to condition a gated residual correction to the action-generating flow.

The paper evaluates TAGER on LIBERO and VLABench, with zero-shot transfer to LIBERO-Plus and LIBERO-PRO. The results suggest useful improvements under distribution shifts, particularly robot-initial-state perturbations. However, the current experiments do not sufficiently isolate the proposed mechanisms from additional capacity, temporal information, and differences in state conditioning.

Critical strengths
  1. The robustness improvements are practically relevant and extend across multiple benchmarks. On LIBERO-Plus, TAGER achieves 87.6% success, compared with 84.6% for the official π₀.₅ baseline and 82.2% for the retrained baseline. The improvement under robot-initial-state perturbations is particularly substantial: 75.5% to 86.6% relative to the official baseline. Improvements also appear on LIBERO-PRO and VLABench, providing evidence beyond a single evaluation setting.

  2. The architecture introduces spatial supervision with a modest reported inference overhead. Teacher networks and reconstruction decoders are removed at inference. Appendix I reports approximately 6.4% additional parameters and 7% additional post-initialization latency. The cumulative ablation in Figure 3 also supports the practical usefulness of both query distillation and the temporal component, although further controls are needed to establish their specific contributions.

Critical weaknesses

W1. The ablations do not sufficiently isolate the mechanisms behind the improvements.

Spatial branches. Figure 3 adds query capacity and teacher supervision together. Without the same architecture trained without distillation, the benefit of teacher knowledge remains confounded with approximately 214M additional parameters.

Preservation of pretrained priors. Appendices A and D.1 acknowledge that reconstruction gradients reach the VLM. This does not invalidate the design, but language-perturbation results and spatial probes do not directly demonstrate preservation of pretrained semantic capabilities.

Temporal component. Figure 3 shows that the module helps, but does not distinguish among:

  • Generic temporal context.

  • Actions or proprioceptive changes alone.

  • Correctly paired actions and visual responses.

  • Additional capacity or residual-head parameterization.

Table 6 partially addresses history length. Further controls should remove actions or query differences, disrupt their pairing, and compare against a conventional history encoder. Since proprioception enters through the history encoder, its availability across variants also needs clarification.

W2. Smaller gains are difficult to assess, and the temporal configuration is inconsistent.

Result reliability. TAGER improves over the strongest listed LIBERO method by 0.3 percentage points and retrained π₀.₅ on LIBERO-PRO by 1.9 points. Without reported variability and clearer baseline comparability, their stability is uncertain. The 2.4-point gap between official and retrained π₀.₅ on LIBERO-Plus also warrants explanation.

Configuration inconsistencies: History length: Table 5 specifies three transitions, requiring four observations; Table 6 reports the main 87.6% result with three observations. One-frame variant: This setting has no historical transitions but achieves 86.1%. Its inputs and difference from the 85.8% query-only model are unclear. Camera views: Appendix E reports wrist-only performance as 85.6%, versus 86.5% in Table 6. Its claim that combining views provides no gain also contradicts the table’s 87.6% versus 86.5%.

The first two inconsistencies affect the definition and configuration of the core temporal module.

Questions for authors
  1. Which history configuration produces the main results? Please reconcile the three transitions in Table 5 with the three observations in Table 6, and explain how historical inputs are constructed for the one-frame variant and at the beginning of a rollout.

  2. Do the proposed mechanisms remain beneficial under matched capacity and input information? Particularly informative controls would include the same query branches without teacher supervision, temporal variants without actions or query differences, disrupted action–response pairing, and a conventional history encoder receiving the same inputs. Please also clarify the proprioceptive information available to each variant.

  3. What explains the gap between official and retrained π₀.₅ on LIBERO-Plus? Are the reported comparisons based on comparable training and evaluation settings?

  4. Is there direct evidence of reduced interference with pretrained visual-semantic capabilities? A controlled comparison with direct feature alignment would strengthen this interpretation. If such evidence is unavailable, the claim should be qualified as an explanation consistent with the behavioral results.

Reviewer_5RkAOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents TAGER, a VLA framework designed to improve a policy’s ability to adapt subsequent actions under perturbations. The method combines two components: Sparse Spatial Query Distillation compresses geometric and semantic features from frozen VGGT and SAM3 teachers into independent query branches that provide additional context to the action expert; Action Response Residual Flow encodes recent executed actions, proprioceptive changes, and query-state differences to condition a gated residual correction. Heavy teacher networks and reconstruction decoders are removed during deployment for faster inference. Experiments cover LIBERO, VLABench, LIBERO-Plus, and LIBERO-PRO, with additional qualitative demonstrations of real-world feasibility.

Critical strengths
  1. The motivation is clear and relevant.
  2. The model design is closely aligned with the goal of incorporating spatial supervision and execution history. Removing teachers and reconstruction decoders at inference also improves deployment practicality. The method achieves high success rates on the reported simulation benchmarks.
Critical weaknesses
  1. Limited novelty. Spatial priors, query-based distillation, separate branches, and temporal conditioning for action generation have been explored in many recent papers, which is not a new topic. Therefore, the proposed design appears incremental, and its new insights require clearer articulation. In particular, the actual contribution should be substantiated by evidence that "the design mitigates interference with pretrained visual-semantic priors while providing useful spatial context for action learning"; however, this claim is not sufficiently supported by existing analyses and comparisons, and high benchmark success rates alone are insufficient to establish it.
  2. Insufficient ablations of the proposed design. The paper introduces several components, but the existing ablations primarily demonstrate performance gains from adding both query distillation and residual flow. They do not sufficiently establish whether these gains depend on the specific mechanisms proposed or how sensitive performance is to the associated hyperparameters. More systematic ablations and sensitivity analyses are needed.
Questions for authors
  1. A more direct, quantitative evaluation of the retention of pretrained visual-semantic capabilities in VLMs should be provided. For example, how does performance on general multimodal VQA benchmarks change before and after SFT of the VLM?
  2. How does Action Response Residual Flow compare with a simpler policy that incorporates the same action, proprioceptive, and visual history through standard conditioning, as evaluated on the simulation benchmarks reported in this paper?
  3. Given the limited real-world evaluation, the authors should provide quantitative comparisons among TAGER, the query-only variant, and more standard baselines, including success rates and an analysis of failure modes. These results would help clarify the method’s practical value for real-world deployment.
Reviewer_1mzSOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes TAGER to improve the robustness of VLA models under distribution shifts. Built on π0.5, the method introduces two components. The first uses separate query transformers to distill spatial and semantic features from VGGT and SAM3, instead of directly aligning the VLM features with the teacher models. The second uses recent actions, robot state changes, and visual feature changes to adjust subsequent action predictions through a residual flow. The authors evaluate TAGER on LIBERO, VLABench, LIBERO-Plus, and LIBERO-PRO, reporting improvements over the π0.5 baseline, particularly under robot-state and visual-noise perturbations.

Critical strengths
  • S1. Strong performance across multiple benchmarks and distribution shifts. The paper evaluates TAGER on LIBERO, VLABench, LIBERO-Plus, and LIBERO-PRO, showing consistent improvements in overall success rates over the π0.5 baseline. The gains under robot-state and visual-noise perturbations are particularly notable. The ablation studies also provide evidence that both spatial query distillation and temporal action feedback contribute to the improvements.
  • S2. Effective integration of spatial knowledge and temporal information. The method combines spatial feature distillation from VGGT and SAM3 with historical action-response information. The separate query branches provide additional spatial representations for action generation, while the residual flow incorporates recent execution feedback. This design offers a practical way to improve VLA robustness without requiring additional fine-tuning on the evaluated perturbations.
Critical weaknesses
  • W1. Lack of direct evidence for preserving pretrained VLM representations. The authors argue that separate query transformers can reduce interference with pretrained VLM features. However, the reconstruction loss still backpropagates through the VLM backbone, as mentioned in Appendix A. The current results show better robustness, but do not directly verify whether the original visual-semantic representations are preserved.
  • W2. Limited comparison for the temporal module. The ablation results show that Action Response Residual Flow improves performance. However, it is unclear whether the improvement comes from the residual flow design or simply from adding historical information. A comparison with a standard history-conditioned action expert would help clarify the contribution of this module.
Questions for authors
  • Q1. Since the reconstruction loss still backpropagates through the VLM backbone, how do the authors verify that the pretrained visual-semantic representations are better preserved? Could the authors provide a direct comparison of feature changes before and after fine-tuning?
  • Q2. Have the authors compared Action Response Residual Flow with a simpler history-conditioned action expert? This would help clarify whether the improvement comes from the residual flow design or from using historical information.
  • Q3. Real-world experiments only show qualitative results. Could the authors provide success rates under different perturbations and compare them with the π0.5 baseline?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.