acceptodds
Under review as a conference paper at ICLR 2027

Paired Residual Adaptation for Vision-Language-Action Policies under Transient Visual Mislocalization

Abstract

A brief error in a target's apparent position can redirect a robot's actions and cause a placement to fail, even after the visual error disappears. We study how to reduce these failures through action-level adaptation of a frozen vision-language-action (VLA) policy. Our approach learns a lightweight residual action policy from paired predictions of the base model. Clean and visually perturbed observations of the same physical state are evaluated with matched sampling noise, and their action difference provides the correction target. Balanced zero-residual supervision discourages unnecessary changes to valid behavior. At deployment, the residual policy uses current images, robot state, and the proposed action chunk to adjust translational actions, without a clean reference or additional base-model queries. We evaluate the method with a frozen π0.5 policy on LIBERO-Spatial, using brief rendering-only target displacements during placement. A fixed residual checkpoint improves task success on held-out initial states. Small development comparisons under matched training budgets also favor the paired residual design over direct action prediction, mismatched references, and unbalanced supervision. These results support targeted action adaptation for transient visual errors, although some clean-condition performance is lost and broader generalization remains to be established.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Vision-Language-Action Models, Residual Policy Learning, Paired Action Supervision, Transient Visual Mislocalization, Robotic Manipulation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_V8RZOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary
  • This paper proposes Paired Residual Adaptation, which is a lightweight method for improving the robustness of VLAs against visual mislocalization. The key idea is to train a residual network to correct the base policy's action predictions using paired clean and visually perturbed observations of the same physical state.
  • The method generates correction targets by computing the difference between clean and perturbed action predictions.
  • The authors evaluate PRA on LIBERO-Spatial using several models. However, the method can reduce performance under clean observations, short perturbations, and physical target displacement.
Critical strengths
  • Simple and practical approach to VLA robustness. I like the idea of adapting VLAs with a lightweight residual network, without requiring additional expert demonstrations. Also, the method adds minimal inference overhead.
  • The authors evaluate PRA with π0.5, π0, and OpenVLA, demonstrating consistent improvements under perturbations. I really appreciate the inclusion of matched-budget ablations.
Critical weaknesses
  • Learning a lightweight residual network to correct a frozen VLA's predicted actions has already been explored in several prior works, including ResVLA and other residual policy learning approaches. While this paper introduces paired clean/perturbed action supervision, it is unclear how the proposed method fundamentally differs from these existing approaches.

  • Limited justification for the visual encoder architecture. The residual network uses a relatively small three-layer CNN trained from scratch on only 2.6k samples. It's unclear whether this architecture can extract sufficiently robust visual representations, especially under distribution shifts. Given the strong pretrained visual encoders such as SigLIP and DINOv2, I wonder whether leveraging pretrained visual features would improve correction accuracy, generalization, and clean-condition performance.

Questions for authors
  • Could the authors compare against a residual model / imitation learning policy trained with standard data augmentation techniques?

  • How does PRA compare with alternative VLA robustness methods? The paper discusses RobustVLA, RoVLA, and Cross-View Action Consistency, but does not provide direct empirical comparisons.

  • Could the authors clarify how PRA differs from existing residual-based VLA methods, particularly ResVLA?

  • Why use a small CNN trained from scratch instead of pretrained visual representations such as SigLIP or DINOv2? Given the limited training data, it would be interesting to see whether frozen pretrained features improve robustness and reduce the clean-performance degradation.

4 comments

Thank you for your questions and suggestions.

1. Standard data augmentation

Our new Aug-Residual control retains PRA’s CNN, frozen π0.5\pi_{0.5}, record count, and training schedule (§§4.3–5). We replace the 587 spatially perturbed records with brightness, contrast, saturation, blur, and noise augmentations, recompute base actions, and retain clean-policy teachers. Other records are unchanged.

All cells use the same 300 starts across ten LIBERO-Spatial tasks. Base/PRA results are reused (§6.1, Fig. 3; clean: §6.3); Aug-Residual and DINOv2 are new. Durations are scheduled control steps, followed by normal rendering if execution continues.

MethodCleanVisual 8 cm, 20 stepsVisual 8 cm, 40 steps
Base π0.5\pi_{0.5}97.67%62.33%47.00%
Aug-Residual97.67%63.33%45.00%
PRA: small CNN91.67%66.00%63.67%
PRA: frozen DINOv2 features94.67%66.67%56.67%

At 40 steps, PRA gains 16.67 pp over Base and 18.67 pp over Aug-Residual (new comparison; Bonferroni-adjusted paired-bootstrap 97.5% CI: [10.67, 26.33]). This supports targeted spatial supervision over the tested augmentations, not pairing alone. The CI covers initial-state uncertainty under fixed tasks and one training seed.

2. Other robustness methods

RobustVLA updates the π0\pi_0 action expert and uses VLM LoRA (App. B.1.1); RoVLA trains selected semantic layers and an action generator (§4.1); Cross-View Action Consistency fine-tunes π0.5\pi_{0.5} with paired-view losses (§4.1).

PRA freezes the entire VLA, including its action head (§§3.1, 4.3; Fig. 2). After target collection, only the external module is trained, without VLA backpropagation or architecture changes. Direct comparisons remain useful but have not been run under our protocol; our controls establish benefits within this frozen-policy setting, not a ranking against these methods.

3. Difference from ResVLA

Restoring observations does not undo actions taken during a disturbance (§3.1; Fig. 1). PRA addresses task completion across this process. Its correction reference comes from clean/perturbed predictions of the same frozen policy at the same physical state and sampling noise (§4.1, Eq. 3), requiring no additional expert action labels or deployment-time clean reference (§3.2).

We appreciate ResVLA’s insightful intent anchoring and residual bridging toward demonstrated actions (§4). PRA instead learns an external correction from paired policy predictions. Three-backbone results support full PRA (§6.1, Fig. 3), not pairing alone (§6.2). Clean and physical-displacement limitations remain (§§6.3–7).

4. CNN versus pretrained features

The CNN predicts corrections from images, state, and base actions with only 0.249M parameters (§4.3). Following your suggestion, we replace it with frozen DINOv2-small CLS features, keeping the VLA unchanged. Both views share an encoder; an MLP combines features, state, and base actions. Training retains 2,601 records and the same schedule (§5).

This new module totals 22.35M parameters, about 90×90\times the CNN, excluding the VLA. Clean success rises from 91.67% to 94.67%, but gains are not uniform. This supports the CNN’s size–performance balance, not general superiority.

We chose DINOv2-small to limit added size; larger variants and SigLIP remain untested. Inputs are 224×224224\times224 for DINOv2 versus 64×6464\times64 for CNN (§4.3), so this tests a specific frozen-CLS replacement, not pretraining in isolation.

Dear Reviewer v8rz,

Thank you for your time and constructive feedback. As the discussion period draws to a close, we would be grateful to know whether the augmentation comparison in our rebuttal addresses your concern or whether any questions remain. If you feel it resolves your concern, we would appreciate your taking this evidence into account when reconsidering your overall assessment.

Thank you again for your time and consideration.

Thank you for your detailed response and additional experiments. I appreciate your efforts to address my concerns and happy to maintain my positive recommendation.

Dear Reviewer V8RZ,

Thank you for considering our response and additional experiments. We appreciate your continued positive recommendation and your constructive suggestions, which helped us strengthen the evaluation. If the additional results have strengthened your confidence in the paper, we would be grateful if you would consider reflecting that in your final rating.

Reviewer_1kr7OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

Paired Residual Adaptation (PRA) freezes a VLA policy and trains a small network to correct its proposed translational actions. Training targets come from the difference between the frozen policy’s predictions on clean and visually shifted observations of the same physical state, using matched sampling noise. Zero-residual examples then encourage preserving the valid actions. Experiments test on rendering-only target displacements on LIBERO-Spatial evals.

Critical strengths

The result is convincing that the fresh-state, 40-step experiments improve success across three backbones: pi0.5, pi0, and OpenVLA.

Also the paper does a good job at distinguishing initial-state uncertainty from seed variance, and finds that pairing and balancing weights have not demonstrated benefits in closed-loop settings.

Critical weaknesses

My main concern of the paper is that, it relies on the assumption that, the kind of scene changes that should be ignored is known ahead of time, e.g., rendering-only target shifts should be corrected and physical target shifts should be preserved. This limits the practicality of the method in realistic deployment settings. Also note the experiments ran without observation history, so the model has a lot of ambiguity in input, and we see how the ambiguity causes performance degradation under physical target shifts (in 6.3).

Also the experimental setting is fairly limited. Only libero sim tasks were considered and there were no real-world evaluations.

Questions for authors

Why did physical-shift performance collapse although there are explicitly zero-residual physical-shift examples in training? How do training and evaluation physical shifts differ?

2 comments

We thank the reviewer for the careful review and for recognizing the fresh-state gains across three backbones. Your comments highlight where we should explain PRA’s training assumptions and deployment requirements more clearly.

Regarding practical applicability. You correctly note that PRA specifies which visual changes to correct during training. We construct supervision from clean and synthetically shifted observations of the same physical state (Sec. 4.1; Fig. 2). Deployment requires neither perturbation-type labels nor clean references, and adds no base-policy queries (Secs. 3.2 and 4.3). This does not by itself resolve your concern about mixed visual and physical changes: our demonstrated setting is visual mislocalization while the physical target remains stationary. We will state this scope explicitly.

Within this setting, PRA provides effective adaptation through a lightweight head while keeping the VLA frozen and requiring no additional expert corrective-action labels. Training uses 2,601 frame-level records (Sec. 5), including 587 visually perturbed records, confirmed by our data audit. The original π0.5 experiment on 300 training-disjoint starts achieves 63.7% success under 40-step visual displacement, versus 47.0% for Base (Sec. 6.1; Fig. 3). Our new matched rebuttal experiment adds a photometric-augmentation residual baseline with the same architecture and training budget. Its success rate is 45.0%, giving PRA an 18.7-percentage-point gain, with a multiplicity-adjusted 97.5% paired interval of [10.7, 26.3]. This further supports the value of targeted correction supervision.

Regarding zero-residual examples and training–evaluation differences. Zero labels encourage preserving base actions (Sec. 4.2) but provide no additional information to distinguish visual mislocalization from genuine motion. They therefore do not guarantee zero correction on unseen, ambiguous inputs. Additional controlled diagnostics on selected cases support interference from learned corrections with tracking genuine motion. Our protocol audit confirms the same 8 cm displacement and approach-phase trigger in training and evaluation. Physical training examples come from 30 trajectories; evaluation uses 30 training-disjoint starts per task. The updated evaluation additionally adjusts some directions under predetermined geometric-validity rules, with shared vectors for Base and PRA and no outcome-based selection. We will make these differences explicit.

Regarding real-world evaluation. We agree that real-robot experiments would further assess applicability. LIBERO allows us to separate visual displacement from physical changes and directly test correction effectiveness (Secs. 3.1 and 5). Real-robot validation is underway, although integration and debugging may prevent completion within the discussion period. We will share complete results if available during this period; current conclusions rely on completed simulation evidence.

Regarding readability. Thank you for this feedback. We will foreground the task assumptions and main results, distinguish training-only information from deployment inputs, and add a concise protocol comparison so that the method’s objective, requirements, and supporting evidence are easier to follow.

Dear Reviewer 1kr7,

Thank you again for your question about the zero-residual training examples. To clarify the diagnostic result mentioned in our response, we replayed 20 selected cases in which Base succeeded and PRA failed. Keeping PRA’s actions up to the physical shift and disabling the residual from that point onward restored success in all 20 cases.

We hope this clarifies how the learned correction interfered with tracking genuine target motion in these cases. If this addresses your concern, we would be grateful if you would take this evidence into account when revisiting your overall assessment.

Reviewer_n4hAOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

A brief error in the apparent position of a target can push a robot off course, and because policies execute action chunks, the consequence outlives the error itself. This paper keeps the pretrained VLA frozen and trains a small residual head to correct the translational part of its proposal, using as supervision the difference between the frozen policy's own predictions on clean and perturbed renderings of one physical state, queried with matched sampling noise. Zero-residual examples teach the head to leave valid proposals alone, and clean references are used only during training. At deployment the head sees only the current images, robot state, and the proposed chunk — no clean reference, no additional query, no history — and adjusts translation at the base policy's native query rate.

Critical strengths
  1. The problem is real, important, and common in deployment. A brief error in the apparent position of a target is a realistic deployment fault, and because policies execute action chunks its consequence outlives the error itself. The gap the paper names — that such faults degrade a frozen policy well after the fault is gone — is worth attacking.
  2. Locating the harm in chunked execution rather than in perception is the right diagnosis, and it is what makes a lightweight post-hoc correction a sensible thing to try.
  3. The deployment form is clean and feasible. At deployment the head sees only current images, robot state, and the proposed chunk — no clean reference, no extra policy query, no history — and it adjusts at the base policy's native query rate. The recipe is cheap and immediately usable.
Critical weaknesses
  1. As the paper defines it, the problem is not identifiable from the head's information set.Rendered displacement and genuine displacement produce identical observations, identical robot state, and — for a deterministic base policy — identical proposals, yet require opposite corrections; no function of the head's inputs can separate them. Trained to always assume the former, the method collapses on the physically displaced group, which is the paper's own proof of the limit.
  2. The one cue that would resolve it is named by the title and removed by the design. The information that would resolve the ambiguity is precisely what the design refuses to supply: time. A rendering fault appears as a discontinuous jump inconsistent with the robot's self-motion and with the rest of the scene; the word "transient" in the title names exactly that solvable cue, and the method discards it. A contemporaneous work closes most of this gap by checking whether visual motion and observation freshness remain consistent with geometry, proprioception, and action transitions.
  3. Both the motivating example and the abstract's framing sit in the regime where the method fails.The motivating episode in the first figure corresponds to the grid cell where the gain is slightly negative. Gains appear only at longer exposures — a fraction of the episode long enough to count as a persistent hallucination rather than a transient one. The abstract's framing describes precisely the regime in which the method fails, so the paper's own motivating material and its own results are in tension.
Questions for authors
  1. Does the motivating episode in Figure 1 correspond to the grid cell where the gain is slightly negative? If so, will the abstract be revised to state that gains appear only at longer exposures?
  2. The mismatched-reference and record-uniform controls both contain zero, and the one contrast that separates is identity versus zero skip initialization — a generic residual-network result. What does the method contribute beyond it?
  3. Please give the net expected effect as a function of the perturbation base rate, and the break-even rate implied by the clean-condition cost. Do you agree that below roughly that rate the method is net harmful?
  4. Why were all mechanistic comparisons run at the cell where the overall gain is indistinguishable from zero? If the pairing and balancing controls were rerun at the longer exposure where the method works, would the attribution change?
1 comment

Dear Reviewer n4hA,

Thank you for your thoughtful questions about the trade-off between clean performance and robustness to sustained visual shifts. As the discussion period draws to a close, we would be grateful to know whether our analysis of when the gains outweigh the clean-performance cost addresses your concern. If it does, we would appreciate your taking this clarification into account when revisiting your overall assessment.

Thank you again for your time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.