acceptodds
Withdrawn from ICLR 2027

Generalizable Action Editor: Sharing Corrections across Tasks for Online Post-Training of Pretrained Robot Policies

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Different manipulation tasks can share local correction structure even when their goals differ. We introduce the Generalizable Action Editor (GAE), a shared policy that learns through online reinforcement learning to edit the action proposals of a frozen pretrained robot policy. Conditioning on local observations and the base proposal allows one editor to express corrections across tasks. To allocate limited training resources, we score each executed correction by its critic-estimated value gain over the same-state base proposal. This score guides CISA replay sampling and score-aware retention within ReSOP, our asynchronous multi-task learning system. A correction-support analysis characterizes how source-context coverage and compatibility of local edits shape transfer. On RoboTwin 2.0 Clean, we train one editor on 46 source tasks and evaluate it across all 50 tasks, including four held-out tasks without editor-parameter updates. Shared editing raises macro success from 90.00% to 90.84%. Four-task comparisons further favor intervention-value-based sampling over uniform and TD-error prioritization under matched learner-update budgets.

Primary area:
applications to robotics, autonomy, planning
Keywords:
robot learning, multi-task reinforcement learning, online policy post-training, cross-task generalization, prioritized experience replay, distributed reinforcement learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_TqY3OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper addresses the problem of correcting small action errors that pretrained multi-task VLA policies make in local phases such as alignment and contact, using online RL. Prior residual RL methods reduce such errors by learning a small corrective policy on top of a frozen base, but they require a separate corrector for each task. The authors observe that tasks with different goals, such as grasping, placing, and inserting, often require similar local corrections, and therefore treat action correction as something that can be shared across tasks. Their goal is a single module that corrects many tasks at once and can be applied to tasks not used in its training without further updates.

The paper proposes three components. First, GAE is a shared module that takes the observation, the robot state, and the base policy's action proposal, and outputs a bounded correction. All tasks share its parameters without task information, and the base policy stays frozen. Second, motivated by the observation that the value of a correction concentrates on a few moments in a trajectory, the authors define CIV as the critic-estimated value difference between the executed corrected action and the base action at the same state. CIV drives CISA, a sampling scheme that prioritizes experiences where the correction mattered, and a retention rule that keeps such experiences longer in replay. Third, ReSOP is an asynchronous system that separates collection, scoring, and learning to run this process across many tasks.

Experiments use 50 RoboTwin 2.0 tasks with a LingBot-VLA base fine-tuned on this benchmark. A single editor trained on 46 tasks raises macro success over all 50 tasks from 90.00% to 90.84%. On the 4 held-out tasks, it moves success from 88.50% to 88.75% without any updates. These results show that one correction module can serve many tasks while improving performance, with a weak generalization gain on unseen tasks. In a 4-task comparison, CISA reaches 86.00%, above uniform sampling at 82.75% and TD-error prioritization at 81.00%.

Critical strengths
  1. A clear formulation of action correction as a shared resource, with a systematic method and system to support it. Prior residual RL uses one corrector per task. This paper explicitly asks whether corrections can be shared across tasks and designs its components consistently around that question. The proposal-conditioned editor handles many tasks with one small module and never modifies the VLA. Since it uses no task information, it can be attached to unseen tasks as is. Together with ReSOP, which runs online RL over 46 tasks asynchronously, the paper offers a practical framework at a time when VLA post-training is becoming expensive.

  2. Substantial gains on some tasks without hurting most others. Because the correction is tightly bounded, success on most of the 50 tasks stays close to the base, while macro success rises from 90.00% to 90.84%. On click_bell, success rises from 75% to 96% and the mean episode length drops from 70 to 48 steps. This points to a real change in behavior rather than noise. stack_bowls_three also improves from 72% to 85%. These results show that a shared editor on a strong base can fix systematic failures on specific tasks while preserving existing capabilities.

Critical weaknesses
  1. The empirical gains are weak and not statistically established.

    • The gains are small. Macro success over 50 tasks improves by only 0.84 pp. The 0.25 pp gain on the held-out tasks amounts to one episode out of 400, and the S10 editor actually lowers held-out success by 1.75 pp.
    • The gains are concentrated in a few tasks. 21 tasks improve and 13 decline. Excluding the top 3, the remaining 47 tasks change by −0.04 pp on average.
    • The room for improvement is structurally limited. The correction scale is fixed at 0.025% of the normalized action range, which narrows what can be fixed, and the base already reaches 90%.
    • There are no real-robot experiments. Evaluation is limited to RoboTwin 2.0 Clean, without domain randomization, and to a single base policy. Local corrections matter most in real settings with sensor delay, calibration error, and contact mismatch. There is no evidence that corrections this small remain meaningful there.
  2. No baseline isolates the effect of sharing.

    • The paper's central question is whether sharing corrections across tasks helps. Yet it includes no comparison with independent per-task editors under the same budget, nor with existing per-task residual RL methods.
    • It is therefore unclear whether gains such as those on click_bell come from sharing or could be obtained equally with per-task training.
    • Although the editor receives no task information, it can identify the task from images. It may therefore simply store per-task corrections in one module rather than learn a shared correction structure.
    • The held-out results are consistent with this. The S1, S4, and S10 editors all fall below the base on held-out tasks. Only S46 recovers to the base level, and its source set includes near-identical siblings of the held-out tasks, such as place_a2b_left for place_a2b_right.
Questions for authors
  • Please see the weaknesses. Could the authors provide real-robot experiments, or results with multiple seeds, to show that the gains are robust?
  • Could the authors compare the shared editor with independent per-task editors under the same budget, to isolate the effect of sharing?
Reviewer_H12YOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces GAE, which can serve as a general actions corrector for general robot policies.

Critical strengths

The idea is interesting that minor local corrections might be similar for different actions or tasks so that a shared GAE can be learned for general action corrections. Continual learning experiments look decent.

Critical weaknesses

While it has an interesting idea, the effectiveness of this work does not seem solid.

  1. Lacks real-world experiments for further evaluation.
  2. The overall improvement compared to the baseline seems very limited (About 1%). Thus it's hard to recognize the effectiveness of the method. Perhaps more visualizations (e.g. how actions are corrected) can be provided to better support. See questions for more.
Questions for authors
  1. It seems only Lingbot-VA is used as the base model for evaluation. Would it be possible to use any other base models such as pi05 with the same GAE? How will it affect the performance, i.e., is GAE applicable in a more general setting with the same checkpoint?
  2. It's reasonable to use RL for training GAE. However, online RL training is expensive in pratice. Is it possible to train GAE with imitation-based methods instead?
  3. I doubt if the training of GAE in this work is sufficient or effetively done. Reward is sparse in many robotic tasks, so RL-based training might be not so helpful, especially when GAE only outputs a small action delta, which is not a strong signal if I got it correctly. I understand CISA can be used to help RL training under this setting, but is it sufficient? In Table 2, we see the improvement is only 1.25% compared to baseline with CISA used.
Reviewer_LUGPOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper provide an single shared module GAE that makes local corrections to the action proposals of a frozen robot policy. Paper also introduce a critic-estimated intervention value, which decides what gets prioritized in replay sampling and what gets retained. On RoboTwin 2.0 Clean, the editor trained on 46 source tasks reaches 90.84% macro success over 50 tasks, against 90.00% for the frozen base.

Critical strengths
  1. The setup is practical and easy to motivate. A shared editor conditioned on the base policy’s proposals is a modular way to improve a strong frozen policy without disturbing its task behavior.
  2. The CIV signal is fairly distinctive. It compares the edited action with the base proposal at the same state, and I appreciate that the paper separates its role in replay sampling from its role in retention. In the four task matched-update ablation, CISA beats both uniform and TD-error sampling.
  3. The 50 tasks benchmark coverage is broad, and the paper gives task-level results and is upfront about protocol limitations, which I found useful.
Critical weaknesses
  1. The main improvements are small and the reliability is skeptical. The improvement is 0.84 over all 50 tasks and only 0.25 on the four GAE-held-out tasks. Each arm is a single training run. Checkpoint selection also uses repeated evaluations on a source task, and that same selected evaluation seems to be reused in the reported averages, so the numbers may be slightly optimistic.
  2. The held-out transfer claim is narrower than it sounds. The base policy was fine-tuned on all 50 tasks, so the four targets are held out from GAE training only, not from the whole pipeline. Everything is also in the Clean setting, with no domain randomization and no physical robot evaluation.
  3. The scaling and reproducibility evidence is incomplete. The source-breadth study changes training updates, expert count and critic capacity at the same time as the number of source tasks, so it doesn’t isolate the effect of adding source tasks.
Questions for authors
  1. Could you report results over multiple training seeds, and evaluate the selected checkpoints on a separate set of episodes, with uncertainty intervals for both the main and held-out results?
  2. Could you test transfer with a base policy that was not fine-tuned on the held-out tasks, and isolate source breadth by fixing the training budget and model capacity?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.