acceptodds
Under review as a conference paper at ICLR 2027

eRLT: Efficient VLA Reinforcement Learning via Action-Relevant Token Routing

11 views

Abstract

Vision-Language-Action (VLA) models provide strong behavioral priors for robotic manipulation, yet efficiently adapting them to downstream tasks remains challenging. Recent work addresses this challenge by adapting frozen VLAs through online reinforcement learning (RL), whose sample efficiency depends on the quality of the state representation used by the actor and critic. Existing methods construct such representations either with VLA-independent visual encoders or through fixed compression of internal VLA representations. Neither design explicitly extracts the task-specific action-relevant VLA features most useful for downstream action refinement and action-value estimation, therefore limiting sample efficiency. To address this limitation, we introduce eRLT, which constructs an effective state representation by routing task-specific action-relevant information across both tokens and layers of the frozen VLA. Specifically, learned routing tokens dynamically aggregate visual-language features at multiple depths, while a lightweight layer router combines these summaries into a fixed-dimensional RL token. The routing module is initialized using expert demonstrations to capture features predictive of expert actions and then refined using critic feedback from online interactions for action-value estimation. Across seven LIBERO and RoboTwin tasks, eRLT improves mean normalized learning-curve AUC by up to 23.7% over representative baselines. Real-robot experiments on USB connector insertion and motherboard ribbon-cable insertion further show AUC improvements of 108.9% and 46.7%, respectively, over the strongest baseline.

Primary area:
applications to robotics, autonomy, planning
Keywords:
VLA, Reinforcement Learning, Post-training

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_6PTYOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies state representations for lightweight online reinforcement learning with a frozen vision-language-action model. eRLT uses learned routing tokens to summarize the model's internal features at several layers, then combines these summaries using task-specific layer weights. The routing module is initialized by predicting expert actions from existing SFT demonstrations and can subsequently be updated through the critic loss. Across seven simulation tasks, eRLT obtains a mean normalized learning-curve AUC of 0.626, compared with 0.585 for DSRL and 0.506 for RLT. Experiments on USB connector insertion and motherboard ribbon-cable insertion also show faster learning within the tested interaction budgets.

Critical strengths
  1. The paper tests a specific representation problem with controlled comparisons. Section 4.1 varies the source layer or token subset while holding the VLA and online RL procedure fixed. The best choices differ between tasks, supporting the motivation for learned aggregation. Table 2 also tests the routing and training components separately, including a final-layer variant that retains the other components. These experiments provide more useful evidence than a comparison of complete methods alone.

  2. The improvement is reasonably consistent across the tested tasks and has practical relevance. In Table 1, eRLT improves on RLT* on all seven tasks and DSRL* on six. The real-world results in Table 4 show substantial AUC improvements over RLT on both insertion tasks. Table 9 additionally shows fewer assisted trajectories under the shared intervention rule: 14 versus 36 on USB insertion and 20 versus 25 on ribbon-cable insertion. This supports the practical value of the representation within the reported collection protocol.

Critical weaknesses
  1. The main simulation results do not clearly report variation across independent training runs. Section 4.1 and Appendix C.5 explicitly use three seeds for the motivating study, but the number of runs underlying Tables 1–3 is not stated clearly, and these tables have no uncertainty estimates. This matters especially for the component conclusions in Section 5.3: removing online adaptation reduces AUC from 0.818 to 0.803, and removing layer-wise routing reduces it to 0.797. The reported values support the overall method, but do not show how consistently these smaller improvements occur across runs.

  2. The real-world experiments do not specify whether the router is updated online. Section 4.5 describes online updates to the routing module, whereas Table 8 gives “/” for the real-world router update interval and target update rate. Appendix C.4 specifies the actor and critic updates but does not explicitly state whether the router is frozen. The distinction matters when interpreting the real-world results as evidence for the two-stage training procedure. Please state which components were updated so that the real-world results can be interpreted correctly.

Questions for authors
  1. How many independent training runs contribute to each entry in Tables 1–3? Please report the per-run AUCs or an uncertainty estimate from those runs, particularly for the online-adaptation and layer-routing ablations in Table 2. This would help determine how much weight to place on the smaller component gains.

  2. Is the routing module updated online in the real-world experiments? Please explain the “/” entries in Table 8. If it is updated, what update interval, learning rate, and target-router update rule are used? If it is frozen, please state this explicitly in Appendix C.4.

  3. Do Figure 5 and Table 4 use the same runs and the same final 10 autonomous episodes? The RLT USB curve appears to end above the 50% reported in Table 4; the ribbon-cable endpoint also appears different from the tabulated value. Please give the corresponding success counts and check the plotted endpoints, episode windows, and the RLT*/RLT labels. This should be resolvable from the existing episode records.

2 comments

Thank you for your careful review and for highlighting the controlled comparisons in Section 4.1, where we varied the source layer or token subset while keeping the VLA and online RL procedure fixed. We also appreciate your close reading of the real-world results and your suggestion to report variation across runs, particularly for the smaller gains in Table 2. Appendix E of the revised paper collects the repeated-run results and training curves, including the additional component ablations.

Q1 and W1. How many independent training runs were used for the main comparisons and component ablations, and how much do the AUCs vary, particularly for the online-adaptation and layer-routing ablations in Table 2? (Appendices E.1, E.2 and E.4)

In the submission, we used one training run per method and task for the main simulation and real-world comparisons, and one per component ablation on Bowl-to-Plate in Table 2. The three seeds reported in Section 4.1 were for the motivating study. We agree that a single run gives little indication of how much a small AUC difference might vary across runs.

We compiled the existing results and completed the remaining runs. We now have three independent training runs per method on all seven simulation tasks and both real-world tasks, and three per component configuration on Bowl-to-Plate and Drawer Opening. The table below gives normalized AUC as the mean ± sample SD for the simulation tasks (Appendix E.1, Table 10) and the real-world tasks (Appendix E.2, Table 12). We have also updated the training curves to show the mean and one sample SD across the three runs for simulation (Appendix E.1, Fig. 10) and real-world insertion (Appendix E.2, Fig. 11). We will incorporate the repeated-run results into the main text.

TaskDSRLRLTeRLT
Drawer Opening0.824 ± 0.0320.686 ± 0.0210.860 ± 0.019
Bowl-to-Plate0.801 ± 0.0370.682 ± 0.0410.828 ± 0.011
Cabinet-Bowl-to-Plate0.165 ± 0.0320.158 ± 0.0180.213 ± 0.049
Two-Moka-Pots0.208 ± 0.0440.204 ± 0.0120.280 ± 0.017
Adjust Bottle0.764 ± 0.0880.822 ± 0.0430.896 ± 0.003
Move Can–Pot0.429 ± 0.2150.315 ± 0.2270.454 ± 0.100
Place Container Plate0.661 ± 0.0120.462 ± 0.2110.636 ± 0.103
USB Insertion (real robot)0.383 ± 0.1210.326 ± 0.0520.615 ± 0.052
Ribbon-Cable Insertion (real robot)0.413 ± 0.1440.558 ± 0.1180.708 ± 0.039

Across the three runs, eRLT achieved the highest mean AUC on six of the seven simulation tasks and on both real-world tasks. The table below gives the three-run results for the component ablations (Appendix E.2, Table 11).

VariantBowl-to-Plate AUCDrawer Opening AUC
eRLT0.828 ± 0.0110.860 ± 0.019
Without offline initialization0.780 ± 0.0270.783 ± 0.043
Without online adaptation0.810 ± 0.0060.812 ± 0.020
Without token-wise routing0.717 ± 0.0330.749 ± 0.113
Without layer-wise routing0.790 ± 0.0060.657 ± 0.050

On Bowl-to-Plate, both gains observed in the original run persisted in the three-run means. The gain from critic-driven adaptation was 0.015 in the original run and 0.018 on average across three runs. The gain from layer-wise routing was 0.021 in the original run and 0.038 on average, with SDs of 0.011 and 0.006, respectively, for full eRLT and the final-layer variant. Both components also improved mean AUC on Drawer Opening (0.048 for critic-driven adaptation and 0.203 for layer-wise routing), and on real-world USB insertion, three runs with layer-wise routing and two without gave a mean AUC gain of 0.128 (Appendix E.4, Table 14 and Fig. 13).

Q2 and W2. Is the real-world router updated online, and what do the “/” entries in Table 8 (Appendix C.2) mean?

The “/” entries noted in the review were unfilled placeholders that we had overlooked. The router was updated online through the critic. Table 8 in Appendix C.2 and the real-world training protocol in Appendix C.4 now specify U = 240 learner updates (once per 15 trajectories), learning rate 3 × 10⁻⁴, and target-router Polyak coefficient 0.005. The latter two match simulation; the VLA remains frozen.

(To stay within the character limit, we’ve included our response to your other questions in the next comment.)

Q3. Do Figure 5 and Table 4 use the same runs and final episode window?

Figure 5 and Table 4 use the same training runs. In Figure 5, we used the first 80 collected trajectories for USB insertion and the first 90 for ribbon-cable insertion. In the original Table 4, we calculated the final success rate using the last ten autonomous episodes in each full recording. Some runs continued beyond the task budget, so we counted episodes that fell outside the budget. We apologize for this error.

We corrected Table 4 by counting successful insertions in the last ten autonomous episodes within each task’s budget. On USB insertion, DSRL succeeded in 4/10 episodes, RLT in 8/10, and eRLT in 9/10. On ribbon-cable insertion, the counts were 3/10, 9/10, and 10/10, respectively. The correction changes RLT’s final success rate from 50% to 80% on USB insertion and from 100% to 90% on ribbon-cable insertion. For DSRL on ribbon-cable insertion, it changes from 50% to 30%. We updated Section 5.4 and corrected the Figure 5 legend to use the same RLT label as Table 4.

The correction narrows the original USB final-SR advantage to 10 percentage points. The AUC advantage is unaffected because the learning curves already used the intended budgets. The corrected results continue to support faster adaptation within these budgets, with a smaller difference in final success. We applied the same cutoff to each repeat. Assisted episodes count toward the interaction budget, and final SR uses the last ten autonomous episodes within that budget (Appendix C.6). Table 12 in Appendix E.2 reports the mean ± sample SD across the original run and two additional runs for each method on each task.

TaskMethodAUCFinal SR (%)
USBDSRL0.383 ± 0.12156.67 ± 20.82
USBRLT0.326 ± 0.05276.67 ± 5.77
USBeRLT0.615 ± 0.05283.33 ± 5.77
Ribbon CableDSRL0.413 ± 0.14463.33 ± 35.12
Ribbon CableRLT0.558 ± 0.11896.67 ± 5.77
Ribbon CableeRLT0.708 ± 0.039100.00 ± 0.00

We hope these results and clarifications address your questions.

Best wishes,

The authors of Submission 29637

Reviewer_g6V3OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes eRLT, a method for constructing task-specific state representations for lightweight online RL with frozen VLA models. Rather than relying on external visual encoders or compressing features from a fixed VLA layer (from the original Pi's design), eRLT introduces learnable routing tokens that aggregate information across visual-language tokens and multiple transformer layers. Experiments cover seven LIBERO and RoboTwin tasks and two real-world precision insertion tasks.

Critical strengths
  1. The overall proposed method is sound and reasonable. Studying the state representation used by the actor and critic in VLA is definitely relevant, and the original method RLT proposed by Physical Intelligence is less intuitive and may discard information useful for downstream adaptation
  2. The paper presents comprehensive empirical study and physical evaluation. The supplementary video is helpful.
  3. The original Pi's paper doesn't provide open source implementation. If authors could provide open source code for baseline and proposed method, this will be a contribution to research community.
Critical weaknesses
  1. The numerical evaluation results do not seem to be strong. For example, in the ablation study (Table 2), the ablated baselines are very close to the original method (0.80 vs 0.82) except the offline initialization. The results confuse me because this suggests that most of the gain seems to come from reconstruction vs action-predictive learned query.
  2. The previous point leads to the weakness of the overall claim about contribution of the paper. The method is a combination of known techniques (prompt-style learned queries / routing, layer scalar mix, critic-trained encoders) applied to frozen-VLA RL, and the components unique to this combination show the smallest measured gains from weakness 1.
  3. The paper's overall writing is confusing and fuzzy, with the use of "routing" vs "query", "The superscript ∗denotes use of only the corresponding state representation in our shared online RL framework". I suggest authors to simplify the language to enhance the readability.
Questions for authors
  1. Are the routing parameters updated through the critic during real-world online RL?
  2. How much of the improvement over RLT comes from the action-prediction objective rather than the routing architecture? and How would authors position the overall paper's novelty based on the evaluation results?
3 comments

Thank you for your careful review and for noting that RLT’s compression may discard features the actor and critic need for downstream adaptation. This concern motivated us to study which layers and tokens of a frozen VLA are most useful for each task. We appreciate your comments on the real-world evaluation and your suggestion to examine how much RLT gains from action prediction when its architecture is held fixed. Appendix E of the revised paper collects the repeated-run results, including the action-prediction comparison and the USB layer-routing ablation.

We will release our eRLT and baseline implementations as soon as we receive institutional approval. We could not submit the code because work involving our robot platform and experimental data requires prior internal review.

Q1. Are the routing parameters updated through the critic during real-world online RL? (Appendices C.2 and C.4)

Yes. On both real-world insertion tasks, we updated the routing tokens and layer weights through the critic loss every 240 learner updates (once per 15 collected trajectories). The router learning rate was 3 × 10⁻⁴, and we updated the target router by Polyak averaging with a coefficient of 0.005. The VLA was frozen throughout. We have filled in the missing entries in Appendix C.2, Table 8, and added these settings to the real-world training protocol in Appendix C.4.

Q2 and W1. How strong is the numerical evidence for eRLT’s gains? In Table 2, several ablated variants score close to full eRLT (about 0.80 vs. 0.82), while removing offline initialization gives a larger drop. Does replacing reconstruction with action prediction explain most of the improvement over RLT, and how much does the routing architecture contribute?

In the original ablation study (Table 2), both offline initialization with action prediction and the learned query for token-wise routing gave clear gains. To assess these gains more reliably, we repeated the ablations on Bowl-to-Plate and ran them on a second simulation task. The table below reports mean normalized AUC ± sample SD for the simulation ablations (Appendix E.2, Table 11) and the layer-wise routing ablation on real-world USB insertion (Appendix E.4, Table 14 and Fig. 13).

VariantBowl-to-PlateDrawer OpeningUSB Insertion (real robot)
eRLT0.828 ± 0.0110.860 ± 0.0190.615 ± 0.052
Without offline initialization0.780 ± 0.0270.783 ± 0.043—
Without online adaptation0.810 ± 0.0060.812 ± 0.020—
Without token-wise routing0.717 ± 0.0330.749 ± 0.113—
Without layer-wise routing0.790 ± 0.0060.657 ± 0.0500.487 ± 0.090

Removing any one component reduced mean AUC on both simulation tasks. Offline initialization and token-wise routing gave consistent gains across the two tasks. The gain from layer-wise routing varied more, from 0.038 on Bowl-to-Plate to 0.203 on Drawer Opening. As our motivating study (Section 4.1) suggests, the most useful layers vary by task, so multi-layer routing may offer less benefit when the final layer already captures the relevant features. We are running full ablations on additional tasks to better estimate the size and consistency of the gain from layer-wise routing. These results support the importance of both the routing architecture and action-prediction initialization in eRLT.

We agree that the original ablations do not isolate the effect of the action-prediction objective. The closest variant, eRLT without layer-wise routing, uses only the final VLA layer, as RLT does. However, it still uses token-wise routing and online adaptation, so a comparison with RLT cannot tell us how much the objective alone contributes. We therefore tested RLT + AP, replacing only RLT’s reconstruction objective with eRLT’s action-prediction objective. The table below gives the results on two simulation tasks and real-world USB insertion (Appendix E.3, Table 13 and Appendix E.4, Table 14). We used three runs per configuration in simulation. On USB, we used three runs each for RLT and full eRLT and two each for RLT + AP and eRLT without layer-wise routing. The USB training curves for RLT and RLT + AP are in Appendix E.3, Fig. 12.

RepresentationBowl-to-Plate AUCDrawer Opening AUCUSB AUC (real robot)
RLT0.682 ± 0.0410.686 ± 0.0210.326 ± 0.052
RLT + AP0.699 ± 0.0410.710 ± 0.0270.472 ± 0.153
eRLT without layer-wise routing0.790 ± 0.0060.657 ± 0.0500.487 ± 0.090
eRLT0.828 ± 0.0110.860 ± 0.0190.615 ± 0.052

RLT + AP achieved a higher mean AUC than RLT on all three tasks. Its average across tasks was still lower than that of eRLT without layer-wise routing, and full eRLT achieved the highest mean AUC on every task. These results support our conclusion that both the routing architecture and the action-prediction objective are important to eRLT’s performance.

(To stay within the character limit, we’ve included our response to your other questions in the next comment.)

Q2 and W2. Given that eRLT combines established techniques and its more distinctive components appear to yield the smallest gains, where does the paper’s novelty lie?

As the repeated ablations above show, each of eRLT’s core components improved mean AUC on both simulation tasks. Together, these results support eRLT as a practical method for improving online RL performance with a frozen VLA.

We agree that learned queries, layer weighting, action prediction, and critic-trained encoders all have precedents. The individual components have limited novelty. Prior work has studied how to aggregate features across tokens and layers for vision-language understanding and action generation. Our methodological contribution lies mainly in adapting similar techniques to address a critical problem with existing online RL methods for VLAs, whose state representations may omit features the actor and critic need for a given task. More fundamentally, we share your view that studying the state representation used by the actor and critic in VLA adaptation is both necessary and important. Our experiments show a clear performance advantage over RLT (Appendix E.1, Table 10 and Appendix E.2, Table 12). eRLT achieves this improvement without requiring more expert demonstrations or online interactions than RLT. We believe this makes eRLT a useful reference for the community.

Our findings in Section 4.1 support your observation that RLT’s compression may discard information useful for adaptation. With the rest of RLT held fixed, the best source layer differed between Drawer Opening and Bowl-to-Plate, as did the best token subset within the final layer. This task dependence motivated us to learn the readout from expert demonstrations for each task.

W3. The routing terminology and baseline notation make parts of the paper difficult to follow.

We agree that the distinction between the routing token and the layer router should be clearer. Reviewer 6PTY also pointed out two errors that made the real-world results harder to interpret. We have corrected the RLT label in Figure 5 and the final success rates in Table 4. We will go through the paper together and revise the wording throughout to make it easier to read.

We hope these results and clarifications address your questions.

Best wishes,

The authors of Submission 29637

Dear Reviewers,

Thank you again for your feedback. Following up on our earlier response, we have completed additional ablations of token-wise and layer-wise routing on three RoboTwin tasks to further assess their contributions across tasks. The table below reports normalized AUC (mean ± SD).

VariantAdjust BottleMove Can-PotPlace Container Plate
eRLT0.896 ± 0.0030.454 ± 0.1000.636 ± 0.103
Without token-wise routing0.807 ± 0.0500.242 ± 0.1650.447 ± 0.133
Without layer-wise routing0.789 ± 0.0710.207 ± 0.0270.516 ± 0.233

We hope these additional results help address your concern about the routing architecture’s contribution: both token-wise and layer-wise routing improve mean AUC on all three RoboTwin tasks, consistent with our earlier LIBERO results. The benefit of layer-wise routing remains task-dependent, ranging from 0.038 on Bowl-to-Plate to 0.247 on Move Can-Pot. This supports our motivation in Section 4.1 and suggests that the small gain on the original task does not fully reflect the architecture’s contribution across tasks.

As the discussion period draws to a close, we kindly invite you to review our responses, and let us know if any questions or concerns remain. If you find that they adequately address your concerns, we would greatly appreciate your reconsideration of the original rating.

Reviewer_Kw5COpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes eRLT, which learns to aggregate tokens from multiple layers of a frozen VLA into a state representation for lightweight RL. The routing module is initialized through expert action prediction and then updated using critic feedback. Experiments on seven simulation tasks and two real-world insertion tasks show improved sample efficiency over the compared representations.

Critical strengths

The paper studies a concrete and useful question: which information from a frozen VLA should be provided to the actor and critic? The controlled comparisons and real-world insertion results make a convincing case that representation design can improve online adaptation without updating the VLA itself.

Critical weaknesses
  • The real-world results lack repeated runs and uncertainty estimates, so it is difficult to judge how consistent the improvements are
  • The component ablations cover only one task, where removing online adaptation or layer routing causes relatively small AUC drops
Questions for authors
  • How many independent runs were used for the main simulation results, component ablations, and real-world experiments, and what is the variation across runs?
  • Do the benefits of layer routing and critic-driven adaptation remain consistent on other tasks, particularly the real-world insertion tasks?
4 comments

Thank you for your careful review and for recognizing the value of studying which features of a frozen VLA should be available to the actor and critic. We appreciate your suggestion to examine how consistently the gains hold across runs and whether the component results extend beyond Bowl-to-Plate. Appendix E of the revised paper brings together the repeated-run comparisons, additional ablations, and training curves.

Q1 and W1. How many independent runs were used for the main simulation results, component ablations, and real-world experiments, and what is the variation across runs? (Appendices E.1, E.2 and E.4)

In the submission, we used one training run per method and task for the main simulation and real-world comparisons, and one per component ablation on Bowl-to-Plate. We agree that a single run is insufficient to reliably assess a method’s typical performance and variability across runs.

We compiled the existing results and completed the remaining runs. We now have three independent runs per method on all nine tasks and three per component ablation on each of two simulation tasks. The table below gives normalized AUC as the mean ± sample SD for the simulation tasks (Appendix E.1, Table 10) and the real-world tasks (Appendix E.2, Table 12). We have also updated the training curves to show the mean and one sample SD across the three runs for simulation (Appendix E.1, Fig. 10) and real-world insertion (Appendix E.2, Fig. 11). We will incorporate the repeated-run results into the main text.

TaskDSRLRLTeRLT
Drawer Opening0.824 ± 0.0320.686 ± 0.0210.860 ± 0.019
Bowl-to-Plate0.801 ± 0.0370.682 ± 0.0410.828 ± 0.011
Cabinet-Bowl-to-Plate0.165 ± 0.0320.158 ± 0.0180.213 ± 0.049
Two-Moka-Pots0.208 ± 0.0440.204 ± 0.0120.280 ± 0.017
Adjust Bottle0.764 ± 0.0880.822 ± 0.0430.896 ± 0.003
Move Can–Pot0.429 ± 0.2150.315 ± 0.2270.454 ± 0.100
Place Container Plate0.661 ± 0.0120.462 ± 0.2110.636 ± 0.103
USB Insertion (real robot)0.383 ± 0.1210.326 ± 0.0520.615 ± 0.052
Ribbon-Cable Insertion (real robot)0.413 ± 0.1440.558 ± 0.1180.708 ± 0.039

The repeated runs show a consistent advantage for eRLT across tasks. It achieved the highest mean AUC on six of the seven simulation tasks and on both real-world tasks. The uncertainty estimates also reveal differences in stability. For all three methods, the average run-to-run SD across the RoboTwin tasks was higher than across the LIBERO tasks. RoboTwin’s higher-dimensional dual-arm controls may make exploration more difficult and contribute to this variability across runs. On both real-world insertion tasks, DSRL’s AUC varied more across runs than that of RLT or eRLT. Under a limited interaction budget, visual variability in real-world scenes may make it difficult for DSRL’s separate visual encoder to learn fine alignment cues consistently.

(To stay within the character limit, we’ve included our response to your second question in the next comment.)

Q2 and W2. Do the benefits of layer routing and critic-driven adaptation remain consistent on other tasks, particularly the real-world insertion tasks? (Appendices C.5, E.2 and E.4)

We agree that the original ablations, with one run per variant on a single task, provided too little evidence to judge how much each component contributes across tasks. This concern is particularly relevant to layer-wise routing and critic-driven adaptation, since both gave relatively small gains on Bowl-to-Plate. Thank you for pointing this out and helping us strengthen the experimental evidence in the paper.

We repeated the original ablations on Bowl-to-Plate, ran the full set of ablations on a second simulation task, Drawer Opening, and tested layer-wise routing on real-world USB insertion. The table below reports mean normalized AUC ± sample SD for the simulation ablations (Appendix E.2, Table 11) and the USB layer-wise routing ablation (Appendix E.4, Table 14). Appendix E.4 also shows the USB training curves with and without layer-wise routing (Fig. 13). Each simulation entry uses three runs. On USB, we completed three runs for full eRLT and two for the variant without layer-wise routing. Given the limited robot time available during rebuttal, we have so far completed only the layer-wise routing ablation on the robot. Our estimates of the gain from critic-driven adaptation therefore come from the two simulation tasks.

VariantBowl-to-PlateDrawer OpeningUSB Insertion (real robot)
eRLT0.828 ± 0.0110.860 ± 0.0190.615 ± 0.052
Without offline initialization0.780 ± 0.0270.783 ± 0.043—
Without online adaptation0.810 ± 0.0060.812 ± 0.020—
Without token-wise routing0.717 ± 0.0330.749 ± 0.113—
Without layer-wise routing0.790 ± 0.0060.657 ± 0.0500.487 ± 0.090

Across the two simulation tasks, removing any one component reduced mean AUC. The gains from critic-driven adaptation were modest, at 0.018 on Bowl-to-Plate and 0.048 on Drawer Opening. Layer-wise routing improved mean AUC by 0.038 on Bowl-to-Plate, 0.203 on Drawer Opening, and 0.128 on USB insertion. As our motivating study (Section 4.1) suggests, the most useful layers vary by task, so multi-layer routing may offer less benefit when the final layer already captures the relevant features. We are running full ablations on additional tasks to assess the size and consistency of these gains.

We hope these additional results and explanations address your questions.

Best wishes,

The authors of Submission 29637

Thank you for the update. The repeated runs strengthen the overall results, particularly on the real robot, but the evidence for critic-driven adaptation remains limited to two simulation tasks, so I’m keeping my rating at 3.

Thank you for taking the time to consider our responses. We appreciate your constructive suggestions to further improve our work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.