- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies state representations for lightweight online reinforcement learning with a frozen vision-language-action model. eRLT uses learned routing tokens to summarize the model's internal features at several layers, then combines these summaries using task-specific layer weights. The routing module is initialized by predicting expert actions from existing SFT demonstrations and can subsequently be updated through the critic loss. Across seven simulation tasks, eRLT obtains a mean normalized learning-curve AUC of 0.626, compared with 0.585 for DSRL and 0.506 for RLT. Experiments on USB connector insertion and motherboard ribbon-cable insertion also show faster learning within the tested interaction budgets.
Critical strengths
-
The paper tests a specific representation problem with controlled comparisons. Section 4.1 varies the source layer or token subset while holding the VLA and online RL procedure fixed. The best choices differ between tasks, supporting the motivation for learned aggregation. Table 2 also tests the routing and training components separately, including a final-layer variant that retains the other components. These experiments provide more useful evidence than a comparison of complete methods alone.
-
The improvement is reasonably consistent across the tested tasks and has practical relevance. In Table 1, eRLT improves on RLT* on all seven tasks and DSRL* on six. The real-world results in Table 4 show substantial AUC improvements over RLT on both insertion tasks. Table 9 additionally shows fewer assisted trajectories under the shared intervention rule: 14 versus 36 on USB insertion and 20 versus 25 on ribbon-cable insertion. This supports the practical value of the representation within the reported collection protocol.
Critical weaknesses
-
The main simulation results do not clearly report variation across independent training runs. Section 4.1 and Appendix C.5 explicitly use three seeds for the motivating study, but the number of runs underlying Tables 1–3 is not stated clearly, and these tables have no uncertainty estimates. This matters especially for the component conclusions in Section 5.3: removing online adaptation reduces AUC from 0.818 to 0.803, and removing layer-wise routing reduces it to 0.797. The reported values support the overall method, but do not show how consistently these smaller improvements occur across runs.
-
The real-world experiments do not specify whether the router is updated online. Section 4.5 describes online updates to the routing module, whereas Table 8 gives “/” for the real-world router update interval and target update rate. Appendix C.4 specifies the actor and critic updates but does not explicitly state whether the router is frozen. The distinction matters when interpreting the real-world results as evidence for the two-stage training procedure. Please state which components were updated so that the real-world results can be interpreted correctly.
Questions for authors
-
How many independent training runs contribute to each entry in Tables 1–3? Please report the per-run AUCs or an uncertainty estimate from those runs, particularly for the online-adaptation and layer-routing ablations in Table 2. This would help determine how much weight to place on the smaller component gains.
-
Is the routing module updated online in the real-world experiments? Please explain the “/” entries in Table 8. If it is updated, what update interval, learning rate, and target-router update rule are used? If it is frozen, please state this explicitly in Appendix C.4.
-
Do Figure 5 and Table 4 use the same runs and the same final 10 autonomous episodes? The RLT USB curve appears to end above the 50% reported in Table 4; the ribbon-cable endpoint also appears different from the tabulated value. Please give the corresponding success counts and check the plotted endpoints, episode windows, and the RLT*/RLT labels. This should be resolvable from the existing episode records.
2 comments
Q3. Do Figure 5 and Table 4 use the same runs and final episode window?
Figure 5 and Table 4 use the same training runs. In Figure 5, we used the first 80 collected trajectories for USB insertion and the first 90 for ribbon-cable insertion. In the original Table 4, we calculated the final success rate using the last ten autonomous episodes in each full recording. Some runs continued beyond the task budget, so we counted episodes that fell outside the budget. We apologize for this error.
We corrected Table 4 by counting successful insertions in the last ten autonomous episodes within each task’s budget. On USB insertion, DSRL succeeded in 4/10 episodes, RLT in 8/10, and eRLT in 9/10. On ribbon-cable insertion, the counts were 3/10, 9/10, and 10/10, respectively. The correction changes RLT’s final success rate from 50% to 80% on USB insertion and from 100% to 90% on ribbon-cable insertion. For DSRL on ribbon-cable insertion, it changes from 50% to 30%. We updated Section 5.4 and corrected the Figure 5 legend to use the same RLT label as Table 4.
The correction narrows the original USB final-SR advantage to 10 percentage points. The AUC advantage is unaffected because the learning curves already used the intended budgets. The corrected results continue to support faster adaptation within these budgets, with a smaller difference in final success. We applied the same cutoff to each repeat. Assisted episodes count toward the interaction budget, and final SR uses the last ten autonomous episodes within that budget (Appendix C.6). Table 12 in Appendix E.2 reports the mean ± sample SD across the original run and two additional runs for each method on each task.
| Task | Method | AUC | Final SR (%) |
|---|---|---|---|
| USB | DSRL | 0.383 ± 0.121 | 56.67 ± 20.82 |
| USB | RLT | 0.326 ± 0.052 | 76.67 ± 5.77 |
| USB | eRLT | 0.615 ± 0.052 | 83.33 ± 5.77 |
| Ribbon Cable | DSRL | 0.413 ± 0.144 | 63.33 ± 35.12 |
| Ribbon Cable | RLT | 0.558 ± 0.118 | 96.67 ± 5.77 |
| Ribbon Cable | eRLT | 0.708 ± 0.039 | 100.00 ± 0.00 |
We hope these results and clarifications address your questions.
Best wishes,
The authors of Submission 29637
Thank you for your careful review and for highlighting the controlled comparisons in Section 4.1, where we varied the source layer or token subset while keeping the VLA and online RL procedure fixed. We also appreciate your close reading of the real-world results and your suggestion to report variation across runs, particularly for the smaller gains in Table 2. Appendix E of the revised paper collects the repeated-run results and training curves, including the additional component ablations.
Q1 and W1. How many independent training runs were used for the main comparisons and component ablations, and how much do the AUCs vary, particularly for the online-adaptation and layer-routing ablations in Table 2? (Appendices E.1, E.2 and E.4)
In the submission, we used one training run per method and task for the main simulation and real-world comparisons, and one per component ablation on Bowl-to-Plate in Table 2. The three seeds reported in Section 4.1 were for the motivating study. We agree that a single run gives little indication of how much a small AUC difference might vary across runs.
We compiled the existing results and completed the remaining runs. We now have three independent training runs per method on all seven simulation tasks and both real-world tasks, and three per component configuration on Bowl-to-Plate and Drawer Opening. The table below gives normalized AUC as the mean ± sample SD for the simulation tasks (Appendix E.1, Table 10) and the real-world tasks (Appendix E.2, Table 12). We have also updated the training curves to show the mean and one sample SD across the three runs for simulation (Appendix E.1, Fig. 10) and real-world insertion (Appendix E.2, Fig. 11). We will incorporate the repeated-run results into the main text.
Across the three runs, eRLT achieved the highest mean AUC on six of the seven simulation tasks and on both real-world tasks. The table below gives the three-run results for the component ablations (Appendix E.2, Table 11).
On Bowl-to-Plate, both gains observed in the original run persisted in the three-run means. The gain from critic-driven adaptation was 0.015 in the original run and 0.018 on average across three runs. The gain from layer-wise routing was 0.021 in the original run and 0.038 on average, with SDs of 0.011 and 0.006, respectively, for full eRLT and the final-layer variant. Both components also improved mean AUC on Drawer Opening (0.048 for critic-driven adaptation and 0.203 for layer-wise routing), and on real-world USB insertion, three runs with layer-wise routing and two without gave a mean AUC gain of 0.128 (Appendix E.4, Table 14 and Fig. 13).
Q2 and W2. Is the real-world router updated online, and what do the “/” entries in Table 8 (Appendix C.2) mean?
The “/” entries noted in the review were unfilled placeholders that we had overlooked. The router was updated online through the critic. Table 8 in Appendix C.2 and the real-world training protocol in Appendix C.4 now specify U = 240 learner updates (once per 15 trajectories), learning rate 3 × 10⁻⁴, and target-router Polyak coefficient 0.005. The latter two match simulation; the VLA remains frozen.
(To stay within the character limit, we’ve included our response to your other questions in the next comment.)