- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes SHARV, a reinforcement learning framework that combines multi-horizon representation and value learning. It introduces Decoupled Dual Representation learning to combine short- and long-horizon representations, and Multi-Start Bootstrapping to provide additional supervision for value learning. The paper presents a theoretical analysis of TD residuals and demonstrates improved aggregate performance over strong baselines across four continuous-control benchmark suites.
Critical strengths
S1. The paper provides extensive evaluations across 34 continuous-control tasks in four benchmark suites, demonstrating strong aggregate performance against recent competitive baselines.
S2. The paper includes component ablations supporting the effectiveness of DDR and MSB, along with theoretical and empirical TD residual analyses supporting MSB design.
Critical weaknesses
W1. Limited methodological novelty. DDR mainly combines existing short- and long-horizon prediction schemes, while MSB resembles the multi-start n-step supervision used in asynchronous n-step Q-learning and A3C. Moreover, although the two components are integrated within the same training pipeline, the paper does not establish a distinct synergistic mechanism between them, making the overall contribution largely incremental.
W2. Insufficient isolation of performance gains. Besides DDR and MSB, SHARV introduces XQC-inspired architectural changes. Although individual component ablations are provided, the paper lacks a baseline removing both components under the same backbone. While Figure 6 compares SS, LL, and SL under the same dual-stream setup, it does not isolate the effect of dual-stream capacity relative to a single-stream model. Furthermore, the dual-stream design may increase model capacity despite preserving the total representation dimension, making the sources of performance improvements difficult to fully disentangle.
Questions for authors
Q1. What are the key methodological contributions of SHARV beyond existing short- and long-horizon representation learning and multi-start supervision in asynchronous n-step Q-learning and A3C? Is there a meaningful connection between DDR and MSB beyond their combination?
Q2. Can the authors provide an ablation removing both DDR and MSB under the same actor-critic backbone, along with parameter counts and computational costs for single- and dual-stream designs, to better isolate the sources of performance gains?
4 comments
W2 / Q2. Insufficient isolation of performance gains
We conducted additional ablation studies and share the results of each experiment below.
- Parameter counts and FLOPs: dual stream (DDR) vs. single stream
To compare the two designs, we report SHARV and SHARV w/o DDR in the tables below. Since observation and action dimensions differ across environments, we report results on Humanoid-Run and Dog-Run from DMC-Hard. The dual-stream design increases representation parameters by 1.44–1.46× and training FLOPs per environment step by 1.33–1.35×.
Parameter counts (Humanoid-Run / Dog-Run)
| Module | SHARV (dual) | w/o DDR (single) | Ratio |
|---|---|---|---|
| Encoder | 3.90M / 4.15M | 2.53M / 2.65M | 1.54× / 1.56× |
| Latent model | 0.17M / 0.17M | 0.30M / 0.30M | 0.56× / 0.56× |
| Total | 4.07M / 4.31M | 2.83M / 2.95M | 1.44× / 1.46× |
FLOPs (forward + backward, measured; Humanoid-Run / Dog-Run)
| Item | SHARV (dual) | w/o DDR (single) | Ratio |
|---|---|---|---|
| SPL update | 30.86G / 32.22G | 18.72G / 19.16G | 1.65× / 1.68× |
| Training per env step (SPL + RL) | 64.45G / 66.33G | 48.33G / 49.03G | 1.33× / 1.35× |
| Inference (one action) | 3.89M / 4.36M | 2.66M / 2.90M | 1.46× / 1.50× |
- SL / SS / LL comparison
To verify that the gain is not simply due to this increased capacity, we compared SL with SS and LL, which share identical parameter counts and compute. This comparison confirms that the performance gain is not simply an ensemble effect of increased parameters.
SS or LL performs well only in certain environments, whereas SL performs consistently well across most environments.
- CKA analysis
To verify that the gain stems from genuinely different representations, we measured the CKA[1, 2] similarity between the two streams in each of SL, SS, and LL.
As training progresses, the two streams in SS and in LL become increasingly similar, whereas the similarity between the two streams in SL gradually decreases. This indicates that the two streams in SL indeed learn different representations.
[1] Song et al. "Feature Selection via Dependence Maximization." JMLR (2012)
[2] Kornblith et al. "Similarity of Neural Network Representations Revisited." ICML (2019)
- Removing both DDR and MSB
To further examine the effect of the actor-critic architecture, we evaluated a variant without both DDR and MSB.
| Suite | Metric | DR.Q | w/o DDR and MSB |
|---|---|---|---|
| Gym MuJoCo | IQM | 1.69 | 1.64 |
| Gym MuJoCo | Mean | 1.61 | 1.58 |
| DMC-Hard | IQM | 0.92 | 0.92 |
| DMC-Hard | Mean | 0.84 | 0.82 |
| HumanoidBench | IQM | 0.43 | 0.41 |
| HumanoidBench | Mean | 0.49 | 0.52 |
As shown above, changing only the architecture yields performance comparable to DR.Q overall. It improves performance in some environments but not in others. We conjecture that self-predictive representation learning already provides a regularization effect similar to that of XQC's normalization. If so, the architectural change mainly adds over-parameterization, which may degrade performance in some environments. In SHARV, this effect does not appear, possibly because SHARV exploits more information from the sampled sequences.
- Controlling for critic supervision
MSB increases the number of supervised state-action pairs by a factor of the sequence length K. A natural question is whether its gain simply comes from this additional supervision.
To control for this, we compared MSB with two variants, both using a K-times larger critic batch. The first simply enlarges the batch without MSB. The second bootstraps over the same range of backup horizons as MSB, but uses non-consecutive samples with horizons drawn uniformly from 1 to K. Both variants underperform SHARV, indicating that neither more samples nor diverse backup horizons alone explain the gain. Instead, the benefit stems from jointly supervising consecutive starting points with a common endpoint, which is consistent with our analysis that this structure controls the within-sequence residuals. In addition, MSB is more efficient in practice, requiring only a single buffer sampling and a single endpoint bootstrap per sequence.
In summary, the performance gains of SHARV do not stem from increased capacity or the architectural change alone. They stem from combining genuinely distinct representations learned under different prediction regimes (DDR) and from jointly supervising consecutive starting points with a common endpoint (MSB).
Dear Reviewer NiEz,
Thank you again for your thoughtful and constructive feedback on our submission. We have posted a detailed response addressing each of your comments, including additional ablations, analysis of representations and clarifications on the methodological novelty.
We would be grateful if you could take a moment to review our rebuttal and let us know whether it addresses your concerns. If you find that our responses have satisfactorily resolved the issues you raised, we would sincerely appreciate it if you would consider revisiting your evaluation of our work. If any points remain unclear, we would be happy to provide further clarification before the discussion period ends.
Thank you for your time and consideration.
Best regards, The Authors
Dear Reviewer NiEz,
Thank you again for your thoughtful and constructive feedback on our submission. We have posted a detailed response addressing each of your comments, including additional ablations, an analysis of the representations, and clarifications on the methodological novelty.
As less than 24 hours remain in the discussion period, we would be grateful if you could take a moment to review our rebuttal and let us know whether it addresses your concerns. If you find that our responses have satisfactorily resolved the issues you raised, we would sincerely appreciate it if you would consider revisiting your evaluation of our work. If any points remain unclear, we would be happy to provide further clarification within the remaining time.
Thank you for your time and consideration.
Best regards, The Authors
We appreciate the reviewer's comments on novelty and the sources of performance gains, and address them below.
W1 / Q1. Methodological novelty and the connection between DDR and MSB
Both components were motivated by the same goal: "Extracting richer supervision from each sampled sequence without additional environment interaction". DDR pursues this goal in representation learning, and MSB pursues it in value learning. We do not claim a direct mechanistic interaction between the two. Rather, we view them as complementary components that address different bottlenecks. The per-task ablations in Appendix F are consistent with this view, as removing each component degrades a largely different set of tasks. We will revise the manuscript to state this framing more clearly rather than implying a synergistic mechanism.
We acknowledge that the multi-start structure itself is not new, and we do not claim it as a contribution. Our motivation was instead to understand why this structure is beneficial in replay-based off-policy learning. In that setting, intermediate pairs would eventually be sampled in later updates, and recent strong baselines rely on single-start or one-step targets. To this end, we showed theoretically that the first-start target error leaves part of the within-sequence TD residuals unconstrained, whereas multi-start target errors bound this part (Prop. 1, Eqs. 16–17). We also confirmed empirically that MSB substantially reduces this residual component even when first-start errors are nearly identical (Table 3), and that removing MSB lowers performance across all proprioceptive suites (Table 2).
Although DDR may appear to be a simple combination, several considerations went into this design decision. We initially considered a weighted average of representations from all prediction horizons. However, this approach raised two concerns: (1) each additional horizon incurs extra computational cost, which becomes excessive, and (2) the weights can only be set heuristically, making them difficult to justify. Instead of weighted averaging, we combined the two most contrasting streams, one-step prediction and fully recursive multi-step prediction up to horizon H, which are learned independently, and let the actor and critic learn how to utilize them. In addition, our SS/LL comparison experiments empirically show that the preferred prediction horizon differs across environments.