acceptodds
Under review as a conference paper at ICLR 2027

SHARV: A Simple Multi-Horizon Approach to Representation and Value Learning in Reinforcement Learning

Abstract

Effective use of collected experience is essential for sample-efficient reinforcement learning. However, existing methods often rely on fixed horizon configurations, limiting their ability to exploit learning signals across environments with different temporal characteristics. We show that short- and long-horizon prediction have environment-dependent effects and that exhaustively combining prediction targets does not consistently improve performance. We present SHARV, a simple approach that exploits multiple horizons for representation and value learning. Our representation learning combines independently learned short- and long-horizon representations to support policy and value learning across diverse environments. For value learning, we characterize internal TD residual components not determined by the first-start target error, motivating Multi-Start Bootstrapping to control these residuals through supervision across multiple starting points. SHARV achieves the highest mean aggregate score among strong baselines on each of four benchmark suites. Ablation studies support the contribution of both components, and observed reductions in internal TD residuals provide empirical support for our theoretical analysis.

Primary area:
reinforcement learning
Keywords:
Reinforcement Learning, Representation, Horizon, Off Policy, Self Predictive Learning, Value

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_NiEzOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes SHARV, a reinforcement learning framework that combines multi-horizon representation and value learning. It introduces Decoupled Dual Representation learning to combine short- and long-horizon representations, and Multi-Start Bootstrapping to provide additional supervision for value learning. The paper presents a theoretical analysis of TD residuals and demonstrates improved aggregate performance over strong baselines across four continuous-control benchmark suites.

Critical strengths

S1. The paper provides extensive evaluations across 34 continuous-control tasks in four benchmark suites, demonstrating strong aggregate performance against recent competitive baselines.

S2. The paper includes component ablations supporting the effectiveness of DDR and MSB, along with theoretical and empirical TD residual analyses supporting MSB design.

Critical weaknesses

W1. Limited methodological novelty. DDR mainly combines existing short- and long-horizon prediction schemes, while MSB resembles the multi-start n-step supervision used in asynchronous n-step Q-learning and A3C. Moreover, although the two components are integrated within the same training pipeline, the paper does not establish a distinct synergistic mechanism between them, making the overall contribution largely incremental.

W2. Insufficient isolation of performance gains. Besides DDR and MSB, SHARV introduces XQC-inspired architectural changes. Although individual component ablations are provided, the paper lacks a baseline removing both components under the same backbone. While Figure 6 compares SS, LL, and SL under the same dual-stream setup, it does not isolate the effect of dual-stream capacity relative to a single-stream model. Furthermore, the dual-stream design may increase model capacity despite preserving the total representation dimension, making the sources of performance improvements difficult to fully disentangle.

Questions for authors

Q1. What are the key methodological contributions of SHARV beyond existing short- and long-horizon representation learning and multi-start supervision in asynchronous n-step Q-learning and A3C? Is there a meaningful connection between DDR and MSB beyond their combination?

Q2. Can the authors provide an ablation removing both DDR and MSB under the same actor-critic backbone, along with parameter counts and computational costs for single- and dual-stream designs, to better isolate the sources of performance gains?

4 comments

We appreciate the reviewer's comments on novelty and the sources of performance gains, and address them below.

W1 / Q1. Methodological novelty and the connection between DDR and MSB

  1. Connection between DDR and MSB

Both components were motivated by the same goal: "Extracting richer supervision from each sampled sequence without additional environment interaction". DDR pursues this goal in representation learning, and MSB pursues it in value learning. We do not claim a direct mechanistic interaction between the two. Rather, we view them as complementary components that address different bottlenecks. The per-task ablations in Appendix F are consistent with this view, as removing each component degrades a largely different set of tasks. We will revise the manuscript to state this framing more clearly rather than implying a synergistic mechanism.

  1. MSB

We acknowledge that the multi-start structure itself is not new, and we do not claim it as a contribution. Our motivation was instead to understand why this structure is beneficial in replay-based off-policy learning. In that setting, intermediate pairs would eventually be sampled in later updates, and recent strong baselines rely on single-start or one-step targets. To this end, we showed theoretically that the first-start target error leaves part of the within-sequence TD residuals unconstrained, whereas multi-start target errors bound this part (Prop. 1, Eqs. 16–17). We also confirmed empirically that MSB substantially reduces this residual component even when first-start errors are nearly identical (Table 3), and that removing MSB lowers performance across all proprioceptive suites (Table 2).

  1. DDR

Although DDR may appear to be a simple combination, several considerations went into this design decision. We initially considered a weighted average of representations from all prediction horizons. However, this approach raised two concerns: (1) each additional horizon incurs extra computational cost, which becomes excessive, and (2) the weights can only be set heuristically, making them difficult to justify. Instead of weighted averaging, we combined the two most contrasting streams, one-step prediction and fully recursive multi-step prediction up to horizon H, which are learned independently, and let the actor and critic learn how to utilize them. In addition, our SS/LL comparison experiments empirically show that the preferred prediction horizon differs across environments.

W2 / Q2. Insufficient isolation of performance gains

We conducted additional ablation studies and share the results of each experiment below.

 

  1. Parameter counts and FLOPs: dual stream (DDR) vs. single stream

To compare the two designs, we report SHARV and SHARV w/o DDR in the tables below. Since observation and action dimensions differ across environments, we report results on Humanoid-Run and Dog-Run from DMC-Hard. The dual-stream design increases representation parameters by 1.44–1.46× and training FLOPs per environment step by 1.33–1.35×.

Parameter counts (Humanoid-Run / Dog-Run)

ModuleSHARV (dual)w/o DDR (single)Ratio
Encoder3.90M / 4.15M2.53M / 2.65M1.54× / 1.56×
Latent model0.17M / 0.17M0.30M / 0.30M0.56× / 0.56×
Total4.07M / 4.31M2.83M / 2.95M1.44× / 1.46×

FLOPs (forward + backward, measured; Humanoid-Run / Dog-Run)

ItemSHARV (dual)w/o DDR (single)Ratio
SPL update30.86G / 32.22G18.72G / 19.16G1.65× / 1.68×
Training per env step (SPL + RL)64.45G / 66.33G48.33G / 49.03G1.33× / 1.35×
Inference (one action)3.89M / 4.36M2.66M / 2.90M1.46× / 1.50×

 

  1. SL / SS / LL comparison

To verify that the gain is not simply due to this increased capacity, we compared SL with SS and LL, which share identical parameter counts and compute. This comparison confirms that the performance gain is not simply an ensemble effect of increased parameters.

SS/LL/SL Ablation Results

SS or LL performs well only in certain environments, whereas SL performs consistently well across most environments.

 

  1. CKA analysis

To verify that the gain stems from genuinely different representations, we measured the CKA[1, 2] similarity between the two streams in each of SL, SS, and LL.

CKA Analysis Results

As training progresses, the two streams in SS and in LL become increasingly similar, whereas the similarity between the two streams in SL gradually decreases. This indicates that the two streams in SL indeed learn different representations.

[1] Song et al. "Feature Selection via Dependence Maximization." JMLR (2012)

[2] Kornblith et al. "Similarity of Neural Network Representations Revisited." ICML (2019)

 

  1. Removing both DDR and MSB

To further examine the effect of the actor-critic architecture, we evaluated a variant without both DDR and MSB.

DDR, MSB Ablation Results

SuiteMetricDR.Qw/o DDR and MSB
Gym MuJoCoIQM1.691.64
Gym MuJoCoMean1.611.58
DMC-HardIQM0.920.92
DMC-HardMean0.840.82
HumanoidBenchIQM0.430.41
HumanoidBenchMean0.490.52

As shown above, changing only the architecture yields performance comparable to DR.Q overall. It improves performance in some environments but not in others. We conjecture that self-predictive representation learning already provides a regularization effect similar to that of XQC's normalization. If so, the architectural change mainly adds over-parameterization, which may degrade performance in some environments. In SHARV, this effect does not appear, possibly because SHARV exploits more information from the sampled sequences.

 

  1. Controlling for critic supervision

MSB increases the number of supervised state-action pairs by a factor of the sequence length K. A natural question is whether its gain simply comes from this additional supervision.

Critic Ablation Results

To control for this, we compared MSB with two variants, both using a K-times larger critic batch. The first simply enlarges the batch without MSB. The second bootstraps over the same range of backup horizons as MSB, but uses non-consecutive samples with horizons drawn uniformly from 1 to K. Both variants underperform SHARV, indicating that neither more samples nor diverse backup horizons alone explain the gain. Instead, the benefit stems from jointly supervising consecutive starting points with a common endpoint, which is consistent with our analysis that this structure controls the within-sequence residuals. In addition, MSB is more efficient in practice, requiring only a single buffer sampling and a single endpoint bootstrap per sequence.

 

In summary, the performance gains of SHARV do not stem from increased capacity or the architectural change alone. They stem from combining genuinely distinct representations learned under different prediction regimes (DDR) and from jointly supervising consecutive starting points with a common endpoint (MSB).

Dear Reviewer NiEz,

Thank you again for your thoughtful and constructive feedback on our submission. We have posted a detailed response addressing each of your comments, including additional ablations, analysis of representations and clarifications on the methodological novelty.

We would be grateful if you could take a moment to review our rebuttal and let us know whether it addresses your concerns. If you find that our responses have satisfactorily resolved the issues you raised, we would sincerely appreciate it if you would consider revisiting your evaluation of our work. If any points remain unclear, we would be happy to provide further clarification before the discussion period ends.

Thank you for your time and consideration.

Best regards, The Authors

Dear Reviewer NiEz,

Thank you again for your thoughtful and constructive feedback on our submission. We have posted a detailed response addressing each of your comments, including additional ablations, an analysis of the representations, and clarifications on the methodological novelty.

As less than 24 hours remain in the discussion period, we would be grateful if you could take a moment to review our rebuttal and let us know whether it addresses your concerns. If you find that our responses have satisfactorily resolved the issues you raised, we would sincerely appreciate it if you would consider revisiting your evaluation of our work. If any points remain unclear, we would be happy to provide further clarification within the remaining time.

Thank you for your time and consideration.

Best regards, The Authors

Reviewer_zfKyOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes SHARV, a continuous-control method that builds on DR.Q's representation learning and an XQC-inspired actor-critic, with two modifications aimed at sample efficiency. First, it trains 2 independent representations, one trained with one-step teacher-forced latent prediction (short) and another with autoregressive multi-step predictions (long). The short and long representations are concatenated together and used as inputs to the actor and critic. Second, the critic is trained on every starting point of a sampled K-step sequence, with n-step targets of decreasing length that all share a single endpoint bootstrap (of the last observation in the sequence). The paper shows that controlling the target errors at all starting points bounds the within-sequence one-step TD residuals, which the first-start target error alone does not. SHARV is evaluated on Gym MuJoCo, DMC-Hard, HumanoidBench, and pixel-based DMC, with ablations of both components.

Critical strengths
  1. The paper is well organized and easy to follow. The problem is well motivated and is demonstrated at the right place. The text and equations are easy to understand and the diagrams are clean.

  2. The paper considers a lot of relevant and SOTA baselines, and uses a wide range of environments to evaluate the proposed method. In addition, the ablations and the choice of hyper-parameters are clear.

Critical weaknesses
  1. Novelty of the 2 proposed modifications. The two modifications the paper proposes are not completely novel on their own. Short and long latent representations have been explored in the past [1]. Similarly, as the paper itself points out in Line 256, the multi-step bootstrapping (MSB) has also been used in the past [2]. The paper should reframe the contributions and novelty claims to be the decoupled-stream design and its integration into an off-policy actor-critic.

[1] Hafner, Danijar, et al. "Learning Latent Dynamics for Planning from Pixels." International Conference on Machine Learning (2019)

[2] Mnih, Volodymyr, et al. "Asynchronous Methods for Deep Reinforcement Learning." International Conference on Machine Learning (2016)

Questions for authors
  1. Could the SS/LL/SL comparison (Fig. 6) be done on other tasks, given that it is the main evidence for combining 2 different representations? In addition, what does the 2 ‘short’s in short-short mean? Is it simply 2 independent encoders trained using the one-step teacher-forced loss?

  2. Can the authors show that the short and long streams actually learn different information (via some method to compare different representations)? This could support the performance gain by using 2 different streams.

  3. In Proposition 1, it seems quite immediate that the first-start target error does not bound the intermediate residuals. What is the key takeaway that warrants presenting it as a formal proposition with a proof?

5 comments

We thank the reviewer for the thoughtful feedback and constructive suggestions. We address each question and concern below.

Q1. Additional SS/LL/SL ablations and Clarification of SS

We have extended the SS/LL/SL ablation study to all 24 proprioceptive tasks. The additional results are available at the following link:

SS/LL/SL Ablation Results

In particular, the DMC-Hard results show a consistent pattern: LL performs better across all dog tasks, while SS performs better across all humanoid tasks (except SL). This further supports the environment-dependent effectiveness of the two prediction objectives.

As the reviewer correctly understood, SS consists of two short-horizon streams with identical architectures but independent parameters, both trained using the one-step teacher-forced prediction objective.

Since the streams have independent parameters, SS may benefit from an ensemble effect. We included SS and LL as controls to examine whether the gains of SL come simply from combining independently parameterized networks or from the diversity induced by different prediction objectives.

Q2. Do the short and long streams actually learn different representations?

We thank the reviewer for this incisive question. To examine whether the two streams learn different representations, we measured their similarity throughout training using CKA with an unbiased estimator [1, 2].

We conducted this analysis for SL(ours), SS, and LL on humanoid-run and dog-run.

CKA Analysis Results

The results show a clear difference. In SS and LL, the similarity between the two streams steadily increases during training. In contrast, CKA similarity in SL decreases substantially, suggesting that the different prediction objectives lead to increasingly distinct representations.

Q3. Key takeaway of Proposition 1

We agree with the reviewer that Proposition 1 is relatively elementary. However, its purpose is not only to show that the first-start target error cannot bound the intermediate residuals.

It motivates the orthogonal decomposition in Equation (16), which separates the residual component determined by the first-start error from the component left unconstrained. This leads to Equation (17), which shows how multi-start supervision bounds these residuals.

Nevertheless, we acknowledge that Proposition 1 itself is straightforward. We will consider presenting it as a lemma or an observation, while placing greater emphasis on the decomposition and the resulting bound.

W1. Limited Methodological Novelty

We appreciate the reviewer's comments on novelty and would like to clarify the motivation behind DDR.

First, we acknowledge that learning predictive representations across multiple horizons is not new. PlaNet[3] explores multi-step latent prediction, which is also reflected in the autoregressive prediction objective of our long stream.

Our hidden motivation for DDR was how to effectively combine representations learned under different prediction horizons.

We initially considered averaging representations from multiple horizons (all possible cases) using heuristic weights. However, this approach raised two concerns: (1) additional computational overhead from handling multiple horizon-specific representations, and (2) the lack of a clear justification for choosing the weights.

Instead, we adopted a simpler design that decouples two contrasting prediction regimes: one-step prediction and fully recursive multi-step prediction up to horizon H. The two streams are trained independently, and their representations are concatenated. This allows the downstream actor and critic to learn automatically how to utilize each representation without manually specified weights.

Nevertheless, we agree that the novelty claims could be framed more precisely. We will consider revising the manuscript to better distinguish our contributions from existing approaches.

References

[1] Song et al. "Feature Selection via Dependence Maximization." JMLR (2012)

[2] Kornblith et al. "Similarity of Neural Network Representations Revisited." ICML (2019)

[3] Hafner, Danijar, et al. "Learning Latent Dynamics for Planning from Pixels." ICML (2019)

Dear Reviewer zfKy,

Thank you again for your thoughtful and constructive feedback on our submission. We have posted a detailed response addressing each of your comments, including additional ablation and analysis of representations.

We would be grateful if you could take a moment to review our rebuttal and let us know whether it addresses your concerns. If any points remain unclear, we would be happy to provide further clarification before the discussion period ends.

Thank you for your time and consideration.

Best regards, The Authors

Dear Reviewer zfKy,

Thank you again for your thoughtful and constructive feedback on our submission. We have posted a detailed response addressing each of your comments, including additional ablation studies and an analysis of the representations.

As less than 24 hours remain in the discussion period, we would be grateful if you could take a moment to review our rebuttal and let us know whether it addresses your concerns. If any points remain unclear, we would be happy to provide further clarification within the remaining time.

Thank you for your time and consideration.

Best regards, The Authors

Reviewer_rgitOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes SHARV, an off-policy actor-critic method that uses multiple horizons for representation and value learning. Decoupled Dual Representation (DDR) independently trains a teacher-forced one-step prediction stream and an autoregressive multi-step prediction stream, then concatenates their features for policy and value estimation while preserving the combined representation dimension. Multi-Start Bootstrapping (MSB) supervises every starting point in a replay sequence using returns of different lengths and a common bootstrap endpoint. The implementation combines these components with an XQC-inspired normalized actor-critic backbone and categorical critics. The theoretical analysis establishes that fitting only the first-start target does not determine all within-sequence residual components, whereas controlling errors across all starts bounds those residuals. This is a local consistency result, not a guarantee of policy-value accuracy or convergence. Evaluation covers 34 continuous-control tasks across Gym MuJoCo, DMC-Hard, HumanoidBench with hands, and DMC-Visual, using 10 seeds. SHARV achieves the highest reported aggregate mean in all four suites, with the largest improvement on HumanoidBench: 0.662 versus 0.489 for DR.Q. Component ablations cover all proprioceptive tasks, and additional experiments examine prediction-stream configurations, residuals, and training time.

Critical strengths
  1. Broad empirical improvements against strong baselines. The evaluation includes 34 tasks, both proprioceptive and visual observations, and 10 seeds, with complete per-task results and uncertainty estimates. SHARV’s highest aggregate mean is corroborated by the IQM results rather than depending solely on one aggregation statistic. The HumanoidBench improvement is particularly compelling: the mean increases from 0.489 for DR.Q to 0.662, and the IQM from approximately 0.43 to 0.60. DMC-Hard also shows a clear aggregate improvement. The visual advantage is modest, and its confidence intervals overlap with DR.Q’s, but the broader empirical evidence provides a meaningful reason for acceptance.
  2. A coherent design with substantive component-level validation. Separating short- and long-horizon prediction into independently trained streams is a simple way to avoid requiring a single representation to satisfy both objectives. The SS/LL/SL comparison directly examines their complementary effects on two tasks. More importantly, removing either DDR or MSB lowers aggregate performance across all three proprioceptive suites under the same actor-critic backbone. For example, removing MSB reduces HumanoidBench’s mean from 0.662 to 0.568. The correctly scoped residual analysis and accompanying diagnostics provide an interpretable motivation beyond a purely empirical combination of components.
Critical weaknesses
  1. The ablations do not fully separate multi-horizon structure from increased capacity and supervision. Appendix C.1 retains each stream’s encoder hidden width while halving its output dimension. Matching the concatenated representation dimension therefore does not establish parameter-count matching. Furthermore, SHARV changes the actor-critic architecture, normalization, and optimization relative to DR.Q. The existing same-backbone component ablations are useful, but a variant with both DDR and MSB disabled is missing, leaving the backbone contribution and component interactions incompletely characterized. The SS/LL/SL control helps address the two-stream explanation, but is limited to two DMC-Hard tasks. MSB also supervises approximately K times as many state-action positions per update as the single-start ablation when sequence count is fixed; the default K is three. Averaging the losses does not equalize this supervision budget. Thus, the results do not fully distinguish the value of the multi-start/common-endpoint structure from simply training on more critic examples. Appendix E appropriately reports the computational cost: 0.030 seconds per step for SHARV versus 0.020 for DR.Q and 0.015 for XQC on humanoid-run. This does not invalidate interaction-based sample efficiency, but a matched-supervision or matched-compute control would strengthen the central explanation of the gains.
  2. The residual mechanism evidence has a limited connection to the implemented categorical objective. Equations (14) and (19) explicitly define the scalar target errors and residual measurements, including the common endpoint. Equations (16)–(17) concern errors relative to unprojected multi-step targets satisfying this endpoint recursion. The implemented critic instead fits categorical targets projected onto the fixed support [-5, 5]. If projection clips mass outside that support, it can change target means, so fitting projected targets does not automatically control the scalar errors appearing in the bound. The paper does not report clipping frequency or the resulting mean bias, leaving the practical importance of this potential mismatch unclear.

Moreover, Table 3 compares agents using their own replay distributions and evolving policies, rather than a shared fixed dataset. Although training uses a running reward scale, I could not verify whether the reported diagnostic errors remain in agent-specific normalized units or have been converted to common reward units; this requires clarification rather than an assumption that their units differ. The authors appropriately describe these measurements as online consistency rather than true value accuracy. Nevertheless, the diagnostics alone do not establish residual reduction as the mechanism responsible for performance gains. This limits the explanatory strength of the contribution without contradicting the algebraic result.

Questions for authors

The following are clarification and targeted validation requests that could materially affect my assessment, rather than suggestions for unrelated future work.

  1. With the same number of supervised critic positions or comparable computation, how does MSB compare with a single-start method using a larger batch? Can a control with the same marginal backup-length distribution distinguish denser supervision from the common-endpoint, multi-start structure?
  2. What are the parameter counts of the single- and dual-stream variants? Can you provide a same-backbone configuration with both DDR and MSB disabled, and evidence that the SL advantage over SS/LL extends beyond the two tasks in Figure 6? This would clarify the architectural contribution and the generality of complementary horizon objectives.
  3. How often does categorical projection clip target mass outside [-5, 5], and what is the resulting change in target means? If clipping is appreciable, how should the projected categorical training loss be related to the errors relative to the unprojected scalar targets in Equation (17)?
  4. Are Table 3’s errors expressed in each agent’s independently normalized reward units? Does the residual advantage persist on a shared fixed replay set with a common reward scale, or after conversion to raw reward units? This would materially strengthen the proposed mechanism evidence.
5 comments

We thank the reviewer for the sharp and insightful review, which has helped us strengthen the paper considerably.

 

W1 / Q1 / Q2. Insufficient ablations of performance gains

  1. Parameter counts and FLOPs: dual stream (DDR) vs. single stream

To compare the two designs, we report SHARV and SHARV w/o DDR in the tables below. Since observation and action dimensions differ across environments, we report results on Humanoid-Run and Dog-Run from DMC-Hard. The dual-stream design increases representation parameters by 1.44–1.46× and training FLOPs per environment step by 1.33–1.35×.

Parameter counts (Humanoid-Run / Dog-Run)

ModuleSHARV (dual)w/o DDR (single)Ratio
Encoder3.90M / 4.15M2.53M / 2.65M1.54× / 1.56×
Latent model0.17M / 0.17M0.30M / 0.30M0.56× / 0.56×
Total4.07M / 4.31M2.83M / 2.95M1.44× / 1.46×

FLOPs (forward + backward, measured; Humanoid-Run / Dog-Run)

ItemSHARV (dual)w/o DDR (single)Ratio
SPL update30.86G / 32.22G18.72G / 19.16G1.65× / 1.68×
Training per env step (SPL + RL)64.45G / 66.33G48.33G / 49.03G1.33× / 1.35×
Inference (one action)3.89M / 4.36M2.66M / 2.90M1.46× / 1.50×

 

  1. SL / SS / LL comparison

To verify that the gain is not simply due to this increased capacity, we compared SL with SS and LL, which share identical parameter counts and compute. This comparison confirms that the performance gain is not simply an ensemble effect of increased parameters.

SS/LL/SL Ablation Results

SS or LL performs well only in certain environments, whereas SL performs consistently well across most environments.

 

  1. CKA analysis

To verify that the gain stems from genuinely different representations, we measured the CKA[1, 2] similarity between the two streams in each of SL, SS, and LL.

CKA Analysis Results

As training progresses, the two streams in SS and in LL become increasingly similar, whereas the similarity between the two streams in SL gradually decreases. This indicates that the two streams in SL indeed learn different representations.

[1] Song et al. "Feature Selection via Dependence Maximization." JMLR (2012)

[2] Kornblith et al. "Similarity of Neural Network Representations Revisited." ICML (2019)

 

  1. Removing both DDR and MSB

To further examine the effect of the actor-critic architecture, we evaluated a variant without both DDR and MSB.

DDR, MSB Ablation Results

SuiteMetricDR.Qw/o DDR and MSB
Gym MuJoCoIQM1.691.64
Gym MuJoCoMean1.611.58
DMC-HardIQM0.920.92
DMC-HardMean0.840.82
HumanoidBenchIQM0.430.41
HumanoidBenchMean0.490.52

As shown above, changing only the architecture yields performance comparable to DR.Q overall. It improves performance in some environments but not in others. We conjecture that self-predictive representation learning already provides a regularization effect similar to that of XQC's normalization. If so, the architectural change mainly adds over-parameterization, which may degrade performance in some environments. In SHARV, this effect does not appear, possibly because SHARV exploits more information from the sampled sequences.

 

  1. Controlling for critic supervision

MSB increases the number of supervised state-action pairs by a factor of the sequence length K. A natural question is whether its gain simply comes from this additional supervision.

Critic Ablation Results

To control for this, we compared MSB with two variants, both using a K-times larger critic batch. The first simply enlarges the batch without MSB. The second bootstraps over the same range of backup horizons as MSB, but uses non-consecutive samples with horizons drawn uniformly from 1 to K. Both variants underperform SHARV, indicating that neither more samples nor diverse backup horizons alone explain the gain. Instead, the benefit stems from jointly supervising consecutive starting points with a common endpoint, which is consistent with our analysis that this structure controls the within-sequence residuals. In addition, MSB is more efficient in practice, requiring only a single buffer sampling and a single endpoint bootstrap per sequence.

 

In summary, the performance gains of SHARV do not stem from increased capacity or the architectural change alone. They stem from combining genuinely distinct representations learned under different prediction regimes (DDR) and from jointly supervising consecutive starting points with a common endpoint (MSB).

W2 / Q3. Categorical projection and the residual bound

The projected target mean deviates from the scalar target only through clipping outside [-5, 5]. To quantify this, we measured the fraction of clipped target mass and the resulting shift in target means during training on humanoid-run.

Clipping Measurement Results

Clipping occurs only transiently in early training, peaking at about 4% of target mass around 100k steps, likely before the running reward scale stabilizes. It drops to nearly zero after about 200k steps and remains negligible thereafter. The resulting mean shift peaks at about 6 × 10⁻³ and likewise vanishes after about 200k steps. In particular, clipping is negligible throughout both intervals reported in Table 3 (200k–400k and 800k–1M steps). Therefore, the projected targets closely match the scalar targets in Eq. 17, and this potential mismatch has little practical effect on our analysis.

W3 / Q4. Units and residual diagnostics

  1. Units and comparison

Yes, the errors in Table 3 are expressed in each agent's independently normalized units. Each agent divides rewards by a running scale c_t = √Var(G) computed from its own returns, and the diagnostics are measured in the same units. Converting to raw reward units multiplies all squared errors at a given time by the same factor c_t². Since agents learn values of different magnitudes, the absolute errors depend on each agent's scale. In contrast, the ratio E⊥ / E_first is largely insensitive to the choice of units, because its numerator and denominator share the same c_t at each time and the factor cancels. This ratio therefore remains essentially unchanged after conversion to raw reward units.

Based on Table 3, this ratio for SHARV is 0.51 in the early interval (200k–400k) and 0.53 in the late interval (800k–1M), compared with 1.21 and 1.51 for SHARV w/o MSB. That is, the ratio for SHARV is 0.43× (early) and 0.35× (late) that of the variant. Thus, largely independent of the choice of units, SHARV's orthogonal residual component relative to its first-start error is less than half that of SHARV w/o MSB. We will add this ratio to Table 3.

  1. Evaluation on a shared fixed replay set

Evaluating trained critics on a shared dataset is computationally feasible. However, such data would be out of distribution for at least one agent, so the results would be dominated by out-of-distribution generalization error rather than the bootstrapping consistency during training. Conversely, training both agents on the same fixed dataset would turn the problem into offline RL, which differs from the online setting that SHARV addresses. In the online setting, each method inevitably shapes its own replay distribution, which is why we describe these measurements as online consistency. To ensure that the difference is not an artifact of particular runs, we used 10 independent seeds per agent, and the results above are averaged over these seeds.

Furthermore, these results are consistent with the critic controls in our response to W1/Q1. There, a variant that matches both the supervision budget and the backup-length distribution of MSB, but without the common endpoint, underperforms SHARV.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.