- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes V-SHARE, an advantage-based value-learning algorithm that combines cross-action value sharing with an importance-weighted Bellman-residual correction. The authors provide asymptotic and finite-time convergence guarantees under Markovian sampling, including decreasing- and two-timescale variants, and evaluate the method on tabular MDPs and deep RL benchmarks.
Critical strengths
1.The paper addresses a concrete theoretical difficulty introduced by cross-action value sharing and provides a coherent convergence analysis for the resulting stochastic updates. In particular, V-SHARE preserves the shared value component while adding an importance-weighted residual correction, and establishes both almost-sure and finite-time guarantees under Markovian sampling.
- The analysis goes beyond a single convergence result by characterizing the role of , developing decreasing- schedules, and providing a two-timescale formulation that removes the single-timescale threshold under timescale separation. These results give a broader picture of the stability–noise tradeoff and the different regimes in which V-SHARE can be analyzed.
Critical weaknesses
1.The main tabular experiments used to illustrate the single-timescale contraction–variance tradeoff are not aligned with the stated sufficient contraction regime. In the reported setup, , there are five actions, and , giving ; Lemma B.2 therefore requires , whereas Figure 1 uses . The experiments demonstrate an interesting empirical trend, but they do not directly validate the regime covered by the corresponding theory.
2.The paper does not fully disentangle the effect of cross-action value sharing from the importance-weighted residual correction. Since the rare-action improvement in the mean dynamics is closely tied to the importance weighting, the mechanism behind the claimed acceleration should be stated more carefully.
Questions for authors
1.Could the authors clarify how the tabular experiments relate to the sufficient contraction condition, given that the reported values appear to lie outside the regime guaranteed by Lemma B.2?
- Could the authors clarify which part of the rare-action improvement should be attributed specifically to cross-action value sharing, rather than to the importance-weighted residual correction?
4 comments
Author Response Part 2
Q1. Could the authors clarify how the tabular experiments relate to the sufficient contraction condition, given that the reported values appear to lie outside the regime guaranteed by Lemma B.2?
We acknowledge that the condition in Lemma B.2 is sufficient rather than necessary, and our experimental observations suggest that it can be conservative. We deliberately chose smaller values of in the original tabular experiments to illustrate the contraction–variance tradeoff more clearly. In particular, varying in this regime allows us to better distinguish its effects on transient convergence and the steady-state error floor.
We observe the same qualitative contraction–variance tradeoff for larger values of satisfying the sufficient condition of Lemma B.2. We have included these additional experiments in Appendix G of the updated manuscript. Thus, the original and additional experiments illustrate the behavior of V-SHARE both outside and within the theoretically guaranteed contraction regime.
Q2. Could the authors clarify which part of the rare-action improvement should be attributed specifically to cross-action value sharing, rather than to the importance-weighted residual correction?
We thank the reviewer for raising this important distinction. The two mechanisms play complementary roles in V-SHARE. The limiting ODE dynamics decompose as
\underbrace{DB(\mathcal TQ-Q)}_{\text{cross-action value sharing}} + \underbrace{\beta D(\mathcal TQ-Q)}_{\text{importance-weighted residual correction}}.$$ The first term propagates information across actions, whereas the second removes the dependence on behavior action probabilities in the diagonal component. The cross-action coupling introduced by value sharing prevents the standard sup-norm contraction argument, while the importance-weighted correction enables us to establish contraction for sufficiently large $\beta$. In particular, the removal of the multiplicative rare-action factor $b_{\min}$ from the leading-order contraction gap in our theoretical comparison is attributable to the importance-weighted residual correction. Moreover, Figure 2(b) partially isolates the contribution of value sharing by varying $r=\alpha_V/\alpha_A$, where $r=0$ removes value sharing while retaining the importance-weighted correction. Increasing $r$ generally accelerates transient convergence, with comparable, occasionally slightly higher, steady-state error floors. Thus, the two mechanisms play complementary roles: cross-action value sharing is central to V-SHARE and contributes to the empirical transient acceleration observed in our experiments, while residual preconditioning enables rigorous convergence guarantees for the combined dynamics. As acknowledged in Section 5, theoretically characterizing the additional acceleration attributable specifically to value sharing, also observed in Dueling Q-learning, remains an interesting open direction.Post-rebuttal assessment
Thank you for the response and the additional experiments in Appendix G. The rebuttal addresses my two main concerns.
- Alignment between theory and tabular experiments (W1 & Q1): The authors acknowledge that the original tabular experiments used parameters outside the sufficient condition in Lemma B.2. The additional evaluations at larger residual weights, including , and with the prescribed two-timescale schedules strengthen the connection between the analysis and the tabular results. They also provide further evidence for the contraction–variance trade-off.
- Value sharing versus importance weighting (W2 & Q2): The response makes the attribution clearer: removing the dependence from the leading contraction term comes from the importance-weighted residual correction, while varying provides empirical evidence that value sharing can further accelerate transient learning. Please carry this distinction into the revised main text.
I continue to view the convergence analysis of coupled value-sharing dynamics under Markovian sampling as the paper’s main contribution. Establishing both asymptotic and finite-sample guarantees is valuable. The deep RL implementation also uses replay buffers, clipping, and regularization beyond the analyzed recursion; the final version should clearly distinguish these implementation choices from the tabular guarantees and streamline the theoretical exposition.
The clarifications and additional experiments support my original assessment. I maintain my rating of 3: Weak acceptance.
Acknowledgement of Reviewer Feedback
We thank the reviewer for their positive assessment of our work and for their constructive feedback throughout the discussion. We are glad that our clarifications and additional experiments have addressed their main concerns.
We will incorporate the suggested improvements into the revised manuscript, particularly by clarifying the distinct roles of cross-action value sharing and importance-weighted residual correction, distinguishing the tabular theoretical guarantees from the neural implementation, and streamlining the theoretical exposition.
Author Response Part 1
We thank the reviewer for their positive assessment of our contributions and for their constructive questions. We address each question below.
W1. Mismatch between the experimental parameters and the sufficient contraction regime.
We acknowledge that the values of β used in the original tabular experiments lie outside the sufficient contraction regime of Lemma B.2. These values were chosen to illustrate the contraction–variance tradeoff more clearly, and the sufficient condition should not be interpreted as necessary for convergence.
To address this concern, we conduct additional experiments with larger values of β satisfying the theoretical contraction condition. As shown in Appendix G, we observe the same qualitative contraction–variance tradeoff in this regime. We discuss these results further in our response to Q1.
W2. Distinguishing value sharing from importance-weighted residual correction.
We appreciate the reviewer's suggestion to clarify the respective roles of the two mechanisms. The central motivation behind V-SHARE is to retain the empirical benefits of cross-action value sharing, which have also been observed in existing advantage-based algorithms, while developing rigorous convergence guarantees for the resulting coupled dynamics.
Value sharing is thus an essential component of the proposed algorithm. However, the cross-action coupling introduced by value sharing prevents the standard sup-norm contraction argument used for asynchronous Q-learning. Our residual-preconditioning approach is designed to overcome this analytical difficulty, enabling us to establish a sufficient contraction regime for the combined dynamics and derive asymptotic and finite-time convergence guarantees under Markovian sampling.
Furthermore, Figure 2(b) provides complementary empirical evidence for the contribution of value sharing. Increasing the value-sharing timescale ratio generally accelerates transient convergence while maintaining comparable steady-state error floors. Thus, the experiments suggest an additional benefit of value sharing beyond the convergence guarantees captured by our current analysis. We clarify the respective roles of the two mechanisms further in our response to Q2.
We continue our response in the next official comment, Author Response Part 2.