- Rating
- 4
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper studies mode connectivity and weight-space ensembling in value-based reinforcement learning. This paper first investigate when Q-networks can be linearly connected in parameter space and show that shared initialization can help achieve such connectivity, while some RL-specific factors, such as different discount factors, may break it. Based on these observations, the authors propose Q-Soup, which averages the parameters of multiple finetuned Q-networks into a single model to reduce inference cost. The method is evalauted on Atari environments with several value-based RL algorithms, including DQN, PQN, and MINTO. The experimental results shows that Q-Soup generally performs better than individual finetuned models and achieves comparable performance to output-space ensembles, while only requiring a single network during inference. This paper also propose Proxy Q-Soup to handle cases where some models are not well connected.
Critical strengths
- The paper systematically investigates mode connectivity in value-based RL, including the conditions where weight averaging works and when it may fail. The analysis provides useful insights into how Q-networks behave in parameter space, which has been less explored comparing to supervised learning.
- Q-Soup provides a simple and practical way to improve the performance of individual Q-networks while maintaining a single model at inference. The experiments are extensive, covering multiple value-based RL algorithms, Atari environments, and different training configurations. The results provide strong empirical suport for the proposed approach, which makes this work a valuable contribution
Critical weaknesses
I do not have substantial concerns about this paper. Overall, I think this is a good practical application of existing techniques to value-based RL, supported by extensive experiments and empirical analysis. The main limitation is the relatively limited algorithmic novelty, as Q-Soup mainly applies existing model averaging techniques to value-based RL. Similarly, several observations about mode connectivity, including the importance of shared initialization, are already well established in previous work.
Questions for authors
The paper claims that Q-Soup achieves comparable performance to output-space ensembles. However, the experimental results seem to show that Q-Soup generally performs slightly worse than output-space ensembles, although it outperforms individual models. Could the authors clarify this claim or use a more precise description of the performance trade-off?
2 comments
Reviewer Vzxy (2/2)
Q1. The paper claims that Q-Soup achieves comparable performance to output-space ensembles. However, the experimental results seem to show that Q-Soup generally performs slightly worse than output-space ensembles, although it outperforms individual models. Could the authors clarify this claim or use a more precise description of the performance trade-off?
We thank the reviewer for this question and agree that a more precise description is needed. By "comparable performance", we meant that Q-Soup offers a favourable trade-off: on average, it recovers most of the improvement of the output-space ensemble over individual models, while requiring a single model at inference time. We did not intend to claim that Q-Soup matches the Output Ensemble in general, and the reviewer is correct that it is typically slightly below it.
To make this trade-off precise, in the homogeneous setting (Fig. 6), Q-Soup trails the Output Ensemble by at most 7.7% across DQN, PQN and MINTO, while recovering 60–89% of the performance gap between the average finetuned model and the Output Ensemble, and it outperforms the best individual finetuned model by at least 12.9%. In the heterogeneous setting (Fig. 7), Q-Soup closely matches the Output Ensemble for PQN and recovers 63–77% of its improvement over the average finetuned model for DQN. When applied to ensemble-based methods, Q-Soup comes within 1.8% and 3.3% of the output-space aggregation of the same shared-initialization members for Maxmin DQN and Online REM, respectively. Compared with the original methods using independent initialization, Q-Soup outperforms Maxmin DQN by 3.5% and trails Online REM by 5.8% (Fig. 8(b), newly added Fig. 18). In return, Q-Soup keeps the memory and compute of a single model, whereas the Output Ensemble grows linearly with the number of models (newly added Sec. A.6). In the revised version, we describe this trade-off more precisely (Abstract, Sec. 1 and Sec. 7): Q-Soup approaches the performance of output-space ensembles, while recovering most of their improvement over individual models and requiring only a single model at inference time.
Reviewer Vzxy (1/2)
We thank the reviewer for the thorough review and the positive assessment of our work, and we appreciate that the reviewer found our analysis of mode connectivity in value-based RL insightful and our experiments extensive. We apologize for the delayed response, as we aimed to address all the points raised with the limited resources available to us. All changes are included in the revised manuscript and highlighted in blue.
We thank the reviewer for the positive assessment. We agree that Q-Soup builds on existing model averaging techniques, as acknowledged in the manuscript, and that the role of shared initialization in mode connectivity is well established in supervised learning. Our contribution therefore lies elsewhere and has two parts: a systematic analysis of how Q-networks relate in parameter space through the lens of mode connectivity, and a practical method that exploits this structure to reduce the inference cost of ensembles in value-based RL. Both are driven by a concrete need. Scientifically, online value-based RL involves components absent from supervised learning, namely bootstrapped targets and data that shift with the policy, and it was unclear whether mode connectivity survives them. Practically, ensembles are widely used in value-based RL, yet their inference cost grows linearly with the number of models.
On the analysis side, our observations progress from the benefit of output-space ensembles (Observation 4.1) and the failure of naive weight averaging (Observation 4.2) to the conditions under which averaging succeeds (Observation 4.3) and under which connectivity emerges (Observation 4.4). While confirming that shared training induces connectivity also in RL, the analysis reveals behaviour specific to value-based learning. Compared with behaviour cloning trained on the same data and network, Q-networks require a substantially larger fraction of shared training to become connected (newly added Fig. 45). Moreover, among the many RL and learning aspects examined in App. D, the discount factor is the one that breaks connectivity, and this cannot be attributed to the Q-value scale alone (newly added Figs. 50–51), but rather to the change in the underlying objective.
On the method side, Q-Soup follows directly from these findings: it merges Q-networks finetuned from a shared checkpoint into a single model. It approaches the output-space ensemble in the homogeneous setting (Fig. 6), outperforms the best individual configuration selected in hindsight in the heterogeneous setting (Fig. 7), and compresses established ensemble-based methods, Maxmin DQN (Fig. 8(b)) and Online REM (newly added Fig. 18), into a single network. Where connectivity breaks, the greedy soup of Model Soups is impractical in RL since every candidate soup must be evaluated in the environment, so we propose Proxy Q-Soup, which relies on training returns instead.
To the best of our knowledge, this is the first systematic study of when Q-networks become linearly connected, what breaks this connectivity, and how it can be exploited to replace output-space ensembles with a single model, which is the central practical goal of this work.