acceptodds
Under review as a conference paper at ICLR 2027

Ensemble Value-Based Reinforcement Learning through the Lens of Mode Connectivity

Abstract

Ensemble methods are a common solution for improving the stability and performance of value-based reinforcement learning (RL) algorithms. However, conventional ensembles operate in the output space and require maintaining multiple models during inference, resulting in inference cost that grows linearly with the ensemble size. In this work, we study the geometric structure of the objective landscape in value-based RL through the lens of mode connectivity and analyze how trained Q-networks relate in parameter space. Our analysis reveals conditions under which weight-space ensembling becomes effective, how linear connectivity can emerge between Q-networks, and which RL-specific conditions can break this connectivity. Building on these insights, we propose Q-Soup, a weight-averaging approach that merges multiple Q-networks trained under identical or diverse configurations into a single model. Empirical results on the Atari benchmark show that Q-Soup achieves performance comparable to traditional output-space ensembles while maintaining a single model at inference time. To the best of our knowledge, this work presents the first systematic investigation of mode connectivity and weight-space ensembling in value-based RL.

Primary area:
reinforcement learning
Keywords:
reinforcement learning, ensemble learning, mode connectivity

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_VzxyOpenReview ↗
Rating
4
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies mode connectivity and weight-space ensembling in value-based reinforcement learning. This paper first investigate when Q-networks can be linearly connected in parameter space and show that shared initialization can help achieve such connectivity, while some RL-specific factors, such as different discount factors, may break it. Based on these observations, the authors propose Q-Soup, which averages the parameters of multiple finetuned Q-networks into a single model to reduce inference cost. The method is evalauted on Atari environments with several value-based RL algorithms, including DQN, PQN, and MINTO. The experimental results shows that Q-Soup generally performs better than individual finetuned models and achieves comparable performance to output-space ensembles, while only requiring a single network during inference. This paper also propose Proxy Q-Soup to handle cases where some models are not well connected.

Critical strengths
  1. The paper systematically investigates mode connectivity in value-based RL, including the conditions where weight averaging works and when it may fail. The analysis provides useful insights into how Q-networks behave in parameter space, which has been less explored comparing to supervised learning.
  2. Q-Soup provides a simple and practical way to improve the performance of individual Q-networks while maintaining a single model at inference. The experiments are extensive, covering multiple value-based RL algorithms, Atari environments, and different training configurations. The results provide strong empirical suport for the proposed approach, which makes this work a valuable contribution
Critical weaknesses

I do not have substantial concerns about this paper. Overall, I think this is a good practical application of existing techniques to value-based RL, supported by extensive experiments and empirical analysis. The main limitation is the relatively limited algorithmic novelty, as Q-Soup mainly applies existing model averaging techniques to value-based RL. Similarly, several observations about mode connectivity, including the importance of shared initialization, are already well established in previous work.

Questions for authors

The paper claims that Q-Soup achieves comparable performance to output-space ensembles. However, the experimental results seem to show that Q-Soup generally performs slightly worse than output-space ensembles, although it outperforms individual models. Could the authors clarify this claim or use a more precise description of the performance trade-off?

2 comments
AuthorsOpenReview ↗

Reviewer Vzxy (1/2)

We thank the reviewer for the thorough review and the positive assessment of our work, and we appreciate that the reviewer found our analysis of mode connectivity in value-based RL insightful and our experiments extensive. We apologize for the delayed response, as we aimed to address all the points raised with the limited resources available to us. All changes are included in the revised manuscript and highlighted in blue.

P1. I do not have substantial concerns about this paper. Overall, I think this is a good practical application of existing techniques to value-based RL, supported by extensive experiments and empirical analysis. The main limitation is the relatively limited algorithmic novelty, as Q-Soup mainly applies existing model averaging techniques to value-based RL. Similarly, several observations about mode connectivity, including the importance of shared initialization, are already well established in previous work.

We thank the reviewer for the positive assessment. We agree that Q-Soup builds on existing model averaging techniques, as acknowledged in the manuscript, and that the role of shared initialization in mode connectivity is well established in supervised learning. Our contribution therefore lies elsewhere and has two parts: a systematic analysis of how Q-networks relate in parameter space through the lens of mode connectivity, and a practical method that exploits this structure to reduce the inference cost of ensembles in value-based RL. Both are driven by a concrete need. Scientifically, online value-based RL involves components absent from supervised learning, namely bootstrapped targets and data that shift with the policy, and it was unclear whether mode connectivity survives them. Practically, ensembles are widely used in value-based RL, yet their inference cost grows linearly with the number of models.

On the analysis side, our observations progress from the benefit of output-space ensembles (Observation 4.1) and the failure of naive weight averaging (Observation 4.2) to the conditions under which averaging succeeds (Observation 4.3) and under which connectivity emerges (Observation 4.4). While confirming that shared training induces connectivity also in RL, the analysis reveals behaviour specific to value-based learning. Compared with behaviour cloning trained on the same data and network, Q-networks require a substantially larger fraction of shared training to become connected (newly added Fig. 45). Moreover, among the many RL and learning aspects examined in App. D, the discount factor is the one that breaks connectivity, and this cannot be attributed to the Q-value scale alone (newly added Figs. 50–51), but rather to the change in the underlying objective.

On the method side, Q-Soup follows directly from these findings: it merges Q-networks finetuned from a shared checkpoint into a single model. It approaches the output-space ensemble in the homogeneous setting (Fig. 6), outperforms the best individual configuration selected in hindsight in the heterogeneous setting (Fig. 7), and compresses established ensemble-based methods, Maxmin DQN (Fig. 8(b)) and Online REM (newly added Fig. 18), into a single network. Where connectivity breaks, the greedy soup of Model Soups is impractical in RL since every candidate soup must be evaluated in the environment, so we propose Proxy Q-Soup, which relies on training returns instead.

To the best of our knowledge, this is the first systematic study of when Q-networks become linearly connected, what breaks this connectivity, and how it can be exploited to replace output-space ensembles with a single model, which is the central practical goal of this work.

AuthorsOpenReview ↗

Reviewer Vzxy (2/2)

Q1. The paper claims that Q-Soup achieves comparable performance to output-space ensembles. However, the experimental results seem to show that Q-Soup generally performs slightly worse than output-space ensembles, although it outperforms individual models. Could the authors clarify this claim or use a more precise description of the performance trade-off?

We thank the reviewer for this question and agree that a more precise description is needed. By "comparable performance", we meant that Q-Soup offers a favourable trade-off: on average, it recovers most of the improvement of the output-space ensemble over individual models, while requiring a single model at inference time. We did not intend to claim that Q-Soup matches the Output Ensemble in general, and the reviewer is correct that it is typically slightly below it.

To make this trade-off precise, in the homogeneous setting (Fig. 6), Q-Soup trails the Output Ensemble by at most 7.7% across DQN, PQN and MINTO, while recovering 60–89% of the performance gap between the average finetuned model and the Output Ensemble, and it outperforms the best individual finetuned model by at least 12.9%. In the heterogeneous setting (Fig. 7), Q-Soup closely matches the Output Ensemble for PQN and recovers 63–77% of its improvement over the average finetuned model for DQN. When applied to ensemble-based methods, Q-Soup comes within 1.8% and 3.3% of the output-space aggregation of the same shared-initialization members for Maxmin DQN and Online REM, respectively. Compared with the original methods using independent initialization, Q-Soup outperforms Maxmin DQN by 3.5% and trails Online REM by 5.8% (Fig. 8(b), newly added Fig. 18). In return, Q-Soup keeps the memory and compute of a single model, whereas the Output Ensemble grows linearly with the number of models (newly added Sec. A.6). In the revised version, we describe this trade-off more precisely (Abstract, Sec. 1 and Sec. 7): Q-Soup approaches the performance of output-space ensembles, while recovering most of their improvement over individual models and requiring only a single model at inference time.

Reviewer_vBEAOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies when Q-networks can be combined through parameter averaging, using mode connectivity to study the performance of policies along interpolations in weight space. The paper reports that averaging independently initialized networks is generally ineffective in the evaluated settings, whereas networks fine-tuned from a shared checkpoint can remain connected by high-return linear paths. Building on these observations, Q-Soup averages networks fine-tuned with either identical or different hyperparameter configurations, producing a single model for inference.

The authors also examine heterogeneous discount factors as a failure case and introduces Proxy Q-Soup, which filters and weights ingredients using training returns without additional evaluation interactions. Experiments on MinAtar and 15 Atari games, including DQN, PQN, MINTO, and an application to Maxmin DQN, suggest that weight averaging can retain a substantial fraction of the gains from output-space ensembles while reducing the number of models required at deployment.

Critical strengths
  1. Maintaining multiple Q-networks at inference can increase memory and computational requirements. Q-Soup addresses this with a straightforward parameter-averaging procedure that requires no additional optimization during merging. The comparison against output ensembles built from the same ingredients usefully quantifies how much ensemble performance is retained in a single model.

  2. The paper goes beyond reporting averaged-model performance by examining independent initialization, shared training trajectories, and heterogeneous configurations. Investigating discount-factor changes as a potential failure case helps identify limitations of uniform averaging. Proxy Q-Soup is also practically motivated, since it reuses training returns rather than requiring additional evaluation interactions for ingredient selection.

  3. The supplementary material includes SWA comparisons, ensemble-size and fine-tuning-duration ablations, Proxy component ablations, and analyses of LayerNorm, exploration schedules, and optimizer resets. These provide useful information that the reported effects are not confined to a single training configuration.

Critical weaknesses
  1. The central procedure closely follows Model Soups, while connectivity after shared training and the learning of network subspaces are established ideas. Applying these ideas to Q-networks is potentially valuable, but the paper needs to distinguish its contribution more clearly from existing model averaging and RL policy merging, including Rewarded Soups. Currently, the methodological advance appears incremental.

  2. The observations establish that connectivity can emerge in value-based RL, but provide limited understanding of how bootstrapping and policy-dependent data collection affect it. The authors explicitly frame the study as empirical; my concern is therefore not the absence of a theorem itself, but the limited explanation accompanying the methodological novelty. Moreover, low barrier depth does not imply that weight averaging improves performance or approximates output ensembling. The discount-factor experiments also conflate changes in learning objectives with changes in Q-value scale. Finally, the piecewise interpolation visualizations support conclusions about sampled combinations, rather than the geometry of an entire high-dimensional flat region.

  3. Comparing Q-Soup with output ensembling over the same ingredients appropriately evaluates compression. However, comparisons with individual ingredients do not establish an advantage under a matched total training budget. A full-budget single-model baseline with checkpoint averaging would strengthen this claim. Actual inference latency and memory measurements are also needed to quantify the deployment benefit.

  4. Comparisons with established value-based ensemble methods, particularly MeanQ (Liang et al., 2022) and online REM (Agarwal et al., 2020), would also strengthen the evaluation. These methods incorporate output-space combinations during learning, providing relevant alternatives to independently fine-tuning and subsequently averaging Q-networks. Matched-budget comparisons would help establish Q-Soup's performance-cost trade-off. Testing shared-initialization and weight-merged variants of these methods would additionally clarify whether the conditions enabling merging preserve the benefits of ensemble training. The effects of changing initialization and subsequently merging should be reported separately: successful compression of a modified ensemble does not necessarily demonstrate preservation of the original method's performance.

  5. Although the Atari evaluation is substantial, Atari and MinAtar offer limited diversity in problem structure. An additional benchmark such as Craftax or Procgen would strengthen the broader applicability claim.

Questions for authors
  1. Under the same total environment-interaction budget, how does Q-Soup compare with a single model using checkpoint averaging?
  2. How does Q-Soup compare with MeanQ and online REM under matched interaction budgets? For these methods, how much does performance change when replacing independent initialization with a shared checkpoint, and how much is subsequently lost through weight averaging?
  3. Does the heterogeneous-discount failure persist after calibrating Q-value scales before merging?
  4. Across games and seeds, does barrier depth predict Q-Soup's improvement over its ingredients, and do training-return proxy scores predict merging compatibility?
  5. Can the main result be reproduced on Craftax or Procgen, with measured inference latency and memory usage?
6 comments
AuthorsOpenReview ↗

Response to Reviewer vBEA (1/6)

We thank the reviewer for the detailed and constructive review, and for recognizing the practical motivation of Q-Soup, our investigation of its failure cases, and the breadth of our supplementary analyses. We apologize for the delayed response, as we aimed to address all the points raised with the limited resources available to us. All changes are included in the revised manuscript and highlighted in blue.

P1. The central procedure closely follows Model Soups, while connectivity after shared training and the learning of network subspaces are established ideas. Applying these ideas to Q-networks is potentially valuable, but the paper needs to distinguish its contribution more clearly from existing model averaging and RL policy merging, including Rewarded Soups. Currently, the methodological advance appears incremental.

We understand the reviewer's concern and are happy to clarify how our work relates to the existing literature. Our contribution consists of two parts: a systematic analysis of ensembles in value-based RL from a parameter-space perspective, through the lens of mode connectivity, and a practical method tailored to value-based RL that reduces the inference cost of ensembles. To the best of our knowledge, this is the first systematic study of when Q-networks become linearly connected, what breaks this connectivity in RL, and how it can be exploited to replace output-space ensembles with a single model. This work is motivated by both a scientific and a practical need. Scientifically, online value-based RL involves moving components that could unpredictably affect the emergence of mode connectivity. Practically, there is a need for ensembles with low inference cost that do not sacrifice the benefits of ensembling.

The first part is a systematic analysis consisting of a series of empirical observations: the benefit of ensembles in value-based RL (Observation 4.1), the failure of weight averaging under independent initialization (Observation 4.2), the conditions under which weight averaging succeeds (Observation 4.3), and the conditions under which linear mode connectivity emerges (Observation 4.4). While linear mode connectivity after shared training is established in supervised learning (Frankle et al. [1]), it was unclear whether it survives the RL-specific aspects of value-based learning, such as non-stationarity and bootstrapping. We show that it does, although it emerges differently than in supervised learning (see our response to P2(a)). We further studied the failure cases that RL-specific components might cause, examining many RL and learning aspects in App. D (Figs. 44–51). We found that varying the discount factor after splitting is the most critical aspect that breaks mode connectivity, and that this cannot be explained by the change in Q-value scale alone (Figs. 50–51; see our response to P3(c)).

The second part is the practical method, Q-Soup, which is inspired by Model Soups [2], as indicated in the original manuscript. We investigated Q-Soup as a test-time replacement for output-space ensembles in both the homogeneous and heterogeneous settings. Moreover, we studied its effectiveness in replacing established ensemble-based methods, Maxmin DQN (Fig. 8(b)) and, in the newly added results, Online REM [3] (Fig. 18). Furthermore, while Model Soups introduced the greedy soup as a more reliable alternative to uniform averaging, this approach is not practical in RL, since evaluating each candidate soup requires additional environment interactions. Hence, we introduced Proxy Q-Soup, which uses training returns as a proxy for ingredient quality, as a heuristic to mitigate the loss of connectivity under varying discount factors (Fig. 8(a)).

Overall, this distinguishes our work from existing model averaging and merging methods. Model averaging methods such as Model Soups [2] improve performance without increasing inference cost, typically assuming linear mode connectivity and verifying it in their target setting, mostly when finetuning large pretrained models in supervised learning, while the mode connectivity literature studies when such connectivity emerges [1]. Our work brings both together for value-based RL. In RL, prior work on weight merging and mode connectivity focuses on policy networks: Rewarded Soups [4] merges policies finetuned from large pretrained models, while Gaya et al. [8] and Hendawy et al. [9] enforce connectivity between policies by design. In contrast, we study when connectivity emerges naturally between Q-networks trained under bootstrapped and non-stationary targets, and exploit it to replace output-space ensembles at test time. We have also added Rewarded Soups [4] to the related work in the revised manuscript (Sec. 2).

AuthorsOpenReview ↗

Response to Reviewer vBEA (2/6)

P2. (a) The observations establish that connectivity can emerge in value-based RL, but provide limited understanding of how bootstrapping and policy-dependent data collection affect it. The authors explicitly frame the study as empirical; my concern is therefore not the absence of a theorem itself, but the limited explanation accompanying the methodological novelty.

We appreciate the reviewer raising this point, as it motivated us to strengthen the manuscript with further insights. As stated in our response to P1, we found that linear mode connectivity still holds in online value-based RL, despite the distribution shift during training and the shifting targets due to bootstrapping, and that the discount factor is the key RL factor that breaks it. We agree with the reviewer that what was missing is a clearer understanding of how these RL-specific aspects affect connectivity compared with supervised learning.

To fill this gap, we added a study comparing linear mode connectivity in DQN with a supervised learning baseline, behaviour cloning (BC), on the same MinAtar games as in Fig. 5. We first train a DQN agent (5 seeds) and use its final replay buffer to train BC with the same network and number of gradient steps, so that the two copies differ only in minibatch order. This removes both bootstrapping and policy-dependent data collection while keeping the network, data and optimization budget fixed. As shown in the newly added Fig. 45, BC becomes linearly connected after sharing only a small fraction of training (around 10% given our discretization), whereas DQN requires a substantially larger fraction (roughly 30–70%, depending on the game). This shows that the RL-specific aspects of value-based learning make connectivity considerably harder to obtain than in supervised learning.

P2. (b) The discount-factor experiments also conflate changes in learning objectives with changes in Q-value scale.
Q3. Does the heterogeneous-discount failure persist after calibrating Q-value scales before merging?

We thank the reviewer for raising this point. To investigate whether the discount-factor failure is due to the change in the learning objective or simply to the Q-value scale, we repeated the study of Observation 4.4 with a calibrated Q-scale and compared it with the uncalibrated version. Since Q-Soup averages parameters rather than Q-values, rescaling the Q-values of trained models before merging leaves the averaged parameters unchanged, unless the scaling is folded into the output layer, in which case all other layers remain as learned under different discount factors. We therefore calibrate the Q-value scale during finetuning, so that the whole network is trained on the calibrated scale. For the calibration, we scale the rewards in the TD target by 1−γ1−γref\frac{1-\gamma}{1-\gamma_{ref}}, where γ\gamma is the discount factor used during finetuning and γref=0.99\gamma_{ref}=0.99 is the one used during pretraining, so that both models learn Q-values on the same scale.

In the first experiment, we consider the same four MinAtar games as in Fig. 5. After splitting, one model is finetuned with γ1=0.99\gamma_1=0.99 and the other with γ2=0.9\gamma_2=0.9, either uncalibrated (Different γ\gamma) or calibrated (Different γ\gamma, Q-scale calibrated). Fig. 50 shows the normalized barrier depth, measured along the linear path between the two models, i.e., on their merged models, as a function of the fraction of training shared. Calibration helps in some games: for example, in Breakout the barrier decreases when a large fraction of training is shared, yet more slowly than in the homogeneous setting (Same γ\gamma). In other games, such as SpaceInvaders, a high barrier remains even at the largest fraction of training shared, although lower than without calibration.

To further test this in a harder setting, we repeated the study with a wider range of discount factors after splitting (γ1=0.85\gamma_1=0.85, γ2=0.999\gamma_2=0.999) on one game where calibration helped (Breakout) and one where it did not (SpaceInvaders). As shown in Fig. 51, with the wider range calibration, it no longer restores linear mode connectivity: the barrier remains high in both games. Overall, calibration can partially reduce the barrier, but it does not resolve the failure, which supports our claim that the main cause is the change in the underlying objective rather than the Q-value scale.

AuthorsOpenReview ↗

Response to Reviewer vBEA (3/6)

P2. (c) Moreover, low barrier depth does not imply that weight averaging improves performance or approximates output ensembling.
Q4. Across games and seeds, does barrier depth predict Q-Soup's improvement over its ingredients, and do training-return proxy scores predict merging compatibility?

We thank the reviewer for this question. We agree that a low barrier depth does not, by itself, imply that weight averaging improves performance: by Definition 3.1, it only guarantees that the interpolated models do not fall below the weaker of the two models, not that they outperform the individual models or approach the output-space ensemble. In our analysis, the barrier depth serves to establish when linear mode connectivity emerges (Observation 4.4).

Barrier depth and the gain of merging. To relate barrier depth to the benefit of merging, we added a new analysis in Sec. B.16. For pairs of models, Fig. 37 reports the barrier depth together with the performance of the averaged model, normalized by the mean performance of the two individual models. When both models keep the same discount factor (Fig. 37(a)), the barrier vanishes as more training is shared, and the averaged model reaches or exceeds the mean of the individual models. With different discount factors (Fig. 37(b)), the barrier remains high and the averaged model stays well below this mean. For the multi-model case, evaluating the full convex hull of the ingredients is prohibitively expensive, so we extended the barrier depth to the two-dimensional slice used in our performance landscapes (App. B.12–B.14). Across 15 games and 5 pretrained seeds in the homogeneous setting (Figs. 31–35), this barrier depth is zero in 74 of the 75 landscapes, and Q-Soup outperforms its mean ingredient in 70 of them, while trailing the output-space ensemble by at most 7.7% (Fig. 6). This is consistent with Observation 4.3 and with our main results: Q-Soup improves over its ingredients in all settings where connectivity is maintained (Figs. 6, 7, 8(b) and newly added Fig. 18), and degrades only when connectivity breaks under varying discount factors (Fig. 8(a)).

Training-return proxy and compatibility. Proxy Q-Soup is a heuristic, but cheap, alternative to the greedy soup of Model Soups, which would require additional environment interactions to evaluate every candidate soup. We proposed it as a first step towards handling the failure case of varying discount factors, rather than only highlighting this failure, and leave more principled approaches for future work. Since training returns measure the quality of each ingredient rather than its compatibility for averaging, Proxy Q-Soup relies on the assumption that incompatible ingredients are typically the weak ones. To test this assumption, we added an analysis of the 75 cases of the discount-factor setting (Sec. B.15). Proxy Q-Soup outperforms uniform averaging in 65 of them and never discards either of the two best ingredients, and the newly added landscapes for this setting (Fig. 36) show that the discarded ingredient is typically the one isolated from the connected high-performing region. However, since the proxy only discards low-scoring outliers, it cannot detect an incompatible ingredient with high return. We observe this in 2 of the 75 cases, with a small loss relative to uniform averaging. We also observe a second failure mode in 2 cases, where uniform averaging outperforms every ingredient but the proxy concentrates its weight on the best one. While the original manuscript already acknowledged the limitations of Proxy Q-Soup, we have stated them more explicitly in the revised version (Sec. B.15 and Conclusion).

Reviewer_nHxBOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies mode connectivity in value-based reinforcement learning and proposes Q-Soup, which averages the parameters of Q-networks independently finetuned from a shared checkpoint. The empirical analysis investigates when weight averaging succeeds and how changing the discount factor can disrupt connectivity. A proxy-based variant uses training returns to filter and weight models. Experiments on MinAtar and 15 Atari games show that Q-Soup improves over individual finetuned models and approaches output-space ensemble performance while requiring only one network at inference.

Critical strengths
  1. The method offers a practical deployment benefit: it retains a substantial fraction of the performance improvement from output-space ensembles while using a single Q-network at inference. Results across DQN, PQN, and MINTO provide useful evidence that this benefit is not specific to one algorithm.

  2. The empirical investigation goes beyond reporting performance gains. The shared-training experiments and discount-factor analysis help identify conditions under which Q-network averaging succeeds or fails. The heterogeneous-configuration experiments and SWA comparison further strengthen the evaluation.

Critical weaknesses
  1. The advantage under a matched total training budget remains insufficiently established. Q-Soup combines several finetuning runs, whereas the individual-model comparisons use only one branch. A single model continued for the same total environment interactions and gradient updates would help determine whether branching and merging are preferable to simply training longer. This does not invalidate the inference-cost advantage over output ensembles, but limits the evidence for the overall training strategy.

  2. The connection between the geometric analysis and Proxy Q-Soup remains heuristic. High training return does not directly establish compatibility for parameter averaging, and pairwise connectivity does not guarantee that a multi-model average performs well. The experiments support the proposed procedure, but stronger evidence relating connectivity measurements to merging outcomes would make the explanation more convincing.

Questions for authors
  1. Under the same total environment-interaction and gradient-update budget, how does Q-Soup compare with a single model trained longer, including a version using SWA? This comparison would materially affect my assessment of the practical benefit.

  2. How consistently does measured connectivity predict the benefit of merging across games and hyperparameter settings? In particular, are there high-return ingredients that remain incompatible for averaging and therefore cannot be identified reliably by the training-return proxy?

2 comments
AuthorsOpenReview ↗

Response to Reviewer nHxB (1/2)

We thank the reviewer for the thorough review and the positive assessment of our work. We apologize for the delayed response, as we aimed to address all the points raised with the limited resources available to us. All changes are included in the revised manuscript and highlighted in blue.

P1. The advantage under a matched total training budget remains insufficiently established. Q-Soup combines several finetuning runs, whereas the individual-model comparisons use only one branch. A single model continued for the same total environment interactions and gradient updates would help determine whether branching and merging are preferable to simply training longer. This does not invalidate the inference-cost advantage over output ensembles, but limits the evidence for the overall training strategy.
Q1. Under the same total environment-interaction and gradient-update budget, how does Q-Soup compare with a single model trained longer, including a version using SWA? This comparison would materially affect my assessment of the practical benefit.

We would like to thank the reviewer for the in-depth review of our submission and for highlighting this point. Although we agree with the reviewer that this would be an ideal comparison, we respectfully disagree regarding its necessity for the following reasons:

  • In this work, we have discussed the benefit of Q-Soup in both the homogeneous (e.g., varying SGD noise) and heterogeneous (e.g., varying hyperparameter values) settings. The reviewer's concern is mainly valid for the homogeneous setting, whereas training for longer (even with checkpoint averaging) is not applicable to the heterogeneous setting. There, we do not know a priori which hyperparameter value is best to train for longer, while Q-Soup achieves better performance on average by running multiple configurations in parallel and then merging them in parameter space (see Fig. 7).
  • Furthermore, Q-Soup has shown great potential with ensemble-based methods, where the ensemble members interact with each other during training, as shown in Fig. 8(b) with Maxmin DQN and in the newly added Fig. 18 with REM. The training-for-longer argument does not apply here either.
  • In addition, our work also provides a systematic study of the emergence of mode connectivity in value-based reinforcement learning (RL). Therefore, our core contribution consists of an analysis of this property and a practical algorithm that can be used in multiple scenarios (homogeneous, heterogeneous, with ensemble-based algorithms, etc.).
  • From a practical perspective and a training time view, finetuning multiple models (e.g., KK models) for NN steps is faster than finetuning a single model for K×NK \times N steps, since the former can be parallelized in practice, while the latter is restricted by its sequential nature.

That said, we are keen to address the reviewer's concern as well as to provide a fair comparison in terms of a matched environment interaction budget (samples). In the newly added Sec. B.8, we compare Q-Soup and the Output Ensemble with single models finetuned for longer, including a variant with checkpoint averaging (SWA). In Fig. 20, Q-Soup and the Output Ensemble combine two models (K=2K=2), each finetuned for N=1.25N=1.25M steps, while the individual models are finetuned for twice as long (K×N=2×1.25K \times N = 2 \times 1.25M steps), with and without checkpoint averaging (SWA with k=2k=2). The figure shows the results for the individual games in (a)–(o) and the aggregated performance over the 15 Atari games in (p). In this setting, Q-Soup still outperforms the models finetuned for longer, with or without checkpoint averaging. Fig. 21 reports a similar comparison with an ensemble size of five and, consequently, longer finetuning for the individual models (5×1.255 \times 1.25M steps). In this case, Q-Soup outperforms the model finetuned for longer, while the checkpoint-averaging variant (SWA) matches the performance of Q-Soup, with both slightly below the Output Ensemble. In the homogeneous setting, these results suggest that, for both Q-Soup and the Output Ensemble, the gain from increasing the ensemble size does not always remain large enough that finetuning for longer cannot match it. However, as stated above, the contribution of this work goes beyond the homogeneous setting. We focus on understanding how ensemble members relate in weight space through the lens of mode connectivity, followed by a practical method that is useful in many settings, including the homogeneous and heterogeneous settings, as well as integration with established ensemble-based methods.

AuthorsOpenReview ↗

Response to Reviewer nHxB (2/2)

P2. The connection between the geometric analysis and Proxy Q-Soup remains heuristic. High training return does not directly establish compatibility for parameter averaging, and pairwise connectivity does not guarantee that a multi-model average performs well. The experiments support the proposed procedure, but stronger evidence relating connectivity measurements to merging outcomes would make the explanation more convincing.
Q2. How consistently does measured connectivity predict the benefit of merging across games and hyperparameter settings? In particular, are there high-return ingredients that remain incompatible for averaging and therefore cannot be identified reliably by the training-return proxy?

We thank the reviewer for this point, which we address in two parts.

Training return as a heuristic. Proxy Q-Soup targets the failure case of varying discount factors, where finetuned models can diverge from the flat region. The greedy soup of Model Soups would require additional environment interactions to evaluate every candidate soup, which can be costly in online RL. We therefore use the training returns already collected during finetuning as a cheap proxy for ingredient quality, implicitly assuming that the ingredients breaking connectivity are typically the weak ones. We test this assumption in the new Sec. B.15. Across the 75 cases of this setting (15 games × 5 pretrained seeds), Proxy Q-Soup outperforms uniform averaging in 65 and never discards either of the two best ingredients, while 64 of its 74 discarded ingredients are the worst one. The new landscapes for 4 games and 5 pretrained seeds (Fig. 36), although only an approximate two-dimensional view, show that the discarded ingredient is typically isolated from the connected high-performing region (e.g., γ\gamma = 0.999 in Boxing, γ\gamma = 0.85 in DemonAttack), while the uniform soup often falls into a low-performing valley.

Nevertheless, the failure case raised by the reviewer is valid but rare (2 of 75 cases): since the proxy only discards low-scoring outliers, it cannot identify a high-return incompatible ingredient. In both cases (Breakout and FishingDerby, the latter with near-random returns), the loss relative to uniform averaging is small. A second failure mode accounts for the largest losses (Assault and VideoPinball): uniform averaging outperformed every ingredient, whereas the proxy concentrated its weight on the best one. A greedy soup would address both, at the cost of extra interactions. We proposed Proxy Q-Soup as a cheap first step beyond only highlighting this failure case, and have stated its limitations more explicitly in the revised version (Sec. B.15 and Conclusion).

Connectivity and the benefit of merging. We agree that pairwise connectivity does not guarantee that a multi-model average performs well. We use the pairwise barrier depth to establish when linear mode connectivity emerges (Observation 4.4, Figs. 5 and 57), and by definition it only ensures that interpolated models do not fall below the weaker model. To relate it to the benefit of merging, the new Sec. B.16 reports the barrier depth together with the performance of the averaged model, normalized by the mean of the two models (Fig. 37). With the same discount factor (Fig. 37(a)), the barrier vanishes as more training is shared and the averaged model reaches or exceeds this mean; with different discount factors (Fig. 37(b)), the barrier remains high and the averaged model stays far below it.

For the multi-model case, one would ideally generalize the barrier depth to the full convex hull of the KK models. However, this hull is (K−1)(K-1)-dimensional, and every point requires a full evaluation in the environment, which makes covering it densely prohibitively expensive. We therefore rely on the performance landscapes (App. B.12–B.14), which approximate this region through a two-dimensional slice, and complement them quantitatively by extending the barrier depth to this slice (Sec. B.16), comparing the weakest ingredient with the worst-performing points within it (using the 5th percentile instead of the minimum to reduce sensitivity to evaluation noise). In the homogeneous setting (15 games × 5 pretrained seeds)(see Figs. 31–35), this barrier depth is zero in 74 of the 75 landscapes, and Q-Soup outperforms the mean finetuned model in 70 of them.

These findings are consistent with Observation 4.3: when the ingredients lie within a connected flat region, its center typically outperforms the average of the individual models. Accordingly, Q-Soup outperforms the average finetuned model in all settings where connectivity is maintained, namely the homogeneous setting (Fig. 6), the heterogeneous settings other than the discount factor (Fig. 7), and ensemble-based methods (Fig. 8(b), newly added Fig. 18). In contrast, when connectivity breaks under varying discount factors, uniform averaging degrades (Fig. 8(a)), which motivates Proxy Q-Soup.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We thank all Reviewers for their constructive feedback, and we apologize for the delayed response, which was due to the time needed to complete the requested experiments given our limited resources. The revised paper highlights all changes in blue; the main additions are:

  • Matched training budget: Q-Soup vs. a single model finetuned for the same total interactions, with and without SWA (Sec. B.8, Figs. 20–21) → nHxB & vBEA.
  • Proxy Q-Soup analysis: when it succeeds and fails across 75 cases, with new landscapes and an explicit limitation (Sec. B.15, Fig. 36) → nHxB & vBEA.
  • Connectivity vs. merging benefit: barrier depth related to averaged-model performance, extended to multi-model landscapes (Sec. B.16, Fig. 37) → nHxB & vBEA.
  • Ensemble-based methods: Q-Soup applied to Online REM, in addition to Maxmin DQN (Sec. B.6, Fig. 18) → vBEA.
  • Inference cost: measured latency, memory and FLOPs on GPU and CPU (Sec. A.6, Table 3) → vBEA.
  • New benchmark: Craftax with a recurrent PQN agent (Sec. B.9, Fig. 22) → vBEA.
  • Mechanistic insights: Q-value scale calibration for the discount-factor failure (Figs. 50–51) and a behaviour cloning comparison (Fig. 45) → vBEA.
  • Positioning and claims: clarified contribution relative to prior work, and replaced "comparable" with "approaches" for the performance trade-off (Abstract, Secs. 1 and 7) → vBEA & Vzxy.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.