- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes incorporating multiple priors into an iterative solver, specifically deep equilibrium models (DEM). The first modification is to add momentum to the outer layer of deep equilibrium network dynamics, and replace the inner layer with a standard multi-step iterative solver. The second modification is within the iterative solver: to combine multiple network residuals together in a Lipschitz-preserving way, using doubly stochastic matrices as a learnable update. Such a method allows for more expressive residuals while (theoretically) preserving convergence of DEM methods, under standard assumptions on the residuals. Experiments are reasonable.
Critical strengths
- Interesting and effective method to combine multiple priors into the DEM framework. Clear motivation as to why to do so, and procedure is sufficiently clearly detailed in the main text and appendix.
Critical weaknesses
- The motivation of the doubly stochastic matrices is entirely unclear, my understanding is that it simply works in practice. There is first a learnable logit matrix , and then another learnable query vector (on the simplex, which is advocated against later in the page!), then the so-called ``fusion weight'' (i.e. weights to combine the residuals) is given by where is projection onto the space of doubly stochastic matrices via Sinkhorn. Why split the training?
- The paragraph on "anti-collapse" is not convincing, neither is the claim in Appendix B.2.
- In Appendix E.4.4, since the relative importance of the softmax and Sinkhorn methods are similar, surely this could also be interpreted as softmax training much faster than Sinkhorn? If you were to continue training, would the Sinkhorn fusion weights tend towards collapse? i.e. how sensitive are these weights to changes in learning rate?
- Black hole experiment seems a bit unfair. Any comparison against DEM methods which allows for this additional adaptation after the model? What model is used for PnP-DM?
- Similar concerns for the imaging problems: are the retrained networks (Appendix C) also applied for the compared baselines?
Questions for authors
- What exactly is being trained? Are the prior models themselves (e.g. DnCNN) being finetuned between the iterations as well, or is it just the stepsize/momentum/logit matrix? Please add some clarification in the main text instead of deferring to appendix.
- Please add a sentence in Section 3.1 saying that the theory is first presented when is one step, then for step.
- Should reference and clarify the differences with [CRDP26] which is also DEQ with inertia (in the inner solver).
- References should be in brackets (maybe \citep?), also there are several instances of "Eq. Equation (xx)".
[CRDP26] Clerc, A., Renaud, M., De Seneville, B. D., & Papadakis, N. (2026). i-DEQ: A stable inertial deep equilibrium model for image restoration. arXiv preprint arXiv:2605.19705.
3 comments
We further track the fusion weights throughout training to investigate whether the difference between Softmax and Sinkhorn is merely due to slower convergence: Both strategies preserve the same ranking of prior contributions, with DnCNN receiving the largest weight. However, their weight distributions evolve differently: Softmax progressively concentrates almost all the weight on DnCNN, reaching approximately 98%, whereas Sinkhorn maintains a substantially more distributed allocation throughout the extended training. The similar ranking therefore does not imply equivalent fusion behavior, nor does it establish that Sinkhorn merely converges more slowly. Instead, the two parameterizations lead to markedly different concentration patterns.
Table R6. Evolution of prior fusion weights during training for Softmax and Sinkhorn-based fusion.
| Epoch | Softmax DnCNN | Softmax DRUNet | Softmax Restormer | Sinkhorn DnCNN | Sinkhorn DRUNet | Sinkhorn Restormer |
|---|---|---|---|---|---|---|
| 1 | 33.5 | 33.2 | 33.3 | 33.4 | 33.3 | 33.3 |
| 50 | 94.5 | 4.2 | 1.3 | 50.8 | 31.0 | 18.2 |
| 100 | 97.2 | 2.4 | 0.4 | 51.2 | 30.9 | 17.9 |
| 200 | 97.8 | 1.9 | 0.3 | 51.9 | 31.2 | 16.9 |
| 300 | 97.7 | 1.9 | 0.4 | 51.7 | 30.6 | 17.7 |
We also examine the sensitivity to the learning rate by varying the base learning rate while keeping the remaining settings unchanged: Across the tested learning-rate range, the maximum Sinkhorn weight remains around 0.51--0.52, and the reconstruction performance varies only slightly. In comparison, Softmax consistently exhibits a highly concentrated distribution, with its maximum weight remaining around 0.96--0.97. These results indicate that the observed difference is not specific to the default learning rate. Learning rates beyond the tested range lead to training-loss divergence. Overall, the additional analyses clarify that the role of the doubly-stochastic constraint is not to guarantee uniform prior usage. It imposes a principled majorization constraint on the fused distribution, while the learnable interaction matrix provides a structured mechanism for adapting the relative contributions of heterogeneous priors. The improved reconstruction performance is therefore associated with the combination of structured interactions and constrained weight redistribution, rather than weight balancing alone.
Table R7. Sensitivity of fusion weights and reconstruction performance to different learning rates.
| Learning Rate | PSNR (dB) | Softmax Max Weight | Sinkhorn Max Weight |
|---|---|---|---|
| 0.1× | 32.94±0.29 | 0.96 | 0.52 |
| 0.5× | 32.85±0.30 | 0.97 | 0.52 |
| 1× | 32.91±0.32 | 0.97 | 0.51 |
| 2× | 32.80±0.21 | 0.97 | 0.51 |
Q2. Experimental Fairness
Black-hole imaging. To address the concern about additional prior adaptation, we introduce DE-Prox and DenseDiff as additional equilibrium-based baselines. Both use exactly the same pretrained denoisers and follow the same preprocessing and experimental protocols as MHC-DEM. The results on synthetic M87 data are:
Table R8. Quantitative comparison of black-hole image reconstruction on synthetic M87 data.
| Method | MSE (×10⁻⁴) | Corr. |
|---|---|---|
| DE-Prox | 1.89 | 0.974 |
| DenseDiff | 1.68 | 0.978 |
| PnP-DM | 3.52 | 0.936 |
| MHC-DEM | 1.32 | 0.996 |
MHC-DEM achieves the best MSE and correlation, demonstrating that its improvement is not merely due to the use of pretrained denoisers or additional prior adaptation.
Regarding PnP-DM, we follow its original configuration. It employs an EDM-parameterized diffusion denoiser trained via denoising score matching, which differs fundamentally from the pretrained discriminative restoration networks used in MHC-DEM. Replacing its denoiser with our prior networks would alter its original formulation; therefore, we retain the original configuration.
Image deblurring. For PnP-based methods, including GS-PnP, DE-Prox, ELDER, and ProxPnP-DRS, we replace their denoisers with the same pretrained DRUNet used in MHC-DEM, ensuring a consistent prior-network basis. For the other baselines, including DPS, DiffIR, DAPS, and DenseDiff, we use the same training data as MHC-DEM while retaining the original configurations and hyperparameters reported in their respective papers. These experimental details, together with the additional M87 results, will be included in the revised manuscript.
Q3. Training and Learnable Parameters
The default heterogeneous prior bank consists of three pretrained denoisers: DnCNN, DRUNet, and Restormer. These denoisers are first trained independently on image restoration datasets and then kept frozen throughout MHC-DEM training, without updating their parameters during either the inner refinement steps or the outer equilibrium iterations. SwinIR is additionally considered in the ablation experiments.
The learnable parameters of MHC-DEM include: (i) non-shared inner-stage step sizes for ; (ii) non-shared interaction logits for , which determine the structured prior interactions through Sinkhorn fusion; and (iii) non-shared query logits for , from which the stage-specific query distributions are obtained as . These parameters are jointly optimized using the equilibrium training objective and implicit differentiation. In addition, the temporal momentum coefficient is parameterized as a shared learnable scalar, . Since the inertial term vanishes at equilibrium, the equilibrium solution is independent of , and thus implicit differentiation does not provide gradients for this parameter. Instead, is optimized using an auxiliary finite-trajectory objective. This separation preserves the stationary fixed-point formulation required by DEQ training while allowing the momentum coefficient to be learned.
We will clarify the initialization, parameter-sharing strategy, and optimization procedures of these components in the main text.
Q4. Differences from i-DEQ [CRDP26]
We thank the reviewer for pointing out the relevance of the concurrent work i-DEQ [CRDP26]. Both i-DEQ and MHC-DEM incorporate inertial information into DEQ dynamics, and thus share the general idea of exploiting historical iterates to improve the convergence behavior of equilibrium models. However, they differ in the placement and role of the inertial mechanism, as well as its integration into the stability analysis.
First, the inertia is introduced at different levels of the equilibrium computation. In i-DEQ, the inertial mechanism is incorporated into the inner solver to accelerate the fixed-point iterations. In contrast, MHC-DEM introduces inertia exclusively into the outer temporal equilibrium dynamics, while deliberately keeping the nested spatial refinement steps free of momentum. This separation distinguishes spatial reconstruction refinement from temporal information propagation across equilibrium iterations.
Second, MHC-DEM employs a learnable shared momentum coefficient, parameterized as , which is optimized through an auxiliary finite-trajectory objective. This allows the strength of temporal inertia to be adapted during training while preserving a stationary fixed-point mapping.
Third, our Theorem 1 explicitly incorporates the effects of heterogeneous prior fusion and temporal inertia into a unified stability condition:
+2\alpha+2\alpha\beta<1$$ Here, the terms involving $\alpha$ and $\alpha\beta$ jointly characterize the contributions of the hyper-connected prior interaction and the inertial coupling, respectively. Thus, the stability analysis accounts for their combined effects within the proposed equilibrium dynamics. In summary, although both methods exploit inertia in DEQ models, i-DEQ introduces inertia into the inner fixed-point solver, whereas MHC-DEM places a learnable inertial mechanism in the outer equilibrium dynamics, together with a momentum-free nested spatial refinement. We will add this discussion and the corresponding reference to better clarify the distinction from [CRDP26]. **Q5. Text Revisions** In Section 3.1, we will clarify that the theoretical analysis first establishes the stability condition for a single inner refinement step ($T=1$), isolating the effects of prior fusion and temporal inertia, and then extends to the general case of $T$ non-shared inner steps, where the stability condition accounts for the cumulative expansion of the composite operator. We will also standardize citations using bracketed reference commands (e.g., \citep{}) and correct equation references throughout the manuscript, eliminating redundant expressions such as “Eq. Equation (xx).”
We thank the reviewer for the insightful comments.
Q1. Motivation of the Doubly-Stochastic Fusion Mechanism
We thank the reviewer for questioning the motivation and optimization behavior of the doubly-stochastic fusion mechanism. We clarify that the two learnable components are jointly optimized end-to-end, rather than trained in separate stages. Specifically, the fusion mechanism is defined as
\mathbf{q}=\operatorname{Softmax}(\mathbf{v}),\quad \mathbf{W}=\operatorname{Sinkhorn}\left(\exp(\mathbf{L}/\tau)\right),$$ where $\mathbf{q}$ represents the base preference over heterogeneous priors, while $\mathbf{W}$ models their structured interactions. The factorization is not intended to increase the representational capacity of the final fusion vector: a direct simplex-valued parameterization can also represent any valid fusion vector. Instead, it introduces a structured parameterization that separates the base preference from the interaction-induced redistribution of prior contributions. This provides an additional inductive bias on how the prior weights are formed. Importantly, the doubly-stochastic constraint has a precise mathematical implication beyond preserving the simplex. Since $\mathbf{W}$ is doubly stochastic, the fused distribution satisfies $$\mathbf{w}=\mathbf{W}\mathbf{q}\prec\mathbf{q},$$ where $\prec$ denotes majorization. Consequently, $$\max_i w_i\leq\max_j q_j,\qquad H(\mathbf{w})\geq H(\mathbf{q}),$$ where $H(\cdot)$ is the Shannon entropy. Thus, the doubly-stochastic transformation cannot make the fused weights more concentrated than the base preference distribution. In contrast, column-stochasticity alone only preserves the sum of the weights and does not ensure this property. For example, a column-stochastic matrix can map a uniform distribution to a one-hot vector. We emphasize, however, that this property is a relative concentration constraint, not a guarantee that the final weights are uniformly distributed or immune to asymptotic collapse: if $\mathbf{q}$ itself approaches a simplex vertex, the fused distribution may also become concentrated. To examine the contribution of this structured constraint, we compare four fusion strategies under identical experimental settings: **Table R5. Comparison of reconstruction performance and weight concentration under different prior fusion strategies.** | Fusion Strategy | PSNR (dB) | Max Weight | | :------------------------ | :------------------: | :--------: | | Softmax (Query-only) | $30.96 \pm 0.16$ | 0.97 | | Column-stochastic + Query | $31.22 \pm 0.35$ | 0.85 | | Uniform Weight | $31.45 \pm 0.10$ | 0.34 | | Sinkhorn (Ours) | **$32.91 \pm 0.32$** | 0.51 | Sinkhorn fusion outperforms Softmax, column-stochastic, and uniform fusion by 1.95, 1.69, and 1.46 dB, respectively. Notably, uniform fusion produces a more balanced weight distribution than Sinkhorn but achieves lower reconstruction performance. This indicates that preventing weight concentration alone does not explain the performance gain. Rather, the results suggest that the structured, learnable interaction matrix enables more effective coordination among heterogeneous priors, while its doubly-stochastic constraint limits excessive concentration in the resulting fusion weights. We have therefore revised the corresponding statements in Section 2.3 and Appendix B.2 to distinguish this mathematical concentration property from the empirical observation of balanced prior contributions.