acceptodds
Withdrawn from ICLR 2027

Toward Solver-Invariant Deep Equilibrium Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Deep equilibrium models define their output through a fixed point of a weight-tied iteration. This formulation motivates solver invariance: predictions should be insensitive to the numerical procedure used for inference. This property need not hold at a finite computation budget. With model weights fixed, replacing the inference solver can reduce the accuracy of a trained MDEQ checkpoint to a fraction of its training-solver accuracy. We investigate the ungated identity path as a source of this sensitivity. FROST removes that bypass, scales the recurrent path with a single learned scalar, and initializes the otherwise free input scale by a self-similar rescaling rule. Without assuming a globally unique fixed point, we give sufficient conditions linking endpoint residuals, state proximity, and readout margins, and measure their premises on trained models. Under identical implicit training, FROST holds a substantially smaller accuracy spread across seven inference procedures than MDEQ. A constant input scale reproduces it, and the result transfers to a non-vision Poisson task.

Primary area:
learning theory
Keywords:
DEQ, solver-invariance, implicit representations

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_zLA9OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper shows that the accuracy of a trained MDEQ [1] depends heavily on which solver is used at inference time, with the weights held fixed. On ImageNet the spread across seven solvers reaches 36.8 points. The authors trace this to the ungated identity path in the cell. Their parameterization, FROST, removes that path, scales the recurrent branch by a single learned scalar λ\lambda and sets the input scale from a self-similarity argument. FROST's accuracy spread is 5.4 points, compared with 36.8 for MDEQ.

Critical strengths
    1. I find the problem setting valuable. Treating the solver as its own evaluation axis at fixed weights is a useful idea, and the protocol is convincing: seven solvers from different families, plus per-example pairwise disagreement rather than aggregate accuracy alone.
    1. The sweep over the identity gain γ\gamma (Table 4) is the most informative experiment in the paper. Only one quantity changes, and it is clear that the problem appears in the γ≥1\gamma \ge 1 regime. I also appreciate that the theoretical results are stated as sufficient conditions and are not presented as guarantees.
Critical weaknesses
    1. The main comparison is run in a regime where both models are noticeably undertrained. MDEQ reaches 56.3% top-1 and FROST 60.1%, while the released MDEQ-small checkpoint reaches 75.4% according to the authors' own Appendix H. The MDEQ fixed-point residual at the returned state is 7.38, so the model is not really at an equilibrium. On the released checkpoint the spread is 16.05, less than half the 36.84 in Table 2. The authors discuss this openly, but I think it is important to see FROST trained to a comparable accuracy.
    1. I would like to understand the mechanism better. After training, λ\lambda drops from 0.5 to about 0.02, the input scale to around 10−310^{-3}, and the secant gain along trajectories is about 0.01. It is possible that FROST converges almost immediately and effectively behaves like a shallow weight-tied network. If so, solver invariance follows largely for free. In addition, the constant input-scale variant does just as well (spread 1.51), so it remains unclear how much the self-similarity argument based on fractional Brownian motion [2] contributes to the final result.

References:

    1. S.Bai, V.Koltun, J.Z.Kolter. Multiscale Deep Equilibrium Models. // Advances in Neural Information Processing Systems (NeurIPS), 2020.
    1. B.B.Mandelbrot, J.W.Van Ness. Fractional Brownian Motions, Fractional Noises and Applications. // SIAM Review, 10(4):422--437, 1968.
Questions for authors
    1. Could the γ\gamma sweep be run on the MDEQ cell (with its normalization and branch fusion) instead of the FROST cell?
    1. How many iterations, or how much wall-clock time, do FROST and MDEQ need to reach the same residual? Does FROST retain meaningful depth? ?
    1. If the secant gain is about 0.01, why does the Broyden-trained FROST lose almost 5 points under the five-step unroll?
    1. Could you report variability and robustness across several runs for the ImageNet results?
Reviewer_m3mgOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies the sensitivity of DEQ to the choice of numerical solvers at inference time. The authors show that under a limited budget, an MDEQ model trained with Broyden’s method could have a large accuracy gap when evaluated with other solvers. To tackle this, they propose FROST, which remove the direct identity bypass in the recurrent cell and uses a self similarity motovated scaling rule for the input.

In addition, the paper gives 2 sufficient conditions relating fixed-point residuals, distance between solver endpoints, and prediction agreement. On experiments on imageNet, CIFAR and a Poisson equation, FROST generally shows a smaller accuracy variance across solvers than the MDEQ baseline.

Critical strengths
  1. The paper finds a practical problem in DEQs that is valuable to study. To be more specific, Table 2 shows changing the inference solver (without changing model weights) can lead to large accuracy drop. I think evaluating a trained DEQ across solvers is useful, since performance under training solver alonge doesn’t tell much about the behavior under other inference procedures.
  2. There’s a obvious empirical improvement. FROST reduces the accuracy spread across solvers on ImageNet. In addition, Table 4 gives some evidence that the residual bypass is related to solver sensitivity.
Critical weaknesses
  1. I find the theoretical contribution very limited, and its connection to FROST is not convincing. To begin with, Prop. 1 essentially follows the mean value argument and lower bound on the smallest singular value of the averaged residual Jacobian. Also, Prop 2 is standard classification margin with lipschitz assumption. Both results are straightforward, but there is very weak connection to FROST. Most importantly, the paper does not verify the key assumptions required by these results for the actual trained models. Overall, I am not convinced that the theoretical analysis provided explains why FROST gives better solver agreement.

  2. I also find self-similarity motivation very weak. The authors derive the scaling rule from time rescaling angle, but I do not see why this is related to the DEQ dynamics. More critically, Section 5.4 shows that a fixed input scale achieves even smaller solver spread (thus better robustness than the dynamically coupled version). This raises lots of questions on how the fractional brownian motion motivation is actually contributing to the method

  3. The experiments show a big improvement, but they do not show what causes it. My main concern is that the “solver invariance” may mostly show improved finite-step convergence, but no distinct mechasim of prediction agreeement across numerical solvers. In table 2, the MDEQ baseline performs more or less similarly under Picard, Anderson, Broyden. The big accuracy drops happen mostly under truncated procedures. FROST also has much smaller fixed point residuals. Therefore, I do not think the experiments sufficiently distinguish improved numerical convergence from the proposed mechanism of solver invariance.

  4. The baseline quality makes the comparison harder to interpret. For example, the MDEQ checkpoint has only 56.29% ImageNet top 1 accuracy under broyden, but the released MDEQ-small reaches 75.42%. The released checkpoint still shows solver sensitivity, but its spread is much smaller than that of the paper’s baseline. This indicates that training quality effects the metric in the paper a lot. I would like to see comparisons at matched accuracy and inference cost, along with convergence curves across solvers. Without them, its hard to determine how much improvement comes from FROST rather than ambiguities in optimization/numerical convergence

Questions for authors
  1. Can authors better distinguish solver invariance from fast numerical convergence? The big accuracy drops in Table 2 mostly come from truncated/poorly converged solvers, while several other solvers give similar predictions for MDEQ.

  2. How sensitive is the main conclusion to the quality of the trained baseline? The released MDEQ checkpoint performs much better than authors’ trained baseline, also with much smaller solver spread. Can you clarify whether FROST still gives a similar advantage when both architectures reach comparable accuracy?

  3. What particular insight does the theoretical analysis provide about FROST that would not apply to a generic DEQ? For example, is there evidence that assumptions you made in Prop 1 and 2 hold for the trained FROST model, beyond the sampled proxies? Also, what role does the self-similarity prior play in the observed robustness?

Reviewer_dqe4OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies the problem of inference with deep equilibrium models (specifically, a multi scale deep equilibrium model or MDEQ) under finite computational budget. The paper notes that MDEQ’s accuracy is sensitive to the choice of solver used when the inference budget is finite. To address this issue, termed in this paper as solver agreement, the paper proposes FROST which is an alternate network parametrization of a recurrent network that introduces a learnable scalar factor, removes the additive residual connection in DEQ and rescales the input image. The method is evaluated on MDEQ architecture across seven different types of solvers, and on CIFAR-10/100, ImageNet and a 2D Poisson dataset.

Critical strengths
  1. The architectural modification in FROST is simple. The paper proposes the update as a learnable parameter λ\lambda times the weight tied block (A(zt)A(z_t)) and another scaled factor (λt+H\lambda^{t+H}) times the static image input xx, where HH is a hyper parameter. The update equation is zt+1=λA(zt)+λ1+Hxz_{t+1} = \lambda A(z_t) + \lambda^{1 + H}x.
  2. The paper evaluates performance on seven different solvers at inference time (unrolled network, Picard, Halpern, Anderson, Broyden, Inertial and Newton).
Critical weaknesses
  1. Poor presentation: The paper is unfortunately not well-written. From reading the paper, I don’t know what the proposed architectural solution “FROST” stands for. The paper suddenly introduces the acronym MDEQ in the abstract without defining it. (I am aware of MDEQs, but a well-written paper should definite all the abbreviations and acronyms correctly.) Similarly, there are many other jargons and concepts that are arbitrarily introduced without sufficient motivation or background. I have expanded on this below. In Appendix B, paper provides proof of Proposition 3 and 4, without stating it. If a reader scrolls all the way down, they would realize that it is in Appendix F. In addition, the mathematical notation in the paper lacks consistency.
  2. Unclear or vague motivation and connections to fractional Brownian motion and state space models (SSMs):
    1. The introduction and Appendix A discuss fractional Brownian motion, however, in Appendix A, it is stated that “The derivation below uses equation 13 only as a deterministic scaling ansatz, stretching time by a while shrinking amplitude by a^{−H}, and draws on no other property of the process.” Therefore, there is no reason to mention fBM in this paper. In fact, DEQs (or other central equations relevant to DEQs or recurrent neural networks) don’t have any Brownian motion. In addition, as shown in Table 4, we don’t need to remove the residual zz from DEQ completely, just damping it works too and reduces the spread.
    2. Line 197 introduces continuous dynamics of SSM, which is discretized with Euler update, however, in the next equation (5), one of the key terms of the update is arbitrarily dropped. Immediately, the paper states that ”this is an architectural construction and not an Euler approximation preserving ODE.” In general, we don’t need background in SSMs to motivate equation 5, therefore this connection is incorrect and seems arbitrary here. Perhaps a better place to mention SSMs could be in related work instead of using it to incorrectly motivate FROST.
  3. The empirical diagnostic after Proposition 5 is probably not sound. The empirical Lipschitz constant LhL_h is estimated by taking the maximum slope across 8 random perturbation directions at the three scales. This would miss the worst case adversarial direction by some factor, which would mean that estimated radius is probably inflated.
  4. I have a hypothesis that FROST achieves solver agreement by collapsing the deep recurrence into a shallow network. Initially, due to randomized weight initialization, the network propagates random features during the forward pass. One “shortcut” that the weight tied network can do in this case to optimize the network is to ignore deeper layers. To do so, the network reduces λ\lambda down to 0.02. Further, the trajectory secant ratio is 0.01 as mentioned in the paper in Line 382. Combining this, it is clear that the recurrence dies out in 2-3 steps, and the model collapses to a shallow network anchored by a static input image x. The results from Table 9 further convinces me of this hypothesis because the accuracy stays almost the same for T>=5 (and also way below the official numbers reported in MDEQ paper). Further, FROST’s shallow network outperforms DEQ with Broyden.
  5. The solver spread on DEQ might be an artifact of the choice of solvers, three of which probably don’t converge to a root under a finite budget. On the four solvers that are convergent (Picard, Anderson, Broyden, Newton), MDEQ performs well and has small spread too (0.69% in Table 2 and 0.06% on official released checkpoint). 5-step unroll is a weak solver that is bound to fail for a traditional DEQ (which is supposed to be accurate close to a fixed point or with small relative residual error). This solver works better for the FROST as it is heavily damped and has a small secant ratio along the trajectory (0.01, Line 382). Inertial solver is also weak because it uses large momentum to an undamped Picard step (unlike damped Picard which actually converges and performs well) while Halpern has an anchor drag towards z0=0z_0=0 which probably prevents the solver from reaching the root in 26 steps, and therefore performs poorly with MDEQ but does well in case of FROST because it is biased towards shallow network.
  6. Lack of theoretical guarantees for FROST: The original DEQ paper by Bai et al. 2019 and follow up works that proposed stabilized DEQ parametrizations such as monDEQ (Winston and Kolter, 2020) and pcDEQ (Gabor et al. 2024) provide proofs such as existence of a fixed point, invertibility of I−JFI - J_F for implicit function theorem, solver convergence, and preservation of expressiveness of the function class. The new parametrization proposed in this work lacks that guarantee. While the authors do mention this limitation in the paper, in my opinion, this further weakens this work. Ideally, Proposition 1 and 2 should analyze FROST’s parametrization and explain why removing the residual (+zt)(+ z_t) from DEQ parameterization and introducing scaling λ\lambda cause different solvers to have smaller residuals under finite budget? Propositions 3 and 4 in the appendix analyze the unnormalized equation without the group norm. This is an issue because group norm normalizes the variance and does not scale the Jacobian by λ\lambda. Thus, the paper does not actually analyze why the actual FROST parameterization results in solver agreement.
Questions for authors
  1. Has the baseline MDEQ correctly converged during training? The reported accuracy of 58.28% on ImageNet-1K is much weaker than the top-1 accuracy reported in the original paper which is at 75.5%.
  2. I hope the authors can address my concerns above.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.