- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper shows that the accuracy of a trained MDEQ [1] depends heavily on which solver is used at inference time, with the weights held fixed. On ImageNet the spread across seven solvers reaches 36.8 points. The authors trace this to the ungated identity path in the cell. Their parameterization, FROST, removes that path, scales the recurrent branch by a single learned scalar and sets the input scale from a self-similarity argument. FROST's accuracy spread is 5.4 points, compared with 36.8 for MDEQ.
Critical strengths
-
- I find the problem setting valuable. Treating the solver as its own evaluation axis at fixed weights is a useful idea, and the protocol is convincing: seven solvers from different families, plus per-example pairwise disagreement rather than aggregate accuracy alone.
-
- The sweep over the identity gain (Table 4) is the most informative experiment in the paper. Only one quantity changes, and it is clear that the problem appears in the regime. I also appreciate that the theoretical results are stated as sufficient conditions and are not presented as guarantees.
Critical weaknesses
-
- The main comparison is run in a regime where both models are noticeably undertrained. MDEQ reaches 56.3% top-1 and FROST 60.1%, while the released MDEQ-small checkpoint reaches 75.4% according to the authors' own Appendix H. The MDEQ fixed-point residual at the returned state is 7.38, so the model is not really at an equilibrium. On the released checkpoint the spread is 16.05, less than half the 36.84 in Table 2. The authors discuss this openly, but I think it is important to see FROST trained to a comparable accuracy.
-
- I would like to understand the mechanism better. After training, drops from 0.5 to about 0.02, the input scale to around , and the secant gain along trajectories is about 0.01. It is possible that FROST converges almost immediately and effectively behaves like a shallow weight-tied network. If so, solver invariance follows largely for free. In addition, the constant input-scale variant does just as well (spread 1.51), so it remains unclear how much the self-similarity argument based on fractional Brownian motion [2] contributes to the final result.
References:
-
- S.Bai, V.Koltun, J.Z.Kolter. Multiscale Deep Equilibrium Models. // Advances in Neural Information Processing Systems (NeurIPS), 2020.
-
- B.B.Mandelbrot, J.W.Van Ness. Fractional Brownian Motions, Fractional Noises and Applications. // SIAM Review, 10(4):422--437, 1968.
Questions for authors
-
- Could the sweep be run on the MDEQ cell (with its normalization and branch fusion) instead of the FROST cell?
-
- How many iterations, or how much wall-clock time, do FROST and MDEQ need to reach the same residual? Does FROST retain meaningful depth? ?
-
- If the secant gain is about 0.01, why does the Broyden-trained FROST lose almost 5 points under the five-step unroll?
-
- Could you report variability and robustness across several runs for the ImageNet results?