- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
Q-MINO maintains a temporal bundle of EMA-filtered surrogate gradients and quantized parameter states. At each iteration, it approximately minimizes the norm of a convex combination of the stored gradients, with a quantized-state drift penalty and a current-gradient alignment constraint. A fixed number of Frank–Wolfe steps produces the direction for a proximal parameter update. The paper claims neighborhood convergence for a smooth surrogate objective through a stochastic Lyapunov/KL analysis. Experiments cover quantized synthetic optimization, a CNN on CIFAR-10 at INT8/INT4/INT2, and synthetic hyperparameter sensitivity studies.
Critical strengths
A coherent algorithmic construction with a feasible fallback. Combining temporal gradient aggregation with a penalty on stale quantized states is a reasonable approach to QAT instability. The current-gradient vertex is feasible for the alignment constraint, including when historical weights are no longer feasible. This gives the subproblem a clear interpretation and a valid fallback, although it does not establish convergence or an empirical advantage.
Critical weaknesses
-
The supplied proof does not establish the central convergence claim. Appendix A, p.14, asserts a contraction of the tracking quantity with forcing only of order . A fixed-decay EMA of nonvanishing stochastic noise retains nonzero variance even at a fixed iterate, so this recursion needs a noise term and a derivation for the changing bundle direction. Equation (22) also contains , which is nonsummable for fixed under the theorem's stepsize assumptions, while p.16 invokes Robbins–Siegmund with instead. The same page assumes compact sublevel containment without a supporting boundedness assumption and concludes distance convergence from a liminf stationarity estimate and vanishing increments, which are insufficient. There is a further implementation/proof mismatch: p.12 uses the norm bound implied by exact subproblem optimality, but the fixed-length Frank–Wolfe solver need not satisfy that bound, as the explicit example below shows. These gaps affect the main result.
-
The empirical scope and related-work comparisons are insufficient to establish practical significance. Experiments are limited to 10-dimensional Rosenbrock/Rastrigin problems, a CNN on CIFAR-10, and synthetic sensitivity studies. There is no large-scale evaluation on ImageNet or larger vision/language models, despite the introduction's motivation from large-model deployment. This matters particularly because Section 5.2 acknowledges storage of twice the bundle size in full parameter-length auxiliary vectors, plus a Frank–Wolfe solve at every update. Without measured peak training memory, throughput, or time to a matched accuracy as model size increases, scalability and the accuracy–cost tradeoff remain unestablished. The acknowledged compute limitations explain the restricted evaluation but do not supply evidence for larger-scale applicability.
Even within the tested setting, Table 1 reports Q-MINO accuracies of 76.63%, 75.06%, and 74.55%, versus Adam+STE's 83.79%, 83.32%, and 76.09% at INT8, INT4, and INT2. A smaller cross-bit accuracy drop does not establish an advantage when absolute accuracy remains lower. Seed variability and a complete training recipe are missing, and the sensitivity studies do not isolate the bundle against EMA/momentum alone or remove the drift and alignment components.
Important related works are also absent: ProxQuant (Bai et al., ICLR 2019), which studies proximal formulations and stability for quantized training; Learned Step Size Quantization (Esser et al., ICLR 2020), an established low-bit QAT approach evaluated on ImageNet; and Oscillation-free Quantization for Low-bit Vision Transformers (Liu et al., ICML 2023), which directly addresses quantization-induced oscillations. The minimum-norm construction also needs positioning relative to Burke, Lewis, and Overton's gradient-sampling method (2005). Its sampled gradients differ from Q-MINO's temporal surrogate-gradient bundle, so its guarantees do not transfer automatically. Nagel et al. (2022) is already cited, but its oscillation-control methods are not experimentally compared. The concern is both incomplete attribution and a lack of evidence that the proposed mechanism adds value beyond relevant existing approaches; these methods are not all interchangeable optimizer baselines.
Questions for authors
-
Convergence clarification. Can you derive a valid conditional recursion for that includes the stochastic-noise contribution, and supply the boundedness and full-sequence convergence argument? How is the term in Equation (22) handled without replacing it by a summable term, and what precise stationary-neighborhood guarantee follows?
-
Finite-solver clarification. How does the implementation ensure the norm bound used on p.12? Take current gradient , historical gradient , , , and initialize at the current-gradient vertex. Both vertices are feasible. With the stated Frank–Wolfe stepsizes and five iterations, the historical weight becomes , giving and , the squared norm of the current gradient. Is an objective safeguard or an inexact-solve term missing from the algorithm and analysis?
-
Empirical scope and related work. What existing measurements support scalability beyond the CIFAR-10 CNN, including peak training memory, throughput, and time to a matched accuracy with bundle storage and the inner solver included? Please provide the training/quantization configuration, tuning protocol, seed variability, and any matched EMA/momentum or oscillation-control comparisons. How does Q-MINO differ from the omitted quantization and minimum-norm methods identified above, and which benefits are established despite its lower accuracy than Adam+STE? Larger-model benchmarks and component ablations would strengthen a revision; the immediate clarification is what the current evidence supports.