- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The manuscript proposes to reverse engineer desired implicit biases using holomorphic functions from complex analysis. The new framework recovers the Hadamard product generalizing the previous implicit bias setups. Moreover, it is showcased to apply to creating a binary bias which then is illustrated on recovering an Ising model with Gaussian gradient noise injection.
Critical strengths
The work provides clearly a novel extension to the implicit bias literature. The connection to Holomorphic functions is interesting. Moreover, I can see the extended framework being used for designing quantization aware training methods as a downstream application.
Critical weaknesses
-
The current theoretical framework does not provide a guarantee for the designed implicit bias. The argument for implicit regularization guarantee informally relies on the noise of the gradient not vanishing leading to drive . Moreover, the standard implicit regularization results characterize the final resulting predictor depending normally on initialization scale or explicitly on the noise trajectory.
-
The relevance of the work is unclear. Implicit regularization classically is concerned with trying to explain the effect of overparameterization in existing or explanatory architectures. Moreover, changing the optimization geometry to promote both sparsity and quantization has been established before for example in [1,2,3]. Similarly in the considered Ising model toy example the explicitly regularized case converges fast and monotonically in contrast to the proposed parameterization, this makes it unclear what practical advantage the binary example offers. Additionally, in the binary construction setting recovers a Riemannian gradient descent step for . This suggests that the design can be done using standard design principles in at least this specific case.
[1] Jacobs, Tom and Rebekka Burkholz. “Mask in the Mirror: Implicit Sparsification.” (ICLR 2025)
[2] Kolb, Chris et al. “Smoothing the Edges: Smooth Optimization for Sparse Regularization Using Hadamard Overparametrization.” Machine Learning 115 (2023): n. pag.
[3] Taner, Sueda et al. “Cauchy-Schwarz Regularizers.” (ICLR 2025)
Questions for authors
-
Can the authors provide implicit regularization guarantees for their binary example?
-
To improve relevance can the work position itself better with respect to literature on optimization geometry design, see W2?
-
Can the authors provide a use case for which the binary inducing reparameterization has a clear advantage over explicit regularization?
3 comments
I thank the authors for their detailed answers. However my concerns regarding the design principle and advantages remain.
The mentioned methods don't just use explicit regularization they also control the implicit bias. For instance, weight decay together with the Hadamard product leads to L1 regularization on product , which matches the implicit bias preference characterized by a mirror flow. Therefore, similarly the fair comparison for binarization would be to take a parameterization that favors the binarization and then regularize this. This would possibly be a route to mitigate the dependency of the claim on the noise as well.
Furthermore, the concerns regarding the design principle remain as it is still possible to take a polynomial preconditioner. The fact that the dynamics are different does not prove an advantage. The motivation for using reparameterizations is to improve the training dynamics while keeping the ability to satisfy the desired constrained such as sparsity or quantization. It remains unclear to me how the principle can provide such an advantage beyond reducing it to a case the more general polynomial preconditioning.
I appreciate the additional empirical results and theoretical clarifications, but I do not believe these concerns have been fully addressed, I will keep my score.
We thank the reviewer for clarifying their concern. We agree that an explicit penalty can reinforce a parametrization’s implicit bias and that different dynamics alone do not establish a performance advantage. We respectfully disagree that an advantage over existing regularizers or polynomial preconditioners is the criterion for assessing our contribution. The paper makes no claim of superior training performance.
Our contribution is a theoretical framework for the inverse design of implicit bias based on holomorphic functions. We start from a preference we want training to exhibit, construct a holomorphic parametrization with the corresponding symmetry structure, and show how training noise can induce that preference under the stated conditions. Sparsity and binarization illustrate this construction. Other methods can favor the same solutions. The novelty lies in showing how to design the parametrization from the desired bias and explain how that bias arises during stochastic training.
W1 and Q1. Guarantees for the binary construction. Appendix D.5, Eq. (103), shows that once SGD reaches a binary representative (0,±1), it stays there. Our reply to Reviewer [w6ap] (W3) addresses when SGD converges to one of these points. For Ising SGD with fresh Gaussian label noise (Section 4.1), if the steps tend to zero, each coordinate converges almost surely to a binary representative.
W2 and Q2.
Relevance of the work: Characterizing the implicit bias of a learning algorithm is generally challenging, and explicit characterizations are available only in few cases. Yet the alignment between an algorithm’s implicit bias and the structure/priors of the learning problem is crucial for generalization.
This motivates our inverse-design perspective, which is the main contribution of the paper: rather than analyzing the bias of a prescribed parametrization, we seek to construct parametrizations with a prescribed geometric preference thus controlling the implicit bias. Our framework provides a specific construction to do so based on the inverse design of a parametrization and its associated symmetry and charge structure, together with an analysis of how stochastic training can select the resulting bias.
About u=0: We agree that the restricted map h(0,v)=(3v−v3)/2 already realizes the binary critical values. Our contribution is to derive this map from the prescribed targets ±1 through a holomorphic construction that gives both the predictor and its associated charge. Therefore the mechanism is completely different. Indeed the two formulations answer different dynamical questions. Fixing u=0 places the iterates on an invariant zero-charge branch from initialization, since χ(0,v)=0 and ∂uh(0,v)=0 for every v. The full parametrization allows us to initialize outside that branch and ask under what conditions training approaches it. The ablation asks whether training u improves binary recovery in this example. The charge analysis asks whether training can select the branch in the first place. Fresh label noise is part of the charge-selection mechanism studied in the paper. It can sustain gradient fluctuations when the clean gradient becomes small. In the Hadamard example, Eq. (34) shows how these fluctuations can sustain charge contraction when the second- and fourth-moment condition holds.
Q3. Advantage over an explicit regularization: Suppose we want to explicitly encourage binarization by adding a regularizer to the model’s loss. In general, however, the model’s implicit bias is unknown and may counteract this explicit regularization. This is a setting in which our method offers a clear advantage. To make this observation concrete, we modified the Ising experiment (binary oracle) by adopting a Hadamard parametrization, which favors sparsity, and adding an explicit binarization regularizer to the loss. So, in training, we have two tendencies, sparsity and binary going one against the other. The results in Table 3 show that the regularized model performs substantially worse than the model with the appropriate parametrization.
Table 3. Hadamard parametrization with the double-well regularizer, in the same setting as Table 2 (in our response to Reviewer w6ap): same randomized runs, modes, and statistics. The model writes w=u∘v and trains both factors on the regularized loss of Table 2, starting from the same latent initialization as the parameterized model. It is not part of the original search, so its hyperparameters (η=1.2, σ=0, λ=0.1) are those with the lowest mean validation MSE over the 40 randomized runs, on 2048 held-out spin configurations.