acceptodds
Under review as a conference paper at ICLR 2027

Inverse design of implicit bias by symmetry, stochasticity and holomorphic parametrizations

Abstract

Can we design a model such that its stochastic training favours a desired class of solutions without enforcing an explicit penalty? Under suitable conditions, the interplay between symmetries of a model's weight parametrization and stochastic training favours particular solutions, inducing an implicit bias. Building on this mechanism, we develop a framework for inverse-designing such biases by constructing novel parametrizations and their associated symmetries. We show how holomorphic functions make this construction and calculation simple and explicit. Specifically, we introduce a new parametrization that biases learned weights toward the binary values . Numerical experiments confirm the theoretical predictions. They also show that our parametrization reproduces the same preference induced by an explicitly regularized model without adding a penalty to the training loss.

Primary area:
learning theory
Keywords:
implicit bias, symmetry, stochastic gradient descent, holomorphic functions, optimization

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_yV8KOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The manuscript proposes to reverse engineer desired implicit biases using holomorphic functions from complex analysis. The new framework recovers the Hadamard product generalizing the previous implicit bias setups. Moreover, it is showcased to apply to creating a binary bias which then is illustrated on recovering an Ising model with Gaussian gradient noise injection.

Critical strengths

The work provides clearly a novel extension to the implicit bias literature. The connection to Holomorphic functions is interesting. Moreover, I can see the extended framework being used for designing quantization aware training methods as a downstream application.

Critical weaknesses
  1. The current theoretical framework does not provide a guarantee for the designed implicit bias. The argument for implicit regularization guarantee informally relies on the noise of the gradient not vanishing leading to drive u=0u=0. Moreover, the standard implicit regularization results characterize the final resulting predictor depending normally on initialization scale or explicitly on the noise trajectory.

  2. The relevance of the work is unclear. Implicit regularization classically is concerned with trying to explain the effect of overparameterization in existing or explanatory architectures. Moreover, changing the optimization geometry to promote both sparsity and quantization has been established before for example in [1,2,3]. Similarly in the considered Ising model toy example the explicitly regularized case converges fast and monotonically in contrast to the proposed parameterization, this makes it unclear what practical advantage the binary example offers. Additionally, in the binary construction setting u=0u = 0 recovers a Riemannian gradient descent step for vv. This suggests that the design can be done using standard design principles in at least this specific case.

[1] Jacobs, Tom and Rebekka Burkholz. “Mask in the Mirror: Implicit Sparsification.” (ICLR 2025)

[2] Kolb, Chris et al. “Smoothing the Edges: Smooth Optimization for Sparse Regularization Using Hadamard Overparametrization.” Machine Learning 115 (2023): n. pag.

[3] Taner, Sueda et al. “Cauchy-Schwarz Regularizers.” (ICLR 2025)

Questions for authors
  1. Can the authors provide implicit regularization guarantees for their binary example?

  2. To improve relevance can the work position itself better with respect to literature on optimization geometry design, see W2?

  3. Can the authors provide a use case for which the binary inducing reparameterization has a clear advantage over explicit regularization?

3 comments

W1 and Q1. Guarantees for the binary construction. Appendix D.5, Eq. (103), shows that once SGD reaches a binary representative (0,±1)(0,\pm1), it stays there. Our reply to Reviewer [w6ap] (W3) addresses when SGD converges to one of these points. For Ising SGD with fresh Gaussian label noise (Section 4.1), if the steps tend to zero, each coordinate converges almost surely to a binary representative.

W2 and Q2.

Relevance of the work: Characterizing the implicit bias of a learning algorithm is generally challenging, and explicit characterizations are available only in few cases. Yet the alignment between an algorithm’s implicit bias and the structure/priors of the learning problem is crucial for generalization.

This motivates our inverse-design perspective, which is the main contribution of the paper: rather than analyzing the bias of a prescribed parametrization, we seek to construct parametrizations with a prescribed geometric preference thus controlling the implicit bias. Our framework provides a specific construction to do so based on the inverse design of a parametrization and its associated symmetry and charge structure, together with an analysis of how stochastic training can select the resulting bias.

About u=0u=0: We agree that the restricted map h(0,v)=(3v−v3)/2h(0,v)=(3v-v^3)/2 already realizes the binary critical values. Our contribution is to derive this map from the prescribed targets ±1\pm1 through a holomorphic construction that gives both the predictor and its associated charge. Therefore the mechanism is completely different. Indeed the two formulations answer different dynamical questions. Fixing u=0u=0 places the iterates on an invariant zero-charge branch from initialization, since χ(0,v)=0\chi(0,v)=0 and ∂uh(0,v)=0\partial_u h(0,v)=0 for every vv. The full parametrization allows us to initialize outside that branch and ask under what conditions training approaches it. The ablation asks whether training uu improves binary recovery in this example. The charge analysis asks whether training can select the branch in the first place. Fresh label noise is part of the charge-selection mechanism studied in the paper. It can sustain gradient fluctuations when the clean gradient becomes small. In the Hadamard example, Eq. (34) shows how these fluctuations can sustain charge contraction when the second- and fourth-moment condition holds.

Q3. Advantage over an explicit regularization: Suppose we want to explicitly encourage binarization by adding a regularizer to the model’s loss. In general, however, the model’s implicit bias is unknown and may counteract this explicit regularization. This is a setting in which our method offers a clear advantage. To make this observation concrete, we modified the Ising experiment (binary oracle) by adopting a Hadamard parametrization, which favors sparsity, and adding an explicit binarization regularizer to the loss. So, in training, we have two tendencies, sparsity and binary going one against the other. The results in Table 3 show that the regularized model performs substantially worse than the model with the appropriate parametrization.

Table 3. Hadamard parametrization with the double-well regularizer, in the same setting as Table 2 (in our response to Reviewer w6ap): same randomized runs, modes, and statistics. The model writes w=u∘vw = u \circ v and trains both factors on the regularized loss of Table 2, starting from the same latent initialization as the parameterized model. It is not part of the original search, so its hyperparameters (η=1.2\eta = 1.2, σ=0\sigma = 0, λ=0.1\lambda = 0.1) are those with the lowest mean validation MSE over the 40 randomized runs, on 2048 held-out spin configurations.

Init seedD1D_1Test MSE
10000 init95.08495.0848.20×10−38.20{\times}10^{-3}
50000 all82.31982.3196.51×10−36.51{\times}10^{-3}
10000 initw97.94797.9479.52×10−39.52{\times}10^{-3}
50000 allw190.222190.2223.64×10−23.64{\times}10^{-2}
10001 init66.63866.6384.27×10−34.27{\times}10^{-3}
50001 all2.90×10−22.90{\times}10^{-2}4.25×10−104.25{\times}10^{-10}
10001 initw114.892114.8921.27×10−21.27{\times}10^{-2}
50001 allw82.47882.4786.85×10−36.85{\times}10^{-3}
10002 init80.20380.2036.09×10−36.09{\times}10^{-3}
Mean ±\pm CI87.260±17.52887.260{\pm}17.528(1.03±0.32)×10−2(1.03{\pm}0.32){\times}10^{-2}
Max190.222190.2223.64×10−23.64{\times}10^{-2}

I thank the authors for their detailed answers. However my concerns regarding the design principle and advantages remain.

The mentioned methods don't just use explicit regularization they also control the implicit bias. For instance, weight decay together with the Hadamard product leads to L1 regularization on product u⊙vu \odot v, which matches the implicit bias preference characterized by a mirror flow. Therefore, similarly the fair comparison for binarization would be to take a parameterization that favors the binarization and then regularize this. This would possibly be a route to mitigate the dependency of the claim on the noise as well.

Furthermore, the concerns regarding the design principle remain as it is still possible to take a polynomial preconditioner. The fact that the dynamics are different does not prove an advantage. The motivation for using reparameterizations is to improve the training dynamics while keeping the ability to satisfy the desired constrained such as sparsity or quantization. It remains unclear to me how the principle can provide such an advantage beyond reducing it to a case the more general polynomial preconditioning.

I appreciate the additional empirical results and theoretical clarifications, but I do not believe these concerns have been fully addressed, I will keep my score.

We thank the reviewer for clarifying their concern. We agree that an explicit penalty can reinforce a parametrization’s implicit bias and that different dynamics alone do not establish a performance advantage. We respectfully disagree that an advantage over existing regularizers or polynomial preconditioners is the criterion for assessing our contribution. The paper makes no claim of superior training performance.

Our contribution is a theoretical framework for the inverse design of implicit bias based on holomorphic functions. We start from a preference we want training to exhibit, construct a holomorphic parametrization with the corresponding symmetry structure, and show how training noise can induce that preference under the stated conditions. Sparsity and binarization illustrate this construction. Other methods can favor the same solutions. The novelty lies in showing how to design the parametrization from the desired bias and explain how that bias arises during stochastic training.

Reviewer_w6apOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper asks the question if it is possible to find a parametrization that favors particular solutions under SGD. The authors provides a geometric construction on how to choose a parametrization to enforce Γ0\Gamma_0 to get larger values at the desired targets via a holomorphic functions analysis, in particular for binary quantization. They provide experiments on this example that supports this construction.

Critical strengths

S1. The paper is well-written and the question is well-motivated

S2. The geometric construction sections 3.2 / 3.3 is nice

Critical weaknesses

W1. The authors do not establish that Γ0\Gamma_0 characterizes the implicit bias of SGD

W2. the role of stochastic GD is not clear, as the results of section 3 seem to extend to GD under similar contraction assumptions.

W3. the end of section 3 is misleading as the authors do not show that vkv_k converges to ±1\pm 1. So a priori we do not know if (S)GD really favors or not ±1\pm 1.

Questions for authors

please comment the weaknesses above

The experiment is done with a fixed init, and the hyperparameters are selected by knowing the true J⋆J^{\star}. Could the authors provide evidence of robustness across different init while keeping the hyperparameters fixed?

1 comment

Response to Reviewer w6ap

Thank you for recognizing our motivation and geometric construction. We clarify charge selection, coarea, and binary dynamics.

W1. What does Γ0\Gamma_0 characterize? We respectfully disagree with the premise. We interpret LIB(w)=−log⁡Γ0(w)L_{\mathrm{IB}}(w)=-\log\Gamma_0(w) (Eq. (11)) as a geometric form of implicit bias. Larger Γ0(w)\Gamma_0(w) gives more parameter volume per unit charge on the selected level, favoring ww. Appendix C states, "Neither this definition nor equation (45) asserts that SGD samples predictors according to Γ0(w)dw\Gamma_0(w) dw." The full SGD bias is much harder to characterize.

We study charge selection under SGD. For Hadamard, Eq. (34) gives a sufficient condition via contraction of the conditional charge second moment, which stochasticity can sustain when the full-batch gradient is negligible.

W2. What distinguishes SGD from GD? As the reviewer notes, the framework applies to any dynamics that contracts charge toward the selected level. For Hadamard, both GD and SGD obey χk+1=(1−η2gk2)χk\chi_{k+1}=(1-\eta^2g_k^2)\chi_k. Under the adiabatic assumption, Eq. (34) gives second-moment contraction.

The title emphasizes stochasticity. Near a clean-loss stationary point (text after Eq. (27)), GD's charge update shrinks with the full-batch gradient and stops when that gradient vanishes. We study SGD because fresh label noise keeps the gradient second moment positive there, allowing contraction to continue (Appendix B.1, Sec. 4.1). The criterion is the moment condition, however it is met.

W3. What ensures binarization? Attraction to u=0u=0 leaves vv undetermined. On this branch, h(0,v)=(3v−v3)/2h(0,v)=(3v-v^3)/2 can be nonbinary, e.g., h(0,0)=0h(0,0)=0 (Appendix D.5, Eq. (98)). The points (0,±1)(0,\pm1) are absorbing under SGD (Eq. (103)). For Ising SGD with fresh Gaussian label noise (Sec. 4.1), vanishing steps imply that each coordinate converges almost surely to one of these points.

Fresh noise can move parameters even at an exact fit (Eqs. (27) and (81)). If the predictor gradient stays away from zero infinitely often, the noise gives a uniformly positive chance of a step bounded away from zero each time. Such steps recur almost surely, so vanishing steps force the gradient to vanish. The iterates approach {(0,1),(0,−1)}\{(0,1),(0,-1)\} (Appendix D.1, Eq. (59)). Small steps prevent switching neighborhoods, so each coordinate converges to one point.

Our contribution is designing a parametrization for prescribed binary values. Whether steps vanish from a given initialization is outside our scope.

Different initializations with fixed hyperparameters. See Table 2.

Table 2. Robustness to problem instance and initialization. Fixed main-run settings are η=σ=0.6\eta=\sigma=0.6 (parameterized, Param.), η=1.2,λ=0.1\eta=1.2,\lambda=0.1 (regularized, Reg.), and η=0.6\eta=0.6 (vanilla, Van.). The first row is the main run (teacher/data/init/training seeds 7/17/27/7). The next nine show the first randomized runs by init seed and mode (10 per mode). init varies initialization only. all also varies teacher J∗J^*, data, and training noise. initw and allw repeat these modes with v0∼N(0,0.12)v_0\sim\mathcal{N}(0,0.1^2), giving every model a random, nonzero starting predictor. Mean, 95% CI (confidence interval) of the mean, and max use only the 40 randomized runs. D1=∑i<j∣wij−Jij∗∣D_1=\sum_{i<j}|w_{ij}-J^*_{ij}| covers all d=1770d=1770 couplings. Test MSE uses 2048 held-out spin configurations. Entries marked ≤\le are below FP-64 resolution. With ε=2−52\varepsilon=2^{-52}, D1≤ε1=dε=3.93×10−13D_1\le\varepsilon_1=d\varepsilon=3.93\times10^{-13} and test MSE ≤ε2=dε2=8.73×10−29\le\varepsilon_2=d\varepsilon^2=8.73\times10^{-29}.

Init seedD1D_1 Param.D1D_1 Reg. (×10−12\times10^{-12})D1D_1 Van.Test MSE Param.Test MSE Reg.Test MSE Van.
27†27^\dagger≤ε1\le\varepsilon_12.49892.964≤ε2\le\varepsilon_2≤ε2\le\varepsilon_20.409
10000 init≤ε1\le\varepsilon_12.60892.964≤ε2\le\varepsilon_2≤ε2\le\varepsilon_20.409
50000 all≤ε1\le\varepsilon_12.68914.162≤ε2\le\varepsilon_2≤ε2\le\varepsilon_20.416
10000 initw≤ε1\le\varepsilon_12.65899.892≤ε2\le\varepsilon_2≤ε2\le\varepsilon_20.413
50000 allw≤ε1\le\varepsilon_12.53925.222≤ε2\le\varepsilon_2≤ε2\le\varepsilon_20.422
10001 init≤ε1\le\varepsilon_12.42892.964≤ε2\le\varepsilon_2≤ε2\le\varepsilon_20.409
50001 all≤ε1\le\varepsilon_12.56909.328≤ε2\le\varepsilon_2≤ε2\le\varepsilon_20.372
10001 initw≤ε1\le\varepsilon_12.57905.178≤ε2\le\varepsilon_2≤ε2\le\varepsilon_20.419
50001 allw≤ε1\le\varepsilon_12.59909.135≤ε2\le\varepsilon_2≤ε2\le\varepsilon_20.378
10002 init≤ε1\le\varepsilon_12.60892.964≤ε2\le\varepsilon_2≤ε2\le\varepsilon_20.409
Mean ±\pm CI≤ε1\le\varepsilon_12.584±0.0452.584\pm0.045905.353±4.757905.353\pm4.757≤ε2\le\varepsilon_2≤ε2\le\varepsilon_20.404±0.0050.404\pm0.005
Max≤ε1\le\varepsilon_12.89940.449≤ε2\le\varepsilon_2≤ε2\le\varepsilon_20.431

†^\dagger Initialization used in the main experiment.

Reviewer_b5dfOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper introduces a parametrization of weights that encourages the learned weights to be binary. It uses an elegant formulation based on holomorphic functions, designing a polynomial from which both the weight value as well as its 'charge' can be extracted. The paper's ising experiments show that this formulation can extract binary weights when compared to a naive version.

Critical strengths

The exposition is easy to follow and insightful. The idea itself is quite elegant and lends itself to an easily testable form. The experiments are easy to understand and replicate.

Critical weaknesses

It is unclear from these experiments whether the (u,v) optimization is actually needed or if h(0,v)=v(3-v^2)/2) would be sufficient for recovering binary weights. I would request the authors to add an ablation with u=0. It also seems that the stochasticity and parametrization are confounded in the experiment since fresh noise is only added for the (u,v) parametrization.

The empirical scope is narrow. Section 4.1 evaluates one synthetic regression setting whose ground-truth coefficients exactly match the targeted binary dictionary. As such, it's hard to tell when this parametrization will recover binary weights and when it might not. An experiment applying the construction to a neural network would strengthen the evidence for broader applicability.

If the authors can show some more applications of this method (or explain why it cannot be applied in more settings), I would be happy to raise the score.

Questions for authors

What are the most general conditions in which this parametrization recovers binary weights? And would this formulation generalize to more values, e.g. 4 bit integers?

3 comments

Is the full (u,v)(u,v) optimization needed? We agree that the restricted map h(0,v)=(3v−v3)/2h(0,v)=(3v-v^3)/2 already realizes the binary critical values. Our contribution is to derive this map from the prescribed targets ±1\pm1 through a holomorphic construction that gives both the predictor and its associated charge. Therefore the mechanism is completely different and the two formulations answer different dynamical questions.

Fixing u=0u=0 places the iterates on an invariant zero-charge branch from initialization, since χ(0,v)=0\chi(0,v)=0 and ∂uh(0,v)=0\partial_u h(0,v)=0 for every vv. However, the full parametrization allows us to initialize outside that branch and ask under what conditions training approaches it. The charge analysis asks whether training can select the branch in the first place.

About fresh noise: Fresh label noise is part of the charge-selection mechanism studied in the paper. It can sustain gradient fluctuations when the clean gradient becomes small. In the Hadamard example, Eq. (34) shows how these fluctuations can sustain charge contraction when the second- and fourth-moment condition holds.

Conditions and extension to four-bit dictionaries and neural network. As discussed in our reply to reviewer [w6ap], we establish local attraction near an interpolating binary representative under stated conditions and characterize the limits of trajectories whose increments vanish. Global convergence from arbitrary initialization is out of the scope of this submission.

Experiment using a neural network: As required by the reviewer, we added a (successful) example of FashionMNIST classification with an MLP (see Table 1). For another example where our method gives a clear advantage, see also the answer to reviewer [yV8K].

Table 1. FashionMNIST classification with a bias-free MLP (two hidden layers of 512 units) trained for 100 epochs by SGD with batch size 100 and cross-entropy loss, with per-step label noise of probability ρ\rho. Entries are means ±\pm 95% confidence intervals over 5 seeds, each with a new initialization, batch order and label noise. For each model the hyperparameters are those with the highest mean validation accuracy over the 5 seeds, on 10000 images held out of the training set, among the settings with no diverged seed; η\eta, ρ\rho and, for the regularized model, λ\lambda are shown in this order. Test metrics are on the 10000 test images. The last four columns describe the distribution of the learned weights, as percentages of all weights: ∣w∣<0.9\lvert w \rvert < 0.9, within 0.10.1 of ±1\pm 1 and ∣w∣>1.1\lvert w \rvert > 1.1, which sum to 100, and, as a measure of how sharp the peaks at ±1\pm 1 are, within 0.010.01 of ±1\pm 1. Param., Reg. and Van. are the parameterized, regularized and vanilla models.

Modelη,ρ,(,λ)\eta, \rho\\,(, \lambda)Test CETest acc. (%)∣w∣<0.9\lvert w \rvert < 0.9 (%)∣w∣∈1±0.1\lvert w \rvert \in 1{\pm}0.1 (%)∣w∣>1.1\lvert w \rvert > 1.1 (%)∣w∣∈1±0.01\lvert w \rvert \in 1{\pm}0.01 (%)
Param.30, 0.20.473±0.0080.473{\pm}0.00889.96±0.3289.96{\pm}0.3266.0±0.266.0{\pm}0.229.4±0.229.4{\pm}0.24.5±0.04.5{\pm}0.09.7±0.19.7{\pm}0.1
Reg.100, 0, 0.010.574±0.0080.574{\pm}0.00890.07±0.2190.07{\pm}0.2160.8±0.560.8{\pm}0.511.9±0.211.9{\pm}0.227.3±0.527.3{\pm}0.51.2±0.01.2{\pm}0.0
Van.100, 01.008±0.0351.008{\pm}0.03590.10±0.0790.10{\pm}0.0764.4±0.664.4{\pm}0.68.2±0.18.2{\pm}0.127.5±0.527.5{\pm}0.50.8±0.00.8{\pm}0.0

About 4-bit integers: For example, the signed four-bit range −8,−7,…,7\\{-8,-7,\ldots,7\\}, we impose F′(ζ)=0F'(\zeta)=0 and F(ζ)=2iζF(\zeta)=2i\zeta at every integer ζ\zeta in the range. The first condition makes each prescribed integer a critical point, following Eq. (18), and the second sets its predictor value. With χ=Re⁡F\chi=\operatorname{Re}F and h=Im⁡F/2h=\operatorname{Im}F/2, each point (u,v)=(ζ,0)(u,v)=(\zeta,0) is therefore a zero-charge critical representative with predictor value h(ζ,0)=ζh(\zeta,0)=\zeta.

As a four-symbol example, consider recovering which base occurs at each position in a DNA sequence from indirect measurements. We encode the bases as A↦1A\mapsto 1, T↦−1T\mapsto -1, C↦2C\mapsto 2, and G↦−2G\mapsto -2. We place the critical points at these four integers by choosing

F′(ζ)=35i36(1−ζ2)(ζ−2)(ζ+2)(ζ−1)(ζ+1).F'(\zeta)=\frac{35i}{36}(1-\zeta^2)(\zeta-2)(\zeta+2)(\zeta-1)(\zeta+1).

Integrating with F(0)=0F(0)=0 gives F(±1)=±2iF(\pm1)=\pm2i and F(±2)=±4iF(\pm2)=\pm4i, so each base has a zero-charge critical representative at its encoded value.

I appreciate the authors' response. Could you please explain why the F-MNIST results seem to show a the majority of weights as < 0.9? This seems to be quite different from the results in the paper, where the recovered weights seem to be in {-1, +1} (Fig 2)

In Fig. 2, the data are generated from couplings that are exactly (±1)(\pm 1), so these values are the ground truth to be recovered. While the objective for F-MNIST is to classify images and this can be achieved with weights of other magnitudes. Our parametrization favors values near (±1)(\pm 1) without enforcing them. So as a result 66.0%66.0\% of the weights have magnitude below 0.90.9 while 29.4%29.4\% are within 0.10.1 of (±1)(\pm 1) compared with 8.2%8.2\% for the vanilla MLP.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Comment

We thank all reviewers for their comments.

We noticed that, although the title, abstract, and introduction clearly indicate that the paper develops a framework for the inverse design of implicit bias, most of the comments assess the contribution primarily by the novelty and competitiveness of the sparsity and binarization examples.

Implicit bias is difficult to characterize, yet its alignment with a problem’s structure is crucial for generalization. Our main contribution is an inverse-design framework that constructs parametrizations with prescribed geometric preferences through their symmetry and charge structure, and analyzes how stochastic training selects the resulting bias. Thus the goal is to control the bias a priori not to study it a posteriori.

As explicitly said in the work, we agree that both sparsity and binarization are well established and can be implemented by other methods, such as mirror descent and explicit regularization (although we have not found an implicit reg for binarization).

We therefore respectfully disagree that the availability of alternative methods diminishes the contribution of the holomorphic inverse-design framework that we proposed, which is the focus and the novelty of the paper. The Hadamard and binary examples simply illustrate this framework.

The remaining reviewers’ questions about convergence, the role of noise, and empirical robustness are important for assessing when the inverse designed bias is realized in training. We answer to them below.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.