acceptodds
Under review as a conference paper at ICLR 2027

Identity-Exact Multi-Bias Activations: Breaking Optimization Symmetry Without Changing the Pretrained Function

Abstract

A nominally three-branch activation can collapse to only two learned knot locations. In the multi-bias parameterization studied here, identity initialization ties its two coefficient- branches. Because these branches evaluate the same feature and receive equal gradients, AdamW's normalized update is identical for the two branches up to its term, preserving the tie throughout optimization. This optimizer-induced branch collapse helps explain why additional branches can match or underperform a same-site control despite the model class's greater expressivity. A multi-bias unit replaces with along a shared preactivation. With learned real coefficients, a ReLU multi-bias unit is a one-dimensional spline, represents XOR with , and, after thresholding, is Boolean complete on finite hypercubes; an exhaustive exact-arithmetic certificate further shows that some five-input Boolean function requires at least eight knots, . The deployed fixed-sign parameterization occupies a degenerate corner of this family: alternating coefficients restrict the available slope increments, while tied biases create the optimizer symmetry. Separating the knots breaks this symmetry but, in a controlled audit, perturbs the pretrained activation by up to in relative . We resolve this conflict with identity-exact nesting, in which one branch starts with coefficient one and additional branches start with coefficient zero at distinct knots. The resulting network exactly preserves the pretrained function at initialization, while the added branches receive distinct coefficient gradients on the first optimization step. Across four frozen ResNet-50 transfer tasks, nested significantly improves over its matched-site control by – accuracy points over three seeds and achieves higher training accuracy in every case. On CIFAR-100, a matched fixed-sign replication is significantly worse than that control ( points), while nested is points better: parameterization decides the effect's sign. Nested is never significantly worse than this control in any tested cell: ViT-B/16 is a four-seed statistical tie, and a 30-epoch sweep peaks at . These results show that expressivity and optimizer reachability can diverge even within a single activation family.

Primary area:
optimization
Keywords:
learnable activation functions, multi-bias activations, parameter-efficient fine-tuning, optimization dynamics, symmetry breaking, identity-preserving initialization, piecewise-linear neural networks, AdamW

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_pj7TOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies activations formed by summing shifted copies of a shared scalar activation. It analyzes restricted slope increments under fixed coefficients and persistent ties between branches, including an AdamW normalization effect for unequal coefficient magnitudes. It proposes initializing one branch with coefficient one and additional branches with zero coefficients at distinct locations, preserving the pretrained function. Supporting results include spline expressivity and a computer-assisted Boolean lower bound. Experiments show improvements over a same-site K=1 control on four frozen ResNet-50 transfer tasks, with limited benefits on ViT and longer-budget Pets experiments.

Critical strengths
  1. The paper provides concrete diagnostics of branch coincidence and distinguishes nominal branch count from the number of distinct learned knots. The explicit Boolean witness and exact-enumeration procedure also make the finite-domain lower bound auditable.

  2. The empirical reporting includes consistent improvements over the selected K=1 control across four ResNet-50 datasets, together with neutral and negative results. The discussion of longer training budgets and larger K helps establish the limitations of the approach.

Critical weaknesses
  1. APL already learns hinge locations and coefficients from a shared scalar input, and SPLASH already initializes the ReLU coefficient to one and the additional coefficients to zero. This paper uses trainable knots and channel-specific parameters, but does not clearly establish their benefit over these existing approaches. In the main (1,-1,1) example, the positive branches also remain tied under SGD; the AdamW-specific result concerns unequal coefficient magnitudes.

  2. Section 4.3 admits a simple counterexample. With fixed coefficients (1,-1,1) and independent biases (d,d,0), where d is nonzero, we have f(z)=sigma(z+d)-sigma(z+d)+sigma(z)=sigma(z). The initial function is preserved, the positive branches are already separated, and the cancelling pair can separate through opposite nonzero bias gradients.

The experiments also change coefficient training and initialization together, so they do not isolate the effect of breaking the tie. Does K=1 learn its coefficient? Table 8 suggests otherwise, and Table 5's equal-parameter-count claim needs reconciliation with Table 3. Table 13 also provides no clear advantage over APL or KAN-style activations. These comparisons leave the source of the reported gains unresolved.

References

Questions for authors
  1. What is the precise methodological advance over APL's learned hinge parameterization and SPLASH's ReLU-preserving zero-coefficient initialization? Which theoretical or experimental result establishes the importance of the remaining differences?

  2. How does the cancellation-pair initialization with coefficients (1,-1,1) and independent biases (d,d,0) affect the claim in Section 4.3? How does this function-preserving fixed-coefficient baseline compare with nested under the same training protocol?

  3. With all coefficients trainable and both initializations preserving the pretrained function, does initializing the added zero-coefficient branches at distinct knots outperform initializing them at tied knots? This comparison would directly test the claimed benefit of symmetry breaking without simultaneously changing coefficient freedom or initial function preservation.

  4. Does the K=1 control learn its coefficient? Please reconcile the parameter counts in Tables 3, 5, and 8 and report whether the improvement persists against K=1 with both a learned coefficient and a learned bias. An additional comparison with fixed versus trainable knots in the nested model would help identify the source of the gain.

1 comment

Thank you for the careful revision and the corrected framing. However, none of the eight primary contrasts is significant (nested vs. learned-amplitude K=1: +0.03/+0.56 with intervals spanning zero), and APL outperforms the method on Pets, CIFAR-100, and Food-101. With the remaining theory largely following from standard results, the revised paper no longer supports a practical contribution, so I will keep my score.

Reviewer_XE8qOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies multi-bias activations for fine-tuning frozen models. It shows that branches with the same sign and bias get locked together under AdamW. To fix this with function preserving, the authors keep one original branch and initialize the extra branches with zero weights at different bias positions.

Critical strengths
  1. The analysis of why AdamW locks same-sign branches together is clear and easy to follow.

  2. Putting zero weights on shifted branches is a simple way to preserve the initial function while breaking symmetry.

Critical weaknesses
  1. Limited comparison and advantages.

a) In Table 3 & 13, on CIFAR-100, the proposed method (82.31%) does not outperform existing methods like APL (82.48%) and KAN (82.40%). Table 2 only compares against K=1 and the authors' previous fixed-sign baseline.

b) Balancing function preservation and symmetry breaking has already been widely studied in previous work, such as Net2Net (Chen et al., ICLR 2016), Splitting Steepest Descent (Wu et al., NeurIPS 2019), GradMax (Evci et al., ICLR 2022), and SPARKLING (Yu et al., ICML 2026). The paper should also discuss and compare with these methods to show the proposed approach is reliable.

  1. Robustness concern and parameter-count discrepancy.

The default setup with gap = 0.5 and 10x bias LR does not seem robust: unpinned BiasNorm drops Pets accuracy by 11% in Table 5, performance drops for K > 1 in Table 6. Also, as far as I understood, table 5's caption claims all non-K=1 rows have the same trainable parameter count, which seems inconsistent with Table 3.

  1. Unverified contribution.

Progressive K growth in Corollary 1 is listed as a main contribution, yet its practical benefit remains experimentally unverified.

  1. Presentation.

The abstract and intro jump straight into the K=3 fixed-sign failure case without first introducing the task background or motivating multi-bias activations, which might confuse readers.

Questions for authors
  1. Is there any comparison with APL, KAN, and parameter-matched K=1 baselines on the other three main datasets besides CIFAR-100?

  2. Could the authors provide comparisons with the symmetry-breaking methods from the above mentioned related work?

  3. Please clarify how sensitive the method is to gap = 0.5, and explain the apparent discrepancy in trainable parameter counts between Table 5's caption and Table 3.

  4. Is there any experimental results for progressively growing K during training?

Reviewer_fV1zOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies an optimization symmetry in multi-bias activations and proposes an initialization that preserves the pretrained function, using distinct knots and branch coefficients initialized to zero. It combines an expressivity analysis with adaptation experiments in which the backbone is fixed.

Critical strengths
  • The symmetry analysis gives useful insight into why tied activation branches can remain redundant. The proposed initialization addresses this while preserving the pretrained function exactly.
  • The method shows consistent improvements on the ResNet transfer tasks. I appreciate the inclusion of neutral and negative results.
Critical weaknesses
  • The main comparison leaves the source of the gains unclear. The K=1 baseline fixes its amplitude, while K=3 learns amplitudes and knots. A K=1 baseline that also learns its amplitude would help isolate the benefit of adding branches.
  • The advantage of the proposed initialization is unclear. Cancelling branches appear to provide a simple alternative to the dilemma in Section 4.3, preserving the function while separating branches with coefficients of the same sign. How does the proposed method compare with this initialization in the mechanism experiment?
Questions for authors
  • Does the improvement remain when the K=1 baseline also learns its amplitude?
  • How does σ(z) − σ(z+c) + σ(z+c), with c ≠ 0 and independently trained biases, fit the claim in Section 4.3? How does this initialization perform in the mechanism experiment?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.