acceptodds
Under review as a conference paper at ICLR 2027

Normalization Couples What the Objective Separates: Representation Selection in Group Composition Networks

Abstract

We study when the representation selected by a normalized network can be predicted by scoring candidate representations independently. The model is a two-layer quadratic network on a finite group's multiplication table, with real Fourier sectors as the candidates. At small logit scale the leading objective has no cross-sector terms, but the sphere-projection multiplier depends on the sum of all sector terms. A sector-local rule may score each candidate from its own Fourier coefficients and from every sector's per-layer magnitudes, but not from another sector's signs, phases or matrix directions. Let the joint input sphere have metric weight and the output sphere weight one; ordinary normalization has . For groups with at least three nontrivial real Fourier sectors, sector-local rules recover the leading selector almost everywhere if and only if , the degree-matched weight. At every other fixed weight, the errors of measurable sector-local rules under uniform initialization are bounded away from zero. The proof reflects the output block of a third sector that wins at neither start and thereby switches the winner between two rivals whose permitted inputs are unchanged. Under ordinary normalization, for each fixed group and finite width and almost every initialization, exact cross entropy inherits the leading labels at its infinite-time endpoints as the scale vanishes. At width one, with three distinct nontrivial real linear characters, the optimal sector-local error of continuous exact cross entropy is positive at every sufficiently small positive scale, even at , where it converges to zero with the scale, and uniformly over any fixed compact set of weights. On two-bit XOR under ordinary normalization, a simple sector-local rule errs on 1.7% of sampled initializations, while the leading selectors of jointly and separately normalized networks have identical label frequencies but disagree on an estimated 18% of paired starts. These examples separate independent representation competition from both approximate prediction and agreement of label frequencies.

Primary area:
learning theory
Keywords:
representation learning, implicit bias, optimization geometry, normalization, representation selection, group composition, Fourier representations, gradient flow, cross entropy, learning theory

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_GFhWOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

The paper asks whether the training endpoint of a normalized two-layer quadratic network on a finite group's multiplication table can be predicted by scoring candidate representations independently at initialization. A sector-local score may read its own complete Fourier blocks and all sectors' per-layer magnitudes, including the trivial sector, but not other sectors' orientations. For exact cross-entropy (CE), the endpoint label maximizes the product of the three per-layer sector energies; the endpoint need not have support strictly confined to one sector. For groups with at least three nontrivial real Fourier sectors, Theorem 3.1 characterizes the width-one leading selector under uniform initialization: exact sector-local ranking is possible almost everywhere if and only if the input-sphere metric weight is ω = 2. Every fixed ω ≠ 2 yields a positive error floor for the entire class. For a fixed group and finite width under ordinary normalization, Theorem 4.2 establishes convergence and leading-label inheritance for almost every initialization below a scale threshold that may depend on that initialization. It covers continuous CE and retracted iterations; the latter threshold may also depend on a fixed upper bound on the physical learning rate. At width one, for continuous CE and groups with a C₂ × C₂ quotient, Theorem 5.1 gives a positive optimal error at every sufficiently small positive scale even at ω = 2, although the error tends to zero as the scale vanishes. Retuning the scalar weight within a fixed compact subset of positive weights also fails to restore exactness. Experiments use XOR, D₄, and Q₈. On XOR, 18.43% is the disagreement between joint and separate normalization, 1.70% is the error of one particular sector-local rule, and approximately 0.0809% is the estimated optimal sector-local error. These are distinct quantities. The CE experiments read labels at a finite horizon and do not directly verify infinite-time endpoint agreement.

Critical strengths
  1. An exact characterization of the full scoring class. Theorem 3.1 constructs paired initializations with identical information available to the two rival sectors and identical magnitudes in every sector. The pair differs only by a reflection of the third sector's output block, which switches the winner between the rivals; the third sector wins at neither start. Appendix C.4-C.6 extends the construction to open sets of the full joint manifold, yielding a positive lower bound shared by all admissible measurable scoring rules. The main novelty is this class-wide converse and its coverage of real, complex, and quaternionic blocks. Matched rates and blow-up-time selection have precedents, as Section 7 acknowledges.
  2. A connection from the leading approximation to exact-loss endpoints. Theorem 4.2 combines potential gaps around Morse-Bott maximum manifolds with analytic convergence to turn finite-horizon closeness into infinite-time label inheritance, including control of discrete steps crossing the trapping boundary. Theorem 5.1 shows that positive-scale corrections destroy exact sector-local ranking at ω = 2, even with scalar metric retuning in a fixed compact set. Selected key local coefficients in Appendix E were recomputed with AI assistance and matched the manuscript; see the AI disclosure for the scope of these checks. These results distinguish accurate small-scale prediction from exact independent ranking at positive scale.
Critical weaknesses
  1. The practical error magnitude remains unclear across groups. Sections 2 and 7 explain the connection to existing selector formulas, but the obstruction concerns this particular information class. Remark F.4 and Proposition H.1 (pp. 50 and 60) imply that, on XOR, the additional exact value of the total objective f allows decoding; no noise-stability guarantee is given. The H.3 estimate of 0.0809%, with a Monte Carlo standard error of about 0.0052 percentage points, applies only to the XOR leading selector. Remark C.2 already bounds the optimal error by q_op, but the gap to the H.2 witness lower bounds remains large. Small witness lower bounds do not establish small optimal errors on D₄ and Q₈. Tighter error estimates are needed to assess the practical predictive impact across groups.
  2. Evidence at positive scale and the scope of the results remain limited. Figure 2(b), Table 3, and H.1 mainly test finite-horizon label inheritance at ω = 1; they do not directly demonstrate the reflected witnesses or boundary displacement of Theorem 5.1 at ω = 2. That theorem covers width one, continuous flow, and groups with a C₂ × C₂ quotient; larger widths and discrete updates remain open. Separately, for the leading selector, exact sector-local ranking at ω ≠ 2 remains unresolved for groups with exactly two nontrivial real sectors (Remark F.3). The Gronwall bound in D.3 is quantitatively uninformative at the reported scales, as acknowledged. Concrete reflected initializations, numerical convergence checks, and boundary displacement across scales at ω = 2 would better substantiate the positive-scale mechanism. The small-logit, full-batch setting also limits extrapolation. These are limitations of evidence and scope, rather than identified errors in the proofs.
Questions for authors
  1. Error magnitude across groups (request for additional evidence). Remark C.2 already bounds the optimal error by q_op, whose Table 2 estimates are 14.465% on D₄ and 15.845% on Q₈. Could the authors provide tighter upper bounds, for example by extending the H.3 raw-block clock rule to these groups and reporting its error with uncertainty estimates? Neither the XOR optimum estimate of 0.0809% nor the small H.2 witness lower bounds locates the optimal errors on these groups. This would affect my assessment of the practical magnitude of the results on noncommutative groups.
  2. The positive-scale mechanism at ω = 2 (request for additional evidence). Could the authors provide a specific reflected pair of initializations for exact CE at a sufficiently small positive scale, and test the distinct endpoint labels using the full system, longer horizons, and refined step sizes or tolerances? If feasible, please also show how the boundary displacement varies with scale. The existing label-inheritance experiments at ω = 1 do not directly address this.
Reviewer_23DdOpenReview ↗
Rating
3(+1)
Confidence
1
Informativeness
2
Presentation
2(+1)
Soundness
3
Read review
Paper summary

The paper asks whether we can predict which pattern a neuron will learn by scoring each possible pattern separately from the initial weights. It studies a simple two-layer network on tasks such as XOR. The main finding is that normalization can make the patterns affect each other during training, even when they contribute separately to a simplified training objective. This can prevent an independent scoring rule from predicting the final choice perfectly.

My comments focus on motivation, potential significance, and presentation. I have not checked the detailed proofs.

Critical strengths
  1. The paper raises an interesting question about how learning works. I find the example in which a pattern that is never selected still changes which of two other patterns is selected particularly helpful. It gives a concrete reason to be careful when explaining training by considering each pattern separately.
Critical weaknesses
  1. The broader motivation for exact representation prediction needs further development: The paper clearly defines the initialization-to-representation prediction problem and connects it to previous selector formulas. However, I would appreciate a more concrete explanation of why predicting this outcome exactly is an important objective, and which conclusions about learning change. For example, does the result change how we should interpret existing explanations of feature selection, or suggest a useful way to model interactions during training?

  2. The presentation is difficult to follow without familiarity with the terminology. The XOR example and Figure 1 are helpful, but the introduction moves quickly from the task to terms such as “characters” and “sectors.” I found it difficult to connect these terms to the actual trainable weights and understand what it means for a neuron to select a pattern. The full network is specified later in Equation (3). Although the necessary information is present, I would find the setup easier to follow if the authors first showed how the inputs are encoded, where the weights appear, and how one neuron produces its output.

Questions for authors
  1. What changes in the network’s behavior when it selects a different pattern? Could you give an example where this changes the network’s predictions, and explain whether different selected patterns can also lead to the same predictions?

  2. Why is perfect prediction important for the question you want to answer? If an independent rule predicts the selected pattern correctly almost all the time, what important aspect of learning would it still miss?

Reviewer_tmXLOpenReview ↗
Rating
4
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper asks whether separate representation scores can predict what training selects. It studies quadratic networks on finite-group multiplication tables, with per-neuron weight-sphere constraints and input/output metric ratio ω. For groups with at least three nontrivial real Fourier sectors, the width-one leading selector is sector-local almost everywhere exactly when ω=2. In this width-one setting, three distinct nontrivial real linear characters imply positive optimal sector-local error under uniform initialization for continuous exact cross-entropy (CE) at ω=2 and every sufficiently small positive scale s. This error nevertheless tends to zero as s→0. Numerical tests compare selectors and check CE label persistence.

Critical strengths
  1. For groups with at least three nontrivial real Fourier sectors, the width-one result identifies precisely when sector-local prediction is possible. At ω≠2, a losing sector's reflection preserves both rivals' permitted information, including all sector masses, but changes their winner, exposing the limitation of independent scoring.
  2. At ω=1 and fixed finite width, the small-scale analysis proves almost-sure label persistence at infinite-time endpoints for continuous CE and retracted iterations with bounded step sizes under absolutely continuous initialization. Trapping neighborhoods and bounds on discrete steps justify the passage from finite-time approximation to endpoints.
Critical weaknesses

The exact impossibility result leaves the accuracy of finite-scale sector-local approximation unclear. At ω=2, the optimal exact-CE error vanishes as s→0, but its finite-scale size is unquantified. XOR's numerically checked 0.081% Monte Carlo estimate concerns the leading selector at ω=1 and cannot determine this quantity.

Questions for authors

For width-one XOR at ω=2, could you plot how often the leading clock rule disagrees with continuous exact-CE endpoint labels across several fixed positive scales? Under uniform initialization, report confidence intervals and unresolved fractions, checking label stability with longer integrations. This would assess a concrete sector-local predictor's finite-scale accuracy.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.