acceptodds
Under review as a conference paper at ICLR 2027

The Cost of Equivariance in Noisy Linear Representations

Abstract

Equivariance is a natural inductive bias for symmetric prediction tasks. Its benefit depends on where the symmetry is imposed. We ask what it costs to impose symmetry on a shared low-dimensional representation when receivers also have their own local features. In a paired Gaussian model, each receiver predicts its partner's coordinate from its own possibly noisy coordinate plus one common energy-constrained noisy linear broadcast; we minimize average risk over all fixed linear encoders and all measurable decoders. For up to half as many channels as input coordinates, we derive the exact minimum and characterize every optimal encoder Gram: equal-energy codes on selected coordinate pairs. On specified cycles and Boolean cubes, we also derive the exact optimum for translation-equivariant linear encoders with orthogonal latent actions. Their risk is strictly higher below the half-dimension threshold whenever local information is informative, so no decoder can recover the shortfall. Averaging optimal Grams over translations can exceed the rank budget; shared randomization instead restores symmetry in distribution. These exact floors separate information loss at the representation from approximation error at the decoder. In a prospectively specified 25-replicate CPU extension following five original replicates, frozen TabPFN-3 and TabICLv2 approach both floors; a separate ten-replicate cube study gives a second implementation check. An invertible post-noise transform preserves the floors while increasing ridge error, and both pretrained decoders retain low excess risk. The result is an exact cost of fixed encoder equivariance under matched budgets within this linear-Gaussian paired family.

Primary area:
learning theory
Keywords:
equivariance, representation learning, Gaussian estimation, linear encoders, receiver side information, Bayes risk

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_VFctOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper considers a stylized linear-Gaussian broadcast model in which nn independent Gaussian sources are compressed into a noisy, rank-rr, power-constrained shared representation. Each receiver has a noisy local observation and aims to predict an assigned partner's source using a Bayes-optimal decoder.

The main contributions are: (i) deriving the optimal average MSE and characterizing all optimal Gram matrices for r≤n/2r \leq n/2 (Lemma 1, Theorem 2), showing that the optimal code concentrates power on selected pairs; (ii) establishing the exact optimum under an invariant-Gram constraint (Theorem 3), which requires uniform power allocation and leads to strictly higher error in the low-rank regime; and (iii) showing that general invertible latent transformations do not improve the invariant optimum (Proposition 6), while random pair selection restores symmetry in distribution (Proposition 8).

The theoretical results are evaluated using closed-form calculations, ridge regression, and two frozen tabular models (TabPFN and TabICL).

Critical strengths
  1. Rigorous characterization of optimality. Rather than simply proposing a non-equivariant construction, the paper proves optimality over all codes with unrestricted decoders and characterizes all optimal Gram matrices (Lemma 1, Theorem 2). This provides a solid theoretical foundation, and the main results (e.g., 5/9 vs. 9/13 MSE per receiver) can be verified analytically.

  2. Well-defined scope. The comparison keeps rank, power, noise, and local information fixed. The authors also acknowledge the model's limitations (§6, Appendix C.3), including cases where correlated sources or different prediction targets may reverse the conclusions.

Critical weaknesses
  1. The main result is somewhat expected from the model, and its broader implications seem overstated. In the low-rank regime, encoding each selected pair as xu+xπ(u)x_u+x_{\pi(u)} allows both receivers to use their local observations to recover information about their partners. This is closely related to analog network coding, as the authors acknowledge. Since the invariant code must distribute power uniformly, its higher average MSE is not particularly surprising. The main technical contribution appears to be the rigorous characterization of optimality.

    More importantly, the conclusion depends on several assumptions. First, the advantage holds for average MSE, but not for worst-case receiver performance: the unconstrained solution leaves half the receivers with no information, whereas the invariant solution treats all receivers equally. Second, Proposition 8 shows that randomization can recover the unconstrained optimum while preserving symmetry in distribution. This connection to symmetry breaking through random features in GNNs deserves discussion. Third, the authors' own experiments show that the ranking can reverse with correlated sources or different prediction targets. These limitations make it difficult to interpret the results as a general cost of equivariance in representation learning. A concrete example of how the theory could inform a practical design choice would strengthen the paper.

  2. The experiments provide limited insight beyond the theoretical results, and their presentation could be substantially improved. Since the Bayes-optimal predictor is linear, ridge regression on raw inputs is sufficient in this setting. The PFN experiments mainly confirm that pretrained tabular models approach the predicted error floors, while the nonlinear-input experiments seem to reflect differences between PFNs and ridge regression rather than the main theoretical claims.

    Sections 5 and Appendix D contain extensive numerical details but relatively little interpretation. For example, the paragraph beginning "Equal floors do not force equal fitted errors" (p. 8) lists numerous estimates and intervals without a clear takeaway. Implementation details, internal variable names, and lengthy reproducibility diagnostics, particularly Appendices D.9–D.10, also distract from the main findings. I would encourage the authors to focus on what the experiments establish, why those findings matter, and where the theoretical conclusions may not apply.

Questions for authors
  1. Under a minimax objective or a fairness constraint across receivers, does the cost of equivariance still hold? If not, would it be more accurate to frame the main result as a cost of uniform allocation under average MSE?

  2. Given Proposition 8, which shows that randomization can achieve the unconstrained optimum while preserving symmetry in distribution, in what practical setting is a deterministic, fixed equivariant encoder necessary? A concrete example involving local side information and rank or power constraints would help clarify the relevance.

  3. Beyond verifying the theoretical error floors, what additional insight do the PFN experiments provide that cannot be obtained from the theory or ridge regression with the known inverse transform?

  4. How does this work relate to existing research on symmetry breaking and random node features in GNNs (e.g., Sato et al., 2021; Abboud et al., 2021), as well as index coding?

Suggestions for improvement

  • I suggest moving implementation and reproducibility details, such as job counts, memory usage, and non-determinism diagnostics, to the appendix. Section 5 would benefit from clearer interpretations of the results rather than extensive numerical reporting.
  • An experiment with learned encoders, such as comparing equivariant and symmetry-broken GNNs on a semi-synthetic graph task, could help establish whether the predicted performance gap persists in practice and how it depends on rank and local information.
Reviewer_nBLtOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper studies a Gaussian paired-demand problem in which each receiver predicts its partner’s coordinate from its own possibly noisy observation and a shared noisy linear broadcast. Under matched dimension and power constraints, it derives the exact minimum population risk for ranks up to half the input dimension and characterizes every optimal encoder Gram as equal-energy rank-one codes on selected demand pairs. It also derives the exact translation-equivariant optimum on specified cycles and Boolean cubes, establishing a strict average-risk penalty below the half-dimension threshold when local information is informative. Extensions address larger local observation sets, shared randomization, and context-based prediction. Synthetic experiments evaluate how closely ridge, TabPFN, and TabICL approach the derived floors, including an invertible presentation change and a prospective replication cohort.

Critical strengths
  1. Exact global optimization with a complete structural characterization. The main contribution goes beyond just exhibiting an effective pair code: it proves that no dense mixture improves on the stated construction and characterizes every optimal Gram. The localization and equality conditions are consequential because equal-power allocation or stationary-point optimization alone would not establish this result.

  2. A precise explanation of the cost of imposing symmetry on a constrained representation. The comparison holds dimension, power, and noise fixed while permitting arbitrary measurable decoding. It therefore isolates an information loss attributable to the encoder constraint. The analysis of rank growth under Gram averaging explains why ordinary symmetrization arguments do not automatically preserve feasibility. The extension to encoder-chosen invertible latent actions further strengthens the optimal-value comparison.

Critical weaknesses

The demonstrated significance is concentrated in a specialized model. The exact characterization relies on white Gaussian inputs, Gaussian noise, paired demands, and a fixed linear broadcast. The larger-neighborhood result supplies a sufficient separation certificate rather than a general characterization. The experiments usefully illustrate extraction of retained information, but provide limited evidence that the mechanism governs practical representation learning beyond this family. In particular, at the principal setting, the cycle and cube raw, canonicalized feature–target distributions agree up to an orthogonal change of latent coordinates. The cube study therefore adds an implementation check within essentially the same statistical setting, leaving generalization to substantially different source distributions or prediction tasks untested. The authors acknowledge these restrictions. My reservation concerns the breadth of the contribution rather than the validity of its stated results, and is the principal reason for recommending weak rather than clear acceptance.

Questions for authors

None that are essential to my current recommendation.

Reviewer_hfo3OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

From what I understood, the paper is arguing that imposing equivariance on a representation can hurt prediction in a specific analytically tractable case. In Section 3 they consider n Gaussian random variables that are paired, and they want one of the components of this pair to predict the value of the other component using its own value and a shared compressed message. They go on to prove (in theorems 2 and 3) that imposing an equivariant representation and a linear encoder will have lower MSE than unconstrained representations. They do some small-scale experiments to test their hypothesis.

Critical strengths
  1. The topic of studying the representational power and expressivity of equivariant models is interesting.
Critical weaknesses
  1. Unfortunately, this paper was not very pleasant to read or easy to understand. Clarity and writing quality need to be significantly improved to meet the standards. It is full of sentences that contain long strings of nonstandard technical words that haven't been introduced before. Here are a few examples"

In a paired Gaussian model, each receiver predicts its partner’s coordinate from its own possibly noisy coordinate plus one common energy-constrained noisy linear broadcast; we minimize average risk over all fixed linear encoders and all measurable decoders.

What is a paired Gaussian model? what do the authors mean by partner’s coordinate? What is energy-constrained noisy linear broadcast Here are some more examples of this:

On specified cycles and Boolean cubes, we also derive the exact optimum for translation-equivariant linear encoders with orthogonal latent actions.

An invertible post-noise transform preserves the floors while increasing ridge error, and both pretrained decoders retain low excess risk

We show that imposing the task’s translations on this bottleneck can strictly increase its Bayes risk, even when the decoder is unrestricted.

Local features are central to the mechanism. A receiver can subtract its own contribution from a broadcast containing both partners, with attenuated subtraction when its observation is noisy. Concentrating the limited channels on a few selected pairs can therefore deliver more than spreading the same energy uniformly across vertices

Pair-sum coding is established analog network coding (Katti et al., 2007); the optimization question is whether any dense mixture can improve on it, and how much enforcing fixed symmetry costs.

These are all examples from the introduction, and this persists throughout the paper and there are other examples of odd writing. There might be something interesting in this paper, but the writing is so unclear that it is impossible for me to assess its contributions.

  1. From what I could understand, it was not clear to me why studying the paired Gaussian case is interesting. The authors need to do a better job of motivating the paper and connecting it to other topics in Machine Learning.
Questions for authors
  1. What do theorems 2 and 3 tell us equivariant representation learning in equivariant CNNs, GNNs and transformers?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.