- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper considers a stylized linear-Gaussian broadcast model in which independent Gaussian sources are compressed into a noisy, rank-, power-constrained shared representation. Each receiver has a noisy local observation and aims to predict an assigned partner's source using a Bayes-optimal decoder.
The main contributions are: (i) deriving the optimal average MSE and characterizing all optimal Gram matrices for (Lemma 1, Theorem 2), showing that the optimal code concentrates power on selected pairs; (ii) establishing the exact optimum under an invariant-Gram constraint (Theorem 3), which requires uniform power allocation and leads to strictly higher error in the low-rank regime; and (iii) showing that general invertible latent transformations do not improve the invariant optimum (Proposition 6), while random pair selection restores symmetry in distribution (Proposition 8).
The theoretical results are evaluated using closed-form calculations, ridge regression, and two frozen tabular models (TabPFN and TabICL).
Critical strengths
-
Rigorous characterization of optimality. Rather than simply proposing a non-equivariant construction, the paper proves optimality over all codes with unrestricted decoders and characterizes all optimal Gram matrices (Lemma 1, Theorem 2). This provides a solid theoretical foundation, and the main results (e.g., 5/9 vs. 9/13 MSE per receiver) can be verified analytically.
-
Well-defined scope. The comparison keeps rank, power, noise, and local information fixed. The authors also acknowledge the model's limitations (§6, Appendix C.3), including cases where correlated sources or different prediction targets may reverse the conclusions.
Critical weaknesses
-
The main result is somewhat expected from the model, and its broader implications seem overstated. In the low-rank regime, encoding each selected pair as allows both receivers to use their local observations to recover information about their partners. This is closely related to analog network coding, as the authors acknowledge. Since the invariant code must distribute power uniformly, its higher average MSE is not particularly surprising. The main technical contribution appears to be the rigorous characterization of optimality.
More importantly, the conclusion depends on several assumptions. First, the advantage holds for average MSE, but not for worst-case receiver performance: the unconstrained solution leaves half the receivers with no information, whereas the invariant solution treats all receivers equally. Second, Proposition 8 shows that randomization can recover the unconstrained optimum while preserving symmetry in distribution. This connection to symmetry breaking through random features in GNNs deserves discussion. Third, the authors' own experiments show that the ranking can reverse with correlated sources or different prediction targets. These limitations make it difficult to interpret the results as a general cost of equivariance in representation learning. A concrete example of how the theory could inform a practical design choice would strengthen the paper.
-
The experiments provide limited insight beyond the theoretical results, and their presentation could be substantially improved. Since the Bayes-optimal predictor is linear, ridge regression on raw inputs is sufficient in this setting. The PFN experiments mainly confirm that pretrained tabular models approach the predicted error floors, while the nonlinear-input experiments seem to reflect differences between PFNs and ridge regression rather than the main theoretical claims.
Sections 5 and Appendix D contain extensive numerical details but relatively little interpretation. For example, the paragraph beginning "Equal floors do not force equal fitted errors" (p. 8) lists numerous estimates and intervals without a clear takeaway. Implementation details, internal variable names, and lengthy reproducibility diagnostics, particularly Appendices D.9–D.10, also distract from the main findings. I would encourage the authors to focus on what the experiments establish, why those findings matter, and where the theoretical conclusions may not apply.
Questions for authors
-
Under a minimax objective or a fairness constraint across receivers, does the cost of equivariance still hold? If not, would it be more accurate to frame the main result as a cost of uniform allocation under average MSE?
-
Given Proposition 8, which shows that randomization can achieve the unconstrained optimum while preserving symmetry in distribution, in what practical setting is a deterministic, fixed equivariant encoder necessary? A concrete example involving local side information and rank or power constraints would help clarify the relevance.
-
Beyond verifying the theoretical error floors, what additional insight do the PFN experiments provide that cannot be obtained from the theory or ridge regression with the known inverse transform?
-
How does this work relate to existing research on symmetry breaking and random node features in GNNs (e.g., Sato et al., 2021; Abboud et al., 2021), as well as index coding?
Suggestions for improvement
- I suggest moving implementation and reproducibility details, such as job counts, memory usage, and non-determinism diagnostics, to the appendix. Section 5 would benefit from clearer interpretations of the results rather than extensive numerical reporting.
- An experiment with learned encoders, such as comparing equivariant and symmetry-broken GNNs on a semi-synthetic graph task, could help establish whether the predicted performance gap persists in practice and how it depends on rank and local information.