acceptodds
Withdrawn from ICLR 2027

Minimax Price of Approximate Equivariance in Linear Regression

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Equivariance is an important source of data efficiency in physical prediction and geometric perception. However, practical symmetries are often only approximate. Exact enforcement can remove predictive departures, while an unconstrained fit sacrifices parameter sharing and increases variance with limited labels. The challenge is to determine how strongly to enforce a known symmetry and what prediction error is unavoidable as the departure grows. In this paper, we formulate this choice as a minimax prediction problem in multivariate linear regression and propose projected shrinkage between exact equivariance and an unconstrained fit. We derive an exact finite-sample risk identity for this family and prove matching upper and lower bounds that characterize the global minimax prediction rate up to universal constants. The result shows that retention is governed by the balance between the possible departure and the variance required to estimate it. We extend this principle to anisotropic covariance, restricted high-dimensional designs, and sampled group averaging. Simulations recover the predicted retention transition, demonstrate the value of residual information in underdetermined designs, and connect projector accuracy to downstream prediction. These results provide a finite-sample answer to how much structure to share through an approximate symmetry when data are limited.

Primary area:
learning theory
Keywords:
approximate equivariance, minimax estimation, linear regression, shrinkage estimation, statistical learning theory, group symmetry

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_JnHKOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

This paper considers an analysis of approximate group equivariance by reformulating it as linear subspace constrained least-squares problem. Under a simple Gaussian model, they exhibit a natural bias-variance trade-off. The model is first isotropic and then made more general, and some synthetic experiments are proposed for illustration.

Critical strengths

I like the idea of reformulating the approximate equivariance problem as a constrained problem (but this idea is not really new, see https://proceedings.mlr.press/v139/finzi21a/finzi21a.pdf).

Critical weaknesses
  • Once the paper reformulates the least-squares problem into a constrained one, this is simply reusing simple traditional least-squares analysis for random design and Gaussian data. This is very classical and should be acknowledged by the authors with references, e.g., the book An Introduction to Multivariate Statistical Analysis by T. W. Anderson (1958). The particularities of equivariances are then ignored. Given that the reformulation is not new, this leads to a strong lack of novelty.
  • In terms of significance, the OLS type analysis has been superseded by more complex ones, in particular ridge regression, without the need for Gaussian data. Moreover, linear regression is too simplistic here.
  • In (7), the selected set of estimators is natural, but it does not seem to be the OLS estimator for the constrained class (unless X'X =I). This makes the math easier, but deviates from what people would do in practice.
Questions for authors
  • Theorem 2 seems to imply that this is a minimax result for all measurable estimator, as defined by M_n(rho), but it is unclear if this is in fact for the one-parameter family. Can you clarify?
  • Section 4 goes beyond isotropic, which is good, but isn't knowing Sigma too much to ask. What would you do in practice?
  • Since you are using least-squares, it seems an approach based on data augmentation would be do able and easy to implement, in particular since only moments are needed. Have you considered it?
Reviewer_Vcu4OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper asks how strongly a learner should enforce a symmetry that holds only approximately. It studies multivariate linear regression with isotropic Gaussian inputs and a known group representation. The representation splits the coefficient matrix into an equivariant part of dimension rr, which is left unrestricted, and a complementary part of dimension ss, whose norm is bounded by ρ\rho.

The proposed estimator keeps the equivariant part of the OLS solution and multiplies the rest by a factor a∈[0,1]a \in [0,1]. Here a=0a = 0 gives hard projection and a=1a = 1 gives plain OLS.

Proposition 1 gives the exact risk of this family and the best factor a⋆a^\star. Theorem 2 shows that the minimax risk is of order σ2r/n+min⁡{ρ2,σ2s/n}\sigma^2 r/n + \min\{\rho^2, \sigma^2 s/n\}, up to constants, for any estimator. Section 4 states extensions to correlated inputs, several blocks, p>np > n, sampled group averaging and an estimated noise level. All experiments are simulations.

W1. Many of the claimed results are not proved in the paper, and the proved part is close to classical theory.

The appendix proves only Proposition 1 and Theorem 2. I could not find proofs, either in the PDF or in the supplementary material (which contains only code), of Theorem 3, the multi-block rate (L248–256), Eqs. (12), (13) and (15), the claim that residual shrinkage beats restricted least squares for every ρ\rho and σ\sigma, the bound with estimated σ\sigma (L319), or the non-adaptivity result (L321–323). Contributions (2) and (3) in the introduction rest on these.

The part that is proved reduces, after a change of basis, to estimation over an unrestricted part plus an ℓ2\ell_2 ball. For this setting, minimax rates and the near-optimality of linear shrinkage are classical (Pinsker 1980; Donoho, Liu & MacGibbon 1990). Appendix E (L906–908) also says the proofs use the representation only through the rank of a projector, so the result would hold for any known subspace and does not really use the group structure.

As a result, much of what would make the paper new cannot be verified from the paper itself.

W2. The experiments are hard to interpret, the setting differs from the motivation, and the writing does not help non-experts.

All experiments are small simulations (for example q=1q = 1, p=24p = 24, r=6r = 6, s=18s = 18, n=96n = 96), with no real dataset and no non-linear or equivariant network.

In Table 1 it is not clear which rows are the proposed method and which are competitors. I believe the method is the "Family oracle" and "Validation" rows, but this should be marked. Ridge has the lowest risk at every radius (for example 0.062 against 0.117 at ρ=0.12\rho = 0.12) because the tested coefficient has a zero equivariant part. So the table measures risk at one point, while the theory is about the worst case. The proposed estimator is also not reported at all on the cyclic-shift task (L466–469). The supplementary code contains only the core estimators (least squares, ridge, shrinkage and group averaging), without the scripts, seeds or settings for the experiments, so none of the tables or figures can be reproduced from it.

Theorem 2 needs n≥2p+2n \ge 2p + 2, while the abstract and introduction stress learning from limited labels. Finally, the abstract opens with general statements before the concrete problem is clear, and the introduction gives few examples and little history of how the question arose, which makes the paper hard to follow for readers outside this sub-area.

Critical strengths

S1. The question is relevant, and the core result is clean and proved. Many works relax equivariance (Finzi et al. 2021; Romero & Lohit 2022; Wang et al. 2022), but they do not say how much relaxation is right for a given amount of data. This paper gives a precise answer in a simple model: keep the symmetry-breaking part when ρ2\rho^2 is larger than the variance of estimating it (about σ2s/n\sigma^2 s/n), and drop it otherwise. Proposition 1 and Theorem 2 are proved in Appendix C, with matching upper and lower bounds that hold for every estimator. I think this gives a clear reference point for the approximate-equivariance literature.

S2. The simulations follow the theory, and results that do not favour the method are reported honestly. The location of the lowest risk moves with ρ\rho, and a⋆a^\star follows it (Fig. 2). Sampled group averaging costs what the formula predicts (Fig. 4), and using an estimated noise level loses very little. The authors also say openly that tuned ridge has the lowest risk (L377) and that the soft penalty wins on the cyclic-shift task (L467). This makes the empirical part trustworthy, even if it is limited.

Critical weaknesses

W1. Many of the claimed results are not proved in the paper, and the proved part is close to classical theory.

The appendix proves only Proposition 1 and Theorem 2. I could not find proofs, either in the PDF or in the supplementary material (which contains only code), of Theorem 3, the multi-block rate (L248–256), Eqs. (12), (13) and (15), the claim that residual shrinkage beats restricted least squares for every ρ\rho and σ\sigma, the bound with estimated σ\sigma (L319), or the non-adaptivity result (L321–323). Contributions (2) and (3) in the introduction rest on these.

The part that is proved reduces, after a change of basis, to estimation over an unrestricted part plus an ℓ2\ell_2 ball. For this setting, minimax rates and the near-optimality of linear shrinkage are classical (Pinsker 1980; Donoho, Liu & MacGibbon 1990). Appendix E (L906–908) also says the proofs use the representation only through the rank of a projector, so the result would hold for any known subspace and does not really use the group structure.

As a result, much of what would make the paper new cannot be verified from the paper itself.

W2. The experiments are hard to interpret, the setting differs from the motivation, and the writing does not help non-experts.

All experiments are small simulations (for example q=1q = 1, p=24p = 24, r=6r = 6, s=18s = 18, n=96n = 96), with no real dataset and no non-linear or equivariant network.

In Table 1 it is not clear which rows are the proposed method and which are competitors. I believe the method is the "Family oracle" and "Validation" rows, but this should be marked. Ridge has the lowest risk at every radius (for example 0.062 against 0.117 at ρ=0.12\rho = 0.12) because the tested coefficient has a zero equivariant part. So the table measures risk at one point, while the theory is about the worst case. The proposed estimator is also not reported at all on the cyclic-shift task (L466–469). The supplementary code contains only the core estimators (least squares, ridge, shrinkage and group averaging), without the scripts, seeds or settings for the experiments, so none of the tables or figures can be reproduced from it.

Theorem 2 needs n≥2p+2n \ge 2p + 2, while the abstract and introduction stress learning from limited labels. Finally, the abstract opens with general statements before the concrete problem is clear, and the introduction gives few examples and little history of how the question arose, which makes the paper hard to follow for readers outside this sub-area.

Questions for authors

Q1. This first question is decisive for my assessment. Can you provide proofs for Theorem 3, the multi-block rate, Eqs. (12), (13) and (15), the strict-improvement claim, the bound with estimated σ\sigma, and the non-adaptivity result?

Q2. This question is about novelty. Does any of your results use the group structure beyond the dimension rr? Is there a statement that would fail for an arbitrary known subspace?

Q3. Which rows of Table 1 correspond to your method? Could you report the worst-case or averaged risk over the parameter set instead of the risk at one coefficient?

Q4. How does the tuned projected-shrinkage estimator perform on the cyclic-shift task? Also, in the supplementary code, residual_shrinkage uses a James–Stein-type rule, max⁡{0,1−(d−2)σ^2/∥⋅∥2}\max\{0, 1 - (d-2)\hat\sigma^2/\|\cdot\|^2\}, while Section 4.2 defines ares=cn/(cn+wn)a_{\text{res}} = c_n/(c_n + w_n), which depends on ρ\rho. Which rule produced Fig. 3, and could you release the experiment scripts?

Q5. As a suggestion: is there a real dataset, or a standard benchmark from the approximate-equivariance literature, where the rule could be tested?

Q6. Also a suggestion: could the abstract and introduction start from a concrete example, such as a physical system where a fixed background breaks the symmetry, so that readers outside this area can follow the motivation?

Minor: "Method I" (L285) is not defined. Theorem 3 has 1/27 as its left constant (L236), but Theorem 2 has 2/27 (L192). Some quantities in Tables 2–3 ("Loewner gap reduction", "pass rate", "runtime proxy") are not defined. The last sentence of the conclusion (L485–486) is ungrammatical.

Reviewer_pXk8OpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies how strongly to enforce a known equivariance constraint in multivariate linear regression. It models approximate equivariance by an unrestricted coefficient component in the equivariant subspace and a bounded complementary component. For isotropic Gaussian designs, it derives an exact risk formula for projected shrinkage and a minimax rate over this cylinder. It also states extensions concerning covariance geometry, underdetermined designs, and sampled group averaging. The isotropic result is proved in Appendix C, and simulations illustrate the proposed tradeoffs and extensions.

Critical strengths
  1. Proposition 1 and Theorem 2 give a clear statistical account of approximate equivariance: the exact bias-variance formula leads to an oracle shrinkage rule, while the additive lower bound establishes minimax optimality up to constants over all estimators. Distinguishing these two claims makes the central result interpretable and well supported by Appendix C.

  2. The cylinder formulation separates an unrestricted equivariant component from a bounded symmetry departure. This makes the cost of enforcing approximate symmetry explicit and explains why a method that performs well at one coefficient need not protect over the full parameter class. The discussion of the ridge comparison appropriately recognizes this distinction.

Critical weaknesses
  1. The proof is incomplete: Appendix C establishes the core isotropic results, but proofs of Theorem 3 and the high-dimensional, calibration, and adaptation claims in Sections 4.2-4.3 are absent from the PDF file and supplement. These extensions form a substantial part of the stated contribution. Their presentation also leaves important details implicit, including the corrected residual estimator, the covariance assumptions for blockwise shrinkage, and the definitions of the covariance diagnostic and filter error. Together, the missing arguments and incomplete specifications prevent a full assessment of the extensions.

  2. The advance over existing statistical theory needs to be clearer. Appendices A-B discuss equivariant regression, soft equivariance, and classical minimax estimation, but do not precisely identify what the present analysis adds. Since the core proof uses symmetry only through a known orthogonal projector, the connection to general subspace shrinkage is particularly important. A direct comparison of assumptions, guarantees, and technical obstacles would help establish which insights and arguments are new.

Questions for authors
  1. Is the excess bound v/Mv/M after equation (15) intended at the oracle choice a∗=ρ2/(ρ2+v)a^* = \rho^2/(\rho^2 + v)? At the same fixed aa, subtracting (8) from (15) gives (1−a)2(v+∥PE⊥Θ∥F2)/M(1-a)^2(v + \|P_{E^\perp}\Theta\|_F^2)/M, which exceeds v/Mv/M when a=0a = 0 and the departure is nonzero. At a∗a^*, the worst-case excess is v2/[M(ρ2+v)]v^2/[M(\rho^2 + v)], so the asserted bound holds. Please clarify the tuning and comparator, including for Figure 4.

  2. Table 2 gives (q,p,n,r,s)=(2,120,80,12,48)(q,p,n,r,s) = (2,120,80,12,48), while s=pq−rs = pq-r gives 228228. Does 4848 denote a smaller experimental departure subspace, or is it a reporting error? Please specify the subspace or group representations, departure direction, and noise scale so that the experiment can be connected to its theoretical reference risk.

  3. Section 4.1 assumes that Σ\Sigma is known, and the estimator attaining Theorem 3's upper bound uses it for whitening and selecting quotient directions. This limits the practical applicability of the anisotropic extension. What guarantees, if any, remain when Σ\Sigma is estimated? Section 5.4 considers this setting empirically, but the reported filter error does not improve consistently as the covariance diagnostic improves. Please define both quantities, explain this behavior, and clarify the additional data or accuracy requirements for the estimated-covariance procedure. If no theoretical guarantee is available, please distinguish the known-covariance benchmark from the empirical procedure and discuss this limitation explicitly.

  4. Could you clarify how much of the main minimax comparison extends to p+1<n<2p+2p+1<n<2p+2? Since the equivariant subspace can be much smaller than the ambient parameter space, it would also help to explain how the restricted-design results in Section 4.2 overcome the main estimator's dependence on full OLS, and which comparable guarantees hold when p≥np\ge n.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.