acceptodds
Under review as a conference paper at ICLR 2027

Subspace Recovery from Repeated Binary Rules

Abstract

We study recovery of a shared latent subspace from repeated binary classification tasks whose unknown rule vectors lie in that subspace. The learner observes Gaussian inputs and their binary labels, but not the rules themselves. With one observation per independently drawn rule, the subspace is unidentifiable. With two observations, we derive the exact likelihood and Fisher information. For fixed ambient dimension and known rank 3 ≤ k < d, we determine the asymptotic minimax constant under normalized squared projector loss and give an estimator that attains it. At rank three, a raw likelihood score can have infinite variance near an incorrect pilot, which requires a regularized correction. We also study a separate synthetic transfer setting with unmarked task boundaries. Retaining temporal spectral information improves early performance on independently drawn target tasks over erased history. This experiment assumes known rank and long histories and does not constitute an empirical verification of the minimax theorem.

Primary area:
learning theory
Keywords:
representation learning, statistical learning theory, subspace estimation, meta-learning, transfer learning

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_3ryROpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper concerns the recovery of an unknown subspace from binary classification rules whose parameters lie in that subspace. The main theoretical contribution is an asymptotic minimax bound on subspace estimation error, together with an estimator that achieves it. It also presents experiments demonstrating improved few-shot learning if historical tasks are based on the same subspace as the new task.

Critical strengths

The overall setup and theory of the paper appears coherent as far as I could evaluate.

Critical weaknesses

The writing is challenging to read The setup, model, and results are not properly explained. Instead it seems the paper assumes the reader to already be familiar with the results of the authors and just lists everything up instead of properly explaining it. Example in case: 066 For every fixed d > k ≥3, we compute the Fisher information Ik in one unit subspace-rotation direction and prove ... - What is d, what is k? This is never explained. Examples like these are many in the text. This makes the paper basically unreadable for a human.

Contribution is limited The authors concede themselves that basically their entire setup (or even a more general version) has been established by previous work. An open question and why it is of particular interest is never defined in the paper, instead the contribution section lists loosely connected mathematical results of no clear broader interest.

Disconnect between theory and experiment The theoretical results show recovery for pairwise observations while the experiment uses blocks of extremely long length and do not make use of the block boundaries. In fact it seems the results do not use the theoretically optimal estimator at all. It is never explained how the experiments are supposed to support the theory. How does this relate to the derived minimax bound? Why is this never evaluated?

Questions for authors

Can you explain why the presented results are interesting to a broader audience?

How do the experimental results relate to the theory?

Reviewer_maf3OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper studies recovery of a shared subspace from binary tasks, in the smallest case where a hidden teacher in the subspace labels two Gaussian inputs with noiseless signs. It computes the Fisher information for rank at least 3, proves the exact asymptotic minimax risk, and gives a one-step estimator that attains it. A simple moment estimator is within a factor k/(k-3) of optimal. A separate synthetic transfer experiment reports a 2.8 point early gain from a spectral memory of unmarked history.

Critical strengths
  1. The theory is complete for the model, with a matching lower bound and estimator.
  2. The information constant does not depend on the ambient dimension and blows up at rank two, which is not obvious.
Critical weaknesses
  1. The contribution is marginal. Gulluk et al. (2021) [1] already showed that the symmetrized label-times-input moment estimator recovers the subspace at the parametric rate for binary tasks with Gaussian inputs. This paper adds the constant for one toy model and shows that estimator was already within k/(k-3) of optimal. The practical conclusion is that the existing method was nearly optimal. The model has no use beyond itself: noiseless signs, uniform teacher, exactly two observations per teacher, known rank, and the paper does not say what changes with label noise or more samples per task.

  2. Claims are not well supported. The theorem is never checked numerically, although a simulation is cheap. The transfer experiment uses a different estimator, data law, and head, so it tests nothing in the theory, and it is weak on its own: the true-subspace oracle beats the learned memory by only 1.3 points, and a single-teacher retrieval baseline leads by about 6 points at shot 32.

[1] Gulluk, H.I., Sun, Y., Oymak, S. and Fazel, M., 2021, June. Sample efficient subspace-based representations for nonlinear meta-learning. In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) (pp. 3685-3689). IEEE.

Questions for authors
  1. Given that the moment estimator of [1] is already within k/(k-3) of the optimal constant, what new knowledge does the exact constant give, and is there any setting, such as noisy labels or more than two samples per task, where the corrected estimator matters in practice?
  2. Since the transfer experiment does not use the estimator from the theorem and does not test the minimax rate, what claim of the paper does it support? A plot of n times the risk of both estimators against n, with the predicted constants, would be the natural check of the theory.

[1] Gulluk, H.I., Sun, Y., Oymak, S. and Fazel, M., 2021, June. Sample efficient subspace-based representations for nonlinear meta-learning. In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) (pp. 3685-3689).

Reviewer_VfQ3OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies the problem where we have a low-dimensional subspace, and in each step, a teacher vector is drawn from the subspace. Based on the teacher vector, pairs of data and responses are generated, where the responses are the binary sign values of the inner product between the input and the teacher vector. The paper answer's the question of what's the best learning guarantee of recovering the subspace. To answer the question, the paper provides upper and lower bound on the recover error.

Critical strengths
  1. The problem setting considered by the paper is fairly new, despite being quite narrow.
  2. The theoretical result is quite strong, providing both upper and lower bound.
Critical weaknesses
  1. Significance. The problem is quite restricted, and it is unclear what significance this subspac recovery setting has. The paper also did not motive the question.

  2. The result is asymptotic, with no sample complexity guarantees.

  3. Experiment does not seem to directly support the claims in the paper. In fact, it is quite confusing about what the experiment section is testing, since the experiment setting's description of the estimator is limited to "We test whether the overlapping signed statistic can improve early target learning." From no where in the paper is this "overlapping signed statistic" formally and explicitly defined.

  4. Section 5 seems disjoint from the previous sections and the purpose is unclear. It suddenly started consider "the transfer assay" with a definition in (5.1) that comes out of nowhere.

  5. Writing of the paper is pretty bad, with unclear notations. The biggest issue is the definition of Rn(d,k)R_n(d,k) in (2.2), where based on the current notation it should be nn independent and hold a constant value of 11 whenever k<d/2k < d/2. I believe what the author is trying to say is something like Rn(d,k)=inf⁡Asup⁡P∈Pd,kEP[L(A(X),P)]R_n(d,k) = \inf_{\mathcal{A}} \sup_{P\in\mathcal{P}_{d,k}}\mathbb{E}_{P}[L(\mathcal{A}(X), P)] for some learning algorithm A\mathcal{A} that maps data samples XX to estimated subpace. The current definition is not rigorous or clear. Also (2.3) suddenly started to use lower-case letters to represent samples. The objective in "experiment (2.1)" is not explicitly defined, and we are not sure which term is "differentiable". This is just a non-comprehensive list, and I suggest the author to double check the whole paper.

  6. Theorem 2.1 should be divided into two parts, with the optimal estimator's part be deferred to Section 4.

  7. Proof has important details left out, which raises soundness issues. For instance, (A.2) and (A.3) are not justified, and are just presented and claimed to be true. Before (B.6), it is unclear why ∥∥G−Ik∥∥op≤∥∥T∥∥op2\|\|G - I_k\|\|_{op}\leq \|\|T\|\|_{op}^2. At the end of page 13: "The boundedness of ZQ,L1Z_Q, L^1 differentiability of the density, continuity of the information on a compact cube, and the vanishing prior boundary terms justify these identities. One may first integrate expectations of bounded statistics over the data, then integrate over the parameter, as in Gassiat and Stoltz (2024). Every product is integrable by Cauchy–Schwarz. The likelihood score has mean zero, so the cross term with the prior score vanishes." I don't think this is how rigorous proofs should be written. These soundness issues are just a subset and it is hard to enumerate all in the review. I suggest the author to double check the proof. Without addressing the proof's soundness issues in detail, I will not consider raise the score.

Questions for authors

Why is (3.5) not good enough? How does λ\lambda compare to Ik\mathcal{I}_k?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.