- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates the data expansion rules in pre-training/meta-learning, that is, why with the increase of meta-training data size, downstream tasks can achieve faster error convergence with fewer samples. The authors propose a complexity minimization framework, no longer directly using the adapted prediction error as the meta-learning objective, but learning a representation that minimizes the optimal downstream model complexity required by different source domains under this representation; specifically, through Lepski's method, the best model complexity of each domain is adaptively estimated from a limited sample set, and then the meta-learner minimizes the worst-case complexity across source domains. The authors further establish an end-to-end theoretical analysis from meta-training to downstream regression, proving that as the meta-training sample size m increases, the downstream convergence index of the learned representation gradually approaches the ideal index, thereby formally obtaining the data scaling law.
Critical strengths
-
The paper presents a significant new theoretical perspective: understanding the quality of representations through downstream model complexity rather than merely downstream loss. The author rephrases meta-representation learning as finding the representation that minimizes the worst-case optimal downstream model complexity across domains, and directly links this complexity to the downstream convergence rate. This aspect is crucial for my overall assessment as it not only presents a new algorithmic detail but also provides a different statistical mechanism for understanding why more pre-training data improves the few-shot sample efficiency. It has the potential to inspire subsequent research on representation complexity and scaling laws.
-
The paper presents a relatively complete end-to-end theoretical chain, rather than merely proposing a sufficient condition or a local bound. The main theorem explicitly links the meta-training data size to the downstream convergence exponent, and further constructs a complete proof through the complexity estimator, meta-learning guarantee, and downstream learner guarantee.
Critical weaknesses
-
The representation-class assumption relied upon by the main theorem has not received any meaningful specific instances to support it. Assumption 2 requires Phi to simultaneously satisfy specific metric-entropy conditions, but the authors explicitly admit that constructing a specific Phi that meets these conditions remains an open problem. This defect has a significant impact on the overall evaluation, as the paper claims to explain the data scaling law in representation learning, but it is currently unclear whether the core theorem is applicable to linear representations, networks subject to norm constraints, or other practical representation classes used in practice.
-
The experiment did not truly validate the complexity-minimization algorithm proposed in the paper, and there is a significant gap in the mechanism between the theory and the experiment. The theoretical method requires using Lepski's method to estimate the domain-wise best complexity and minimizing its worst-case value by the meta-learner; however, the experiment merely added spectral-norm and other elements to the existing meta-learning method. The intuition that "complexity regularization may improve sample efficiency" cannot directly prove that the complexity minimization in the theory is the mechanism causing the observed scaling behavior. This weakens the closed loop between theory and empirical evidence.
Questions for authors
-
Assumption 2 is of crucial importance to the main theorem of this paper. Could the author provide at least one non-trivial and interpretable representation family Phi (such as a certain type of linear representation, a finite-dimensional parameterized model, or a neural network class subject to norm constraints) that satisfies this assumption, or at least clearly state in which actual related model classes this assumption can be verified?
-
The current experiment employs spectral/L1-norm regularization instead of the Lepski-based complexity-minimization algorithm mentioned in the text. Could the authors establish a clearer theoretical or empirical correspondence between the two, for instance, directly estimating the downstream convergence exponent under different pre-training sizes m, or implementing an approximation version that is closer to domain-wise complexity minimization?
We thank all reviewers for their careful reading and insightful comments. The reviews share the following concerns, which we answer below; the revised paper gives the details.
1. A concrete class satisfying Assumption 2 (qc8Z, To9V, kp5w)
We would like to highlight that Assumption 2 is not vacuous. The revised paper gives a non-trivial and interpretable representation class, in which the representation decides how the head complexity is shared among the input coordinates, and proves both conditions of Assumption 2 for every task allowed by Assumption 1, without assuming a smoothness class. In this class, the exponent genuinely depends on the representation, and the best representation improves it. Hence, the main theorem applies to a setting where pre-training improves the downstream exponent, which is the phenomenon the paper aims to explain.
For standard neural network classes, the first condition of Assumption 2 holds when the extractor and the heads are Lipschitz in finitely many parameters. The second condition, which controls how the exponent varies over the representation class, remains open: verifying it requires the approximation exponent of each regression function under each network extractor, which is not available for ReLU networks.
2. The analysed algorithm and the exponent (qc8Z, To9V, kp5w)
We implemented the analysed algorithm itself, with the Lepski base-learner, the worst-domain meta-learner and the downstream Lepski selector, on tasks whose exponents are known. The estimated complexity tracks the population one, the meta-learner finds the representation with the ideal exponent, and the estimated exponent of the downstream error improves with m towards the ideal one. This confirms the claim of the main theorem, an improvement of the exponent, and not only a lower error at a fixed n.
The norm penalties in the neural network experiments are heuristic proxies for J. There is no formal link between the spectral or ℓ1 norm and J, and we do not claim one.
3. Assumption 1 and exactly representable targets (qc8Z)
We thank Reviewer qc8Z for pointing out the exactly representable case. We extended the analysis to exactly representable targets with the main theorem unchanged.
4. Gap between the theoretical setting and practice (To9V)
The theorem is for regression with exact sieved ERM over nested head classes. It establishes a provable mechanism by which pre-training improves the downstream exponent, not a model of classification with SGD and full fine-tuning. Extending complexity minimization to gradient-based optimization and fine-tuning is an important direction for future work.