acceptodds
Under review as a conference paper at ICLR 2027

Knowledge Distillation for Improved Generalization in Two-Stage Classification

Abstract

Large-scale classification often uses a two-stage architecture: a lightweight candidate generator selects a small subset of classes, and a stronger target classifier scores them. Recent work has improved these systems through cross-stage training, using techniques such as knowledge distillation and joint optimization. Despite these empirical advances, the effect of the candidate-generator distribution on target-classifier training remains poorly understood. To explain this mechanism, we formulate sampled softmax training as importance sampling SGD, using the candidate generator distribution as the proposal. We also develop a variance reduction analysis of cross-stage distillation. In the context of cross-stage information asymmetry, the output distribution of the target classifier is minimax-optimal for the unbiased estimator. For the practical self-normalized estimator, alignment controls the variance and bias bounds. These results demonstrate how distillation enhances training efficiency and convergence. In addition to providing training samples, the candidate generator determines which classes are available during inference. To analyze this role, we derive conditions for teacher quality and candidate generator capacity under which positive weight distillation strictly tightens the coverage loss upper bound relative to independent training. Combining these results, we obtain an end-to-end generalization bound that considers both coverage loss and target-classifier error. Experiments on real-world datasets demonstrate that distillation yields the lowest gradient variance among non-Oracle proposals. It also improves final accuracy across three model combinations. Convergence experiments on two combinations further demonstrate faster training.

Primary area:
learning theory
Keywords:
Knowledge Distillation, Two-Stage Classification, Generalization Bounds, Variance Reduction, Importance Sampling

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_DC5ROpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the role of cross-stage knowledge distillation in two-stage classification systems from the perspectives of sampling variance and candidate coverage. The authors formulate sampled softmax training as an importance sampling problem and derive variance-optimal and minimax-optimal proposal distributions. They further analyze the bias and variance of the practical self-normalized estimator and establish coverage-loss and end-to-end generalization bounds under distribution alignment. Experiments on two recommendation datasets with three model combinations demonstrate reduced sampling variance and improved ranking performance.

Critical strengths

1.The paper studies a practically relevant problem in two-stage classification systems. Analyzing the candidate-generator distribution as both a sampling proposal during training and a candidate selector during inference provides a useful perspective on cross-stage knowledge distillation.

2.The paper presents a relatively comprehensive theoretical framework, including variance-optimal sampling, minimax analysis, self-normalized importance sampling, and candidate coverage bounds. The attempt to connect training efficiency with end-to-end prediction performance is meaningful.

3.The experiments cover multiple model combinations and demonstrate consistent reductions in sampling variance and improvements in final ranking performance. The controlled variance measurements provide useful empirical evidence supporting the benefits of distribution alignment.

Critical weaknesses

1.The theoretical novelty is somewhat limited. Several central results are derived from established importance sampling and variance reduction principles, and the paper does not sufficiently clarify the additional insights provided by its two-stage formulation.

2.The connection between the theoretical analysis and practical implementation requires further justification. The minimax-optimal proposal may not be achievable under the information constraints of the candidate generator, while the sampled candidate-set KL objective used in practice differs from the full-distribution alignment assumed in the theoretical analysis.

3.The experimental comparisons do not fully isolate the contribution of knowledge distillation. The Distill setting updates both stages, whereas the other baselines keep the candidate generator frozen. Consequently, the observed performance improvements may partially result from additional candidate-generator training rather than distribution alignment alone.

Questions for authors

1.Could the authors further clarify the main theoretical insights introduced by the proposed framework beyond existing importance sampling and variance reduction principles?

2.How does the minimax-optimal proposal relate to the candidate generator's limited information and representation capacity? Could the authors further explain how the practical distillation objective supports the theoretical conclusions?

3.Could the authors provide a comparison in which both stages are continuously trained without knowledge distillation under the same training budget? This would help clarify the actual contribution of distribution alignment.

Reviewer_GFAjOpenReview ↗
Rating
2
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies cross-stage knowledge distillation in two-stage classification, where the candidate generator serves as both the proposal for sampled-softmax training and the retriever at inference. It derives an optimal proposal for an unbiased importance-sampling estimator and a minimax solution under gradient-norm uncertainty, then bounds the variance and bias of a practical self-normalized estimator. A separate analysis bounds candidate coverage loss and combines it with teacher error in a population top-1 risk decomposition. Experiments on two recommendation datasets and three model pairs report lower non-Oracle sampling variance and improved final ranking metrics with distillation.

Critical strengths
  1. Separating the candidate generator's training-proposal role from its inference-coverage role is useful. The unbiased importance-sampling analysis is explicit about gradient-norm information, and the coverage analysis states teacher-quality and model-capacity conditions rather than claiming universal benefit from distillation (Theorems 4.2–4.3 and 5.1–5.3).

  2. RQ1 provides a meaningful controlled comparison: the ranker is fixed and Distill and Retriever receive matched retriever updates. Across all six dataset–model settings, Distill has the lowest measured non-Oracle sampling variance and empirical chi-square divergence (Table 1; Appendix E.1). This supports the proposed mechanism empirically within those settings.

Critical weaknesses
  1. The connection from the unbiased minimax result to the practical estimator and implemented distillation is overstated. Theorem 4.3 minimizes a worst-case second-moment criterion; Proposition 4.4 minimizes a bound, not the actual SNIS variance. For Eq.(7), with P=(0.8,0.2), score gradients (1,-1), positive class 1 and K=20, exact enumeration gives conditional variance 0.029025 at Q=P, versus 0.018854 at Q=(0.5,0.5). This does not contradict the stated minimax second-moment theorem, but rules out an unrestricted practical variance-optimality reading of the surrounding text. Moreover, Appendix E.1 uses KL between distributions renormalized on 100 sampled candidates, whereas the theory controls full-distribution chi-square quantities and population KL objectives. Decreasing KL does not in general guarantee decreasing chi-square divergence. The paper needs a more precise scope and a justified bridge to its implementation. Related adaptive sampled-softmax analyses by Blanc and Rendle (ICML 2018) and Rawat et al. (NeurIPS 2019) should also be discussed explicitly.

  2. RQ2 confounds the mechanism with additional retriever adaptation and changed inference candidates. Distill updates both stages, whereas Retriever, Uniform and Top-M freeze the pretrained retriever (p.9; Appendix E.1.5). Consequently, the final metric and convergence improvements cannot be attributed specifically to variance reduction. A matched continued-training retriever without distillation, and a comparison holding inference candidates fixed, would materially strengthen the claim. The experiments also use catalogs of approximately 8k–9k items, so they do not establish computational scalability to the much larger class spaces motivating the work.

Questions for authors
  1. Which optimality claim is intended for Eq.(7): actual conditional variance, a minimax objective, or the displayed upper bound? Please address the finite-K example above and distinguish your estimator from conventional sampled-softmax estimators with different positive/negative corrections.
  2. What justifies applying full-distribution KL/chi-square guarantees to the sampled candidate-set KL objective? Can you quantify full-distribution alignment and weight tails, and clarify whether repeated sampled items retain multiplicity in Eq.(6) and the implementation?
  3. Can you report RQ2 with matched retriever updates without distillation and with fixed inference candidate sets, together with wall-clock costs? Please also give seed counts and dispersion for all model pairs and distinguish Table 8's peak validation recall from held-out test performance.
Reviewer_DszdOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

Considering the effect of the candidate-generator distribution on target-classifier training, this manuscript develops a variance reduction analysis of cross-stage distillation. In the context of cross-stage information asymmetry, the output distribution of the target classifier is minimax-optimal for the unbiased estimator. For the practical self-normalized estimator, alignment controls the variance and bias bounds. Then, authors derive conditions for teacher quality and candidate generator capacity under which positive weight distillation strictly tightens the coverage loss upper bound relative to independent training.

Critical strengths

This manuscript presents a series of theoretical analyses on cross-stage distillation and gradient variance reduction. An end-to-end error decomposition links candidate coverage to target-classifier error.

Critical weaknesses

The theoretical analysis of this manuscript has not been fully verified through experiments.

Questions for authors

1.The assumptions underlying some theorems and corollaries are not stated sufficiently clearly, such as Theorem 4.1.

2.The proof of Theorem 4.3 utilizes the way of the inner maximization and then the outer minimization. What justifies this optimization order? Is this approach reasonable? Does it adequately reflect real-world data?

3.Many theoretical results are derived under specific assumptions. However, do these assumptions align with the actual distribution of big data? How can you prove it?

4.The manuscript motivates its study by the computational challenges of full softmax in settings with millions of classes. However, the experimental item catalogs contain only 9,175 and 8,360 items. Could the authors discuss the applicability of their theoretical and empirical findings to larger class spaces, and clarify how the effectiveness of variance reduction and the computational cost of training may change as the number of classes increases?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.