- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates the role of cross-stage knowledge distillation in two-stage classification systems from the perspectives of sampling variance and candidate coverage. The authors formulate sampled softmax training as an importance sampling problem and derive variance-optimal and minimax-optimal proposal distributions. They further analyze the bias and variance of the practical self-normalized estimator and establish coverage-loss and end-to-end generalization bounds under distribution alignment. Experiments on two recommendation datasets with three model combinations demonstrate reduced sampling variance and improved ranking performance.
Critical strengths
1.The paper studies a practically relevant problem in two-stage classification systems. Analyzing the candidate-generator distribution as both a sampling proposal during training and a candidate selector during inference provides a useful perspective on cross-stage knowledge distillation.
2.The paper presents a relatively comprehensive theoretical framework, including variance-optimal sampling, minimax analysis, self-normalized importance sampling, and candidate coverage bounds. The attempt to connect training efficiency with end-to-end prediction performance is meaningful.
3.The experiments cover multiple model combinations and demonstrate consistent reductions in sampling variance and improvements in final ranking performance. The controlled variance measurements provide useful empirical evidence supporting the benefits of distribution alignment.
Critical weaknesses
1.The theoretical novelty is somewhat limited. Several central results are derived from established importance sampling and variance reduction principles, and the paper does not sufficiently clarify the additional insights provided by its two-stage formulation.
2.The connection between the theoretical analysis and practical implementation requires further justification. The minimax-optimal proposal may not be achievable under the information constraints of the candidate generator, while the sampled candidate-set KL objective used in practice differs from the full-distribution alignment assumed in the theoretical analysis.
3.The experimental comparisons do not fully isolate the contribution of knowledge distillation. The Distill setting updates both stages, whereas the other baselines keep the candidate generator frozen. Consequently, the observed performance improvements may partially result from additional candidate-generator training rather than distribution alignment alone.
Questions for authors
1.Could the authors further clarify the main theoretical insights introduced by the proposed framework beyond existing importance sampling and variance reduction principles?
2.How does the minimax-optimal proposal relate to the candidate generator's limited information and representation capacity? Could the authors further explain how the practical distillation objective supports the theoretical conclusions?
3.Could the authors provide a comparison in which both stages are continuously trained without knowledge distillation under the same training budget? This would help clarify the actual contribution of distribution alignment.