- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces GSRT, that determines how much recommendation system should use a related data source and whether that information should be shared globally, across groups or individually. It uses statistical tests to avoid harmful transfers while keeping the system efficient. Their method works well on synthetic data, finds no useful transfer on KuaiRand-Pure and improves recall on production data when original model has little training data.
Critical strengths
- The authors test pre-defined hypotheses, report all results and use controls to distinguish real improvements from simple recalibration. They also identity results caused by data leakage and make their findings more trustworthy.
- Their design is practical and efficient. Production study shows that choosing weights based on ranking performance rather than prediction accuracy can be more effective in extremely data-sparse setting -- ranking based selection improves recall while recalibration helps maintain prediction quality.
Critical weaknesses
- Only positive real-data results depends on a weak, frozen model that could be improved by retraining.
- Paper uses ranking-based method rather than the brier-gated GSRT analyzed in the theory and there is no online evaluation or evidence of improvement to an actual production system.
- The theoritical predictions are broad and match the best method in only 56% of experimental cases, the predictive power of the theory is weak.
Questions for authors
- Please report per-user recall@k or recall within each group, in addition to global top-1% recall, to see whether the gains remains without cross-user reordering and better reflect performance in a candidate-generation setting.
- It is unclear whether using placement-based selection with GSRT evidence gate reproduce the same regime pattern and H1/H2 boundary results.
- The definition of weight makes it depend on H, while the proof assumes it is independent of H. Please clarify the definition.
7 comments
W2: Connecting the Ranking Variant to Its Actual Evidence
Question:
The paper uses a ranking-based method rather than the Brier-gated GSRT analyzed in the theory, and there is no online evaluation or evidence of improvement to a real production system.
Revised locations: §3.2–3.4; §4.1–4.2; Appendices I–N
Response:
The revised method separates Brier-selected and placement-selected GSRT. They share the additive score and inner-product factorization but optimize different objectives. The ranking implementation selects by AUC placements, uses count pooling, and calibrates probabilities separately; its group-gated arm does not search the full Brier hierarchy.
Section 3.4 adds the exact within-user contrast , explaining why a global allocation change can differ from a user's candidate-ordering change. The weighted-projection theory remains a probability-risk characterization, while the placement screen is an approximate test of its stated statistic. It is not a guarantee of recall or calibration improvement. Appendix M now tests the loss distinction directly in a fixed 180-cell sweep: the placement- and Brier-screened family choices differ in 32 cells. Of 64 ranking admissions, one has negative probability-weighted AUC gain. The complete results therefore characterize the ranking procedure rather than asserting that it inherits the squared-loss boundary.
The original Appendix I ranking experiment, the production-score subset evaluation and the additional 120-cell nonlinear-model study in Appendix L provide offline validation on real production data. KuaiRand and the added Ali-CCP official-sample diagnostic supply complementary public-data evaluations. The latter shows why the loss/decision distinction matters empirically: the exploratory complete policy increases mean within-user recall while reducing global AUC and quota recall, with stronger global metrics for the all-data refit. The industrial SSE figures describe a bundled system and are not attributed to GSRT. In Appendix I, the headline group candidate is ungated and the Brier gains are reported probability-based expected-loss diagnostics, which remain distinct from empirical observed-label losses. No actual GSRT online trial or creator-growth effect is claimed. The revision addresses the request for evidence on real production data while distinguishing offline validation from an online system improvement.
W3: Exact-Family Prediction and Risk Proximity Are Different Claims
Question:
The theoretical predictions are broad and match the best method in only 56% of cases, so the theory's predictive power is weak.
Revised locations: §3.3; §4.3–4.4; Appendices B, F and M
Response:
We agree that 56% agreement does not justify describing the theory as an accurate exact-family selector. The revised paper retains this original result, together with the 92% within-five-per-million result, and identifies them as two different criteria: predicting the identity of the best family and being close to its risk. The second number does not repair a weak claim about the first.
The revised analysis separates the population approximation benefit of a resolution from the local cost of estimating it. In particular, the nested-partition projection identity exactly characterizes the surrogate-risk reduction from refining an observable grouping. Turning that identity into the scalar information criterion uses additional assumptions; finite grids, shrinkage, calibration and the selection procedure introduce further differences from the simple prediction. The implemented method therefore continues to use held-out outcomes rather than replacing selection with the theoretical budget.
The original controlled figures, tables and 56% statistic remain the record of that campaign. We now label the dashed line as a local reference. The finite- condition in Theorem 3.4 instead uses , with the factor retained in . For a fixed design, tends to the global-reference information as ; identifying this limit with the plotted budget requires matching the reference probabilities, observed row set and two-group parameterization. We clarify this distinction in Section 3.3, Figures 2 and 6, Sections 4.4-4.5 and the retained H1 hypothesis. The old line is not asserted to be the exact finite- boundary. These changes correct the interpretation without recomputing the curves, experimental results or 56% statistic.
Appendix M adds an independent ranking-resolution sweep, where the fitting diagnostic and validation choice are recorded separately. It is an additional loss-specific experiment, not a recalculation of the original 56% statistic or a reproduction of that campaign.
W1: Validating on Real Production Data and Retaining the Retraining Comparison
Question:
Revised locations: §1; §4.1–4.2; Appendices G, I, J, K and N
Response:
We agree that a positive result from a limited base must be assessed against feasible retraining. We evaluate real production data offline in Appendix I and the 20-cell production-score subset study, reporting both positive transfer outcomes and evidence that a stronger base can be preferable.
We compare against a stronger base on real production data: a 3M-row target-only model exceeds the adapted 1M-row candidate on AUC in that evaluation. Likewise, the production-score subset evaluation shows a positive smaller-base upload within-user recall change, a negative camera change, and no mean user-recall improvement for the larger upload base. These are conditional offline measurements, not evidence that preserving a weak model is a sound production choice.
The existing KuaiRand study does not establish a robust positive transfer result after accounting for the shared neural base and item-feature ablation. The new real Ali-CCP sample experiment adds a same-feature all-data refit. In its exploratory convex diagnostic, the complete policy increases mean user Recall@1 from 0.02564 to 0.04359, but global AUC falls from 0.57074 to 0.56593 and quota recall from 0.02381 to 0.01905. The all-data refit reaches AUC 0.58817 and quota recall 0.04762. Thus the new experiment measures a limited benefit at one decision unit while directly preserving evidence in favor of retraining on other metrics. We retain those findings and position GSRT around the constrained adaptation problem and its statistical/serving analysis. The real-production-data validations likewise retain the achievable retraining alternative; an online deployment advantage remains unestablished.