- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies when historical feedback can support the change of recommendations. The authors propose a new theoretical framework to decide whether a new slate of recommendations is sufficiently better than the old one. Based on the framework, the authors propose a new method called Shared Calibration, which combines historical borrowing and two checks to ensure a safe change in recommendations. Experiments on KuaiRand-Pure show the effectiveness of the new method and verify the authors' analysis.
Critical strengths
- The paper addresses a meaningful problem, how to safely update recommendation updates with historical feedback. The author propose a theoretic framework to solve it and evaluate on real-world data with several baseline methods.
Critical weaknesses
- The related work section just briefly discusses how the proposed method differs from existing work at a high level. The authors do not clearly explain why existing methods cannot be applied to the setting in this work, and why they combine certain existing techniques. It is hard for me to know which parts of this work's framework are new and how the authors came to this framework.
- The flow of Section 3 can be improved. I believe that Appendix A.1 connects Equations (1) and (2) to Equation (4), which should be moved into Section 3.1 or 3.2 to improve readability. Some experimental results, like Figure 3, can be moved to the Appendix instead.
- The design of the three experimental settings is not that comprehensible. For example, I do not know why the transition setting is needed or what it is intended to demonstrate. The analysis of the experimental results is also quite brief.
Questions for authors
- Could the authors explain more clearly why existing methods cannot be directly applied to the setting in this work? What motivated you to develop the proposed framework?
- In Figure 2, it is understandable that Shared performs worse than Stage-aware in the current-rich setting, since more current labels are available. However, why does Shared also perform worse in the transition setting? Could the authors provide more analysis to explain this result?
4 comments
Questions For Authors:
1. Could the authors explain more clearly why existing methods cannot be directly applied to the setting in this work?
Direct slate evaluation and prediction-powered inference apply when representative target viewing outcomes are available. We compare both in Section 4.4. Before those outcomes arrive, exposure weighting can correct which videos were seen, and standard label-shift correction can handle a change in outcome prevalence if the feedback distribution within each class stays stable. Our setting allows the like rates within those classes to change (Section 2, Appendix D.2).
Theorem 2 explains why this matters even with perfect knowledge of the target like rates. Suppose a candidate leads by 0.2 percentage points in likes, with equal contexts and no deployment error. If likes are ten percentage points more common after long views than short ones, this implies a two-point viewing gain. If the gap is forty points, the same target likes imply only half a point. Only the first clears a one-point target. The two cases can have identical historical and target-feedback distributions. Current labeled views distinguish them, which motivates calibrating the feedback relation before deciding.
2. However, why does Shared also perform worse in the transition setting?
The extra matched records in transition also strengthen Stage aware. Moving from 448 to 512 blocks adds 128 current labels to each method's fit. At the same time, target feedback falls from 16,384 to 8,192 records per video, reducing the information available to compare the slates directly (Appendix B.3).
Table 6 shows why the resulting difference is 0.25 points. Zero fits the separate-stage model while keeping Shared's error allocation for searching three borrowing levels. In transition, allowing borrowing raises mean gain from 12.64 for Zero to 14.16 for Shared, a gain of 1.52 points. Stage aware reaches 14.41 with its single-level error allocation, 1.77 points above the same Zero. The extra 1.52 points from borrowing fall just short of 1.77. In the calibration window the corresponding gains are 6.85 and 4.11, which explains Shared's lead there. These within-window comparisons use the same observations for all three methods.
In the revision, we will distinguish the new joint guarantee from its statistical ingredients in Section 2, bring the derivation into Section 3, and add the comparison with Zero beside Figure 2. We appreciate your positive assessment and would be grateful if you would revisit your rating and update it if these explanations resolve your remaining concerns.
Invitation to discussion
Dear Reviewer ,
Thank you again for your detailed review. We have provided responses to each of your comments and a revised manuscript. The main updates are summarized in our “General Response to All Reviewers.”
Could you please review these updates during the discussion period and let us know whether they address your concerns? We would be grateful if you could reconsider your assessment and update your rating as appropriate. If any points need further clarification, we would be happy to discuss them.
Best regards,
The authors
I thank the authors for their response. However, I am still not convinced by their explanation of why existing methods cannot work. For example, off-policy evaluation methods can be used to alleviate certain types of distribution shift problems in direct slate evaluation. I still feel that the discussion of related work remains insufficient and will maintain my score.
Thank you for pointing out where the derivation would help readers and for asking why the ordering of the methods changes in Figure 2.
Critical Weaknesses:
1. The related work section just briefly discusses how the proposed method differs from existing work at a high level.
The new result is a bound on the chance of approving a slate whose viewing gain falls short, after choosing both the slate and how much history to borrow. Pooling earlier records can improve precision. If likes after short or long views have changed, pooling can also bias the decision. High-confidence policy improvement therefore needs a bound on how that change affects the estimated viewing gain. The starting points for borrowing and safe policy improvement are described by Viele et al. (2014) and Thomas et al. (2015).
Equation (4) shows that the same two changes in like rates determine the bias in both the gain check and the current slope. Estimating them jointly gives one confidence set for every slate and borrowing level. In Table 5 this takes two calibration directions for Shared, compared with 121 for calibrating every distinct comparison separately. Theorem 1 combines that shared set with sampling bounds that account for repeated observations from each user and the final choice among the candidates.
2. The flow of Section 3 can be improved.
We will place the short derivation from Appendix A.1 before Equation (4). Subtracting the mean like rates of two equal-sized slates cancels their common intercept. Equations (1) and (2) then give d⊤r=aDΔ+xd⊤β+d⊤b. Subtract the context contribution xd⊤β and the required gain in like-rate units, aDτ. What remains is aD(Δ−τ)+d⊤b. Once the residual d⊤b is accounted for, a positive score establishes the desired viewing gain when the slope aD is positive. This explains why Equation (4) estimates both quantities. We will move Figure 3's full price curves to Appendix D and keep the main cost comparison in Section 4.4.
3. The design of the three experimental settings is not that comprehensible.
The three settings ask when historical records help a recommender decide whether to change a slate. The first has many target feedback records and 192 ordinary current fitting labels. The transition setting keeps those 192 labels, adds matched calibration records, and halves the target feedback per video. This lets us ask whether a better estimate of how the like–view relationship has changed can offset having fewer observations for comparing the videos. The third setting keeps the calibration and target-feedback budgets at their original levels and increases the ordinary current fitting labels from 192 to 4,096 (Appendix B.3).