acceptodds
Under review as a conference paper at ICLR 2027

When is Historical Feedback Enough to Change a Recommendation?

Abstract

Historical feedback can help a recommender decide whether to change its recommendations. However, more likes need not mean more sustained viewing, and the relation between these responses can change over time. We study when historical observations provide enough evidence for an improvement despite this uncertainty. Our method estimates changes in the feedback relation jointly and uses the same calibration set for every proposed recommendation. It checks both whether the estimated gain is large enough and whether the current feedback still points in the right direction. The decision accounts for repeated observations from a user and for choosing how much history to use. Under the stated assumptions, the rule bounds the chance of accepting a change that falls short of the required gain. A KuaiRand-Pure study shows that the benefit is regime dependent: shared calibration is useful in a calibration-limited matched setting, while fixed borrowing, separate-stage fitting or additional current labels can be preferable elsewhere. These findings connect the value of historical feedback to the information still needed before a recommendation changes.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
Recommender Systems, Uncertainty Quantification, Safe Decision Making, Historical Data Borrowing, Calibration

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_i5Z3OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies when historical feedback can support the change of recommendations. The authors propose a new theoretical framework to decide whether a new slate of recommendations is sufficiently better than the old one. Based on the framework, the authors propose a new method called Shared Calibration, which combines historical borrowing and two checks to ensure a safe change in recommendations. Experiments on KuaiRand-Pure show the effectiveness of the new method and verify the authors' analysis.

Critical strengths
  1. The paper addresses a meaningful problem, how to safely update recommendation updates with historical feedback. The author propose a theoretic framework to solve it and evaluate on real-world data with several baseline methods.
Critical weaknesses
  1. The related work section just briefly discusses how the proposed method differs from existing work at a high level. The authors do not clearly explain why existing methods cannot be applied to the setting in this work, and why they combine certain existing techniques. It is hard for me to know which parts of this work's framework are new and how the authors came to this framework.
  2. The flow of Section 3 can be improved. I believe that Appendix A.1 connects Equations (1) and (2) to Equation (4), which should be moved into Section 3.1 or 3.2 to improve readability. Some experimental results, like Figure 3, can be moved to the Appendix instead.
  3. The design of the three experimental settings is not that comprehensible. For example, I do not know why the transition setting is needed or what it is intended to demonstrate. The analysis of the experimental results is also quite brief.
Questions for authors
  1. Could the authors explain more clearly why existing methods cannot be directly applied to the setting in this work? What motivated you to develop the proposed framework?
  2. In Figure 2, it is understandable that Shared performs worse than Stage-aware in the current-rich setting, since more current labels are available. However, why does Shared also perform worse in the transition setting? Could the authors provide more analysis to explain this result?
4 comments

Thank you for pointing out where the derivation would help readers and for asking why the ordering of the methods changes in Figure 2.

Critical Weaknesses:

1. The related work section just briefly discusses how the proposed method differs from existing work at a high level.

The new result is a bound on the chance of approving a slate whose viewing gain falls short, after choosing both the slate and how much history to borrow. Pooling earlier records can improve precision. If likes after short or long views have changed, pooling can also bias the decision. High-confidence policy improvement therefore needs a bound on how that change affects the estimated viewing gain. The starting points for borrowing and safe policy improvement are described by Viele et al. (2014) and Thomas et al. (2015).

Equation (4) shows that the same two changes in like rates determine the bias in both the gain check and the current slope. Estimating them jointly gives one confidence set for every slate and borrowing level. In Table 5 this takes two calibration directions for Shared, compared with 121 for calibrating every distinct comparison separately. Theorem 1 combines that shared set with sampling bounds that account for repeated observations from each user and the final choice among the candidates.

2. The flow of Section 3 can be improved.

We will place the short derivation from Appendix A.1 before Equation (4). Subtracting the mean like rates of two equal-sized slates cancels their common intercept. Equations (1) and (2) then give d⊤r=aDΔ+xd⊤β+d⊤bd^\top r=a_D\Delta+x_d^\top\beta+d^\top b. Subtract the context contribution xd⊤βx_d^\top\beta and the required gain in like-rate units, aDτa_D\tau. What remains is aD(Δ−τ)+d⊤ba_D(\Delta-\tau)+d^\top b. Once the residual d⊤bd^\top b is accounted for, a positive score establishes the desired viewing gain when the slope aDa_D is positive. This explains why Equation (4) estimates both quantities. We will move Figure 3's full price curves to Appendix D and keep the main cost comparison in Section 4.4.

3. The design of the three experimental settings is not that comprehensible.

The three settings ask when historical records help a recommender decide whether to change a slate. The first has many target feedback records and 192 ordinary current fitting labels. The transition setting keeps those 192 labels, adds matched calibration records, and halves the target feedback per video. This lets us ask whether a better estimate of how the like–view relationship has changed can offset having fewer observations for comparing the videos. The third setting keeps the calibration and target-feedback budgets at their original levels and increases the ordinary current fitting labels from 192 to 4,096 (Appendix B.3).

Questions For Authors:

1. Could the authors explain more clearly why existing methods cannot be directly applied to the setting in this work?

Direct slate evaluation and prediction-powered inference apply when representative target viewing outcomes are available. We compare both in Section 4.4. Before those outcomes arrive, exposure weighting can correct which videos were seen, and standard label-shift correction can handle a change in outcome prevalence if the feedback distribution within each class stays stable. Our setting allows the like rates within those classes to change (Section 2, Appendix D.2).

Theorem 2 explains why this matters even with perfect knowledge of the target like rates. Suppose a candidate leads by 0.2 percentage points in likes, with equal contexts and no deployment error. If likes are ten percentage points more common after long views than short ones, this implies a two-point viewing gain. If the gap is forty points, the same target likes imply only half a point. Only the first clears a one-point target. The two cases can have identical historical and target-feedback distributions. Current labeled views distinguish them, which motivates calibrating the feedback relation before deciding.

2. However, why does Shared also perform worse in the transition setting?

The extra matched records in transition also strengthen Stage aware. Moving from 448 to 512 blocks adds 128 current labels to each method's fit. At the same time, target feedback falls from 16,384 to 8,192 records per video, reducing the information available to compare the slates directly (Appendix B.3).

Table 6 shows why the resulting difference is 0.25 points. Zero fits the separate-stage model while keeping Shared's error allocation for searching three borrowing levels. In transition, allowing borrowing raises mean gain from 12.64 for Zero to 14.16 for Shared, a gain of 1.52 points. Stage aware reaches 14.41 with its single-level error allocation, 1.77 points above the same Zero. The extra 1.52 points from borrowing fall just short of 1.77. In the calibration window the corresponding gains are 6.85 and 4.11, which explains Shared's lead there. These within-window comparisons use the same observations for all three methods.

In the revision, we will distinguish the new joint guarantee from its statistical ingredients in Section 2, bring the derivation into Section 3, and add the comparison with Zero beside Figure 2. We appreciate your positive assessment and would be grateful if you would revisit your rating and update it if these explanations resolve your remaining concerns.

AuthorsOpenReview ↗

Invitation to discussion

Dear Reviewer i5Z3\textbf{i5Z3},

Thank you again for your detailed review. We have provided responses to each of your comments and a revised manuscript. The main updates are summarized in our “General Response to All Reviewers.”

Could you please review these updates during the discussion period and let us know whether they address your concerns? We would be grateful if you could reconsider your assessment and update your rating as appropriate. If any points need further clarification, we would be happy to discuss them.

Best regards,
The authors

I thank the authors for their response. However, I am still not convinced by their explanation of why existing methods cannot work. For example, off-policy evaluation methods can be used to alleviate certain types of distribution shift problems in direct slate evaluation. I still feel that the discussion of related work remains insufficient and will maintain my score.

Reviewer_s7iZOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper talks about when historical feedback can provide evidence to support changing a recommendation, especially when the relationship between proxy feedback and the target outcome may shift over time. The authors propose a shared calibration framework that jointly accounts for historical-current distribution shifts, sampling uncertainty, and the selection over candidate recommendations and borrowing levels. The method provides a finite-sample guarantee for avoiding insufficient recommendation changes, and experiments on one dataset show that historical borrowing is useful when current outcome labels are limited, while direct use of current labels can be preferable in other regimes.

Critical strengths

1, The proposed problem is fresh and interesting. It studies when historical feedback is sufficient to support a recommendation change rather than directly optimizing a standard recommendation objective.

2, The proposed shared calibration makes good use of the low-dimensional structure of the historical-current shift. By sharing the calibration set across different candidate recommendations and borrowing levels, the method avoids calibrating each decision separately and provides a relatively clean way to control the uncertainty introduced by historical borrowing.

Critical weaknesses

1, The method relies on assumptions about the measurement model.

The proposed framework critically depends on the measurement model in Equation (2), where the relationship between the proxy feedback and the target outcome is assumed to follow a conditional mean structure. However, in real-world recommender systems, the feedback outcome relationship may exhibit more complex user, item, context, or time-dependent shifts. It would be helpful to further discuss the robustness of the proposed method when the historical-current discrepancy is higher-dimensional.

2, The task setting and its practical role in current recommender systems could be motivated more clearly.

The proposed task is less standard than commonly studied recommendation tasks such as next item prediction or ranking. Therefore, the paper would benefit from a more concise and intuitive explanation of where this decision problem arises in a real-world recommendation pipeline, at what stage the proposed method would be applied, and how it could be integrated with existing recall, ranking, and re-ranking workflows. This would make the practical availability of the problem easier to understand.

3, The empirical generalization is currently limited.

The experiment is conducted on a single dataset with one specific proxy–outcome pair, like and long view. It is currently unclear whether the same conclusions would hold for other recommendation scenarios and other feedback. Additional experiments, or at least a more detailed discussion of when the assumptions and observed conclusions are expected to generalize, would strengthen the paper.

4, The presentation, especially some figures, could be made more concise and visually informative.

Some figures contain a relatively large amount of textual explanation while providing limited additional visual intuition. For example, Figure 1 includes many text-heavy components and currently feels closer to a structured table or workflow description than a conceptual illustration.

Questions for authors

1, How sensitive is the proposed method to violations of the measurement model in Eq 2? In particular, what happens if the feedback–outcome relation or the historical-current shift is more complex than the assumed low-dimensional form?

2, Could you clarify where this task would fit into a real-world recommendation pipeline? For example, at what stage would this decision rule be applied, and how would it interact with existing ranking and re-ranking pipeline?

3, Do you expect the same conclusions to hold for other feedback–outcome pairs or recommendation settings? What properties of the signals are necessary for the method to work well?

6 comments

Thank you for your thoughtful review and for asking how the method would fit into an existing recommendation system.

Critical Weaknesses:

1, The method relies on assumptions about the measurement model.

Consider an interface change that makes users more likely to like a video on phones than on desktop. A device feature can capture a persistent difference between the two. If the interface changes the size of that difference between periods, Appendix A.2 allows a separate period-by-device term. The analyst chooses this term on development data and uses labeled observations from both periods to calibrate it. The same construction can retain several such context effects. The main experiment estimates two changes, in like rates after short and long views. Proposition 1's bound of four pairs of calibration directions applies to that two-dimensional model.

2, The task setting and its practical role in current recommender systems could be motivated more clearly.

The method fits immediately after reranking, at the point where a platform decides whether to replace the slate it currently displays. Suppose the reranker proposes changing just the third video in a three-video slate. To improve the slate's average long-view rate by more than one percentage point, the new video must beat the displaced video by more than three points. The rule uses the earlier likes and views to translate the observed difference in likes into evidence for that specific replacement. The existing system continues to generate the alternatives.

3, The empirical generalization is currently limited.

The feature that carries over to other settings is the ability to calibrate what an early response means for a later outcome. The experiment varies that relation while retaining the same logged video task. In the controlled larger-shift condition, Shared achieved 14.98 points of mean gain and Stage aware 12.54. In the weak-signal condition, all eight rules had zero mean gain (Table 6). This gives a concrete reason to expect different outcomes across applications. A sizable change from history can still permit borrowing if labeled records identify it precisely. A proxy that barely distinguishes the two outcome classes supplies little evidence for changing a recommendation.

4, The presentation, especially some figures, could be made more concise and visually informative.

In the revision, we will simplify the figures and tables so readers can see how the method works and understand the results more easily.

Questions For Authors:

1, How sensitive is the proposed method to violations of the measurement model in Eq 2?

There are two ways to protect a decision against a more complex shift. With labels for the affected groups, the extra changes can be modeled and calibrated jointly (Appendix A.2). For an omitted effect with a known bound, Appendix A.7 subtracts its largest possible contribution from each decision margin, accounting for records reused in calibration and fitting.

Table 6 shows why that subtraction matters. When we introduced a smooth context effect, using its known bound gave Shared 1.22 points of mean gain. Setting its allowance to zero gave 11.20 points, because decisions could pass without paying for that source of error. Only the first calculation includes the protection required by Theorem 1. As the number of changing effects grows, sufficiently precise labels or residual bounds are needed for both margins to stay positive.

2, Could you clarify where this task would fit into a real-world recommendation pipeline?

Retrieval and ranking produce candidates as usual. The reranker proposes slates using items with logged observations. We fix this list on independent development data before examining the feedback for the decision. Our rule reads the current slate and those alternatives, uses earlier labeled records to estimate the feedback relation, and uses target likes to compare videos. Among candidates that pass both checks at an allowed borrowing level, it returns the one with the largest score margin. If none passes, it returns the current slate (Algorithm 1). Equation (1) evaluates a change in slate membership. Moving the same videos to different positions gives them the same value under that objective.

3, Do you expect the same conclusions to hold for other feedback–outcome pairs or recommendation settings?

For example, the same decision could use saves as the early response and purchase within seven days as the outcome. Completed purchase records would supply the save rates for buyers and nonbuyers at comparable contexts. A clear positive gap gives direction to a difference in saves between two proposed slates. If buyers and nonbuyers save at similar rates, even a precise increase in saves provides little evidence for a purchase gain.

More generally, the rule takes bounded feedback, labeled examples of both outcome classes, records covering the proposed items, and an allowance for change into the target audience. Borrowing helps when the earlier records reduce sampling uncertainty enough for the gain and slope checks both to pass after accounting for the historical shift.

In the revision, we will add the three-video example to Section 3, simplify the figures and tables, and explain how a practitioner can include a changing context effect. If these clarifications address your concerns, we would appreciate your reconsideration of the paper and an updated rating if appropriate.

AuthorsOpenReview ↗

Invitation to discussion

Dear Reviewer s7iZ\textbf{s7iZ},

Thank you again for your detailed review. We have provided responses to each of your comments and a revised manuscript. The main updates are summarized in our “General Response to All Reviewers.”

Could you please review these updates during the discussion period and let us know whether they address your concerns? We would be grateful if you could reconsider your assessment and update your rating as appropriate. If any points need further clarification, we would be happy to discuss them.

Best regards,
The authors

Reviewer_vuF2OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies when historical proxy feedback can safely justify changing a recommendation slate despite shifts in the proxy-outcome relationship. The proposed Shared method jointly calibrates these shifts, adaptively borrows historical data, and provides finite-sample protection for the selected recommendation. Experiments on KuaiRand-Pure show benefits in a matched, calibration-limited setting, but the advantage disappears when sufficient current labels are available.

Critical strengths
  1. The paper gives a principled finite-sample formulation of recommendation replacement under a changing proxy-outcome relationship. It accounts for selection over both recommendation slates and borrowing levels, as well as repeated observations from the same user. This is a meaningful strength because these sources of uncertainty are often ignored in simpler historical-borrowing procedures.
  2. The empirical analysis includes useful mechanism checks and does not present Shared as universally superior. The comparisons with Stage aware, fixed borrowing, alternative calibration regions, and additional label acquisition help identify the specific regime in which shared calibration is beneficial.
Critical weaknesses
  1. The empirical evidence is too narrow to establish the practical generality of the method. The main result is based on one logged dataset, a fixed support of 16 videos, three-item slates, and 120 predefined alternatives.
  2. The guarantee depends on strong assumptions that are difficult to validate using the information available at decision time. The paper sets the deployment allowance to η=0.001\eta=0.001, but does not provide a general data-driven procedure for selecting or validating it without access to target outcomes. The stress tests also show that false acceptance increases when this allowance is violated. This substantially limits the operational meaning of the guarantee.
Questions for authors
  1. How should a practitioner select and validate the deployment allowance η\eta when target outcomes are unavailable at decision time? Would the theoretical guarantee remain valid if uncertainty in the estimation of η\eta were explicitly incorporated?
  2. How representative are users who satisfy the four required stage-by-class conditions for matched calibration? Could selection into this matched subset introduce a population shift relative to the target users?
9 comments

Thank you for your careful review and for distinguishing the statistical guarantee from the evidence needed to use it in practice.

Critical Weaknesses:

1. The empirical evidence is too narrow to establish the practical generality of the method.

We tested the effect of the candidate menu in four controlled extensions of the KuaiRand-Pure study (Table 11). These use 32 or 40 videos, five or ten videos per slate, and between 65 and 5,000 alternatives. Shared had higher mean viewing gain than Stage aware in all four. In the largest comparison, the gains were 15.96 and 12.23 percentage points, respectively, over 40 runs. This directly tests whether the result depends on the original three-video slates or the particular set of 120 alternatives.

The practical application is to evaluate a shortlist from an existing recommender. Theorem 1 allows any finite shortlist fixed before inference on items with adequate observations. To apply the rule to another service, the inputs to replace are the local labeled feedback used to estimate the proxy–outcome relation and the target feedback for the proposed items. For instance, a service using saves to anticipate purchases would calibrate the save rates of buyers and nonbuyers from its own completed purchase records.

2. The guarantee depends on strong assumptions that are difficult to validate using the information available at decision time.

At decision time, we know how often the new audience liked each video. More likes could mean that people watched longer, or that they were more inclined to click the like button. The assumption concerns how much this relationship can change. A practitioner can check earlier groups whose viewing outcomes are now known. For each video, they can use its observed long-view rate and the model from the preceding period to predict its like rate, then compare that prediction with the actual like rate. These past discrepancies help the practitioner choose an allowance for the new group. Theorem 1 controls the risk of approving too small a viewing gain when the chosen allowance covers the new group's discrepancy for every video.

The allowance affects which replacements qualify. For the replacement in Table 3, doubling the allowance from 0.1 to 0.2 percentage points per video still leaves enough evidence to approve the change. At 0.3 percentage points, the same replacement no longer qualifies. We also tested what happens when the actual discrepancy exceeds the chosen allowance. At five times the allowance, Shared made six false approvals in 200 runs (Table 11).

Questions For Authors:

1. How should a practitioner select and validate the deployment allowance η when target outcomes are unavailable at decision time?

We would choose η\eta before comparing slates, using earlier cohorts whose viewing outcomes have since arrived (Appendix D.3). Treat each as a past deployment. For each video, use its observed viewing rate and the preceding period's model to predict its like rate. Compare that prediction with the actual like rate. Use upper confidence bounds on the absolute discrepancies across videos, accounting for uncertainty in both the fit and the observed rates. A later completed cohort can be held out to check the proposed range. For deployment, the range must cover the expected drift into the new cohort, including any additional allowance justified by changes in the audience or interface.

Yes, uncertainty in this step can enter the guarantee. If the bounds cover all target-video discrepancies with probability at least 1−αη1-\alpha_\eta, the proof of Theorem 1 gives total false-approval probability at most αC+αs+αa+αη\alpha_C+\alpha_s+\alpha_a+\alpha_\eta. For a 5% total, for example, reserve 1% for the allowance and use 4% for the original three checks.

2. How representative are users who satisfy the four required stage-by-class conditions for matched calibration?

The matched panel consists of repeat viewers who supply both a long and a short view in each period. This requirement can favor frequent viewers. Their role is specific. They estimate how the like rate changed after long views and after short views. The like rates used to compare the candidate videos come from the separate target sample (Section 3.1, Appendix B.2).

Selection matters if either of those two changes differs between eligible viewers and the broader population. A direct check would estimate each change twice at comparable contexts, once in matched users and once using separate users from the four period and viewing groups. If the audit gives simultaneous upper bounds on both differences, the calibration set can be widened by those amounts before testing a slate, allowing for the audit in the error budget. Alternatively, ordinary calibration uses separate users throughout and removes the four-record requirement. Section 4.2 reports that this version selected positive borrowing in 26 of 200 runs. The screening rate of 0.8 in Appendix D.1 enters only the cost calculation.

In the revision, we will highlight the larger-menu results in the main text, explain in Section 3.4 how earlier groups can inform the deployment allowance and how it affects a replacement decision, and explain in Section 5 how to check for selection into the matched panel. We would be grateful if you would reconsider your assessment in light of these clarifications and update your rating if appropriate.

AuthorsOpenReview ↗

Invitation to discussion

Dear Reviewer vuF2\textbf{vuF2},

Thank you again for your detailed review. We have provided responses to each of your comments and a revised manuscript. The main updates are summarized in our “General Response to All Reviewers.”

Could you please review these updates during the discussion period and let us know whether they address your concerns? We would be grateful if you could reconsider your assessment and update your rating as appropriate. If any points need further clarification, we would be happy to discuss them.

Best regards,
The authors

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response to All Reviewers

We thank all reviewers for their careful reading, thoughtful comments, and constructive suggestions. We have provided a revised manuscript incorporating the clarifications in our individual responses. We summarize the main points below.

What the method contributes. Our method decides whether to replace a set of recommended items after an existing recommender has proposed alternatives. More likes can reflect longer viewing or a change in how readily users click the like button. Shared accounts for changes in this relationship and carries their uncertainty into two checks: whether the viewing gain is large enough and whether likes still point in the right direction. Section 3.1 now explains where this decision fits in the recommendation process, and Section 3.2 includes a short derivation showing why both checks are needed. Under the stated assumptions, Theorem 1 bounds the chance of approving an insufficient improvement, including after selecting both the recommendation and how much history to use.

What the experiments establish. In the matched calibration setting, Shared improves mean viewing gain over Stage aware by 2.74 percentage points, with an adjusted confidence interval of [1.40, 4.08] (Table 2). Section 4.1 now explains more directly how the three settings vary calibration records, current fitting labels, and target feedback. Section 4.2 also separates the benefit of borrowing from the cost of selecting a borrowing level. The four controlled extensions reach 40 videos, ten videos per slate, and 5,000 alternatives; Shared has higher mean gain than Stage aware in all four (Table 11).

How the assumptions enter a decision. Section 3.4 now explains the deployment allowance before giving a numerical example. This allowance specifies how much the relationship between likes and viewing may change for the target audience. A larger allowance requires stronger evidence before a replacement passes. Appendix D.3 describes how earlier groups with completed viewing outcomes can inform this choice, and Appendix A.8 explains how uncertainty in the allowance enters the total error budget. We also clarify how changing context and differences between matched users and the fitting population can be included in the uncertainty bounds.

Presentation. We have added Figure 3 to illustrate the two checks and the final decision for the worked example in Table 3. We provide Algorithm 1 in Appendix A.8 to clarify the decision procedure. The full curves for the data purchase comparison appear in Appendix D.1, while the main text retains the key comparisons.

We again thank the reviewers for their time and constructive suggestions. We would be grateful if they would reconsider their assessments in light of our responses and the revised manuscript, and update their ratings if appropriate.

AuthorsOpenReview ↗

Summary of the Authors' Responses

To facilitate discussion, we summarize the reviewers’ main concerns, our corresponding responses and revisions, and the current discussion status below.

Concerns and author responses

ConcernvuF2s7iZi5Z3Response providedOur response and its location
Scope of the empirical evaluation✓✓—✓Discussed four controlled extensions reaching 40 videos, ten videos per slate, and 5,000 alternatives. Shared has higher mean gain than Stage aware in all four settings (Section 4.5; Table 11).
Choosing the deployment allowance and accounting for its uncertainty✓——✓Explained how earlier groups with completed outcomes can inform the allowance and how its uncertainty enters the total error budget (Section 3.4; Appendices D.3 and A.8).
Selection into the matched calibration group✓——✓Clarified the roles of matched users and the separate target sample. Described how population differences can be assessed and included in the calibration bounds, alongside the ordinary calibration alternative (Sections 4.2 and 5; Appendix B.2).
More complex feedback relations and context changes—✓—✓Explained how to include additional changing context effects and account for bounded effects omitted from the model, including when calibration observations also enter fitting (Appendices A.2 and A.7).
Role in an existing recommendation pipeline—✓—✓Clarified that the rule evaluates candidate slates proposed by an existing recommender and decides whether to replace the current slate. Added a concrete video replacement example (Section 3.1).
Applicability to other feedback–outcome pairs—✓—✓Used saves and purchases as an example to explain the required labels, target feedback, and signal properties, and when historical borrowing can support a decision (Section 5).
Contribution relative to existing methods——✓✓Clarified how shared calibration accounts for uncertainty in both checks and the final selection. Explained when direct evaluation and prediction-powered inference apply (Sections 2 and 4.4; Theorem 1; Appendix D.2).
Derivation and explanation of the decision rule——✓✓Added the short derivation connecting the measurement model to the two checks in Section 3.2. Algorithm 1 is provided in Appendix A.8.
Purpose of the experimental settings and the transition result——✓✓Clarified the data budgets and the comparison with Zero. The respective gains of 1.52 and 1.77 points explain the 0.25-point difference between Shared and Stage aware in transition (Sections 4.1–4.2; Table 6).
Clarity of figures and presentation—✓✓✓Added Figure 3 to illustrate the two checks using the example in Table 3. The full data purchase curves appear in Appendix D.1, with the main comparisons retained in Section 4.4.

A tick in a reviewer column indicates that the reviewer raised the concern. A tick in “Response provided” indicates that we supplied a corresponding answer or clarification. Detailed responses are available in the individual discussion threads.

Ratings and reviewer follow-up

The following status reflects the discussion available as of 18 November 2026.

ReviewerCurrent ratingReply to our responsesLatest follow-up status
vuF22 — Weak rejectionYesConfirmed keeping the rating. Our subsequent request to clarify the remaining concerns is awaiting a reply.
s7iZ2 — Weak rejectionNo reply yetAn invitation to review our responses and revised manuscript has been posted.
i5Z33 — Weak acceptanceNo reply yetAn invitation to review our responses and revised manuscript has been posted.

We thank all reviewers and the area chair for their time and constructive feedback. We welcome further comments on any remaining concerns and would be grateful if the reviewers could reconsider their assessments and update their ratings as appropriate in light of our responses and the revised manuscript.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.