acceptodds
Under review as a conference paper at ICLR 2027

Scheduled Information Value for Delayed Selective Labels

Abstract

Selective-label policy learning is challenging because current decisions affect both immediate payoffs and the outcomes available for future learning. This problem becomes harder under delayed feedback, where previously selected but unresolved outcomes may already provide substantial information in the near future. Because queue size alone does not reveal when such feedback will arrive, we then develop Scheduled-aware Information Value (SAIV) for sequential decision making under delayed selective feedback. SAIV combines a survival-based uncertainty baseline, which accounts for information expected from the pending queue, with a scheduled-information term that uses elapsed waiting times to capture timing information beyond a count-aware reference. This allows the policy to adapt exploration to both how much feedback is pending and when it is likely to arrive. Under a Beta-Bernoulli model, we derive the associated Bayes-risk reduction, establish a memoryless-delay null result, and bound sensitivity to delay-kernel estimation. Simulations and a sequential lending replay show consistent gains over exploration and delayed-feedback baselines, with mechanism experiments confirming that the improvements arise from exploiting the timing structure of unresolved feedback.

Primary area:
reinforcement learning
Keywords:
Delayed Selective Labels, Scheduled Information Value, Survival Analysis, Sequential Policy Learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_yaZtOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies sequential decision-making with delayed selective labels, where selecting an instance produces both immediate utility and feedback that may inform future decisions. It proposes Scheduled-aware Information Value (SAIV), which combines a count-aware uncertainty bonus with an elapsed-time correction based on the residual maturation probabilities of pending labels. Under a Beta–Bernoulli model, the authors derive marginal posterior-variance reductions, establish that the correction vanishes under memoryless delays, and bound sensitivity to delay-kernel estimation. Validation includes synthetic experiments, matched timing interventions, sensitivity analyses, and a sequential replay of bank lending data.

Critical strengths
  1. Two states with the same posterior and queue size can have different prospective information because pending labels have different residual delivery times. This is a meaningful consideration for exploration under delayed feedback.
  2. The paper makes a substantial effort to examine the proposed mechanism.
Critical weaknesses
  1. Section 2.1 assumes that delay is conditionally independent of the outcome. However, Appendix G and Table 3 indicate that delinquent outcomes are revealed at first delinquency, whereas non-delinquent outcomes are revealed at month 36. Before month 36, delivered labels therefore consist exclusively of delinquent outcomes, and remaining unresolved itself provides outcome information. Updating a Beta posterior only from delivered labels generally does not produce the stated posterior under this mechanism.

  2. The main simulation does not adequately establish the contribution of schedule-aware learning.

Questions for authors
  1. How is the outcome-dependent feedback mechanism in Table 3 reconciled with the conditional independence assumption in Section 2.1?

  2. How does SAIV compare with a validation-tuned static threshold on initial prior means, and with baselines given the same tuning budget?

Reviewer_KVmVOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper looks at sequential decisions where only selected cases reveal outcomes, and those outcomes take time to arrive. The idea is to consider not just how many labels are pending, but also how long they have been waiting. Two queues of the same size may provide very different amounts of information soon, which affects whether further exploration is worthwhile. SAIV estimates when pending feedback may arrive and adds a timing-based correction to the exploration score. Beliefs are updated only when outcomes arrive. Under a Beta–Bernoulli model, the authors analyze the information from an additional label, show that the correction vanishes under memoryless delays, and study sensitivity to delay-estimation errors. Simulations and paired experiments support the benefit of timing information. The bank replay also shows a modest gain over Bayes-UCB, from 43.98 to 44.68 utility per 1,000 decisions.

Critical strengths
  1. I think the paper identifies a meaningful distinction: how many labels are pending and when they are likely to arrive can affect exploration differently. Even with the same posterior and queue size, different waiting times can change the value of acquiring another label. SAIV turns this information into a computable correction and updates the posterior only when outcomes actually arrive. The motivation is clear and addresses a real issue in delayed-feedback decisions. This is what I find valuable about the method itself.

  2. The paired mechanism experiments are convincing. The authors start from the same posterior and pending queue, expose both policies to the same future realizations, and change only whether the timing correction is enabled. They then check whether it changes actions and improves subsequent utility. The memoryless-delay analysis and experiment also help establish when timing information is useful. Across 100 paired simulation seeds, SAIV improves utility from CAIV’s 337.46 to 354.03 and exceeds Bayes-UCB’s 339.58. Together, these results give me more confidence that waiting-time information is useful and make the source of improvement clearer.

Critical weaknesses
  1. I am concerned about the connection between the theoretical assumptions and the bank experiment. The theory assumes that delay is independent of the outcome given customer information, but delinquent outcomes are revealed at the first delinquency, while non-delinquent outcomes are revealed at month 36. An unresolved outcome may therefore already indicate that the customer has not yet become delinquent. Updating the posterior using only arrived outcomes could also introduce bias. The authors acknowledge informative delay as a limitation, but it appears directly relevant to this dataset. I am therefore unsure how much the bank results support the mechanism studied under the theoretical assumptions. This directly affects my assessment of the application results.

  2. The comparison with external baselines is not fully convincing to me. The appendix explicitly describes some baselines as simplified implementations: Bayes-UCB uses a fixed 0.75 posterior quantile, and Delayed-UCB uses a fixed 0.05 uncertainty bonus. SAIV has several adjustable exploration parameters, with its bank configuration selected on validation data. The matched CAIV experiments support the timing correction itself, but assessing the advantage over other exploration methods requires clarification of whether those baselines received comparable tuning budgets. This matters particularly because the bank replay gain over Bayes-UCB is only about 1.6%; stronger baseline tuning could affect that conclusion. I am therefore cautious about the strength and generality of the reported performance advantage.

Questions for authors
  1. In the bank experiment, delinquency is revealed when it first occurs, while non-delinquency is revealed at month 36. An unresolved outcome may therefore already contain information about the outcome, which differs from the outcome-independent delay assumption. How is this handled in the current experiment? Could updating the posterior only from arrived outcomes introduce bias because delinquent outcomes arrive earlier? I would like to understand how this affects SAIV and the baselines.

  2. Which parameters were tuned for the external baselines, over what ranges, and with what validation budgets? The appendix uses a fixed 0.75 quantile for Bayes-UCB and a fixed 0.05 uncertainty bonus for Delayed-UCB, while SAIV’s parameters are selected on validation data. How much of the gap remains with comparable baseline tuning? This matters particularly for the modest gain in the bank replay.

  3. Why does the timing correction retain only the positive part? If waiting times indicate that substantial feedback will arrive soon, the value of further exploration should decrease, but the current method sets the correction to zero without lowering the CAIV score. Is this one-sided design derived from the decision objective or an empirical choice? Do you have a comparison retaining both positive and negative corrections? I would like to understand whether this choice limits how the method uses timing information.

  4. How do decision steps map to calendar months in the bank replay? Multiple customers may apply within a month, while feedback delays are measured in months. When does the queue advance and release labels? Across the 100 replay seeds, does only customer order change, or do application times also change? This directly affects how much feedback is expected to arrive at each decision, so I would like to clarify how the replay’s timing structure is constructed.

Reviewer_HPUROpenReview ↗
Rating
3
Confidence
1
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

I gave it to Claude to read and explain to me. This paper is about the policy optimization problem of who to land money or not based on the observations of if they reimbursed or not. As far as I understood they propose to take into account since how long we are waiting for a reimbursement in order to decide.

Critical strengths
  1. The idea is simple and makes sense: two banks can have the same number of loans waiting for an answer, but if the loans are "old" the answers will come soon and if they are "young" they will not.

  2. The code is available and it works. Claude ran it and got exactly the numbers of Table 1

Critical weaknesses

I don't understand much of this field, so I asked Claude to read the paper and to run the authors' released code. Here is what it told me

"1. The gain does not seem to come from the timing. I ran a 'fake timing' control: the exact SAIV formula, but every pending label gets the same fixed maturation probability q = 0.02 instead of the one computed from its elapsed time. This policy only uses the number of pending labels, so it belongs to the count-aware class Π^cnt. Over the same 1,000 seeds it reaches 355 utility, versus 354 for SAIV. Under memoryless (geometric) delays it reaches 359, while SAIV and CAIV are identical there at 328. The best tuned CAIV (ρ from 0 to 2) stays at 339. So the improvement comes from the shape of the bonus γκ[√I − √I^cnt]_+ as an exploration bonus, not from the elapsed-time information. This undermines the central claim and the mechanism experiments of Section 3.3, which only compare SAIV against CAIV with the same ρ."

"2. The bank application breaks the model's key assumption. The theory requires the delay D to be independent of the outcome Y. In the bank data, delinquent customers are observed at their first delinquency (median month 14, IQR 6–24), while non-delinquent customers are only resolved at month 36 (Table 3). The delay is therefore highly informative about the outcome. In addition, the reported gain is small: 44.68 vs 43.98 utility per 1,000 decisions against Bayes-UCB."

I trust these numbers because Claude first reproduced Table 1 exactly from the authors' code

Questions for authors

Eq. 8 uses m̄ = L·a0 (count only), but in policy.py the bonus uses the sum of the a_i, which depends on the ages. Which version gives the results in the paper?

How do you justify the independence between delay and outcome in the bank data?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.