acceptodds
Under review as a conference paper at ICLR 2027

When Delayed Feedback Changes the Objective: Benchmark Validity in Sequential Decision Making

Abstract

Modern recommender, advertising, and online experimentation systems often act several times before a consequential outcome matures. Delayed-learning theory usually treats this lag as an information constraint while taking the evaluation loss as given. We study the prior question: when does the route from delayed observations to action-level losses define a valid benchmark? We separate structural decision time from observation time and show that validity is governed by preservation of action gaps, not by delay length. For any realized action sequence, route-level performance transfers to the structural benchmark up to the cumulative distortion of action gaps. When structural margins are present, this gap-control certificate also recovers rankings and optimal actions; when persistent conflict crosses those margins, a learner may optimize the routed objective while accumulating structural regret. We further prove that fully unlabelled delayed observations can leave route validity non-identified even under adaptive data collection, motivating evidence-qualified audits rather than assumption-free corrections. Controlled experiments stress-test the predicted transfer and failure regimes; logged-data studies diagnose attribution sensitivity and illustrate the distinction between predictive fit and decision recovery on observed support. Reliable delayed-feedback learning therefore requires two certificates: optimization of the operational route and validation of the benchmark it induces.

Primary area:
causal reasoning
Keywords:
Causal inference

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_738XOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies the problem that delayed feedback may cause the actual optimization objective of a learner to deviate from the true objective. The authors distinguish Structural Loss and Routed Loss, introduce the Action-gap Distortion metric, and use three theorems to discuss regret transfer, objective conflict, and non-identifiability. Finally, they conduct experiments in simulated environments and on real-world datasets.

Critical strengths
  1. The distinction between Structural Loss and Routed Loss is reasonable, and helps to discuss the difference between feedback attribution and the true optimization objective.
  2. The mathematical formalization is relatively complete, and the main results also provide their corresponding conditions of applicability.
Critical weaknesses
  1. The authors seem to write many intuitive or even obvious conclusions as theorems. Theorem 1 is essentially just a perturbation bound between two loss vectors, and it can hold even without delayed feedback. Theorem 2 proves that two objectives with persistent conflict cannot both be optimized well, which is almost directly determined by its assumptions. I think the authors seem to be assigning mathematical symbols to common sense, and then proving that these symbols satisfy the expected relations.
  2. The main regret bound compares the results of the same action sequence under two loss functions. It does not analyze how incorrect feedback changes the learner's future decisions. In addition, the validation method proposed by the authors requires knowing enough true outcomes, but obtaining such information is exactly one of the most difficult parts in real-world applications. Although the authors spend a lot of space on mathematical derivations, they do not really solve the most difficult part of the problem. I am confused about the practical value of this framework.
  3. The experiments seem more like verifying answers that have already been written in advance. For example, Experiment 1 artificially creates feedback misalignment, and then shows that misalignment causes regret. Experiment 2 uses different advertising attribution rules and finds that different rules produce different results. These phenomena are really not surprising, and the latter experiment does not even provide evidence about which attribution rule is correct. Overall, these experiments look more like unit tests for the authors' definitions, rather than discovering any new scientific phenomena.
Questions for authors

The main theoretical results of this paper seem to be quite straightforward, and some conclusions do not even depend on delayed feedback. Could the authors explain more clearly what new theoretical insights these theorems actually provide compared with existing studies?

Reviewer_J1R2OpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This work studies the problems that arise when theoretical analysis of sequential decision systems assumes that delayed feedback is simple lag. They highlight that given a sequence of actions, the "cumulative action-gap distortion" sharply controls the difference between regrets when comparing a specified structural loss map and a loss map that accounts for the route of actions (Thm 1). At the same time, these regret notions are fundamentally incompatible: route regret may be vanishing while structural regret is linear (Thm 2). Additionally, they provide further results on route validity identification and run empirical validations.

Critical strengths

The conceptual premise of the work is interesting. That is, the study of issues in modeling delayed feedback as simple lag in a decision making problem is appealing. Based on the length of results and modeling that was not included in the main body, I would suggest that following a rewriting (see weaknesses), authors consider a venue such an INFORMS journal.

Critical weaknesses

Broadly, my recommendation of giving this paper a clear rejection comes from three major weaknesses: (1) poor exposition/writing, (2) poor structuring, and, somewhat relatedly, (3) an incomplete AI Use Statement. I elaborate on these below:

  1. Poor writing. The main body of the paper is extremely hard to read (from a human reader perspective). This is particularly true of Sections 1 and 3, which are required to understand in order to follow the rest of the paper because S1 and S3 present the model used by theoretical and experimental results. The writing issues are split into two rough (slightly overlapping) categories: (A) logical problems and (B) lack of background problems. For (A), the work repeatedly includes sentences that, even when read closely are difficult to follow. For example, on line 040, the authors state "The prior problem is benchmark validity, not delay alone." However, in the context of the previous sentences, it is not actually immediately clear which problem this may refer to: an outcome scoring the wrong action, a worsening of the intended intervention, both, or something else. Additionally, by this point, the meaning of "benchmark validity" is also not clear, rendering the sentence inscrutable. For (B), there are many places where concepts or words are introduced, but never properly explained, despite being non-obvious to readers from relevant communities. For example, "records a conversion" (line 035), "online experiment adapts assignments" (line 036). Both of these are used in the description of sequential decision systems receiving delayed feedback, but it is not explained what an advertising conversion or online experiment is. If these are standard examples from background literature, these should be referenced. These examples both come from Section 1, but writing problems falling into (A) and (B) continue throughout the work.

  2. Poor Structuring. The main body of the paper suffers heavily from lack of clarity due to the amount of important details left to the appendix. Notably, the entire problem set-up (Appendix C) is left out of the main body. The lack of proper model presentation makes the theoretical results very difficult to parse and modeling that is in the main body is often lacking. For example, A\mathcal{A}, which first appears on line 157 is never formally introduced in the main body.

  3. AI Use Statement. As described further in my summary of the ethics statement, the (required) AI Use Statement is fairly lacking. The AI policy for authors (https://iclr.cc/Conferences/2027/AIPolicyForAuthors) lists required subtasks that are not described in the authors' statement and also recommends elaboration, which the authors have not done. Additionally, though I concede that this is speculation, much of the quality of the writing/structuring I have listed above leads me to believe that AI tools were much more heavily utilized in the writing than such a short statement implies. I have used both Pangram and GPTZero on a few selections of text (lines 146-159, 765-781, and 034-053) and the results (which, again, I acknowledge are not necessarily accurate) indicate extremely high (94-100%) likelihood that text is entirely AI-generated. While this is not a weakness in and of itself, I feel that the AI Use Statement as currently written (i.e., "the authors used AI-based tools to aid language editing", line 488) does not reflect this, if it is true.

Questions for authors

Given the significant rewriting/structuring required of the work in order to achieve acceptable form and the limited format of rebuttals, I think it is unlikely that I would increase my score to a weak accept or higher, but as I have described various weaknesses involving potential AI misuse, I would like to provide authors a few specific questions with respect to these concerns to ensure there is space to address any of my claims that may be untrue:

  1. Could you please elaborate on your AI Use Statement in the format suggested by the AI Policy for Authors (https://iclr.cc/Conferences/2027/AIPolicyForAuthors) specifically noting the use of AI for each of the required subtasks?
  2. To what extent are my statements about text generation in lines 146-159, 765-781, and 034-053 correct?
Reviewer_vgKoOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper studies sequential decision problems in which observed outcomes must first be mapped to action-level losses, and this mapping may be misspecified. Decision-making must therefore account for that misspecification. The mapping is formalized through the notion of a “route”, i.e. an attribution rule that turns observations into losses for each action.

Its main results show that regret measured under the route transfers to the true (structural) regret up to the cumulative distortion of action gaps. Conversely, persistent disagreement between route-optimal and truly optimal actions forces linear structural regret, even when route regret vanishes. It further shows that, under fully unlabelled feedback, route validity may not be identifiable, and it proposes audit procedures to certify validity from additional data.

Critical strengths

The paper aims at defining a new scientific perspective (and relevant underlying problems) around the notion of reward misspecification (implied for instance through delayed feedback). These questions might lead to new and relevant results around learning with model /reward misspecification.

Critical weaknesses
  • At first reading, I had a very difficult time to understand the story. I think one reason is that, contrary to what the title suggests and the first sections present, the core results are essentially about optimizing any proxy loss instead of the true one. Theorem 1, the margin results and Theorem 2 hold for any two loss sequences L̃ and Lᶜ on a common action sequence. Delay never appears in statements or proofs. The proxy-reconstruction and fitted-model routes (Prop. D.21) have nothing to do with attribution at all. They are just model misspecification. The audit section is generic finite-population estimation of a discrepancy between two loss maps. So the whole paper is very difficult to understand because the vocabulary and the stories told are misleading in my opinion. The honest framing would be reward misspecification. And that might point towards another body of literature : Goodhart effects in bandits, corrupted rewards, STARC-type reward distances and evolving observations, which the authors only partially cite.

  • Then, I m under the impression that the framework is too general to obtain deep results. The theorems are quite elementary Lipschitz and pigeonhole facts with delay playing no structural role, and the central quantity δ mixes misrouting with information loss despite the paper’s stated decomposition.

  • The paper treats the concept of route (which took some time to grasp) as a fixed map from observations to action-level losses and compares route and structural regret along a given action sequence. This abstracts away the central difficulty: the bias a route induces is policy-dependent. (See question 1)

  • The motivating examples (retention, delayed endpoints) are exactly cases where actions affect future state, and that is outside the benchmark. The paper acknowledges this, but it limits relevance.

Questions for authors

Question 1: The biais is policy dependent and that should be the central scientific question addressed.

Take a stationary bandit with fixed delay τ and last-touch attribution. The bias on arm a is Σ_b P(A_{t−τ} = b | A_t = a)(μ(b) − μ(a)), which depends entirely on the learner's switching behaviour:

  • Slow-switching make even this crude route nearly valid, with at most (number of switches) × τ misattributed samples.
  • Exploring or alternating policies can reverse the action ranking entirely.
  • Exploiting learners can lock in on an arbitrary arm, because exploration credits every arm with the current arm's losses.

The real problem is therefore a closed loop: route → estimates → policy → switching pattern → route bias. The relevant questions are which (learner, route) pairs admit bad fixed points, and how exploration can be designed so that a given route stays valid. The paper's main results do not address this loop. Theorem 1 is a Lipschitz bound valid for any two loss sequences, and Theorem 2 a pigeonhole argument. Both hold for arbitrary action sequences and so say nothing about it. Delay plays no essential role in either. As a result, the framework identifies when a fixed route is harmless but cannot explain or predict the failures it uses as motivation.

Question 2: Can you give at least one learning result under routed bandit feedback;

Question 3 can you characterize positive identification conditions;

Question 4 (or suggestion) Maybe you should consider repositioning the paper as a framework paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.