acceptodds
Under review as a conference paper at ICLR 2027

The Reward Horizon Problem: Learning From Delayed Feedback

Abstract

People increasingly use language models to pursue objectives that are realized over long time horizons. However, post-training rewards are typically computed over short horizons, which creates a mismatch between the reward signal and the ultimate objective. For example, while a short-horizon training signal may reward code that runs, does it reward code that stays stable? We study this _reward horizon problem_ in a software engineering setting, where the consequences of a code change may not be observed for several days or months. Using code corrections as a source of delayed feedback, we find substantial differences between the signals available at short and long horizons. Short-horizon feedback omits information about long-horizon objectives and encodes a different distribution of label sources. We then analyze how critic models trained on long-horizon outcomes may distill some of this delayed feedback into short-horizon rewards.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
reward modeling, delayed feedback, misalignment

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 2 reviews, above 85% of the others.
Reviewer_kfrmOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies the "reward horizon problem" where short-horizon rewards RhR_h may diverge from long-horizon outcomes RHR_H and favor different actions when the expected delayed feedback Dh(x,a)=E[RH−Rh∣x,a]D_h(x, a) = E[R_H - R_h \mid x, a] varies across actions. To study this empirically in software engineering, the authors assemble 257,788 merged pull requests (PRs) at least one year old across 11 open-source GitHub repositories (2011–2025) and use GPT-5.6 Terra to identify 13,176 PRs (5.1%) with an observed, explicitly attributed corrective follow-up (10,050) or revert (3,126) within 365 days.

Across four empirical analyses, they show that 52.4% of one-year observed corrections occur more than 7 days after merge; train L2-regularized logistic regression critics on Qwen3-Embedding-8B embeddings of PR titles/bodies and code diffs across cumulative (C7, C30, C365) and disjoint (D7, D7-30, D30-365) horizon targets, finding that C365 achieves higher AUROC than C7 across all three cumulative evaluation horizons; analyze GPT-5.6 Terra multi-label taxonomies of Issue Source and Reason Not Caught across correction horizons; and compare how critic scores correlate with complexity proxies when using code-change embeddings alone versus adding PR title/body embeddings

Critical strengths
  1. Large-scale longitudinal dataset of post-merge software corrections. The paper assembles 257,788 merged PRs across 11 major open-source repositories (2011–2025) with a chronological train/evaluation split separated by a 365-day gap intended to reduce potential lookahead bias. Figure 1 clearly shows that 52.4% of one-year observed corrections arrive more than a week after merge (with repository medians ranging from 4.9 days in scikit-learn to 62.4 days in Django), providing concrete evidence that short post-merge windows miss a large share of observed post-merge corrections

  2. Transparent prompts, uncertainty reporting, and worked examples. The appendices provide the exact prompts for GPT-5.6 Terra and GPT-5.4 Nano, bootstrap confidence intervals for the pooled critic and taxonomy comparisons, repository-filtered evaluations for the cumulative critics (Tables C.4–C.5), repository-fixed-effect adjustments for the taxonomy means (Figure D.2), and concrete PR-pair examples for every substantive taxonomy category displayed in Figure 3

Critical weaknesses
  1. Repository composition and training-sample variability limit interpretation of the disjoint-critic results:

    Although C365 has higher AUROC point estimates than C7 in most repository-filtered evaluations (Tables C.4–C.5), the positive-count-matched evidence for distinct horizon signals comes from the pooled disjoint critics (D7, D7-30, D30-365; Figure 2 and Tables C.1–C.2). Repository correction rates (~3%–10%) and median correction horizons (4.9–62.4 days) differ substantially. Matching 1,025 positives per critic does not ensure or verify matched repository composition across windows, so repository identity potentially encoded in the embeddings could partly explain the horizon-specific advantages. Per-repository evaluation or matched repository compositions would help distinguish this explanation from distinct within-repository defect signals. Moreover, the evaluation-row bootstrap holds the fitted critics fixed (Appendix C, lines 708–709), so its intervals do not capture variability from positive subsampling and model refitting. No metadata-only baseline, such as repository indicators and diff size, is reported

  2. Limited label validation and selective taxonomy coverage constrain interpretation of feedback differences:

    The dataset measures explicitly attributed corrections, whose coverage across horizons and repositories remains unquantified. Appendix B validates 200 candidate edges sampled equally from Terra's predicted-positive and predicted-negative sets, but reports only aggregate pairwise agreement and Cohen's κ\kappa (Table B.1). These statistics do not reveal how disagreements are distributed between the two prediction strata. The taxonomy analyses are appropriately framed as exploratory, but five of the six substantive Issue Source categories have reported confidence intervals on the horizon difference that include zero (Table D.1). For Reason Not Caught, only 10.5% of corrections enter the mention-gated cohort, and 31.1% of that cohort is subsequently labeled Unknown. Human–human exact-label-set agreement also has κ=0.28\kappa = 0.28 for both taxonomies, with heterogeneous per-label agreement (Figure B.2, Table B.2). These limitations leave the robustness of the feedback-composition findings to annotation disagreement and cohort selection unclear

Questions for authors

This is a borderline submission with a valuable longitudinal dataset. The following clarifications and analyses would materially inform my assessment, particularly whether the disjoint-critic findings persist after accounting for repository composition:

  1. Do the disjoint-critic results (D7, D7-30, D30-365; Figure 2 and Tables C.1–C.2) hold within individual repositories or after matching repository composition across horizon windows in both training and evaluation? Are they stable across repeated positive subsampling and model refitting? As supplementary checks, if available, comparisons with a metadata-only baseline and sensitivity to clustered resampling by shared correcting artifacts would also be informative.

  2. Were the human annotators involved in taxonomy or prompt development (Appendix E), and were they blinded to Terra's labels? Can you provide candidate-edge confusion matrices against each human annotator, or against adjudicated reference labels if available? As a supplementary analysis, how sensitive are the Reason Not Caught horizon differences in Table D.1 to mention-gating and the inclusion of Unknown cases?

5 comments

Thank you for your detailed review and suggestions. We have added experiments addressing repository composition, training variability, metadata baselines, and bootstrap uncertainty, along with additional label-validation results. Changes in the revised manuscript are marked in blue.

W1 / Q1.

Repository composition and training-sample variability limit interpretation of the disjoint-critic results:

Do the disjoint-critic results (D7, D7-30, D30-365; Figure 2 and Tables C.1–C.2) hold within individual repositories or after matching repository composition across horizon windows in both training and evaluation? Are they stable across repeated positive subsampling and model refitting? As supplementary checks, if available, comparisons with a metadata-only baseline and sensitivity to clustered resampling by shared correcting artifacts would also be informative.

Thank you for raising these concerns about repository composition and training variability. We added within-repository evaluations of the disjoint critics. Consistent with our previous findings, D30–365 outperforms D7 on corrections after 30 days in all eleven repositories (Appendix C, Table C.6, p. 19, lines 975–998).

We also repeated positive sampling and training across ten seeds. Consistent with our previous findings, D7 outperforms D30–365 on first-week corrections, while D30–365 outperforms D7 on both later windows in all ten runs (Appendix C, Table C.10, p. 20, lines 1073–1079).

We trained additional critics using only repository identity and log lines changed and found that they outperform the embedding critics in pooled evaluation. To assess whether these features explain the embedding critics’ early-versus-late pattern, we also evaluated the critics within repositories and groups of similarly sized PRs (Appendix C, Figure C.4 and Table C.11, p. 21, lines 1104–1130). Consistent with our previous findings, D7 outperforms D30–365 on first-week corrections, while D30–365 outperforms D7 on corrections after 30 days. Repository identity and coarse differences in diff size alone therefore do not explain the embedding models’ AUROC performance pattern.

We also recomputed confidence intervals using a clustered bootstrap, resampling together PRs sharing their first correcting PR or commit. Consistent with our previous findings, D7 outperforms D30–365 on first-week corrections, while D30–365 outperforms D7 on both later windows. All three paired 95% confidence intervals exclude zero (Appendix C, Table C.12, p. 22, lines 1141–1152).

W2 / Q2.

Limited label validation and selective taxonomy coverage constrain interpretation of feedback differences:

Were the human annotators involved in taxonomy or prompt development (Appendix E), and were they blinded to Terra's labels? Can you provide candidate-edge confusion matrices against each human annotator, or against adjudicated reference labels if available? As a supplementary analysis, how sensitive are the Reason Not Caught horizon differences in Table D.1 to mention-gating and the inclusion of Unknown cases?

Thank you for requesting this breakdown. We added correction-label confusion matrices against each human labeler (Appendix B, Table B.4, p. 14, lines 725–733). Of 100 Terra-positive candidates, the humans judged 83 and 71 to be corrections. Of 100 Terra-negative candidates, they judged 100 and 90 not to be corrections. Table B.3 (p. 14, lines 708–719) reports agreement by horizon and Terra label.

The human labelers are authors. One labeler developed the taxonomies and labeling prompts and had previously seen a handful of Terra labels. However, neither labeler was shown Terra’s labels during annotation. We clarified these details in Appendix B (p. 12, lines 613–617).

We repeated the Reason Not Caught comparisons after excluding Unknown cases. All six differences retain their signs, and the same three 95% confidence intervals exclude zero (Appendix D, Table D.3, p. 25, lines 1301–1315).

We agree that restricting the analysis to documented explanations limits its coverage. Section 4.3 (p. 6, lines 310–323) already explicitly states that “we do not claim that these labels are representative of all corrections” and frames the analysis as descriptive and exploratory. We therefore interpret these results as differences among documented explanations, rather for all corrections.

Thank you again for your feedback! We hope the additional experiments and clarifications have sufficiently addressed your concerns. If so, we hope you’ll consider increasing your score.

Thank you for running these additional experiments and adding the detailed breakdowns in the appendix. They address most of my initial concerns, and I am leaning toward raising my score. Before finalizing my rating, I would appreciate a brief discussion of the following three points in the main text:

  1. Metadata baseline (Figure C.4)

    In pooled evaluation, the simple baseline using only repo ID and log lines changed gets higher mean AUROC than the embedding critics across all matching settings (e.g., 0.712 vs. 0.664 for D30–365 on the 30–365d window). Please discuss this comparison in Section 4.2 and what it means for interpreting the embedding results

  2. Disjoint-critic nuances (Figure C.4, Table C.11)

    It is good to see that the early-vs.-late crossover between D7 and D30–365 still holds within repo and size groups. However, on the (7, 30]d evaluation window, D7 outperforms D30–365 under grouped evaluation, reversing their ordering in pooled evaluation. D7–30 also achieves the highest mean AUROC across all three windows under grouped evaluation. Please mention these nuances in Section 4.2 when describing the horizon-specific signal

  3. Human validation of correction labels (Table B.4)

    Thank you for sharing the confusion matrices and annotation details. Out of 100 Terra-positive candidate edges, the two human labelers judged 17 and 29 to be non-corrections; out of 100 Terra-negative candidate edges, they judged 0 and 10 to be corrections. Since no adjudicated reference labels are reported, these counts describe disagreement between Terra and the human annotators and highlight uncertainty in the correction labels. Briefly summarizing them in Section 3 or 4.1 would help readers judge label reliability

Thank you for the follow-up and for clarifying what would help readers interpret these results. As before, changes in the revised manuscript are marked in blue.

We added the metadata comparison and the grouped evaluation results to Section 4.2 (pp. 5–6, lines 266–269 and 307–311):

“To assess how well repository identity and diff size alone predict corrections, we trained critics using only repository identity and log lines changed. These critics have higher mean AUROC than the embedding critics in pooled evaluation (Figure C.4, left panel). We next evaluate the embedding critics on similarly sized PRs within each repository to assess whether the relative performance of D7 and D30–365 across horizons depends on these characteristics (Figure C.4, right panel). We find that D7 still outperforms D30–365 on first-week corrections, while D30–365 still outperforms D7 on corrections after 30 days. On the (7, 30]d window, D7 slightly outperforms D30–365, reversing their pooled ordering (Table C.11). Additionally, D7–30 has the highest mean AUROC on all three grouped windows (Figure C.4). Taken together, these results suggest that different horizons carry both shared and distinct predictive signals.”

We also added the human validation results to Section 3 (p. 3, lines 125–127):

“Two authors independently reviewed a sample of candidate edges labeled by GPT-5.6 Terra. Their agreement with the model ranged from 71% to 83% for edges it labeled as corrections and from 90% to 100% for those it labeled as non-corrections (Appendix B, Table B.4).”

Thank you again for all of your feedback!

Reviewer_bm9NOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper operationalizes the mismatch between short-horizon rewards and long-horizon objectives as subsequent corrections to software changes. Of 257,788 merged PRs across 11 open-source repositories, 13,176 received an explicitly attributed correction within one year; 52.4% of these appeared only after the first week. The authors compare critics trained on 7/30/365-day labels (embedding + logistic regression) and find that long-horizon labels improve prediction, using equal-positive-count, taxonomy-labeled, and context-conditioning analyses to examine feedback composition.

Critical strengths

1.The research question has a clear operational definition, spanning 11 repositories with different workflows and scales. The one-year maturation window, chronological split with a 365-day gap, repository stratification, and the explicit boundary of "only visible and explicitly attributed corrections" are all transparently documented (Sections 3–4, Figure 1, Appendix A). 2.The comparisons account for both prediction quantity and label type: the cumulative-label critic C365 improves AUROC over C7 on the one-year target by +0.031 [0.028, 0.035]; the disjoint-window critics trained on equal positive counts fit their respective windows better, supporting the claim that feedback composition varies with horizon (Section 4.2, Figure 2, Appendix C).

Critical weaknesses

1."Long-horizon rewards are better for training" remains a predictive association rather than decision evidence. Only corrections that explicitly cite an earlier PR become positives; uncorrected, unattributed, and post-year problems can all fall into the negative class. C365's gains may come from positive count, detection rate, and historical workflow differences simultaneously. Using critic scores as rewards could shift the policy distribution, and the paper does not provide offline policy ranking or real code-quality validation (Section 3, Section 4.2, Section 5). 2.The reliability of the taxonomy label interpretation is weaker than the main results: only 7,639/1,385 corrections with recordable descriptions enter the two mechanism analyses, and Appendix B's two-annotator agreement is very low for several categories (e.g., "logic error" and "signal lost" κ ≈ 0.19/0.12). The specific failure-source ordering in Section 4.3 should therefore remain exploratory, and needs verification under repository, time, and PR-size controls.

Questions for authors

1.Can you estimate the recall of "explicitly attributed corrections" relative to all true defects on a human-reviewed random sample, with sensitivity analyses by delay and repository? Do Figure 1 and the critic results hold under a stricter definition of uncorrected negatives? 2.If C365 were used to rank candidate patches for the same requirement, would it match expert quality judgments or actual subsequent correction rates better than C7? How much of the cumulative critic's gain survives equal positive counts, repository fixed effects, and time stratification?

2 comments

Thank you for your review and thoughtful suggestions. We have run additional experiments based on your comments and clarified the scope of our claims. Changes in the revised manuscript are marked in blue.

W1 / Q2.

“Long-horizon rewards are better for training” remains a predictive association rather than decision evidence.

If C365 were used to rank candidate patches for the same requirement, would it match expert quality judgments or actual subsequent correction rates better than C7?

Thank you for highlighting this. We agree that our experiments do not establish that using these critics as rewards improves code generation. We clarified in Section 4.2 (p. 5, lines 257–260) that “whether optimizing against it improves code generation remains untested,” and added this evaluation as future work in Section 5 (p. 9, lines 440–442). Our contribution is to show how feedback changes across horizons and that delayed correction outcomes can be learned from historical examples and predicted on later code changes, providing a natural starting point for post-training work.

W1 / Q1.

Can you estimate the recall of “explicitly attributed corrections” relative to all true defects on a human-reviewed random sample […]? Do Figure 1 and the critic results hold under a stricter definition of uncorrected negatives?

We agree, as discussed in Sections 4.1 (p. 3, lines 143–151) and 5 (p. 8, lines 425–428). Explicit attribution provides a documented link between changes and subsequent corrections, but does not capture every defect. Estimating recall would require independently identifying defects and tracing their corrections, including cases without explicit attribution. This is difficult even for human reviewers: a defect may be caused by several interacting changes, and reviewing a PR does not guarantee finding every defect. In Section 5 (p. 8, lines 427–428) we acknowledge that our labels “likely understate the true incidence of issues.”

We agree that testing a stricter definition of uncorrected negatives is useful. We added this comparison in Appendix C (Table C.8, p. 20, lines 1033–1046), excluding PRs whose first observed correction occurs after one year from both training and evaluation. Consistent with our previous findings, C365 continues to outperform C7 across cumulative windows. Its one-year AUROC advantage is +0.023 (95% CI [0.019, 0.027]), compared with +0.031 originally. Similarly with the disjoint critics: D7 continues to perform better on first-week corrections, while D30–365 performs better on later corrections.

W1 / Q2.

How much of the cumulative critic's gain survives equal positive counts, repository fixed effects, and time stratification?

Thank you for suggesting this. The cumulative experiment measures the benefit of longer feedback windows, including additional positives, while the disjoint experiment holds positive and negative counts fixed. We additionally tested cumulative critics with equal positive counts, balanced across correction windows, over ten sampling seeds (Appendix C, Figure C.3, p. 18, lines 942–955). Consistent with our original late-window results, C365 exceeds C7 by 0.042 AUROC on corrections after 30 days (95% CI [0.022, 0.061]). Its one-year cumulative gain is 0.020, with an interval spanning zero ([−0.001, 0.041]). These intervals resample the positive-sampling seeds, holding evaluation PRs fixed.

To address your concern about differences across repositories and time, we also evaluated the original critics within each repository and PR merge year (Appendix C, Tables C.6–C.7, p. 19, lines 975–1024). C365 has higher one-year AUROC than C7 in 10 of 11 repositories and all 6 years. D30–365 exceeds D7 on the (30,365]d target in all 11 repositories and 5 of 6 years.

W2.

[…] only 7,639/1,385 corrections with recordable descriptions enter the two mechanism analyses […] two-annotator agreement is very low for several categories […]. The specific failure-source ordering […] should therefore remain exploratory, and needs verification under repository, time, and PR-size controls.

Thank you for raising this concern. Section 4.3 (p. 6, lines 310–323) already describes the taxonomy analysis as “descriptive, exploratory” and limits its findings to when explanations are documented. We added a direct reference to Table B.2 (Section 4.3, p. 7, lines 324–325) to make the annotation-agreement results more visible.

The original submission already adjusted for repository differences. To futher address your concerns, we controlled for PR merge year and PR size. Consistent with our previous findings, the directions remain unchanged for every label (Appendix D, Figure D.2, p. 27, lines 1446–1454; Table D.2, p. 24, lines 1258–1280).

Thank you again for your feedback! We hope our responses and additional experiments have sufficiently clarified the scope of our claims and addressed your concerns. If so, we hope you’ll consider increasing your score.

Thank you for your reply. I will keep my score unchanged.

Comments on the paper

1 comment

Thank you for your careful review and suggestions. We have added a context-only comparison, reported human validation by horizon, and clarified the critic inputs. Changes in the revised manuscript are marked in blue.

W1 / Q3.

The connection between predicting corrections and providing a useful reward for code generation remains incomplete. The current evaluation compares historical PRs from different tasks. […] Still, a controlled evaluation of alternative implementations under a fixed task would provide substantially stronger support for the proposed reward-modeling application.

How well does a context-only critic perform under the same evaluation protocol? If available, results comparing alternative implementations of the same task would also be helpful. These would clarify how much of the predictive signal comes from implementation quality rather than differences between tasks.

Thank you for suggesting this comparison. To address your concerns, we trained a critic using only the PR title and body, with the same training and evaluation protocol. The PR-context critic, which uses both the code change and the PR title and body, outperforms this critic by 0.018 AUROC (0.729 versus 0.711; 95% CI [0.015, 0.021]). This indicates that the code change provides predictive information beyond the PR title and body. We added these results in Section 4.4 (p. 8, lines 406–409) and Appendix C (Table C.9, p. 20, lines 1055–1063).

We agree that comparing alternative code changes for the same task would be valuable, but this is beyond the scope of this paper. Our contributions concern how feedback changes across horizons and whether critics can predict delayed corrections, rather than establishing improved code generation.

W2.

I am less convinced by the interpretation of feedback categories in Section 4.3. Human–human agreement is low for some relevant categories: κ is 0.19 for incorrect logic and 0.21 for contract change. Their estimated horizon differences also have confidence intervals that touch or cross zero. The “Reason Not Caught” analysis covers only 1,385 of the 13,176 corrections. These results can support an exploratory discussion, but I would be cautious about drawing a broader distinction between simple failures captured early and more involved failures captured late. For the core correction labels, the pooled validation results also leave open whether identification errors differ between early and late windows.

Thank you for highlighting these limitations. We agree that the taxonomy results should be interpreted as exploratory. Section 4.3 (p. 6, lines 310–323) explicitly calls this a “descriptive, exploratory analysis” and states that “we do not claim that these labels are representative of all corrections.” We added a direct reference to the per-label agreement results in Table B.2 (Section 4.3, p. 7, lines 324–325).

W2 / Q1.

For the core correction labels, the pooled validation results also leave open whether identification errors differ between early and late windows. Can you report the correction-label validation separately for early and late horizons? Even a modest stratified audit would help assess whether differences in labeling accuracy affect the horizon comparisons.

Thank you for the question. We added human judgments by horizon and Terra label (Appendix B, Table B.3, p. 14, lines 708–719). Agreement appears higher in the first-week sample, but the samples are too small to establish any systematic differences in labeling errors. We explicitly acknowledge this limitation in Appendix B (p. 14, lines 702–705).

Q2.

Are the PR titles and bodies used by the critics taken from their state at merge time or at data collection time? Please also clarify whether each PR’s diff is selected using a uniform rule that is independent of its subsequent correction links. The chronological split addresses label maturity, but the timing of the input information needs clarification.

Thank you for raising this question. PR titles and bodies are taken at the time of data collection. We clarified this in Section 4.2 (p. 4, lines 210–212). Each critic input uses the diff between the PR’s merge commit and its first parent. We apply this rule uniformly, independently of subsequent correction links. We also checked the edit histories of all 13,176 corrected PRs and identified 159 with edits after the correcting PR was merged, including 75 evaluation PRs (1.13% of evaluation positives). We manually inspected around 50 of these edits and found that edits mostly involved typos, checklist updates, documentation updates, or release-note changes. We saw no clear indication of correction leakage.

Thank you again for your feedback! We hope our additional experiments and clarifications have addressed your concerns about the critic inputs, predictive evidence, and label validation. If so, we hope you’ll consider increasing your score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.