- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies the "reward horizon problem" where short-horizon rewards may diverge from long-horizon outcomes and favor different actions when the expected delayed feedback varies across actions. To study this empirically in software engineering, the authors assemble 257,788 merged pull requests (PRs) at least one year old across 11 open-source GitHub repositories (2011–2025) and use GPT-5.6 Terra to identify 13,176 PRs (5.1%) with an observed, explicitly attributed corrective follow-up (10,050) or revert (3,126) within 365 days.
Across four empirical analyses, they show that 52.4% of one-year observed corrections occur more than 7 days after merge; train L2-regularized logistic regression critics on Qwen3-Embedding-8B embeddings of PR titles/bodies and code diffs across cumulative (C7, C30, C365) and disjoint (D7, D7-30, D30-365) horizon targets, finding that C365 achieves higher AUROC than C7 across all three cumulative evaluation horizons; analyze GPT-5.6 Terra multi-label taxonomies of Issue Source and Reason Not Caught across correction horizons; and compare how critic scores correlate with complexity proxies when using code-change embeddings alone versus adding PR title/body embeddings
Critical strengths
-
Large-scale longitudinal dataset of post-merge software corrections. The paper assembles 257,788 merged PRs across 11 major open-source repositories (2011–2025) with a chronological train/evaluation split separated by a 365-day gap intended to reduce potential lookahead bias. Figure 1 clearly shows that 52.4% of one-year observed corrections arrive more than a week after merge (with repository medians ranging from 4.9 days in
scikit-learnto 62.4 days inDjango), providing concrete evidence that short post-merge windows miss a large share of observed post-merge corrections -
Transparent prompts, uncertainty reporting, and worked examples. The appendices provide the exact prompts for
GPT-5.6 TerraandGPT-5.4 Nano, bootstrap confidence intervals for the pooled critic and taxonomy comparisons, repository-filtered evaluations for the cumulative critics (Tables C.4–C.5), repository-fixed-effect adjustments for the taxonomy means (Figure D.2), and concrete PR-pair examples for every substantive taxonomy category displayed in Figure 3
Critical weaknesses
-
Repository composition and training-sample variability limit interpretation of the disjoint-critic results:
Although
C365has higher AUROC point estimates thanC7in most repository-filtered evaluations (Tables C.4–C.5), the positive-count-matched evidence for distinct horizon signals comes from the pooled disjoint critics (D7,D7-30,D30-365; Figure 2 and Tables C.1–C.2). Repository correction rates (~3%–10%) and median correction horizons (4.9–62.4 days) differ substantially. Matching 1,025 positives per critic does not ensure or verify matched repository composition across windows, so repository identity potentially encoded in the embeddings could partly explain the horizon-specific advantages. Per-repository evaluation or matched repository compositions would help distinguish this explanation from distinct within-repository defect signals. Moreover, the evaluation-row bootstrap holds the fitted critics fixed (Appendix C, lines 708–709), so its intervals do not capture variability from positive subsampling and model refitting. No metadata-only baseline, such as repository indicators and diff size, is reported -
Limited label validation and selective taxonomy coverage constrain interpretation of feedback differences:
The dataset measures explicitly attributed corrections, whose coverage across horizons and repositories remains unquantified. Appendix B validates 200 candidate edges sampled equally from
Terra's predicted-positive and predicted-negative sets, but reports only aggregate pairwise agreement and Cohen's (Table B.1). These statistics do not reveal how disagreements are distributed between the two prediction strata. The taxonomy analyses are appropriately framed as exploratory, but five of the six substantiveIssue Sourcecategories have reported confidence intervals on the horizon difference that include zero (Table D.1). ForReason Not Caught, only 10.5% of corrections enter the mention-gated cohort, and 31.1% of that cohort is subsequently labeledUnknown. Human–human exact-label-set agreement also has for both taxonomies, with heterogeneous per-label agreement (Figure B.2, Table B.2). These limitations leave the robustness of the feedback-composition findings to annotation disagreement and cohort selection unclear
Questions for authors
This is a borderline submission with a valuable longitudinal dataset. The following clarifications and analyses would materially inform my assessment, particularly whether the disjoint-critic findings persist after accounting for repository composition:
-
Do the disjoint-critic results (
D7,D7-30,D30-365; Figure 2 and Tables C.1–C.2) hold within individual repositories or after matching repository composition across horizon windows in both training and evaluation? Are they stable across repeated positive subsampling and model refitting? As supplementary checks, if available, comparisons with a metadata-only baseline and sensitivity to clustered resampling by shared correcting artifacts would also be informative. -
Were the human annotators involved in taxonomy or prompt development (Appendix E), and were they blinded to
Terra's labels? Can you provide candidate-edge confusion matrices against each human annotator, or against adjudicated reference labels if available? As a supplementary analysis, how sensitive are theReason Not Caughthorizon differences in Table D.1 to mention-gating and the inclusion ofUnknowncases?
5 comments
Thank you for running these additional experiments and adding the detailed breakdowns in the appendix. They address most of my initial concerns, and I am leaning toward raising my score. Before finalizing my rating, I would appreciate a brief discussion of the following three points in the main text:
-
Metadata baseline (Figure C.4)
In pooled evaluation, the simple baseline using only repo ID and log lines changed gets higher mean AUROC than the embedding critics across all matching settings (e.g., 0.712 vs. 0.664 for D30–365 on the 30–365d window). Please discuss this comparison in Section 4.2 and what it means for interpreting the embedding results
-
Disjoint-critic nuances (Figure C.4, Table C.11)
It is good to see that the early-vs.-late crossover between D7 and D30–365 still holds within repo and size groups. However, on the (7, 30]d evaluation window, D7 outperforms D30–365 under grouped evaluation, reversing their ordering in pooled evaluation. D7–30 also achieves the highest mean AUROC across all three windows under grouped evaluation. Please mention these nuances in Section 4.2 when describing the horizon-specific signal
-
Human validation of correction labels (Table B.4)
Thank you for sharing the confusion matrices and annotation details. Out of 100 Terra-positive candidate edges, the two human labelers judged 17 and 29 to be non-corrections; out of 100 Terra-negative candidate edges, they judged 0 and 10 to be corrections. Since no adjudicated reference labels are reported, these counts describe disagreement between Terra and the human annotators and highlight uncertainty in the correction labels. Briefly summarizing them in Section 3 or 4.1 would help readers judge label reliability
Thank you for the follow-up and for clarifying what would help readers interpret these results. As before, changes in the revised manuscript are marked in blue.
We added the metadata comparison and the grouped evaluation results to Section 4.2 (pp. 5–6, lines 266–269 and 307–311):
“To assess how well repository identity and diff size alone predict corrections, we trained critics using only repository identity and log lines changed. These critics have higher mean AUROC than the embedding critics in pooled evaluation (Figure C.4, left panel). We next evaluate the embedding critics on similarly sized PRs within each repository to assess whether the relative performance of D7 and D30–365 across horizons depends on these characteristics (Figure C.4, right panel). We find that D7 still outperforms D30–365 on first-week corrections, while D30–365 still outperforms D7 on corrections after 30 days. On the (7, 30]d window, D7 slightly outperforms D30–365, reversing their pooled ordering (Table C.11). Additionally, D7–30 has the highest mean AUROC on all three grouped windows (Figure C.4). Taken together, these results suggest that different horizons carry both shared and distinct predictive signals.”
We also added the human validation results to Section 3 (p. 3, lines 125–127):
“Two authors independently reviewed a sample of candidate edges labeled by GPT-5.6 Terra. Their agreement with the model ranged from 71% to 83% for edges it labeled as corrections and from 90% to 100% for those it labeled as non-corrections (Appendix B, Table B.4).”
Thank you again for all of your feedback!
Thank you for your detailed review and suggestions. We have added experiments addressing repository composition, training variability, metadata baselines, and bootstrap uncertainty, along with additional label-validation results. Changes in the revised manuscript are marked in blue.
W1 / Q1.
Thank you for raising these concerns about repository composition and training variability. We added within-repository evaluations of the disjoint critics. Consistent with our previous findings, D30–365 outperforms D7 on corrections after 30 days in all eleven repositories (Appendix C, Table C.6, p. 19, lines 975–998).
We also repeated positive sampling and training across ten seeds. Consistent with our previous findings, D7 outperforms D30–365 on first-week corrections, while D30–365 outperforms D7 on both later windows in all ten runs (Appendix C, Table C.10, p. 20, lines 1073–1079).
We trained additional critics using only repository identity and log lines changed and found that they outperform the embedding critics in pooled evaluation. To assess whether these features explain the embedding critics’ early-versus-late pattern, we also evaluated the critics within repositories and groups of similarly sized PRs (Appendix C, Figure C.4 and Table C.11, p. 21, lines 1104–1130). Consistent with our previous findings, D7 outperforms D30–365 on first-week corrections, while D30–365 outperforms D7 on corrections after 30 days. Repository identity and coarse differences in diff size alone therefore do not explain the embedding models’ AUROC performance pattern.
We also recomputed confidence intervals using a clustered bootstrap, resampling together PRs sharing their first correcting PR or commit. Consistent with our previous findings, D7 outperforms D30–365 on first-week corrections, while D30–365 outperforms D7 on both later windows. All three paired 95% confidence intervals exclude zero (Appendix C, Table C.12, p. 22, lines 1141–1152).
W2 / Q2.
Thank you for requesting this breakdown. We added correction-label confusion matrices against each human labeler (Appendix B, Table B.4, p. 14, lines 725–733). Of 100 Terra-positive candidates, the humans judged 83 and 71 to be corrections. Of 100 Terra-negative candidates, they judged 100 and 90 not to be corrections. Table B.3 (p. 14, lines 708–719) reports agreement by horizon and Terra label.
The human labelers are authors. One labeler developed the taxonomies and labeling prompts and had previously seen a handful of Terra labels. However, neither labeler was shown Terra’s labels during annotation. We clarified these details in Appendix B (p. 12, lines 613–617).
We repeated the Reason Not Caught comparisons after excluding Unknown cases. All six differences retain their signs, and the same three 95% confidence intervals exclude zero (Appendix D, Table D.3, p. 25, lines 1301–1315).
We agree that restricting the analysis to documented explanations limits its coverage. Section 4.3 (p. 6, lines 310–323) already explicitly states that “we do not claim that these labels are representative of all corrections” and frames the analysis as descriptive and exploratory. We therefore interpret these results as differences among documented explanations, rather for all corrections.
Thank you again for your feedback! We hope the additional experiments and clarifications have sufficiently addressed your concerns. If so, we hope you’ll consider increasing your score.