acceptodds
Under review as a conference paper at ICLR 2027

When Success Is a Lossy Record: Matched-Success Continuation Evaluation for Persistent Agents

Abstract

Agent benchmarks reduce a completed episode to a verdict such as PASS, discarding the environment that later tasks inherit. We formalize this compression failure as continuation insufficiency: the excess Bayes risk of predicting future outcomes from the verdict rather than the state under declared continuation conditions. Matched-success evaluation forks equally accepted states into identical frozen continuations and separates transition, preservation, downstream utility, and cost. Across 360 controlled coding pairs, later success differs by 27.50 percentage points; a complementary 240-pair enterprise study shows an 8.75-point shift in the same direction. Among 64 naturally produced successful states, a 2.73-point reference-free separation survives crossed randomization (p = 0.0023). On 144 held-out executions in 18 repositories, retaining continuation information avoids 13.19 points of verdict-only choice regret and recovers the historical-state baseline. Replay and recovery identify history compatibility as the principal omitted variable, while a common-snapshot control isolates downstream divergence in Jsonpickle. Success is therefore a lossy record of an agent's work: our protocol detects the missing information, locates its mechanism, and prices the cost of discarding it.

Primary area:
datasets and benchmarks
Keywords:
AI agents, agent evaluation, persistent state, continuation evaluation, software engineering agents, benchmark validity

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_gipHOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies whether a pass/fail verdict is enough to judge an agent’s work when later tasks build on it. It runs identical follow-up tasks from different passing states and finds that outcomes differ. The core contribution is a quantity, "continuation insufficiency", together with a protocol for estimating it.

Critical strengths

The experiment protocol is careful, and is structured into a logical sequence of research questions.

Critical weaknesses
  • It's unclear what this framework adds beyond evaluating agents sequentially without resets. The benefits of isolation (forking passing states into identical future tasks) are reasonable in principle, but the paper never directly evaluates to what extent such careful isolation might change rankings beyond a single repository case study (jsonpickle).
  • Section 2 is decorative. Eq 1 is the law of total variance. Eq 2 is just Jensen's inequality. These two sections could have been described much more simply.
Questions for authors
  • Is there any setting where evaluating agents sequentially without resets would reach a different conclusion about which agent or state is better than your proposed scheme?
  • Which empirical claims depend on the formalism in Sec 2?
2 comments
AuthorsOpenReview ↗

Response to Reviewer gipH (1/2): W1/Q1

W1/Q1. What matched-success evaluation adds to sequential evaluation

Author response

Thank you for asking what the matched comparison adds to sequential evaluation and whether it can change a preference between accepted states. Our protocol adds a matched comparison to sequential evaluation. By holding the current verdict and continuation conditions fixed while varying the accepted starting state, it tests whether the verdict preserves the information needed to predict what happens next.

A sequential trajectory records how work proceeds from its particular starting state. To evaluate the information retained by PASS, we also need to observe what happens from other states that passed the same current-task test, under the same future tasks, executor, verifier, and budget. A sequential benchmark that performs this comparison implements our test. Matching makes differences within a shared verdict measurable, while recording the failure stage reveals whether they arise during transition, preservation of earlier behavior, or completion of the later task.

A ranking reversal is one possible consequence of these differences. A stable performance gap also shows that the shared verdict omits information relevant to choosing a starting state. Our controlled and natural-state experiments test for such within-verdict differences. Whether the choice of evaluation procedure changes the ranking is an additional empirical question, which we investigated directly.

In the Jsonpickle pilot, historical versus alternative success changed from 4/6 versus 2/6 under the matched condition to 4/6 versus 5/6 after a preceding edit. However, the sequential condition included an additional editing opportunity, so this comparison cannot isolate the effect of inheriting the preceding edit. We therefore added comparisons with equal editing opportunities. The small reversal observed in h11 disappeared when the same tests imported their exception explicitly. Across two task panels in three JavaScript projects, all 192 terminal paths passed both verifiers, leaving the starting states tied on that endpoint.

These follow-up experiments leave the evidence for robust ranking reversals beyond Jsonpickle limited. We revised the manuscript to make that scope explicit. The matched-success comparison remains a direct test of whether states assigned the same verdict have different outcomes under the specified continuation conditions.


Changes in the revised manuscript

Section 3.1, “Relation to sequential evaluation” (p. 4). We explain how the matched comparison relates to sequential execution.

“A sequential trajectory measures its realized history; crossing accepted states with identical continuations implements the matched comparison.”

The same paragraph summarizes the additional Python and JavaScript comparisons and reports that the h11 reversal disappears under the explicit-import check. The detailed results appear in Appendices N–O (pp. 34–38).

Appendix O (p. 38). We report the outcome of the JavaScript extension.

“None of the three additional JavaScript projects reproduced a strict state-ranking reversal across both case panels. All 192 terminal paths passed both rules, so this panel cannot distinguish the two starters’ terminal difficulty.”

AuthorsOpenReview ↗

Response to Reviewer gipH (2/2): W2/Q2

W2/Q2. Role of the formalism

Author response

We agree that the conditional-variance decomposition and Jensen’s inequality are established results, and that the original presentation devoted too much space to their derivation. We shortened Section 2, explicitly identified these results as standard, and placed the derivations in Appendix A.

The formal definition specifies what we mean by information lost from a success record. Under a fixed distribution of accepted states and continuation conditions, it measures the additional optimal prediction error incurred when the predictor observes only the verdict rather than the concrete state. The standard squared-loss identity expresses this quantity as variation in expected continuation outcomes among states sharing that verdict. Jensen’s inequality then relates a nonzero matched population contrast to positive information loss.

To answer which empirical claims depend on this formalism, the reported success rates, paired differences, selection outcomes, and their uncertainty are computed directly from the execution records. The formalism supplies the interpretation connecting a population difference within a shared verdict to predictive information omitted by that verdict. The natural-state result of 2.73 points is a null-adjusted mean pairwise absolute separation, reported on its own scale. We clarified this distinction in Section 3.2.


Changes in the revised manuscript

Section 2.2 (p. 3). We clarify which results are computed from execution records and how the formalism supports their interpretation.

“Success rates, paired differences, and selection outcomes are all computed from execution records. The equations define and interpret the information omitted by the success record.”

Section 3.2 (p. 4). We clarify the relationship between the natural-state statistic and the formal quantity.

“This null-adjusted L¹ separation is the primary natural-state endpoint, not a numerical estimate of 𝓘₂.”

The derivations of the conditional-variance identity and Jensen bound are provided in Appendix A (p. 16).

Reviewer_52EuOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper asks whether states receiving the same current-task success verdict remain equivalent for later tasks. It defines continuation insufficiency as excess prediction risk from retaining the verdict rather than the state, and proposes matched continuations that separate transition, preservation, downstream utility, and cost. Validation includes 600 controlled pairs, 64 unique naturally produced successful states, an 18-repository held-out selection panel, and recovery and common-snapshot studies. The 13.19-percentage-point selection gain over uniform choice matches the historical-state baseline and is largely explained by history compatibility.

Critical strengths
  1. The paper turns an intuitive limitation of pass/fail evaluation into a precise, context-dependent measurement question. Assigning identical continuations to equally accepted states makes the omitted information testable without asserting a universal ranking of states.

  2. The evidence ladder addresses meaningful alternative explanations. Natural states complement designed contrasts; recovery separates transition failures from downstream failures; and the paper explicitly reports the failed selection gate and equality with the historical baseline. These distinctions support the scoped existence claim.

Critical weaknesses
  1. The demonstrated operational benefit is limited. The frozen selector does not improve on retaining historical-state identity, and its gain over shortest patch lacks exact-test support. Consequently, the headline regret reduction establishes the cost of a deliberately compressed record rather than an improvement over those available state-aware defaults.

  2. Evidence for downstream utility after removing history asymmetry is concentrated in Jsonpickle. The enterprise contrast and aggregate common-snapshot intervals include zero, while the corrected natural-state separation is 2.73 points. These results support existence but leave the breadth and practical magnitude of the phenomenon uncertain.

Questions for authors
  1. Assessment-relevant accounting: What development-rollout cost is required for the frozen selector, including failed transitions, and how does held-out performance change with that budget? Since the selector matches the historical default, this accounting would determine whether the proposed measurement is practical when selecting among accepted states.

  2. Clarification of formal contribution: Grimm et al., “The Value Equivalence Principle for Model-Based Reinforcement Learning” (NeurIPS 2020), not cited in the manuscript, defines equivalence relative to selected functions and policies. What is the precise relationship to your context-relative record criterion, and which formal or experimental test is added by conditioning on a shared accepted record? Please distinguish the established squared-loss decomposition from the contribution of the matched-success protocol.

https://papers.nips.cc/paper/2020/hash/3bb585ea00014b0e3ebe4c6dd165a358-Abstract.html

2 comments
AuthorsOpenReview ↗

Response to Reviewer 52Eu (1/2): W1-W2

W1. Practical benefit relative to available defaults

Author response

We agree that the practical benefit of the selector depends on what the practitioner already knows. When the previously merged version is known, our results support retaining it. Across 144 held-out executions in 18 repositories, retaining that version achieved the same 37.50% success rate as our selector and required no development rollouts. Choosing the shorter patch achieved 34.72%. The observed 2.78-percentage-point advantage over that rule was not supported by the exact two-sided test (p = 1.0).

The design of the repository study helps explain this result. The follow-up changes were written against the version the maintainers had merged. When we replayed those changes on an alternative that had passed the same original test, 48 of 72 executions stopped at a source conflict before the follow-up task began. The 13.19-percentage-point difference compares two specific choices: retaining the merged version achieved 37.50% success, whereas choosing uniformly between the two passing versions achieved 24.31%. For an evaluation that follows a project’s actual history, the practical implication is to record which passing version was merged. On this panel, that information alone recovered the full observed gain over uniform choice.


Changes in the revised manuscript

Abstract (p. 1). The revised abstract reports the historical-version and shortest-patch comparisons alongside the gain over uniform choice:

“In a subsequently frozen 18-repository stress test, continuation evidence gains 13.19 points over uniform choice but exactly matches the free keep-history policy and gains only 2.78 points over shortest patch (p = 1.0); 48/72 candidate executions stop at replay conflicts.”

Section 5.2 (p. 7), paragraph comparing the selector with the historical version and the shortest patch. After reporting the three comparisons, we state what the result establishes for a practitioner who already knows which version was merged:

“Thus the experiment establishes the cost of deleting state identity, not an advantage over a deployment that already retains history.”

Discussion and Conclusion (p. 10). We make the practical audit output and available default explicit.

The practical output is an audit of what a verdict omits and where continuation fails.

When historical identity is available in the repository setting tested here, retaining it achieves the selector's outcome without development rollouts.

W2. Breadth and practical magnitude of downstream differences

Author response

We agree that the five-repository common-snapshot study provides limited evidence about how broadly downstream differences persist once history replay is removed. We checked this directly by excluding Jsonpickle. With all five repositories included, the estimated differences were 10.0 and 3.33 percentage points for the two executors, respectively, and both intervals included zero. After excluding Jsonpickle, the aggregate difference was zero for both executors. Jsonpickle therefore provides a concrete case in which the starting state mattered under a common snapshot. This panel does not establish how frequently that occurs across repositories.

The natural-state experiment provides separate evidence about magnitude under its own test conditions. Among 64 model-produced programs that passed the current task, the mean pairwise difference in continuation outcomes exceeded the randomization baseline by 2.73 percentage points (p = 0.0023). One of six task families survived correction for multiple comparisons. The enterprise estimate was 8.75 points, with a 95% interval from −1.25 to 19.58. These results support the conclusion that a shared PASS verdict can conceal differences relevant to later tasks. Their frequency and typical size in a new repository remain open.


Changes in the revised manuscript

Section 5.3, common-snapshot analysis (p. 9). We report how the result changes when Jsonpickle is excluded:

“Removing Jsonpickle leaves zero aggregate difference for both executors.”

Abstract (p. 1). We report the magnitude observed in model-produced states and the extent of the evidence across task families:

“Among 64 model-produced successful states, reference-free separation is modest but detectable (2.73 points, p = 0.0023), with one of six families surviving multiplicity correction.”

AuthorsOpenReview ↗

Response to Reviewer 52Eu (2/2): Q1-Q2

Q1. Development cost and performance under reduced budgets

Author response

Thank you for requesting this accounting. The selector’s development phase comprised 72 attempted transitions. Thirty stopped at replay conflicts before the continuation agent ran, and 42 reached a continuation session, recording 708 tool calls. All 30 failed transitions are included in the attempt count.

We assessed sensitivity to the number of development attempts by recomputing the selection rule using each of the two repeats separately. Each reduced-attempt variant used 36 transitions and 21 continuation sessions. Both achieved 37.50% success on the same held-out evaluation, matching the full selector and the historical-version default. This is a retrospective analysis of the recorded runs. It halves the number of development attempts, while incomplete timing for failed transitions prevents a corresponding claim about total elapsed time or monetary cost.

For this repository panel, retaining the historical version achieved the same held-out performance with zero development rollouts. The cost accounting therefore supports using that available default when choosing between these accepted versions.


Changes in the revised manuscript

Section 5.2, “Evaluation cost” (p. 8). We report the development attempts, including failures before the continuation agent ran.

“The formal development ledger contains 72 attempted transitions: 30 stop at replay failure and 42 reach a consumer, recording 708 tool calls. Failed transitions remain in the cost denominator.”

The same paragraph reports the reduced-attempt analysis.

“Retrospectively using either one of the two repeats reduces development to 36 attempts and 21 consumer sessions; both reduced-budget rules still achieve 37.50% heldout success, equal to the full rule and to the zero-development historical default.”

Appendix G (p. 24) provides the detailed development-cost accounting, held-out evaluation costs, and timing coverage.

Q2. Relationship to value equivalence and the contribution of matched-success evaluation

Author response

Thank you for pointing us to Grimm et al. (2020), which we now cite in Section 6. We share their premise that equivalence depends on the intended use. Grimm et al. compare two environment models by asking whether they produce the same Bellman updates for a specified set of policies and value functions. We ask whether a benchmark record preserves the information needed to predict a later task’s outcome. For a specified future task, executor, verifier, and budget, does knowing which accepted state was retained improve prediction beyond knowing its recorded verdict?

Conditioning on a shared verdict makes this question testable. We retain distinct implementations that passed the same current-task test and give each the same continuation. The recorded verdict then provides the same information for both implementations. A reproducible difference in their expected outcomes under that continuation shows what the verdict omitted. By recording whether each implementation reached the later task, preserved the earlier behavior, and completed the new task, we can also locate where the difference arose.

Equation 1 uses the standard squared-loss decomposition to express the additional prediction error from retaining only the verdict. Our contribution is the matched-success protocol that tests the adequacy of that verdict under specified continuation conditions.


Changes in the revised manuscript

Section 6, “Context-relative equivalence” (p. 10). We added the citation to Grimm et al. and explained how conditioning on the same verdict leads to an experimental comparison:

“Conditioning on a shared record enables a within-record intervention test: vary the accepted state while assigning the same continuation.”

The following sentence distinguishes the established mathematical result from our experimental contribution:

“It does not add a new Bellman-equivalence theorem, and the squared-loss decomposition is standard.”

Reviewer_GtKhOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper makes the case that current benchmarks only investigate the verdict of an agent and the boolean result of whether the agent has passed or failed the task. However, as the paper points out, different agent trajectories and eventual final states can result in the same verdict. These different final states can alter its future usability. The authors formalize this concept as continuation efficiency: the excess Bayes risk of predicting future outcomes from the result/verdict rather than the states. To investigate this, the paper proposes a matched-success evaluation where equally accepted states are forked into identical frozen continuations and outcomes are compared, separating transition, preservation, downstream utility, and cost. The authors execute a series of experiments and analyses across the Jsonpickle repository.

Critical strengths

S1. The questions the authors pose are significant and relevant to the research field.

S2. The failure-stage decomposition is careful and potentially exportable to other benchmarks.

Critical weaknesses

W1. The results shown in the abstract seem exaggerated to me. On the one hand, the 27.5pp gap comes from pairs hand-designed as “extensible vs. narrower”, so I believe it would be fair to say that the authors measure their own construction. On the other hand, the enterprise arm has a gap of 8.75pp but at p=0.052: a statistically insignificant result. In the wild, the phenomenon seems to have a much lower gap (2.73pp).

W2. The mechanism (section 5.3) localizes where the failures occur, but does not explain or study the properties of the state that cause the divergence. As a result, the finding is hard to act upon.

W3. The paper does not address any pretraining contamination issues, which I believe strongly would affect the results of the study. The results come from essentially one producer (Codex used in 12 out of 18 repositories) and are tested in Python repositories only. As a result, the generalizability of the findings is contestable.

W4. Resources such as code haven't been released, hindering reproducibility.

W5. Semi-Minor comment: The paper is dense and hard to parse. For example, multiple times (lines: 044, 070, 072, 088) variables are used without being introduced in advance, and notions like “maintainer” (line 112) are used without explaining what they refer to.

Questions for authors

I do not have further questions for the authors, but I would greatly appreciate a response to the weaknesses I have mentioned.

3 comments
AuthorsOpenReview ↗

Response to Reviewer GtKh (1/2): W1-W2

W1. Scope of the 27.50-point gap.

Author response

Thank you for pointing out that the original abstract could lead readers to interpret 27.50 points as a typical gap between model-produced solutions. In the 360 designed coding pairs, an extensible implementation and a narrower implementation both passed the same current-task test. We then measured their success on identical follow-up tasks, obtaining 68.33% and 40.83%, respectively. The 27.50-point gap quantifies the difference that the shared PASS verdict concealed in this comparison.


Changes in the revised manuscript

In the revised abstract (p. 1), we now identify the designed comparison explicitly, report the uncertainty in the enterprise estimate, and distinguish both from the result for model-produced states. The revised sentences read:

“Designed coding pairs establish existence with a 27.50-point continuation gap; the enterprise contrast is directionally positive but unresolved (8.75 points, 95% CI [−1.25, 19.58]). Among 64 model-produced successful states, reference-free separation is modest but detectable (2.73 points, p = 0.0023), with one of six families surviving multiplicity correction.”

W2. Which properties of the starting state explain the divergence?

Author response

We agree that the original Section 5.3 located the divergence in Jsonpickle but left its cause unresolved. The successful agent edits used an existing mechanism for preserving references to dictionaries. Our description suggested that these edits succeeded because this mechanism was available only in the alternative starting state. We added two experiments to test that explanation.

A source audit first showed that all three starting states contained the dictionary-reference mechanism. We then varied three source-code properties that actually differed across the states, producing eight starting versions. All eight passed the original task and failed the follow-up task before repair. Each of two fixed repairs passed the follow-up task on all eight versions. We also tested whether the agents’ own edits depended on where they had been written. We took all 18 original patches, including unsuccessful ones, and tried each on all three starting states. Of the 54 transfers, 42 applied and 12 encountered textual conflicts. Four patches passed the follow-up task on all three states, accounting for all 12 successful transfers. Among patches that applied, none changed from passing to failing, or vice versa, when moved to another state.

These results rule out a specific explanation for the observed divergence: the agents that failed did not start from code that lacked a mechanism required by the successful patches. What remains unexplained is why an agent produced a successful edit in some runs and an unsuccessful edit in others. A starting-state feature could influence which edit an agent attempts even when the same repair works across states. Our experiments do not establish such an effect for any particular feature.


Changes in the revised manuscript

We revised Section 5.3, in the paragraph on common-snapshot execution (p. 9), to narrow the mechanism claim. The revised text reads:

“The separation is localized to Jsonpickle. New source interventions show that the tested successful repairs work in every initial state; the divergence does not identify a uniquely available repair primitive (Appendix K).”

We added the source interventions and patch-transfer results in Appendix K (p. 28). The revised text reports:

“Four donor patches (three Sol, one Terra) pass the future in every background, giving 12 successful transfers; the other 30 applicable transfers fail it.”

AuthorsOpenReview ↗

Response to Reviewer GtKh (2/2): W3-W5

W3. Pretraining exposure, producer coverage, and language scope.

Author response

We agree that the use of public repositories leaves pretraining exposure unresolved. The hidden tests restrict what an agent can access during evaluation. Because the models’ training corpora are unavailable, we cannot determine whether they had previously seen a repository or its later commits, or quantify the effect of such exposure.

We audited the provenance of the alternative implementations in the 18-repository study. Codex produced 14 and Qwen produced four. We repeated the comparison separately for each producer. In the Codex repositories, selection based on development continuations and simply retaining the historical implementation both achieved 39.29% held-out success. In the Qwen repositories, both achieved 31.25%. Thus, the tie with the historical baseline was present in both groups. The four Qwen repositories provide limited evidence about variation across producers.

We applied the same evaluation protocol to three independently maintained JavaScript projects. Across both task panels, the two starting implementations tied on the terminal checks: all 192 paths passed both verifiers.


Changes in the revised manuscript

We added the producer-stratified results in Appendix K.1 (Table 9, p. 29) and the JavaScript experiments in Appendix O (pp. 37–38). The revised Discussion (p. 10) states:

“Public repository histories and unavailable training corpora prevent excluding pretraining exposure; hidden tests address evaluation-time leakage only.”

Appendix O (p. 38) reports the outcome directly:

“None of the three additional JavaScript projects reproduced a strict state-ranking reversal across both case panels. All 192 terminal paths passed both rules, so this panel cannot distinguish the two starters’ terminal difficulty.”

W4. Code availability and reproducibility

Author response

Thank you for raising this concern. We have made the evaluation code, frozen task specifications and tests, archived model outputs, and analysis scripts available in the anonymous repository. After downloading the repository, reviewers can run make reproduce-review to recompute the reported scores from the archived outputs without making new model calls.

We verified the release by downloading the public ZIP without signing in and running the command in a clean environment. On a second machine, we reproduced all 768 primary checks on the 288 archived Python outputs. For the two JavaScript panels, the replay reproduced the archived scores exactly for all 288 outputs.


Changes in the revised manuscript

The Introduction (p. 1, footnote) links directly to the anonymous repository. The Reproducibility Statement (p. 15) describes the release contents and how to recreate the results. The revised text reads:

“Code and frozen evaluation artifacts are available in the anonymous code repository (ID: matched-success-continuation-evaluation-B117).”

W5. Readability and introduction of notation

Author response

Thank you for identifying the places where readers encounter notation or terminology before they know what it means. We revised the Introduction and Figure 1 so readers can follow the comparison before encountering its formal definition, and we now define the historical comparator when it first appears in the Jsonpickle example.


Changes in the revised manuscript

  • Introduction (original lines 044 and 088, revised pp. 1–2). We removed the formal risk notation from the opening explanation and replaced the context tuple with the four conditions held fixed: the future tasks, executor, tests, and budget.
  • Figure 1 caption (original lines 070 and 072, revised p. 2). We removed the context tuple and the expression for the expected outcome, and explained the profile marked V in the figure. The caption now describes the comparison directly:

“We give both states the same future tasks, executor, tests, and budget, then compare how often each succeeds.”

  • Section 2, opening example (original line 112, revised p. 3). We define both the people involved and the version of the code used for comparison:

“Repository maintainers are the people who review and merge changes into a project; the historical state is the version they merged for the original issue.”

For readability elsewhere, Section 3.1 (p. 4) now explains what each of the three matching steps holds fixed, and Appendix A (p. 16) contains the derivations of the Jensen bound and the context-free regret calculation.

Reviewer_GtKhOpenReview ↗

Response to authors

Thank you for the detailed response and the substantial additional work. I appreciate the candour of the revision: the claims now match the evidence much more closely.

Resolved. The revised abstract addresses W1: the designed-pair gap, the unresolved enterprise contrast, and the modest natural-state effect are now clearly distinguished. Releasing the code addresses W4. The Introduction and Figure 1 changes improve W5, and thank you for correcting my count of the producer mix (14 Codex / 4 Qwen).

Remaining concerns.

  • W2 (mechanism). The source interventions and patch-transfer experiments are well designed, but as you note, they rule out one explanation without finding the cause. With Jsonpickle excluded, the common-snapshot difference is zero for both executors. So the evidence for downstream divergence after removing history replay rests on a single repository whose cause is unknown. This makes the finding hard to act on and hard to generalize from.

  • W3 (generalizability). Pretraining exposure is acknowledged but untested. The Qwen stratum (4 repositories) is too small to say much about variation across producers. The JavaScript extension uses designed tasks and scripted refactors, a different executor setup (one call, no tools), and every one of the 192 terminal paths passes both verifiers. By your own account, it cannot distinguish the states, so it adds no evidence for or against generalization.

  • Significance. With claims scoped correctly, the empirical picture is: a large gap on pairs built to differ, a small effect on natural states (2.73pp, one of six families after correction), a selector that fails its preregistered gate and in the 18-repository study only matches the free keep-history baseline, and downstream divergence in one repository. I find the question important and the failure-stage decomposition useful. But I am not yet convinced the evidence shows that this omission matters in practice beyond constructed settings.

  • The abstract. The revised abstract is harder to read than the original. Consider:

    “Under equal two-call budgets, h11 shows a sparse reversal that disappears when the same tests import their exception explicitly.”

    A reader has no way to know what h11 is, what a “sparse reversal” is, which exception is meant, or why importing it matters. To be brutally honest, the sentence reads as if it was generated and not checked by anyone for whether it communicates the intended meaning clearly. The same goes for “keep-history policy”, “reference-free separation” and “common-snapshot control”, none of which are defined where they appear. The authors disclose that language models were used for manuscript drafting. That is acceptable, but the authors are still responsible for the text being readable, and I am sorry to state this, but unfortunately this abstract is not.

I have kept my scores, and I maintain my overall rating. I thank the authors again for an engaged and transparent rebuttal.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.