- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper studies whether a pass/fail verdict is enough to judge an agent’s work when later tasks build on it. It runs identical follow-up tasks from different passing states and finds that outcomes differ. The core contribution is a quantity, "continuation insufficiency", together with a protocol for estimating it.
Critical strengths
The experiment protocol is careful, and is structured into a logical sequence of research questions.
Critical weaknesses
- It's unclear what this framework adds beyond evaluating agents sequentially without resets. The benefits of isolation (forking passing states into identical future tasks) are reasonable in principle, but the paper never directly evaluates to what extent such careful isolation might change rankings beyond a single repository case study (jsonpickle).
- Section 2 is decorative. Eq 1 is the law of total variance. Eq 2 is just Jensen's inequality. These two sections could have been described much more simply.
Questions for authors
- Is there any setting where evaluating agents sequentially without resets would reach a different conclusion about which agent or state is better than your proposed scheme?
- Which empirical claims depend on the formalism in Sec 2?
2 comments
Response to Reviewer gipH (2/2): W2/Q2
W2/Q2. Role of the formalism
Author response
We agree that the conditional-variance decomposition and Jensen’s inequality are established results, and that the original presentation devoted too much space to their derivation. We shortened Section 2, explicitly identified these results as standard, and placed the derivations in Appendix A.
The formal definition specifies what we mean by information lost from a success record. Under a fixed distribution of accepted states and continuation conditions, it measures the additional optimal prediction error incurred when the predictor observes only the verdict rather than the concrete state. The standard squared-loss identity expresses this quantity as variation in expected continuation outcomes among states sharing that verdict. Jensen’s inequality then relates a nonzero matched population contrast to positive information loss.
To answer which empirical claims depend on this formalism, the reported success rates, paired differences, selection outcomes, and their uncertainty are computed directly from the execution records. The formalism supplies the interpretation connecting a population difference within a shared verdict to predictive information omitted by that verdict. The natural-state result of 2.73 points is a null-adjusted mean pairwise absolute separation, reported on its own scale. We clarified this distinction in Section 3.2.
Changes in the revised manuscript
Section 2.2 (p. 3). We clarify which results are computed from execution records and how the formalism supports their interpretation.
“Success rates, paired differences, and selection outcomes are all computed from execution records. The equations define and interpret the information omitted by the success record.”
Section 3.2 (p. 4). We clarify the relationship between the natural-state statistic and the formal quantity.
“This null-adjusted L¹ separation is the primary natural-state endpoint, not a numerical estimate of 𝓘₂.”
The derivations of the conditional-variance identity and Jensen bound are provided in Appendix A (p. 16).
Response to Reviewer gipH (1/2): W1/Q1
W1/Q1. What matched-success evaluation adds to sequential evaluation
Author response
Thank you for asking what the matched comparison adds to sequential evaluation and whether it can change a preference between accepted states. Our protocol adds a matched comparison to sequential evaluation. By holding the current verdict and continuation conditions fixed while varying the accepted starting state, it tests whether the verdict preserves the information needed to predict what happens next.
A sequential trajectory records how work proceeds from its particular starting state. To evaluate the information retained by PASS, we also need to observe what happens from other states that passed the same current-task test, under the same future tasks, executor, verifier, and budget. A sequential benchmark that performs this comparison implements our test. Matching makes differences within a shared verdict measurable, while recording the failure stage reveals whether they arise during transition, preservation of earlier behavior, or completion of the later task.
A ranking reversal is one possible consequence of these differences. A stable performance gap also shows that the shared verdict omits information relevant to choosing a starting state. Our controlled and natural-state experiments test for such within-verdict differences. Whether the choice of evaluation procedure changes the ranking is an additional empirical question, which we investigated directly.
In the Jsonpickle pilot, historical versus alternative success changed from 4/6 versus 2/6 under the matched condition to 4/6 versus 5/6 after a preceding edit. However, the sequential condition included an additional editing opportunity, so this comparison cannot isolate the effect of inheriting the preceding edit. We therefore added comparisons with equal editing opportunities. The small reversal observed in h11 disappeared when the same tests imported their exception explicitly. Across two task panels in three JavaScript projects, all 192 terminal paths passed both verifiers, leaving the starting states tied on that endpoint.
These follow-up experiments leave the evidence for robust ranking reversals beyond Jsonpickle limited. We revised the manuscript to make that scope explicit. The matched-success comparison remains a direct test of whether states assigned the same verdict have different outcomes under the specified continuation conditions.
Changes in the revised manuscript
Section 3.1, “Relation to sequential evaluation” (p. 4). We explain how the matched comparison relates to sequential execution.
The same paragraph summarizes the additional Python and JavaScript comparisons and reports that the h11 reversal disappears under the explicit-import check. The detailed results appear in Appendices N–O (pp. 34–38).
Appendix O (p. 38). We report the outcome of the JavaScript extension.