- Rating
- 2
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper investigates coding agents that tackle a sequence of dependent requests. This paper proposes BOBA-Bench, which has 280 chains of 20 consecutive requests. Data are constructed from 140 Python repositories, and each request is graded by hidden developer tests. This paper evaluates five frontier agents in settings that separate who writes the underlying code from whether that code is verified. Agents complete 48% of chains on average while passing 94-98% of individual requests, fail the next request far more often after an unverified mistake, and gain 8-16 points of chain completion when simply told that verification failed.
Critical strengths
-
This is a carefully engineered benchmark that measures something single-request evaluation cannot. Coding agents can tackle a sequence of dependent requests. This paper helps measure beyond single requests, constructing data from history commits of 140 repositories.
-
This paper performs informative experiments. Because runs can resume from the exact failed state, the paper can compare matched arms. The results are quantitative and informative for actionable analysis.
Critical weaknesses
- The paper does not establish that a silent miss is the agent's mistake, and its own evidence suggests that a substantial share are not.
In Table 21, it shows 25.2% of the 575 first-attempt misses (about 145) as "Benchmark-side, suspected: test stricter than the specification, broken or ambiguous". The share is 31% for both Fable 5 and GPT-5.6. Yet the main text says a miss "can be attributed to the agent" in section 3.1, and its root-cause list in section 4.2 gives four agent-side causes without mentioning this category.
Also, section D.5 states that "no human-labeled adequacy sample exists yet". Claude Opus models, with GLM-5.2 as an outage fallback, write the specifications. GLM-5.2, Fable 5, and GPT-5.6 audit them with the hidden tests visible, and GPT-5.6 and Fable 5 rewrite the rejected ones. So three of the five evaluated agents audit the specifications they are tested on, and two of them also rewrite some.
There's also other evidence on this, such as:
- 886 benchmark specifications (20.1%) are never solved blind by either model in 16 attempts (Table 14). They touch 204 chains, and 9 chains consist entirely of them.
- When two models fail the same chain, they fail at the same request 53.5% of the time, against 5.2% if failure positions are independent (F.7).
- The post hoc R15 round removes 14 chains whose hidden tests presuppose the human implementation of an earlier request. That defect class is found only where a model happened to expose it.
So all the main findings assume that misses are agent errors: the completion rates, the "mistakes they cannot see", and above all the near-zero reviewer discrimination (Youden J of 0.01-0.15, Table 11). Near-zero discrimination is exactly what to expect if many "wrong" implementations are wrong only relative to tests the specification does not determine. So this can decrease Fable 5's per-request miss rate, yet greatly raise the constant-rate prediction of its completion.
- Main claims go beyond what the designs can support.
The chain ends at the first miss, so a constant per-request miss rate already predicts the "48% of chains versus 94-98% of requests" gap. F.7 shows that (1−h)^20 gives 67.9/53.3/48.2/45.9/30.7% against the observed 65.7/52.1/48.2/43.9/27.9%, and per-quarter miss rates rise only slightly over the chain (Table 24). The abstract says "the degradation comes from building on unverified code". But in the autonomous setting, no request is graded on an unverified state.
Also, Appendix J itself warns against treating a non-significant result as equivalence, and several conditions here favor finding no gap, such as:
- 72-91% of boundaries pass 8/8 from both prefixes (Table 9)
- 40-64 of the 140 repositories never produce a certified agent prefix and drop out
- Agent prefixes are retried until they are certified
- R15 removes 14 chains selected because the agent prefix failes where the human prefix passes, which is the contrast being measured
Section 4.2 states in bold that the odds of failing roughly double "with every additional unverified mistake", while F.4 calls the progression "descriptive, not a controlled dose". The claim that "a repaired state is no longer a broken assumption" (in section 5.1) rests on repaired subsets selected on outcome (F.3, K.1).
So the abstract's two conclusions "degradation comes from building on unverified code" and "verified agent code is as good a base as human code" are the paper's conceptual contribution, and neither is yet supported as stated.
Questions for authors
-
How can you validate that a silent miss is the agent's mistake, as the evidence suggests that a substantial share are not?
-
Would you be able to validate and explain the contradictory evidence in weakness 1?
-
What is the per-request miss rate by specification class? Such as hard but fair, repaired pending validation, rewritten by GPT-5.6 or Fable 5, and original text? How do you reconcile 20% of specifications going 0/16 blind with per-request pass rates of 94-98%?
-
In arm D, how many own-work failures at k+1 occur in tests that call symbols introduced or changed by request k? How is "request overlaps buggy code" defined in Figure 4?
-
The noisy supervisor raises false alarms but doesn't miss a true failure? What happens to completion when the supervisor also misses true failures, for example, at recall 0.5-0.9, or when the supervisor is one of the section 4.3 model reviewers? The findings and conclusions in section 5 depend on this.