- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
SeamBench studies append-only code handoffs: model A's answer to a HumanEval/MBPP task is cut at a line boundary, model B writes only the remaining text, and the combined program is executed. It contains 2,488 such seams from 428 tasks both models can solve alone, plus a matched same-model restart control and a BigCodeBench validation. The paper claims that switching models does not increase failure, that most failures are structural and caught by simple rules, and that an LLM judge benefits more from seeing the task than from seeing where the cut was.
Critical strengths
- A clearly defined problem: the prefix is immutable, the combined program is executed, and structural, test, and append-contract failures are recorded separately.
- Careful design: a matched same-prefix control, a BigCodeBench cohort fixed before outcomes were seen, and negative results kept.
- Reporting detection at fixed false-rejection budgets, rather than only AUC, fits an accept/reject decision.
- An unusually complete release, with every seam, label, and judge record plus offline scorers.
Critical weaknesses
-
"Switching is not the cost" is stronger than the evidence. The data support "no increase was detected," not "never increases": the intervals still allow increases of up to about 13 points, there is no equivalence test, the small MBPP follow-up in Appendix B.6 shows a large increase for GPT-4o-mini to Qwen, and the one large decrease does not survive the paper's own Holm correction. The control also compares two different models on the same prefix, so it mixes model ability with any cost of continuing someone else's code. It covers only two pairs, and none where Sonnet is the replacement, which is where failures concentrate.
-
Most failures are formatting or structural breakage, but the paper never says which kinds, so the headline rate may partly reflect the protocol. These are genuine append-only failures, but nearly all failures where Sonnet continues are of this kind (a leading Markdown fence alone counts), and no breakdown by type (fences, duplicate definitions, overlaps) is given. There is also no fresh full-solution baseline, so part of the 17–26% restart failure may be ordinary regeneration failure rather than anything prefix-specific.
-
Headline detector recall is dominated by checks that overlap the failure label. The rules share their parse and entry-point checks with the label, so their 93% recall mostly restates how common structural failures are. The structural-pass subset is the key test of semantic detection, and there the paper says rules catch none of the 37 failures, while Table 10 shows one. The main judge is not given the task description, so its weak recall on semantic failures says little about judges in general, and 37 core failures are too few for reliable per-direction comparisons. The core was also used during detector development, so the BigCodeBench results (Table 6) are the main held-out evidence.
-
The judge-information result is too small to call a "first-order design variable." In Table 7, showing the task changes detections by two to four examples and showing the cut by at most one, there are no sampling intervals, and only one judge, prompt, and model pair are tested. Appendix B.8 itself says no general advantage for revealing the cut is established.
-
The setting is narrow. Cuts are only at line boundaries, while real interruptions can fall mid-line, and the paper's own cursor-state pilot finds open strings and unfinished blocks at the cut as a recurring failure pattern. The HumanEval/MBPP core consists of short public Python functions that may have been seen in pretraining, and BigCodeBench, while more complex, covers only one model pair.
-
A reporting inconsistency. GPT-5-mini requests in the two older, largest cohorts used effective temperature 1 (Appendix B.1), although Section 6.1 says all generations used temperature zero.
Questions for authors
- For the GPT-4o-mini/Qwen pair, where both models continue both models' prefixes, is there a prefix-source × continuer interaction? Can you add matched arms with Sonnet or GPT-5-mini as the replacement?
- What is the failure rate of a fresh standalone regeneration on the matched anchors?
- How do the structural failures break down by type, and how many survive fence stripping and overlap removal?
- What margin would make the switching cost negligible, and does an equivalence test pass against it?
- Why does the Sonnet judge reject 42% of correct Sonnet-continued outputs in one direction?