acceptodds
Under review as a conference paper at ICLR 2027

SeamBench: Evaluating Append-Only Model Handoffs in Code

Abstract

When an LLM service switches models mid-response, the replacement must finish code it did not write and cannot revise, and the client must decide whether to accept the result. We introduce SeamBench, a benchmark of 2,488 such append-only code handoffs from 428 HumanEval and MBPP tasks, built only from tasks both models solve on their own and labelled by executing the combined program. The acceptance decision turns out to be the hard part, and what a detector is allowed to see is a first-order design variable: in a controlled re-judging of 517 BigCodeBench handoffs, giving the judge the task description raises recall in every condition, while revealing where the switch occurred lowers AUC without the task and raises it with the task. Continuations fail on 10–26% of prefixes; 91% of failures are caught by syntax and entry-point checks, but the remaining semantic failures are caught by no rule and by a content-only judge fewer than half the time, at a false-rejection cost that varies from 2% to 42% across model directions. Switching itself is not the source of the failures: on 261 matched prefixes with fresh same-model restarts, replacing the continuation model never increases failure, and in one direction lowers it from 26% to 10%. All seams, labels and scorers are released under MIT and CC-BY licenses.

Primary area:
datasets and benchmarks
Keywords:
LLM handoff, model switching, code generation, benchmark, LLM generation continuation, LLM reliability, output verification

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_boe9OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

SeamBench studies append-only code handoffs: model A's answer to a HumanEval/MBPP task is cut at a line boundary, model B writes only the remaining text, and the combined program is executed. It contains 2,488 such seams from 428 tasks both models can solve alone, plus a matched same-model restart control and a BigCodeBench validation. The paper claims that switching models does not increase failure, that most failures are structural and caught by simple rules, and that an LLM judge benefits more from seeing the task than from seeing where the cut was.

Critical strengths
  • A clearly defined problem: the prefix is immutable, the combined program is executed, and structural, test, and append-contract failures are recorded separately.
  • Careful design: a matched same-prefix control, a BigCodeBench cohort fixed before outcomes were seen, and negative results kept.
  • Reporting detection at fixed false-rejection budgets, rather than only AUC, fits an accept/reject decision.
  • An unusually complete release, with every seam, label, and judge record plus offline scorers.
Critical weaknesses
  1. "Switching is not the cost" is stronger than the evidence. The data support "no increase was detected," not "never increases": the intervals still allow increases of up to about 13 points, there is no equivalence test, the small MBPP follow-up in Appendix B.6 shows a large increase for GPT-4o-mini to Qwen, and the one large decrease does not survive the paper's own Holm correction. The control also compares two different models on the same prefix, so it mixes model ability with any cost of continuing someone else's code. It covers only two pairs, and none where Sonnet is the replacement, which is where failures concentrate.

  2. Most failures are formatting or structural breakage, but the paper never says which kinds, so the headline rate may partly reflect the protocol. These are genuine append-only failures, but nearly all failures where Sonnet continues are of this kind (a leading Markdown fence alone counts), and no breakdown by type (fences, duplicate definitions, overlaps) is given. There is also no fresh full-solution baseline, so part of the 17–26% restart failure may be ordinary regeneration failure rather than anything prefix-specific.

  3. Headline detector recall is dominated by checks that overlap the failure label. The rules share their parse and entry-point checks with the label, so their 93% recall mostly restates how common structural failures are. The structural-pass subset is the key test of semantic detection, and there the paper says rules catch none of the 37 failures, while Table 10 shows one. The main judge is not given the task description, so its weak recall on semantic failures says little about judges in general, and 37 core failures are too few for reliable per-direction comparisons. The core was also used during detector development, so the BigCodeBench results (Table 6) are the main held-out evidence.

  4. The judge-information result is too small to call a "first-order design variable." In Table 7, showing the task changes detections by two to four examples and showing the cut by at most one, there are no sampling intervals, and only one judge, prompt, and model pair are tested. Appendix B.8 itself says no general advantage for revealing the cut is established.

  5. The setting is narrow. Cuts are only at line boundaries, while real interruptions can fall mid-line, and the paper's own cursor-state pilot finds open strings and unfinished blocks at the cut as a recurring failure pattern. The HumanEval/MBPP core consists of short public Python functions that may have been seen in pretraining, and BigCodeBench, while more complex, covers only one model pair.

  6. A reporting inconsistency. GPT-5-mini requests in the two older, largest cohorts used effective temperature 1 (Appendix B.1), although Section 6.1 says all generations used temperature zero.

Questions for authors
  1. For the GPT-4o-mini/Qwen pair, where both models continue both models' prefixes, is there a prefix-source × continuer interaction? Can you add matched arms with Sonnet or GPT-5-mini as the replacement?
  2. What is the failure rate of a fresh standalone regeneration on the matched anchors?
  3. How do the structural failures break down by type, and how many survive fence stripping and overlap removal?
  4. What margin would make the switching cost negligible, and does an equivalence test pass against it?
  5. Why does the Sonnet judge reject 42% of correct Sonnet-continued outputs in one direction?
Reviewer_4ftyOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

In the context of a general coding agent, it is usually applicable to switch the backend LLMs and continue working freely (adding, deleting, modifying existing content, replacing the same context window or directly working on consolidated files); in this paper, the authors focused on a narrower but real-world scenario where the coding agent is in a streaming service and is only allowed to append upon existing former-LLM-emitted code (where prefix made immutable). Investigating such code handoffs with constraints, the authors introduced a benchmark called SeamBench and made the following claims: (1) switching LLMs is not the cost (does not increase failure rate) ; (2) the cost comes from continuation where a lot of detectable failures with static checks are due to the invalidity of structural rules such as syntax parsing; (3) remaining failures are semantic and hard to detect with LLM-as-judge; (4) informing the judge task description is better than informing where the boundary of the handoff is.

Critical strengths

S1. Counter-intuitive insight: switching is not the cost. This main claim is the best contribution from this paper. It is a counter-intuitive insight; if true, it provides the research community new knowledge and may practically guide engineering efforts in industry. (However, this finding might not be perfectly valid - I explain the details in the weaknesses section, and expect the authors to respond to the related questions and concerns)

S2. Practical problem formulation. This paper is likely rare in tackling this particular problem where the coding agent is in a streaming service and is only allowed to append existing former-LLM-emitted code. Compared to existing works on agent handoff, this work adds a low-latency streaming constraint. If done well, it could effectively fill a research gap.

Critical weaknesses

W1. Confounding factor and selection bias (Major). This evaluation seems to be confounded: the outcome of switching from A to B depends not only on whether the switching itself is the cost (the inherent handoff penalty) but also on whether model B is inherently better or worse than A in completing a coding task. Switching from a stronger model to a weaker one (due to cost reduction purposes), or from a weaker model to a stronger one (an upgrade for stronger performance) are both realistic scenarios to investigate and discuss. In table 2, it would be more interesting and persuasive if it adds such seams.

In this paper, the dual-success filtering does not resolve the issue of model capability gaps, as it brings in a selection bias - problems remained after this filtering are easy ones even for the weaker model, which explains why the handoff failure rate from Sonnet(stronger) to GPT4o-Mini (weaker) is only around 10%. This might cover the possible “disastrous failures” (even repo-level impair and degrade) when switching from a stronger model to a weaker model in a real-world scenario.

W2. Main claim supported by potentially flawed settings (Major). HumanEval and MBPP are both single-file python function completion tasks, historically used to measure an LLM(without an agent harness)'s code completion capabilities. They are much easier than practical coding tasks on real-world multi-file repositories. It is totally valid if this paper only focuses on the scope of single-file completion scenarios; but importantly, this nature of task being too easy might cover the potential failures when switching from a stronger model to a weaker model, the same issue mentioned in W1, regardless of the scenarios. Furthermore, since tasks mainly come from HumanEval and MBPP (plus BigCodeBench, which is good), they are not contamination-free. Contemporary models have potential pretraining exposure to them, and may have strong priors; this could also make both model A and model B, strong or weak, having similar performance on tasks. All of the these factors weaken the main claim that "switching LLMs is not the cost".

W3. Validity limitations (Minor). Dual-success filtering seems to only ensure a task is solvable by two models. But what if for model A, the pass@k is a lot higher than B's pass@k? Does this fully eliminate the factor of task difficulty? Also, SeamBench is simulated/artificial; does this fully represent real-world switching scenarios? I am worried the simulated nature of the benchmark does not fully validate the real streaming failures. In the paper, the authors acknowledge these limitations.

Questions for authors

Q1. Regarding W1&W2. Could the authors address my concerns in W1&W2? If the claim is indeed flawed, would the authors narrow “switching itself is not the cost” to a valid statement?

Q2. Following Q1. Dual-success filtering excluded 97/187 BigCodebench tasks. What is the distribution for A-only, B-only, and neither-success? Out of the A-only tasks, what is the success continuation rate for A→A handoff and A→B handoff, respectively?

My main concern is: what if a stronger model A could build a good foundation, and the weaker model B, although might fail in a standalone generation by itself, could solve it based on the foundation of the stronger model A? Conversely, model B might fail to complete a code snippet that A could successfully continue? The dual-success filtering fully eliminate such cases, both possibilities remain unexamined. Could this be biased in reaching the conclusions or affect the applicability of the conclusions for practical model handoffs?

Q3. Minor clarification, does not change assessment. In table 5, the 95% CI for Qwen→GPT-4o-mini on BigCodeBench, although including zero, allows up to +9.6 pp increase in failure rate. Does this contradict with the statement “switching is not the cost”? This sounds stronger than “not statistically significant”.

Reviewer_9Rg2OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces SeamBench, a benchmark for evaluating append-only code generation handoffs between large language models (LLMs). The benchmark considers scenarios where one model is interrupted during code generation and another model must continue from the existing prefix without modifying previously generated content. The authors investigate two main questions: (1) whether switching to a different model increases the probability of code generation failure compared with restarting the original model, and (2) how reliably different detectors can identify invalid continuations before they are accepted.

SeamBench contains 2,488 code handoffs derived from 428 HumanEval and MBPP tasks across six model-switching directions. The evaluation distinguishes structural validity, task correctness, and append-contract compliance. The authors further conduct matched same-model versus cross-model continuation experiments on 261 prefixes and validate selected findings using 517 additional handoffs from BigCodeBench. They compare rule-based detectors, LLM-based judges, and hybrid approaches, and investigate how providing task descriptions and handoff boundary information affects detection performance.

The results show that 569 of the 2,488 core handoffs fail the combined validation criteria, with approximately 91% of these failures detectable through structural checks. However, detecting the remaining semantic failures proves considerably more challenging. In the primary matched experiments, switching models does not produce a clear increase in failure rates relative to same-model restarts, although the statistical uncertainty and the limited coverage of model pairs constrain this conclusion. The authors also find that providing task information generally improves LLM-based detection, whereas the effect of revealing the handoff boundary depends on the available context. The paper releases benchmark records, evaluation labels, and scoring tools to support further research on model handoffs and continuation validation.

Critical strengths
  1. The paper divides failures into structural failures, task-test failures and append-contract violations, and this division gives very useful information. Of the 569 unsafe core seams, 520 (91%) already fail the public structural checks. Thus, low-cost deterministic checks can stop most of the handoff failures that the paper observes. The remaining semantic failures show the limits of the current LLM judges, mainly when the evaluation also counts false rejections. The paper also compares detectors under different false-rejection budgets. This comparison gives more practical information than a single aggregate AUC.
  2. The matched judge-information control is a useful addition: the judge sees or does not see the task description and the numeric cut position. The results show that detector performance depends on information access, not only on the detector architecture. The paper also gives much detail about the prompts, the validation procedures, the recorded model identities and the experiment settings. The authors state that they release all seams, labels and offline scorers, and the appendix describes how to reproduce the results. Thus, SeamBench can be a useful resource for future research.
Critical weaknesses
  1. The paper states many times that switching models is not the source of continuation failures. However, the matched restart controls do not show statistical equivalence between same-model restarts and cross-model continuations. In Table 4, the largest improvement for cross-model continuation (−16.5 percentage points) has a Holm-adjusted p-value of 0.115, so it is not significant. For the other two directions, the 95% confidence intervals include increases in failure that can be important (up to +13.3 and +4.4 points). In Table 5, both BigCodeBench intervals also include zero, and one direction includes an increase of up to 9.6 points. Thus, the results show no clear increase in failure, but they do not show that switching has no cost.
  2. The dual-success condition isolates continuation failures, but the population of tasks that both models solve on their own can differ from real failover cases. In deployment, timeouts, resource limits or partial failures can cause a handoff, also in generations that the original model would not complete correctly. Also, the core benchmark cuts only at line boundaries near 25%, 50% and 75%, but real interruptions can occur at any token position. A cut inside a line, for example in the middle of an expression or a token or an incomplete structure can make the continuation difficulty and the error distribution very different. The paper mentions some limits but does not systematically study how cut position, prefix length and code complexity change failure rates and detector performance. An evaluation on real interruptions, or on a wider distribution of cut positions, will help to show that the benchmark represents practical conditions.
Questions for authors
  1. The matched restart controls do not show statistical equivalence between same-model restarts and cross-model continuations. Can you report an equivalence test (e.g., TOST) with a margin that you specify before the analysis? Which increase in failure rate do you consider practically important, and how many anchors are necessary to exclude it?
  2. The core benchmark cuts each solution near 25%, 50% and 75% of its length. Can you report failure rates and detector performance separately for each cut position? Can you also report them by prefix length and code complexity?
  3. The current cuts are only at line boundaries. Appendix B.9 shows that some of these cuts already fall inside open multiline strings. Did you evaluate cuts inside a line, for example in the middle of an expression or a token, or naturally occurring interruptions?
  4. The dual-success condition keeps only tasks that both models solve on their own. In deployment, a handoff can also occur in a generation that the original model would not complete correctly. Can you report results on a population without this condition, for example tasks that only one model solves?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.