acceptodds
Under review as a conference paper at ICLR 2027

Snapshot-correct, replay-wrong: Grading generated data pipelines over time

Abstract

Agent-written data pipelines can pass the test they ship with and still corrupt the tables they maintain: executed a second time on the same records, or on records delivered late, twice or out of order, they go silently wrong, and a benchmark that runs a generated artifact once cannot see it. Grading eleven model arms on 40 tasks we built against a fixed snapshot, as seven of nine agent benchmarks we audit do, passes 86.0–100.0% of the pipelines that run; re-executing those same artifacts under duplicate, delayed, reordered and retried delivery finds 7.0–79.2% of the passing ones wrong (1.7–78.1% without five tasks an independent reviewer found ambiguously worded). dbt's default grain tests, placed on each task's key, catch 4 of 110 such failures: a uniqueness test on a business key is satisfied by exactly the upsert that double-counts. In exploratory analyses, the algebra of the update predicts which hazard class a pipeline fails, though not how often: additive folds double-count a redelivered record where naturally idempotent ones absorb it, a gap of +42.2 [+26.6, +58.4] points that survives a port to Python. The defect also has a locus we can move: rewriting uniqueness to the event-arrival boundary raises idempotency pass rates 74.7 and 67.0 points on DuckDB and PostgreSQL where an entity-keyed rewrite and a syntax-preserving one do not. Telling a model about the property narrows the gap without closing the spread between models: the share of passing pipelines that are wrong falls in all 11 arms (significantly in 10 after adjusting for multiplicity) and reaches zero on 3, yet all-in (silent-or-crash) failure still spans 0.0–71.0%, and on Sonnet 5 much of the fall is silent wrongness traded for crashes on a constraint the model itself declares. Evaluating stateful generated code therefore means perturbing its execution history rather than grading one snapshot of it.

Primary area:
datasets and benchmarks
Keywords:
benchmark, LLM agents, code generation, data pipelines, data engineering, evaluation methodology, replay invariance, idempotence, metamorphic testing, stateful code, silent failures, program repair

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_ubz2OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper examines whether snapshot tests overestimate the correctness of LLM-generated stateful data pipelines. The authors evaluate 11 models on 40 tasks, testing the same stored pipelines first on a snapshot and then under duplicate delivery, late and out-of-order arrivals, crash retries, and backfills. Outputs are checked against batch recomputation over the complete event log. Without hazard hints, snapshot pass rates among runnable pipelines range from 86.0% to 100.0%, while certified-wrong rates under replay range from 7.0% to 79.2%. The latter measures silent replay failures among artifacts that passed the snapshot test, averaged across tasks. Further experiments examine hazard hints, the algebraic properties of updates, and the placement of uniqueness constraints. Hazard hints lower silent-failure rates in most models, but some also show higher crash rates. Moving deduplication to the event-arrival boundary repairs a substantial fraction of idempotency failures. The main contributions are an evaluation protocol that exposes failures missed by snapshot tests and experiments that explain several of their causes.

Critical strengths
  1. Re-evaluating the same code makes the central comparison convincing. Both evaluations use the same stored code, so variation between separate model generations cannot explain the gap. Replay failures remain after excluding the five tasks flagged as ambiguous and when each batch uses a fresh database session. The results show across multiple models how high snapshot scores can conceal failures under other execution histories.

  2. Independent correctness checks and targeted interventions support the findings. Correctness labels come from execution against a batch oracle over the complete event log. A second implementation that shares no grading code produces identical verdicts on both database engines. The code-rewrite experiments also connect the observed failures to a concrete implementation choice: enforcing uniqueness at the event-arrival boundary improves idempotency outcomes while leaving ordering outcomes unchanged. The entity-keyed and pass-through controls support the link between uniqueness scope and the observed idempotency failures.

Critical weaknesses
  1. The task suite has limited scope. All tasks were constructed within the authors’ framework and assume a single arrival interface and stable record identities. How performance on this suite translates to real data-engineering work remains unclear. The public dbt example shows the mechanism outside the benchmark, but the project’s own tests catch its failure, leaving the prevalence of silent failures in practice unresolved. Under +hint, three models also reach zero certified-wrong, so this metric no longer distinguishes them in that condition.

  2. Script changes and baseline timing complicate the instruction comparisons. Some instruction comparisons involve revised starter scripts or baselines from earlier runs. The authors have checked the 30 unchanged tasks and rerun five baselines. What remains unclear is which effects persist when script version and baseline timing are considered together. This matters most for interpreting the reported shift from silent failures to crashes.

Questions for authors
  1. How much replay failure remains on the common task subset? Could you report per-model results on the intersection of the 30 tasks with unchanged starter scripts and the tasks whose wording was not flagged as ambiguous? Please give the number of retained tasks and the certified-wrong rates under both the unhinted and +hint conditions. Alongside these, please report silent-failure, crash, and any-failure rates using a common denominator. I am particularly interested in how much unhinted failure remains for the three models that reach zero certified-wrong under +hint.

  2. Which instruction effects persist after accounting for both script version and baseline timing? For the experiments described in Table 5 as “randomised between arms,” what was the randomisation unit and procedure, and how were the baseline and instruction runs scheduled? For models with later baseline runs, could you report instruction effects on tasks with unchanged starter scripts, using those baselines? Please give changes in silent-failure, crash, and any-failure rates, with task-clustered confidence intervals. Which models still show fewer silent failures alongside more crashes, and is overall failure reduced?

3 comments
AuthorsOpenReview ↗

Response to Reviewer ubz2 (1/2): Q1 and weakness 1

Thank you for the precise questions. Both are answered from the stored generations, with no new model calls.

Q1. Replay failure on the common subset. 26 tasks remain: the 30 with unchanged starter scripts, less the four among them that our independent task reviewer flagged as ambiguously worded. Every arm is graded on all 26.

Certified-wrong is the paper's per-task mean over certified cells. The other columns are per cell over all cells of the arm, so they share a denominator; any failure means not passing the snapshot and every hazard. Tasks certified counts the tasks with at least one certified cell, unhinted / +hint. The last three columns read unhinted → +hint. Intervals are 95%, clustered by task.

ModelTasks certifiedCertified-wrong, unhintedCertified-wrong, +hintSilentCrashAny failure
GPT-5.6 Sol26 / 262.3 [0.0, 6.2]0.0 [0.0, 0.0]2.3 → 0.00.0 → 0.02.3 → 0.0
Opus 526 / 2618.8 [7.2, 32.3]0.0 [0.0, 0.0]17.7 → 0.03.8 → 0.821.5 → 0.8
Kimi K326 / 2636.5 [20.8, 53.5]0.0 [0.0, 0.0]33.1 → 0.03.8 → 0.037.7 → 0.0
DeepSeek V4 Pro26 / 2640.6 [24.8, 57.1]16.3 [8.3, 24.9]35.4 → 12.313.1 → 20.848.5 → 34.6
Sonnet 525 / 2257.6 [37.6, 76.0]14.9 [3.8, 28.9]50.8 → 10.010.0 → 46.960.8 → 56.9
MiniMax M326 / 2650.8 [36.0, 65.4]10.6 [4.9, 17.2]40.8 → 8.515.4 → 19.256.9 → 27.7
Haiku 4.524 / 2549.2 [30.0, 67.5]29.9 [16.0, 45.1]31.5 → 18.533.1 → 31.565.4 → 51.5
DeepSeek V4 Flash26 / 2649.3 [33.0, 66.0]23.9 [10.9, 39.2]36.2 → 13.823.8 → 32.361.5 → 48.5
Gemma 4 26B24 / 2261.0 [42.3, 79.2]24.2 [12.1, 37.1]43.8 → 11.530.8 → 46.974.6 → 59.2
Qwen3-Coder-Next26 / 2676.4 [62.1, 89.1]59.0 [42.6, 74.0]52.3 → 27.726.9 → 43.880.0 → 73.1
Codestral26 / 2484.6 [71.5, 95.2]78.8 [62.5, 92.5]53.8 → 56.218.5 → 17.787.7 → 85.4

Of the three arms that reach zero under +hint, GPT-5.6 Sol has almost no unhinted failure here (2.3% certified-wrong and 2.3% any failure, against 7.0% on all 40), so its zero says little and its fall is no longer significant (Holm-adjusted p = 0.37). Opus 5 keeps 18.8% certified-wrong and 21.5% any failure, and Kimi K3 36.5% and 37.7%, so their falls to zero still carry information.

Of the 13 multiplicity-adjusted conclusions that hold on all 40 tasks, 12 hold on these 26 and all 13 on the 30 unchanged tasks. Table 4 and Table S11 print these rates.

Weakness 1. Scope and prevalence. We agree. The revision adds evidence on each limit you name.

  • Tasks we did not construct. On 13 tasks written by two data engineers outside the project, 28.5–92.4% of the passing pipelines of three smaller models are wrong under replay (Section 5.4).
  • Record identity and arrival interface. We rebuilt 9 of our additive or mixed tasks without event_id (Table S14) and ran the same 9 through a landing table that keeps every delivery (Table S15). Unhinted arms leave 88.9–100.0% and 8.9–100.0% of passing pipelines wrong. Under +hint, once the contract says how redelivered batches are numbered, the three strongest arms leave 0.0%, 0.0% and 4.4% (Kimi K3) without a record identity and 0.0% each on the landing table. Keyed on _batch_id, our repair cuts all-in failure on the tasks without a record identity from 75.6% to 19.9%, fixing 199 of 357 programs and breaking none.
  • Prevalence. Two raters labelled 60 public incremental models. Weighted back to the 1,162 mined, an estimated 15.9% [10.0, 21.8] keep an additive or mixed value on a key other than the record's, and in an estimated 58.1% [21.5, 79.6] of those dbt's own unique test cannot see a double count. In the commit histories of 1,109 models, 4.4% have a commit that fixes duplicates or double counting. That is a lower bound, since a failure nobody noticed leaves no commit. These measure exposure and fixes, not how often redelivery happens in production, which needs incident data we do not have.
  • Separation at the top. With the hazards listed, three arms reach zero, so at the top the suite is a diagnostic rather than a leaderboard, and the revision says so.

One correction to a strength you list. The second oracle shares no ground-truth or comparison code with the harness, but it reuses the harness's schedule generator and copies its crash rule, so agreement cannot reveal an error in how a schedule is built. Section 4 and Appendix S8 now say so.

AuthorsOpenReview ↗

Response to Reviewer ubz2 (2/2): Q2 and weakness 2

Q2. "Randomised" was the wrong word. Nothing was randomised.

  • Unit. An arm is one model under one prompt. Every arm covers all 40 tasks at 5 samples each, so every task receives every condition and nothing is assigned by chance.
  • Procedure. Each arm is one job that visits the tasks in a fixed order and draws samples 0–4 per task, sample 0 at temperature 0 where the provider allows it and the others at 1.0 (Opus 5, Sonnet 5 and GPT-5.6 Sol ran at the provider default throughout).
  • Scheduling. The unhinted baselines, the four early +hint arms (Haiku 4.5, Sonnet 5, Opus 5, GPT-5.6 Sol) and the +arch arms predate our rewrite of ten starter scripts on 30 August. The +mech arms date from 7 September, the seven later +hint arms from 21–22 September, and five models' later baselines from 13–14 September. For Haiku 4.5 and Opus 5 (+hint) and Haiku 4.5 and Sonnet 5 (+mech), we drew a second set of five samples (5–9) for arm and baseline in one session.

The revision describes a controlled comparison between arms generated at different times, and says that the starred "same-time" baselines of the submitted Table 4 came six days after the +mech arms.

Weakness 2: which effects persist with script version and baseline timing taken together. We agree that this was unclear. Taken together, the shift from silent failure to crashes persists only for Sonnet 5. The two tables below give each comparison.

Does +hint trade silent failure for crashes, that is, is the silent interval below zero and the crash interval above it? A dash means no baseline of that kind exists.

ModelStored, 40 tasksStored, 30 unchangedLater baseline, 30 unchangedSame session, 30 unchangedFresh session per batch
GPT-5.6 Solnono––no
Opus 5nono–nono
Kimi K3nono––no
DeepSeek V4 Proyesyes––no
Sonnet 5yesyesyes–yes
MiniMax M3yesnono–no
Haiku 4.5nonoyesnono
DeepSeek V4 Flashnonoyes–no
Gemma 4 26Byesyes––no
Qwen3-Coder-Nextyesyesno–no
Codestralnono––no

The comparisons against later and same-session baselines, on the 30 unchanged tasks. Instruction minus baseline, per cell, in points, with 95% task-clustered intervals.

Model, instructionBaseline (generated)Arm generatedΔ silentΔ crashΔ any failure
Haiku 4.5 +hintlater, 13 Sepbefore 30 Aug-20.0 [-30.0, -10.7]+7.3 [+0.7, +14.7]-11.3 [-21.3, -2.0]
Haiku 4.5 +hintsame session, 13–14 Sepsame session-16.0 [-25.3, -7.3]+5.3 [-4.0, +14.7]-10.0 [-21.3, +1.3]
Sonnet 5 +hintlater, 13 Sepbefore 30 Aug-38.7 [-54.7, -23.3]+27.3 [+13.3, +40.7]-11.3 [-26.7, +3.3]
Sonnet 5 +mechlater, 13 Sep7 Sep-31.3 [-47.3, -16.0]+46.0 [+31.3, +60.7]+14.7 [+0.7, +29.3]
Sonnet 5 +mechsame session, 13 Sepsame session-28.7 [-44.0, -13.3]+48.7 [+35.3, +62.0]+20.0 [+6.7, +34.0]
Opus 5 +hintsame session, 14 Sepsame session-21.3 [-34.7, -10.0]-4.0 [-8.0, -0.7]-25.3 [-39.3, -12.7]
Qwen3-Coder-Next +hintlater, 13 Sep21–22 Sep-20.7 [-31.3, -10.0]+10.7 [-1.3, +22.0]-9.3 [-18.0, -0.7]
DeepSeek V4 Flash +hintlater, 13 Sep21–22 Sep-18.0 [-28.7, -7.3]+16.0 [+8.0, +24.7]+0.0 [-11.3, +11.3]
MiniMax M3 +hintlater, 14 Sep21–22 Sep-34.7 [-49.3, -20.0]+6.0 [+0.0, +12.0]-28.7 [-43.3, -13.3]
  • Sonnet 5 trades under every comparison: stored and later baselines, a fresh session per batch and, for +mech, the same session. Its overall failure does not fall under +hint and rises under +mech.
  • DeepSeek V4 Flash trades against its later baseline, generated a week before its +hint arm. With a fresh session per batch its crash change is −0.5 [−8.0, +6.5].
  • Haiku 4.5 trades only against its later baseline, generated five weeks after its +hint arm. In the same session its crash change is +5.3 [−4.0, +14.7], so we claim no trade for Haiku.
  • Qwen3-Coder-Next, MiniMax M3, DeepSeek V4 Pro and Gemma 4 26B trade only against stored baselines, and none trades with a fresh session per batch.
  • Is overall failure reduced? Against these baselines, any failure falls for Opus 5 (−25.3, same session), MiniMax M3, Qwen3-Coder-Next and Haiku 4.5 (later baselines), and not for Sonnet 5 or DeepSeek V4 Flash.

On the 26 tasks of Q1 the later and same-session results are unchanged, and Sonnet 5 still trades everywhere. The revision states the trade as a Sonnet 5 result (Table 4, Appendix S18).

We would be glad to hear whether the 26-task rates in part 1 and these comparisons answer your two questions and weakness 2.

Thank you for the additional experiments and analysis. They addressed my concern, so I’ve decided to raise my score.

Reviewer_g255OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper argues that agents for data pipelines run the generated artifact once, but can fail under realistic settings. For example, corruption under duplicate, late, out-of-order, retried, or backfilled delivery. The author builds 40 tasks with a recomputation oracle and grades 11 models. Results demonstrate this degradation.

Critical strengths
  1. The problem is relevant, and reflect relastice requirment.
  2. The evaluation is done across a decent number of models.
  3. The Oracles help ensure data quality. The hazards selected seem to be relevant.
Critical weaknesses
  1. The scope, size, and procedure of the data generation pipeline are limited.
  2. The scope of the work mostly focuses on the perturbation under the set of modifications, together with the data creation, which is not well motivated given that the focus is on the modification effect, not the data design.
  3. The paper has large parts that are hard to parse, making some of the authors' claims less clear.
Questions for authors

Probably the main point for me for addressing the writing issue is establishing common ground and better articulating the current state, the gap, and the suggested solution. Given that, it will be much easier to assess the contribution and consider it sufficient for the conference requirements.

4 comments
AuthorsOpenReview ↗

Response to Reviewer g255

Thank you. We agree that the paper was harder to read than it should be. Our general comment sets out the problem in the terms you suggest, and the revision does the same.

Current state, gap and approach, on page 1. The introduction now opens with three short paragraphs.

  • What exists. Agents write pipeline scripts, and benchmarks run each script once.
  • What that misses. A script that is correct once can corrupt its table when the same records arrive again, late or out of order.
  • What we do. We run the same stored script again under those delivery schedules and compare the result with a recomputation from the complete log.

A worked example follows. Our hand-written control adds each batch's sales to a stored total. It is correct when run once on three sales, and counts the last sale twice when its batch arrives again. The abstract is shorter, and its longest sentence falls from 87 words to 35.

What a task is (weaknesses 1 and 2). The paper did not make this clear early enough. Each task asks the model to write one SQL script that maintains a table. The model receives a description of the table, the source and target schemas, and a test. The event logs are small synthetic inputs whose only role is to define the correct table. They are not a dataset we contribute, and the paper does not study data generation. Its subject is the effect of re-running the same script under different delivery histories, as you say. The tasks exist because no benchmark has stateful tasks that can be replayed against a recomputed answer.

The revision says this at the start of Section 3, adds a table of the tasks and of what each delivery schedule does (Table 2), and prints one task as the model receives it (Appendix S12).

Scope and size. The revision adds tasks built outside our procedure. Two data engineers who had not seen our hypotheses wrote 13 tasks, and 28.5–92.4% of the passing pipelines of three smaller models are still wrong under replay on them (Section 5.4). Nine of our tasks also run without a record identity and through a second arrival interface (Appendix S18).

Hard-to-parse passages (weakness 3). Terms are now defined before use, internal labels and registry codes are gone from the main text, and the densest results paragraphs are shorter and open with their point.

We would be glad to hear whether the new first page answers your concern.

Thank you for your response. I still do not think the paper is clear enough. I think it can provide a decent contribution, but for that, it will require a more significant revision.

Thank you for taking the time to read our response and for coming back to us. We appreciate that you see a contribution in the work, and we agree that it only counts if the paper is easy to follow.

We take the point that the revision has not gone far enough on clarity. We will make a further pass over the manuscript and aim to upload it before the revision deadline on 19 November. If there are particular sections or passages you found hardest to follow, we would be grateful to know which, so that we can start there. Any pointer, however brief, would help us.

For reference, the changes to the writing so far are a new first page that sets out what exists, what it misses and what we do, with a worked example; a shorter abstract; a description of a task at the start of Section 3, with a table of the tasks and delivery schedules (Table 2); and one task printed as the model receives it (Appendix S12). We recognise that more is needed and are grateful for your patience.

AuthorsOpenReview ↗

Further revision for clarity

Thank you again for pushing us on clarity. We have uploaded a further revision focused on the writing:

  • The terms the paper uses (arm, unhinted, +hint, cell, silent failure, crash and certified-wrong) are now defined in one paragraph at the end of the introduction, before Section 3 uses them.
  • The contributions are shorter and state one point each.
  • Section 3 defines replay invariance in the equation's own terms, without notation used nowhere else.
  • The first paragraph of Section 5.1 and the typed-outcomes paragraph of Section 6.1 now open with their main point.
  • Section 6.2 is regrouped into what the repair is, what it does, what it costs and its limits.

No result, number or claim has changed. We hope this makes the paper easier to follow.

Reviewer_qfTCOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

Paper evaluates generated data pipelines beyond a single successful execution. Across 40 tasks and 11 model arms, many pipelines pass snapshot tests but produce wrong tables under duplicate, delayed, reordered or retried delivery. Authors also study hazard-aware prompts and show that deduplication at event-arrival boundary repairs a substantial part of failures.

Critical strengths

Same stored artifact is evaluated under both snapshot and replay, so main gap is directly observable rather than inferred from different generations. Separating silent failures from crashes makes results much more useful. Event-level rewrite, entity-level control and pass-through control on two engines give good evidence for a specific failure mechanism.

Critical weaknesses
  1. All tasks have a single arrival interface and stable record identity, which also makes proposed repair easier to apply. Independent review checks task correctness, but does not establish coverage of real pipeline work. Three strongest hinted arms reach zero, so benchmark currently has limited ability to distinguish capable models once requirements are explicit.

  2. Some prompt comparisons change more than instruction: seven later hinted arms also saw rewritten starter scripts on ten tasks, and several comparisons use stored baselines. Authors report checks on unchanged tasks and later baselines, which helps, but effect sizes and uncertainty for these cleaner comparisons should be easier to find in main paper.

Questions for authors
  1. On 30 unchanged tasks, what are per-model hint effects and intervals for silent, crash and all-in failure? Do all multiplicity-adjusted conclusions still hold?
  2. Against same-time baselines, which silent-to-crash effects remain supported? Please separate these from comparisons using stored generations.
  3. Can authors include a small independently authored task set to test whether failure patterns and repair gains transfer beyond current task construction?
2 comments
AuthorsOpenReview ↗

Response to Reviewer qfTC (1/2): Q1, Q2 and weakness 2

Thank you for the review and the specific questions. Q1, Q2 and weakness 2 are answered from the stored generations. Q3 and weakness 1 led to new runs, which part 2 reports.

Q1. Hint effects on the 30 unchanged tasks. +hint minus unhinted, per cell, in points, with 95% task-clustered intervals (Table S10). The Holm columns say whether the fall in certified-wrong stays significant after Holm adjustment over the paper's 14-test family, recomputed on each subset.

ModelΔ silentΔ crashΔ all-inHolm, 30 tasksHolm, 26 tasks
GPT-5.6 Sol-9.3 [-18.7, -2.0]+0.0 [+0.0, +0.0]-9.3 [-18.7, -2.0]holdsdoes not (p = 0.37)
Opus 5-17.3 [-28.7, -7.3]-2.7 [-6.7, +0.7]-20.0 [-32.0, -9.3]holdsholds
Kimi K3-41.3 [-56.7, -26.7]-3.3 [-8.0, +0.0]-44.7 [-60.7, -28.7]holdsholds
DeepSeek V4 Pro-26.7 [-40.0, -14.7]+7.3 [+0.7, +14.0]-19.3 [-31.3, -8.0]holdsholds
Sonnet 5-45.3 [-60.0, -30.7]+32.7 [+20.7, +44.7]-12.7 [-27.3, +2.0]holdsholds
MiniMax M3-35.3 [-48.7, -22.7]+4.7 [-3.3, +12.7]-30.7 [-45.3, -16.0]holdsholds
Haiku 4.5-13.3 [-22.7, -5.3]-1.3 [-8.7, +6.7]-14.7 [-25.3, -4.7]holdsholds
DeepSeek V4 Flash-21.3 [-32.0, -10.7]+8.0 [-2.7, +19.3]-13.3 [-26.0, +0.0]holdsholds
Gemma 4 26B-32.7 [-46.7, -19.3]+18.7 [+3.3, +33.3]-14.0 [-28.7, +0.7]holdsholds
Qwen3-Coder-Next-24.0 [-35.3, -13.3]+16.7 [+6.0, +27.3]-7.3 [-14.7, +0.0]holdsholds
Codestral+1.3 [-7.3, +9.3]+0.0 [-7.3, +8.0]+1.3 [-6.7, +8.7]does not (p = 0.19)does not (p = 0.37)

The family has 14 tests: the 11 falls in certified-wrong, the rank correlation between the two leaderboards (Goodman–Kruskal γ), and Sonnet 5's silent and crash shares under +hint. The last three hold on both subsets. All 13 tests that hold on the 40 tasks hold on these 30. On the stricter 26 tasks that ubz2 asked for, 12 hold: GPT-5.6 Sol's fall does not, because Sol has almost no unhinted failure left there (2.3%).

Q2. Same-time against stored baselines. The starred baselines of the submitted Table 4 were not same-time: they came six days after the +mech arms. The table below gives every comparison against a later or a same-session baseline, on the 30 unchanged tasks (also in Table 4 and Table S12). Instruction minus baseline, per cell, in points.

Model, instructionBaseline (generated)Arm generatedΔ silentΔ crashΔ any failure
Haiku 4.5 +hintlater, 13 Sepbefore 30 Aug-20.0 [-30.0, -10.7]+7.3 [+0.7, +14.7]-11.3 [-21.3, -2.0]
Haiku 4.5 +hintsame sessionsame session-16.0 [-25.3, -7.3]+5.3 [-4.0, +14.7]-10.0 [-21.3, +1.3]
Sonnet 5 +hintlater, 13 Sepbefore 30 Aug-38.7 [-54.7, -23.3]+27.3 [+13.3, +40.7]-11.3 [-26.7, +3.3]
Sonnet 5 +mechlater, 13 Sep7 Sep-31.3 [-47.3, -16.0]+46.0 [+31.3, +60.7]+14.7 [+0.7, +29.3]
Sonnet 5 +mechsame sessionsame session-28.7 [-44.0, -13.3]+48.7 [+35.3, +62.0]+20.0 [+6.7, +34.0]
Opus 5 +hintsame sessionsame session-21.3 [-34.7, -10.0]-4.0 [-8.0, -0.7]-25.3 [-39.3, -12.7]
Qwen3-Coder-Next +hintlater, 13 Sep21–22 Sep-20.7 [-31.3, -10.0]+10.7 [-1.3, +22.0]-9.3 [-18.0, -0.7]
DeepSeek V4 Flash +hintlater, 13 Sep21–22 Sep-18.0 [-28.7, -7.3]+16.0 [+8.0, +24.7]+0.0 [-11.3, +11.3]
MiniMax M3 +hintlater, 14 Sep21–22 Sep-34.7 [-49.3, -20.0]+6.0 [+0.0, +12.0]-28.7 [-43.3, -13.3]

The trade that survives every comparison we have is Sonnet 5's: stored and later baselines, a fresh session per batch and, for +mech, the same session. In the same session, Opus 5 lowers silent failure and crashes together, and Haiku 4.5's crash change spans zero. DeepSeek V4 Flash and Haiku 4.5 trade against a later baseline but not with a fresh session per batch. The other arms that trade do so only against stored baselines. We no longer describe the trade as a general effect of the instruction.

Weakness 2. We agree. Table 4 now gives the clean-subset rates and which arms trade under each baseline, and Tables S10 and S12 give every effect with its interval.

AuthorsOpenReview ↗

Response to Reviewer qfTC (2/2): Q3 and weakness 1

Q3. Independently authored tasks. We ran this test (Section 5.4 and Table 5; Table S13 gives every interval). Two practising data engineers who had not seen our hypotheses wrote 13 tasks (6 and 7) from a brief that names no hazard and no result, each with its prompt, schema, event log and reference query. We added a per-batch gold and a control, and fixed the procedure before any model call.

The gap reappears. For Haiku 4.5, Qwen3-Coder-Next and Gemma 4 26B, certified-wrong is 64.6%, 92.4% and 28.5% (52.2%, 73.7% and 54.7% on our tasks), although 11 of the 13 prompts ask for deduplication or mention replays and ours never do. The new tasks are also harder to get running: snapshot pass is 86–95% (99% on ours) and 31–55% of pipelines crash, most often on a scratch table created without IF NOT EXISTS, the commonest crash on our tasks too.

The repair transfers weakly. RR fixes 10 pipelines and breaks none, but lowers all-in failure only from 73.3% to 68.2% (51.7% to 25.1% on ours): most failures here are crashes, which it does not address.

The frontier arms give a mixed answer. Under +hint, GPT-5.6 Sol, Opus 5 and Kimi K3 pass all 1,560 verdicts. On the prompts as written, Sol passes all 520, while Opus 5 leaves 6.2% [0.0, 16.9] and Kimi K3 13.5% [5.0, 23.1].

Weakness 1. We agree that one arrival interface and a stable record identity limit what the suite shows. We tested both on the same 9 source tasks, and registered each set of runs before any graded arm ran.

No record identity (Table S14). We dropped event_id; a redelivered batch keeps its _batch_id. Unhinted, all five arms leave 88.9–100.0% of passing pipelines wrong. Under +hint the three arms first left 37.8% (Opus 5), 73.3% (GPT-5.6 Sol) and 17.8% (Kimi K3), and 54 of their 58 failing programs would pass if a redelivered batch kept its arrival numbers: our contract sentence let them read those as unique per row. A registered rerun with one clarifying sentence gives 0.0%, 0.0% and, for Kimi K3, 4.4% [0.0, 11.1]; both of its failing pipelines also fail the in-order stream, which carries no hazard. Keyed on _batch_id, the repair (RbR_b) cuts all-in failure from 75.6% to 19.9%, fixing 199 of 357 programs and breaking none, so it needs an identity only at the level of the batch.

A landing table (Table S15). Each batch is appended to a table that keeps every earlier delivery, as an ELT loader feeds a dbt model. Under +hint the three arms leave 0.0%. Unhinted, every arm leaves some wrong (8.9–100.0%). GPT-5.6 Sol rises from 26.7% on the same 9 tasks in the corpus to 82.2%, and 25 of its 45 programs read the table behind a mark on the arrival numbers, which a redelivered batch passes.

Separation at the top. On prompts written by outside engineers, the suite still separates Kimi K3 from zero. Once the hazards are listed, three arms at zero limit what certified-wrong can separate, so at the top the suite is a diagnostic rather than a leaderboard, and Limitations says so. The independent review shows that the tasks are well formed, not that they cover real pipeline work.

We would be glad to hear whether the comparisons in part 1 and the new runs here answer your questions.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response to all reviewers

We thank the three reviewers. We have uploaded a revised PDF and a revised code release, which holds every new task, generation, verdict and script. This comment says what is new; each reply answers its reviewer's questions. The changes are in the abstract and introduction, the start of Section 3 with Table 2, Table 4, the new Section 5.4, Limitations, and Appendices S14 and S18.

The problem, the gap and our approach (g255). Language-model agents now write data pipelines: scripts that keep a table up to date as records arrive. Benchmarks grade such a script by running it once on a fixed input. In production it runs once per delivery, and records can arrive twice, late or out of order, so a script that is correct once can silently corrupt its table later. We re-run each stored script under schedules that duplicate, delay, reorder or retry its input and compare the final table with a recomputation from the complete log. Page 1 now says this, with a worked example.

New evidence. The first three items are new runs, each registered before any graded arm ran. The last two are analyses of stored data.

  • Tasks written by others (qfTC Q3, ubz2, g255). Two data engineers who had not seen our hypotheses wrote 13 tasks. On them, 28.5–92.4% of the passing pipelines of three smaller models are wrong under replay. With the hazards listed, GPT-5.6 Sol, Opus 5 and Kimi K3 pass every verdict. On the prompts as written, Sol still does, while Opus 5 and Kimi K3 leave 6.2% and 13.5% wrong (only Kimi K3's interval excludes zero). Section 5.4, Table 5.
  • No record identity (qfTC, ubz2). On 9 of our tasks rebuilt without one, all five unhinted arms leave 88.9–100.0% of passing pipelines wrong. Under +hint, once the contract says how redelivered batches are numbered, Opus 5 and GPT-5.6 Sol leave 0.0% and Kimi K3 4.4% [0.0, 11.1]; both of Kimi K3's failing pipelines also fail the in-order stream, which carries no hazard. Keyed on the batch in place of the record, our repair fixes 199 of 357 programs and breaks none. Table S14.
  • A second arrival interface (qfTC, ubz2). Through a landing table that keeps every delivery, unhinted arms leave 8.9–100.0% wrong and the three arms under +hint leave 0.0%. Table S15.
  • Prevalence in public dbt code (ubz2). Two raters' labels on 60 public incremental models give an estimated 15.9% [10.0, 21.8] at risk: they keep an additive or mixed value on a key other than the record's. In the commit histories of 1,109 models, 4.4% have a commit that fixes duplicates or double counting, a lower bound. Appendix S14.
  • Cleaner comparisons (qfTC Q1 and Q2, ubz2 Q1 and Q2), from the stored generations. Most effects hold on the 26 and 30 cleaner tasks. Two statements weaken, and the revision says so: GPT-5.6 Sol's fall under +hint is no longer significant on 26 tasks, and the silent-to-crash trade is robust only for Sonnet 5. Table 4, Appendix S18.

Corrections to the submission.

  • Table 5 of the submission said "randomised between arms". Nothing was randomised: the instruction is assigned by design to whole arms generated at different times.
  • Sample 0 was not at temperature 0 in every arm: Opus 5, Sonnet 5 and GPT-5.6 Sol ran at the provider default throughout.
  • The algebra paragraph printed pooled margins beside a difference of per-task means; it now prints the per-task means.
  • The entity-keyed control breaks the in-order multi-batch stream, so it is a different program rather than a matched placebo.
  • The ethics statement said the release held the human participants' scripts; it held their per-person verdicts under pseudonyms, and the revised release holds neither.
  • The second oracle shares no ground-truth or comparison code with the harness, but it reuses the harness's schedule generator.

Certified-wrong is now defined in full in Section 4; it always excluded crashing cells, so no number changes.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.