- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper examines whether snapshot tests overestimate the correctness of LLM-generated stateful data pipelines. The authors evaluate 11 models on 40 tasks, testing the same stored pipelines first on a snapshot and then under duplicate delivery, late and out-of-order arrivals, crash retries, and backfills. Outputs are checked against batch recomputation over the complete event log. Without hazard hints, snapshot pass rates among runnable pipelines range from 86.0% to 100.0%, while certified-wrong rates under replay range from 7.0% to 79.2%. The latter measures silent replay failures among artifacts that passed the snapshot test, averaged across tasks. Further experiments examine hazard hints, the algebraic properties of updates, and the placement of uniqueness constraints. Hazard hints lower silent-failure rates in most models, but some also show higher crash rates. Moving deduplication to the event-arrival boundary repairs a substantial fraction of idempotency failures. The main contributions are an evaluation protocol that exposes failures missed by snapshot tests and experiments that explain several of their causes.
Critical strengths
-
Re-evaluating the same code makes the central comparison convincing. Both evaluations use the same stored code, so variation between separate model generations cannot explain the gap. Replay failures remain after excluding the five tasks flagged as ambiguous and when each batch uses a fresh database session. The results show across multiple models how high snapshot scores can conceal failures under other execution histories.
-
Independent correctness checks and targeted interventions support the findings. Correctness labels come from execution against a batch oracle over the complete event log. A second implementation that shares no grading code produces identical verdicts on both database engines. The code-rewrite experiments also connect the observed failures to a concrete implementation choice: enforcing uniqueness at the event-arrival boundary improves idempotency outcomes while leaving ordering outcomes unchanged. The entity-keyed and pass-through controls support the link between uniqueness scope and the observed idempotency failures.
Critical weaknesses
-
The task suite has limited scope. All tasks were constructed within the authors’ framework and assume a single arrival interface and stable record identities. How performance on this suite translates to real data-engineering work remains unclear. The public dbt example shows the mechanism outside the benchmark, but the project’s own tests catch its failure, leaving the prevalence of silent failures in practice unresolved. Under +hint, three models also reach zero certified-wrong, so this metric no longer distinguishes them in that condition.
-
Script changes and baseline timing complicate the instruction comparisons. Some instruction comparisons involve revised starter scripts or baselines from earlier runs. The authors have checked the 30 unchanged tasks and rerun five baselines. What remains unclear is which effects persist when script version and baseline timing are considered together. This matters most for interpreting the reported shift from silent failures to crashes.
Questions for authors
-
How much replay failure remains on the common task subset? Could you report per-model results on the intersection of the 30 tasks with unchanged starter scripts and the tasks whose wording was not flagged as ambiguous? Please give the number of retained tasks and the certified-wrong rates under both the unhinted and +hint conditions. Alongside these, please report silent-failure, crash, and any-failure rates using a common denominator. I am particularly interested in how much unhinted failure remains for the three models that reach zero certified-wrong under +hint.
-
Which instruction effects persist after accounting for both script version and baseline timing? For the experiments described in Table 5 as “randomised between arms,” what was the randomisation unit and procedure, and how were the baseline and instruction runs scheduled? For models with later baseline runs, could you report instruction effects on tasks with unchanged starter scripts, using those baselines? Please give changes in silent-failure, crash, and any-failure rates, with task-clustered confidence intervals. Which models still show fewer silent failures alongside more crashes, and is overall failure reduced?
3 comments
Response to Reviewer ubz2 (2/2): Q2 and weakness 2
Q2. "Randomised" was the wrong word. Nothing was randomised.
- Unit. An arm is one model under one prompt. Every arm covers all 40 tasks at 5 samples each, so every task receives every condition and nothing is assigned by chance.
- Procedure. Each arm is one job that visits the tasks in a fixed order and draws samples 0–4 per task, sample 0 at temperature 0 where the provider allows it and the others at 1.0 (Opus 5, Sonnet 5 and GPT-5.6 Sol ran at the provider default throughout).
- Scheduling. The unhinted baselines, the four early
+hintarms (Haiku 4.5, Sonnet 5, Opus 5, GPT-5.6 Sol) and the+archarms predate our rewrite of ten starter scripts on 30 August. The+mecharms date from 7 September, the seven later+hintarms from 21–22 September, and five models' later baselines from 13–14 September. For Haiku 4.5 and Opus 5 (+hint) and Haiku 4.5 and Sonnet 5 (+mech), we drew a second set of five samples (5–9) for arm and baseline in one session.
The revision describes a controlled comparison between arms generated at different times, and says that the starred "same-time" baselines of the submitted Table 4 came six days after the +mech arms.
Weakness 2: which effects persist with script version and baseline timing taken together. We agree that this was unclear. Taken together, the shift from silent failure to crashes persists only for Sonnet 5. The two tables below give each comparison.
Does +hint trade silent failure for crashes, that is, is the silent interval below zero and the crash interval above it? A dash means no baseline of that kind exists.
| Model | Stored, 40 tasks | Stored, 30 unchanged | Later baseline, 30 unchanged | Same session, 30 unchanged | Fresh session per batch |
|---|---|---|---|---|---|
| GPT-5.6 Sol | no | no | – | – | no |
| Opus 5 | no | no | – | no | no |
| Kimi K3 | no | no | – | – | no |
| DeepSeek V4 Pro | yes | yes | – | – | no |
| Sonnet 5 | yes | yes | yes | – | yes |
| MiniMax M3 | yes | no | no | – | no |
| Haiku 4.5 | no | no | yes | no | no |
| DeepSeek V4 Flash | no | no | yes | – | no |
| Gemma 4 26B | yes | yes | – | – | no |
| Qwen3-Coder-Next | yes | yes | no | – | no |
| Codestral | no | no | – | – | no |
The comparisons against later and same-session baselines, on the 30 unchanged tasks. Instruction minus baseline, per cell, in points, with 95% task-clustered intervals.
| Model, instruction | Baseline (generated) | Arm generated | Δ silent | Δ crash | Δ any failure |
|---|---|---|---|---|---|
Haiku 4.5 +hint | later, 13 Sep | before 30 Aug | -20.0 [-30.0, -10.7] | +7.3 [+0.7, +14.7] | -11.3 [-21.3, -2.0] |
Haiku 4.5 +hint | same session, 13–14 Sep | same session | -16.0 [-25.3, -7.3] | +5.3 [-4.0, +14.7] | -10.0 [-21.3, +1.3] |
Sonnet 5 +hint | later, 13 Sep | before 30 Aug | -38.7 [-54.7, -23.3] | +27.3 [+13.3, +40.7] | -11.3 [-26.7, +3.3] |
Sonnet 5 +mech | later, 13 Sep | 7 Sep | -31.3 [-47.3, -16.0] | +46.0 [+31.3, +60.7] | +14.7 [+0.7, +29.3] |
Sonnet 5 +mech | same session, 13 Sep | same session | -28.7 [-44.0, -13.3] | +48.7 [+35.3, +62.0] | +20.0 [+6.7, +34.0] |
Opus 5 +hint | same session, 14 Sep | same session | -21.3 [-34.7, -10.0] | -4.0 [-8.0, -0.7] | -25.3 [-39.3, -12.7] |
Qwen3-Coder-Next +hint | later, 13 Sep | 21–22 Sep | -20.7 [-31.3, -10.0] | +10.7 [-1.3, +22.0] | -9.3 [-18.0, -0.7] |
DeepSeek V4 Flash +hint | later, 13 Sep | 21–22 Sep | -18.0 [-28.7, -7.3] | +16.0 [+8.0, +24.7] | +0.0 [-11.3, +11.3] |
MiniMax M3 +hint | later, 14 Sep | 21–22 Sep | -34.7 [-49.3, -20.0] | +6.0 [+0.0, +12.0] | -28.7 [-43.3, -13.3] |
- Sonnet 5 trades under every comparison: stored and later baselines, a fresh session per batch and, for
+mech, the same session. Its overall failure does not fall under+hintand rises under+mech. - DeepSeek V4 Flash trades against its later baseline, generated a week before its
+hintarm. With a fresh session per batch its crash change is −0.5 [−8.0, +6.5]. - Haiku 4.5 trades only against its later baseline, generated five weeks after its
+hintarm. In the same session its crash change is +5.3 [−4.0, +14.7], so we claim no trade for Haiku. - Qwen3-Coder-Next, MiniMax M3, DeepSeek V4 Pro and Gemma 4 26B trade only against stored baselines, and none trades with a fresh session per batch.
- Is overall failure reduced? Against these baselines, any failure falls for Opus 5 (−25.3, same session), MiniMax M3, Qwen3-Coder-Next and Haiku 4.5 (later baselines), and not for Sonnet 5 or DeepSeek V4 Flash.
On the 26 tasks of Q1 the later and same-session results are unchanged, and Sonnet 5 still trades everywhere. The revision states the trade as a Sonnet 5 result (Table 4, Appendix S18).
We would be glad to hear whether the 26-task rates in part 1 and these comparisons answer your two questions and weakness 2.
Thank you for the additional experiments and analysis. They addressed my concern, so I’ve decided to raise my score.
Response to Reviewer ubz2 (1/2): Q1 and weakness 1
Thank you for the precise questions. Both are answered from the stored generations, with no new model calls.
Q1. Replay failure on the common subset. 26 tasks remain: the 30 with unchanged starter scripts, less the four among them that our independent task reviewer flagged as ambiguously worded. Every arm is graded on all 26.
Certified-wrong is the paper's per-task mean over certified cells. The other columns are per cell over all cells of the arm, so they share a denominator; any failure means not passing the snapshot and every hazard. Tasks certified counts the tasks with at least one certified cell, unhinted /
+hint. The last three columns read unhinted →+hint. Intervals are 95%, clustered by task.+hintOf the three arms that reach zero under
+hint, GPT-5.6 Sol has almost no unhinted failure here (2.3% certified-wrong and 2.3% any failure, against 7.0% on all 40), so its zero says little and its fall is no longer significant (Holm-adjusted p = 0.37). Opus 5 keeps 18.8% certified-wrong and 21.5% any failure, and Kimi K3 36.5% and 37.7%, so their falls to zero still carry information.Of the 13 multiplicity-adjusted conclusions that hold on all 40 tasks, 12 hold on these 26 and all 13 on the 30 unchanged tasks. Table 4 and Table S11 print these rates.
Weakness 1. Scope and prevalence. We agree. The revision adds evidence on each limit you name.
event_id(Table S14) and ran the same 9 through a landing table that keeps every delivery (Table S15). Unhinted arms leave 88.9–100.0% and 8.9–100.0% of passing pipelines wrong. Under+hint, once the contract says how redelivered batches are numbered, the three strongest arms leave 0.0%, 0.0% and 4.4% (Kimi K3) without a record identity and 0.0% each on the landing table. Keyed on_batch_id, our repair cuts all-in failure on the tasks without a record identity from 75.6% to 19.9%, fixing 199 of 357 programs and breaking none.One correction to a strength you list. The second oracle shares no ground-truth or comparison code with the harness, but it reuses the harness's schedule generator and copies its crash rule, so agreement cannot reveal an error in how a schedule is built. Section 4 and Appendix S8 now say so.