- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
- Problem: when an LLM task is split into a chain of separate calls (a "pipeline"), each later stage only sees what the previous stage wrote. The paper asks how much accuracy is lost at these hand-offs ("interfaces").
- Method: run the same pipeline twice (same model, prompts, token budget). "Strict" arm: each stage sees only the previous stage's output and is told it does not have the original problem. "Full" arm: every stage also gets the original problem ("re-grounding"). The accuracy gap is the "decomposition tax"; re-grounding one stage at a time shows which stage pays.
- Scale: 21 open-weight models (1B–24B), GSM-Hard and MATH-500, n = 200 items per cell, 445 cells, paired McNemar tests, bootstrap intervals, BH and Holm corrections.
- Main results: largest tax 40.5 points (gemma-3-12B on MATH-500: 15.0% strict vs 55.5% full); 70 of 118 primary tests survive BH (54 Holm). A placebo that keeps the extra tokens but masks the problem recovers nothing (GSM-Hard, 3 models). Condensing interfaces (a list of quantities) cost ~4x more than preserving ones (prose restatement): 0.1516 vs 0.0387, 34 of 39 pairs. Adding "keep every relationship stated between them" lowers the tax on 9 of 9 models on MATH-500 (gemma-3-12B: 36.5 → 7.0). With one lossy interface, re-grounding the stage after it beats re-grounding the stage before it on 7 of 7 models.
- Predictions were preregistered; several failed and are reported openly. Code, data and registration are released.
Critical strengths
- Unusually careful, transparent measurement: single-factor manipulation (only "which stages see the problem" changes), 21 models, paired tests with two multiplicity corrections, a placebo plus six registered controls, and honest reporting of failed predictions. This matters because the reported effect sizes can be trusted as measurements, which is rare in prompt-engineering papers.
- Actionable results for pipeline builders: the "relationships" clause (9/9 models), the repair-placement rule (7/7 models, holding on the three newest models), and condense-vs-preserve (34/39 pairs). If the setting is accepted, these are concrete and cheap to apply.
Critical weaknesses
- The "strict" arm is an extreme, constructed regime that the paper never shows to be common in practice. In strict, every stage loses the original problem AND all earlier history, and its prompt explicitly says it lacks the problem. Appendix I admits the strict arm "is the lossy end rather than the common case" and that the regime is one "we take to be common in deployed systems and do not demonstrate here". So "LLM pipelines lose up to 40 points" really means "a pipeline deliberately denied the problem loses up to 40 points", largely by construction. This is decisive: the headline becomes a near-tautology, and its magnitude is mostly a property of the authors' own hand-written prompts (one instruction rewrite moves the tax from 4.5 to 36.5 points, Section 3.2; 40.5 is the maximum over 118 tests).
- Limited practical relevance and scope. Even the fully re-grounded pipeline loses to a single call by 0.12 on the headline shape and wins only 17 of 104 cells (Limitation 1), so on these tasks the real lesson is "do not decompose", which the main text never says. Evidence covers only two math benchmarks; the single non-math probe (LastLetter, 2 models) shows a smaller tax on 4 of 4 comparisons. There is no compute-matched control (Limitation 6; placebo only on GSM-Hard, 3 models). The preregistration also protects less than it seems: the original confirmatory family was withdrawn after the campaign, most of its predictions were contradicted, the family partition used for correction was fixed post hoc, and the registration listed "the decomposition tax, measured" as a framing it did not license (Appendix D).
Questions for authors
- Which deployed frameworks or published pipelines actually withhold the original task from later stages and drop all history? Concrete evidence that the strict regime is common would substantially strengthen the framing.
- The strict-arm prompt tells each stage it does not have the original problem. Have you ablated that sentence? Could priming the model to "not know" explain part of the tax?
- Can you add an arm that keeps all earlier stage outputs but not the problem (least-to-most style), to separate history loss from problem loss (Appendix I says TAX currently contains both)? And a compute-matched control (strict arm given the full arm's extra tokens) plus a placebo on MATH-500?
- Applying the registered quality gates excludes the headline cell (Appendix D table). What are the headline tax and survivor counts with the gates applied exactly as registered?
- Were the A_asset/B_asset prompts iterated against results before being frozen? Given the extreme prompt sensitivity, how can a reader know the 40.5-point cell is not a selected worst case? And is there any cell where a re-grounded pipeline beats the single call at matched budget?
3 comments
The strict arm in the revised PDF
You called the strict arm constructed: later stages lose the problem and all earlier history, and their prompt says the problem is missing. Section 3.1, where this arm is called message-only, tests both features (Table 11). For Qwen2.5-7B and gemma-4-12B on both math benchmarks, deleting that sentence leaves the tax in place. With every earlier output forwarded, showing the problem still adds 10.0 to 36.5 points for these two models and for Qwen2.5-32B. Does this answer your concern?
Thank you for the engagement and the revision. It improves the manuscript noticeably but does not address all of my concerns.
The new controls help. The title is very misleading, more so now. Only a handful of cases reach 40 points, the median is 24, and for API models 6.5 to 25. That is still significant without overselling it. The repaired pipeline losing to a single call should be in the main text, currently it is on the last appendix page. It should be clear that in this setting, for the math tasks, it is better not to split them.
Given the improvements to the manuscript and the criticisms mirrored by the other reviewers, I remain in the "weak acceptance" bracket with my recommendation (even before that it was more towards "weak rejection" rather than "strong acceptance". However, I want to stress that I think the authors are on to something with the concept/further exploration and quantification of the decomposition tax. Also, that the manuscript markedly improved, simply not to the degree that it moves to the next bracket.
Reply to Reviewer qL78
We thank the reviewer for the careful reading.
LangChain's legacy SimpleSequentialChain passes each step only the previous step's output. We also found the opposite default: on a handoff in the OpenAI Agents SDK, by default the receiving agent gets the whole prior conversation, which is close to our re-grounded condition. So the message-only setting exists, but we cannot show it is common, and we will call it the lossy end of the range. Note that a tax remains when either feature that makes this end extreme is removed: with every earlier output passed along, showing the problem still adds 0.100 to 0.365, and with the sentence about the missing problem deleted, the tax is 0.100 to 0.470.
We have now. Deleting it from every later stage did not reliably shrink the tax in any of the four configurations: it is 0.470, 0.250, 0.390 and 0.100 without the sentence, against 0.380, 0.265, 0.355 and 0.095 with it (Qwen2.5-7B on MATH-500 and GSM-Hard, then Gemma-4-12B). On Qwen's MATH-500 it grew by 0.090.
All three are done, the compute match as a three-sample vote rather than one longer answer; the table is in our results comment. With every earlier output passed along, showing the problem still adds 0.365 and 0.230 on Qwen2.5-7B and 0.170 and 0.100 on Gemma-4-12B, each with an interval above zero. In these four configurations history alone recovers little of the tax, except on Gemma's MATH-500, where it recovers 54% (an exploratory run on four more models is mixed; see our reply to RszT). For the compute match, the message-only pipeline got three samples and a majority vote and spent more tokens than re-grounding in every configuration (2738 vs 1609 per item on Qwen's MATH-500), and re-grounding stayed ahead by 0.380, 0.260, 0.325 and 0.090. The MATH-500 placebo scores below the message-only pipeline on both models.
With the message-only condition parsing at least 0.99 of its answers, no empty stage outputs, and one test per configuration (a rule of ours that was not registered), 84 tests remain. Of these, 43 survive Benjamini-Hochberg and 37 survive Holm, and the largest tax is 0.380, in two MATH-500 configurations. The 40.5-point configuration is excluded because its re-grounded condition has two empty stage outputs. Two registered rules are not applied: the 0.99 parse rate in the other conditions, and 180 distinct messages per stage, which over the intermediate stages removes one more test. The abstract will lead with the gated four-stage median, 24.0 points.
The prompts were frozen in the registration before any of the 21 models was run and were not changed afterwards. They came out of earlier development work for which we kept no log, so we cannot exclude that results from that work shaped them. The 40.5 points are the largest of the 118 primary tests, i.e. a maximum, and we will call it one.
Counting prompt plus completion tokens, no MATH-500 or GSM-Hard configuration at temperature 0 has the re-grounded pipeline ahead of the single call while spending no more. Counting completion tokens only, the budget the paper fixes, there are 30 such configurations, e.g. Gemma-4-12B on GSM-Hard (0.855 vs 0.775, with 522.6 vs 744.5 completion tokens per item).
We agree the main text should say this and will add it: over the 444 configurations that have a single-call condition, the fully re-grounded pipeline is 0.070 below the single call on average (0.120 on the four-stage pipeline you cite). The paper's results are for pipelines that have to decompose, because of a context budget or a role boundary.
On the registration, the new tests have the same weakness you describe: we wrote their criteria down before running but did not post them. We will describe the registration as a record and keep the withdrawn confirmatory family in the appendix.