acceptodds
Under review as a conference paper at ICLR 2027

The Decomposition Tax: LLM Pipelines Lose Up to 40 Accuracy Points at Their Own Interfaces

Abstract

A four-stage LLM pipeline gives up as much as 40.5 accuracy points at its own interfaces (gemma-3-12B on MATH-500, Holm-corrected p = 1.66e-19; the largest tax in the primary family). We hold model, problem, stages, stage prompts and completion budget fixed, vary only whether each stage can still see the original problem, and call the accuracy difference the decomposition tax. Across 21 open-weight models from nine organisations, on GSM-Hard and MATH-500 at n = 200 paired items per cell, 70 of 118 primary-family tests survive Benjamini-Hochberg correction and 54 survive Holm. On GSM-Hard, a placebo recovers nothing: it carries at least 60% of the extra tokens and at most one word of the problem. Builders design a pipeline one stage at a time, and its bill arrives at the interfaces between stages. Rewriting one stage's instruction moves gemma-3-12B's tax from 4.5 to 36.5 points, and adding "every relationship stated between them" to a stage that lists the numerical quantities lowers the tax on 9 of 9 models on MATH-500. Re-grounding, which shows a stage the original problem again, belongs after the loss. With one lossy interface, re-grounding the stage after it beats re-grounding the stage before it on 7 of 7 models on both benchmarks; on MATH-500 the earlier repair is worse than none on 7 of 7. Newer models still pay: gemma-4-12B gives up 37.0 points, and the repair holds on all three of the newest models we test. A sealed held-out test refuted a stronger rule we registered, which predicted the paying stage from the interface and receiver types, so we locate the tax by measuring one stage at a time. The prescription has two parts: re-ground the stage after the lossy interface, and if a stage must list the quantities, tell it to keep the relationships.

Primary area:
foundation or frontier models, including LLMs
Keywords:
large language models, multi-stage pipelines, task decomposition, prompting, compound AI systems, context, evaluation, pre-registration

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_qL78OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary
  • Problem: when an LLM task is split into a chain of separate calls (a "pipeline"), each later stage only sees what the previous stage wrote. The paper asks how much accuracy is lost at these hand-offs ("interfaces").
  • Method: run the same pipeline twice (same model, prompts, token budget). "Strict" arm: each stage sees only the previous stage's output and is told it does not have the original problem. "Full" arm: every stage also gets the original problem ("re-grounding"). The accuracy gap is the "decomposition tax"; re-grounding one stage at a time shows which stage pays.
  • Scale: 21 open-weight models (1B–24B), GSM-Hard and MATH-500, n = 200 items per cell, 445 cells, paired McNemar tests, bootstrap intervals, BH and Holm corrections.
  • Main results: largest tax 40.5 points (gemma-3-12B on MATH-500: 15.0% strict vs 55.5% full); 70 of 118 primary tests survive BH (54 Holm). A placebo that keeps the extra tokens but masks the problem recovers nothing (GSM-Hard, 3 models). Condensing interfaces (a list of quantities) cost ~4x more than preserving ones (prose restatement): 0.1516 vs 0.0387, 34 of 39 pairs. Adding "keep every relationship stated between them" lowers the tax on 9 of 9 models on MATH-500 (gemma-3-12B: 36.5 → 7.0). With one lossy interface, re-grounding the stage after it beats re-grounding the stage before it on 7 of 7 models.
  • Predictions were preregistered; several failed and are reported openly. Code, data and registration are released.
Critical strengths
  • Unusually careful, transparent measurement: single-factor manipulation (only "which stages see the problem" changes), 21 models, paired tests with two multiplicity corrections, a placebo plus six registered controls, and honest reporting of failed predictions. This matters because the reported effect sizes can be trusted as measurements, which is rare in prompt-engineering papers.
  • Actionable results for pipeline builders: the "relationships" clause (9/9 models), the repair-placement rule (7/7 models, holding on the three newest models), and condense-vs-preserve (34/39 pairs). If the setting is accepted, these are concrete and cheap to apply.
Critical weaknesses
  • The "strict" arm is an extreme, constructed regime that the paper never shows to be common in practice. In strict, every stage loses the original problem AND all earlier history, and its prompt explicitly says it lacks the problem. Appendix I admits the strict arm "is the lossy end rather than the common case" and that the regime is one "we take to be common in deployed systems and do not demonstrate here". So "LLM pipelines lose up to 40 points" really means "a pipeline deliberately denied the problem loses up to 40 points", largely by construction. This is decisive: the headline becomes a near-tautology, and its magnitude is mostly a property of the authors' own hand-written prompts (one instruction rewrite moves the tax from 4.5 to 36.5 points, Section 3.2; 40.5 is the maximum over 118 tests).
  • Limited practical relevance and scope. Even the fully re-grounded pipeline loses to a single call by 0.12 on the headline shape and wins only 17 of 104 cells (Limitation 1), so on these tasks the real lesson is "do not decompose", which the main text never says. Evidence covers only two math benchmarks; the single non-math probe (LastLetter, 2 models) shows a smaller tax on 4 of 4 comparisons. There is no compute-matched control (Limitation 6; placebo only on GSM-Hard, 3 models). The preregistration also protects less than it seems: the original confirmatory family was withdrawn after the campaign, most of its predictions were contradicted, the family partition used for correction was fixed post hoc, and the registration listed "the decomposition tax, measured" as a framing it did not license (Appendix D).
Questions for authors
  1. Which deployed frameworks or published pipelines actually withhold the original task from later stages and drop all history? Concrete evidence that the strict regime is common would substantially strengthen the framing.
  2. The strict-arm prompt tells each stage it does not have the original problem. Have you ablated that sentence? Could priming the model to "not know" explain part of the tax?
  3. Can you add an arm that keeps all earlier stage outputs but not the problem (least-to-most style), to separate history loss from problem loss (Appendix I says TAX currently contains both)? And a compute-matched control (strict arm given the full arm's extra tokens) plus a placebo on MATH-500?
  4. Applying the registered quality gates excludes the headline cell (Appendix D table). What are the headline tax and survivor counts with the gates applied exactly as registered?
  5. Were the A_asset/B_asset prompts iterated against results before being frozen? Given the extreme prompt sensitivity, how can a reader know the 40.5-point cell is not a selected worst case? And is there any cell where a re-grounded pipeline beats the single call at matched budget?
3 comments
AuthorsOpenReview ↗

Reply to Reviewer qL78

We thank the reviewer for the careful reading.

Which deployed frameworks or published pipelines actually withhold the original task from later stages and drop all history?

LangChain's legacy SimpleSequentialChain passes each step only the previous step's output. We also found the opposite default: on a handoff in the OpenAI Agents SDK, by default the receiving agent gets the whole prior conversation, which is close to our re-grounded condition. So the message-only setting exists, but we cannot show it is common, and we will call it the lossy end of the range. Note that a tax remains when either feature that makes this end extreme is removed: with every earlier output passed along, showing the problem still adds 0.100 to 0.365, and with the sentence about the missing problem deleted, the tax is 0.100 to 0.470.

The strict-arm prompt tells each stage it does not have the original problem. Have you ablated that sentence?

We have now. Deleting it from every later stage did not reliably shrink the tax in any of the four configurations: it is 0.470, 0.250, 0.390 and 0.100 without the sentence, against 0.380, 0.265, 0.355 and 0.095 with it (Qwen2.5-7B on MATH-500 and GSM-Hard, then Gemma-4-12B). On Qwen's MATH-500 it grew by 0.090.

Can you add an arm that keeps all earlier stage outputs but not the problem (least-to-most style), to separate history loss from problem loss (Appendix I says TAX currently contains both)? And a compute-matched control (strict arm given the full arm's extra tokens) plus a placebo on MATH-500?

All three are done, the compute match as a three-sample vote rather than one longer answer; the table is in our results comment. With every earlier output passed along, showing the problem still adds 0.365 and 0.230 on Qwen2.5-7B and 0.170 and 0.100 on Gemma-4-12B, each with an interval above zero. In these four configurations history alone recovers little of the tax, except on Gemma's MATH-500, where it recovers 54% (an exploratory run on four more models is mixed; see our reply to RszT). For the compute match, the message-only pipeline got three samples and a majority vote and spent more tokens than re-grounding in every configuration (2738 vs 1609 per item on Qwen's MATH-500), and re-grounding stayed ahead by 0.380, 0.260, 0.325 and 0.090. The MATH-500 placebo scores below the message-only pipeline on both models.

What are the headline tax and survivor counts with the gates applied exactly as registered?

With the message-only condition parsing at least 0.99 of its answers, no empty stage outputs, and one test per configuration (a rule of ours that was not registered), 84 tests remain. Of these, 43 survive Benjamini-Hochberg and 37 survive Holm, and the largest tax is 0.380, in two MATH-500 configurations. The 40.5-point configuration is excluded because its re-grounded condition has two empty stage outputs. Two registered rules are not applied: the 0.99 parse rate in the other conditions, and 180 distinct messages per stage, which over the intermediate stages removes one more test. The abstract will lead with the gated four-stage median, 24.0 points.

Were the A_asset/B_asset prompts iterated against results before being frozen? Given the extreme prompt sensitivity, how can a reader know the 40.5-point cell is not a selected worst case? And is there any cell where a re-grounded pipeline beats the single call at matched budget?

The prompts were frozen in the registration before any of the 21 models was run and were not changed afterwards. They came out of earlier development work for which we kept no log, so we cannot exclude that results from that work shaped them. The 40.5 points are the largest of the 118 primary tests, i.e. a maximum, and we will call it one.

Counting prompt plus completion tokens, no MATH-500 or GSM-Hard configuration at temperature 0 has the re-grounded pipeline ahead of the single call while spending no more. Counting completion tokens only, the budget the paper fixes, there are 30 such configurations, e.g. Gemma-4-12B on GSM-Hard (0.855 vs 0.775, with 522.6 vs 744.5 completion tokens per item).

so on these tasks the real lesson is "do not decompose", which the main text never says

We agree the main text should say this and will add it: over the 444 configurations that have a single-call condition, the fully re-grounded pipeline is 0.070 below the single call on average (0.120 on the four-stage pipeline you cite). The paper's results are for pipelines that have to decompose, because of a context budget or a role boundary.

On the registration, the new tests have the same weakness you describe: we wrote their criteria down before running but did not post them. We will describe the registration as a record and keep the withdrawn confirmatory family in the appendix.

AuthorsOpenReview ↗

The strict arm in the revised PDF

You called the strict arm constructed: later stages lose the problem and all earlier history, and their prompt says the problem is missing. Section 3.1, where this arm is called message-only, tests both features (Table 11). For Qwen2.5-7B and gemma-4-12B on both math benchmarks, deleting that sentence leaves the tax in place. With every earlier output forwarded, showing the problem still adds 10.0 to 36.5 points for these two models and for Qwen2.5-32B. Does this answer your concern?

Thank you for the engagement and the revision. It improves the manuscript noticeably but does not address all of my concerns.

The new controls help. The title is very misleading, more so now. Only a handful of cases reach 40 points, the median is 24, and for API models 6.5 to 25. That is still significant without overselling it. The repaired pipeline losing to a single call should be in the main text, currently it is on the last appendix page. It should be clear that in this setting, for the math tasks, it is better not to split them.

Given the improvements to the manuscript and the criticisms mirrored by the other reviewers, I remain in the "weak acceptance" bracket with my recommendation (even before that it was more towards "weak rejection" rather than "strong acceptance". However, I want to stress that I think the authors are on to something with the concept/further exploration and quantification of the decomposition tax. Also, that the manuscript markedly improved, simply not to the degree that it moves to the next bracket.

Reviewer_TcVnOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

The paper designs a pipeline of stages to solve hard maths, looks at the "tax" impact of condensing information at stage boundaries, ways to reverse that tax.

Critical strengths

Pre-registering of expectations is a good practice. The analysis of 21 open models of different sizes is a good size/spread. The use of 2 datasets in the core analysis is ok.

Critical weaknesses
  • Tasks: Both core tasks are maths, so it's note clear whether the findings are more general
  • Models as planners: There is no analysis of whether the first model in the pipeline is a good planner. If the first model creates a poor planned set of steps, or misunderstand the problem, the pipeline is doomed.
  • Pipeline structure: The pipeline structure seems a bit arbitrary. Is a pipeline of stages the best approach for maths problems? Maths seems like a bad domain to test the "condense" approach on, as every maths fact can be vital for a correct answer. A pipeline is always at risk of compounding errors through successive stages.
  • Use of hindsight: The “re-ground the stage after the lossy interface” approach assumes the pipeline/controller already knows which interface is lossy. If the pipeline controller knows this, then perhaps a better solution is to redesign the pipeline or use a different model or not condense. So this instruction is unlikely to be used in practice.
  • Core result not novel: The core result amounts to “model X performs CONDENSE in a way that loses important information, degrading pipeline performance.” This also explains "With one lossy interface, re-grounding the stage after it beats re-grounding the stage before it"
Questions for authors

If feel the use of AI on the research and writing of this paper exceeds that reported in "AI USE STATEMENT". Is this fair?

2 comments
AuthorsOpenReview ↗

Reply to Reviewer TcVn

Thank you for the review. We reply to the weaknesses in order.

Tasks: Both core tasks are maths, so it's note clear whether the findings are more general

This is fair. We have added HotpotQA in the distractor setting, where the evidence is ten paragraphs that a pipeline can condense. On a condensing four-stage pipeline the tax is 0.430 on Qwen2.5-7B and 0.435 on Gemma-4-12B, both with intervals above zero. One finding did not carry over: there the preserving pipeline pays about as much as the condensing one, so we will state that gap for the mathematical pipelines only. Outside mathematics the evidence is thin, two models each on HotpotQA and LastLetter, and we will state that scope.

Models as planners: There is no analysis of whether the first model in the pipeline is a good planner.

No stage in our pipelines plans. The first stage runs a fixed prompt that lists or restates the problem, and the message-only and re-grounded conditions receive the identical first-stage output. So the first stage's quality is held fixed across the two conditions, and the tax measures what the later stages lose when that output is all they have, including whatever it dropped or got wrong. What the first stage keeps is in fact the variable we study: adding "every relationship stated between them" to its instruction in our three-stage design lowers gemma-3-12B's tax from 36.5 points to 7.0, and lowers the tax on all nine models tested on MATH-500 (Section 3.2).

Is a pipeline of stages the best approach for maths problems?

Probably not, and we do not claim that it is. Over 444 configurations a fully re-grounded pipeline is on average 0.070 below a single call, which we will state in the main text. The paper is about pipelines that must decompose. Errors that compound across stages occur in both of our conditions, since they share their stages and prompts; whether they compound faster when the problem is hidden is a separate question that we did not test.

The "re-ground the stage after the lossy interface" approach assumes the pipeline/controller already knows which interface is lossy.

We agree that a designer who knows which interface loses information should first try to stop the loss, and Section 3.2 measures that option (the clause above). The placement result is for the case where the sender cannot be changed, for example a fixed component, or a context budget that forces it to condense. That case needs no hindsight: the lossy interface is the one whose instruction says to condense, and the instruction can be read before anything runs. On the mathematical pipelines, condensing interfaces cost 3.92 times what preserving ones cost (0.1516 against 0.0387 over 118 tests); before we ran the preserving pipeline on the full roster, we had written down that the ratio would be at least three. We did not test a pipeline that picks the repaired stage from the instruction text, and we will say so.

Core result not novel: The core result amounts to "model X performs CONDENSE in a way that loses important information, degrading pipeline performance."

That removing information hurts is not new, and the paper credits Radhakrishnan et al. (2023) for a 3.8-point cost of isolating context. However, an information-loss account does not predict where the repair works. On MATH-500, re-grounding the stage just before the lossy hand-off, which is the intuitive choice, does worse than no repair on 7 of 7 models, whereas re-grounding the stage after it beats re-grounding the stage before it on all seven models and both benchmarks (Section 3.3; these are point estimates). The new controls also argue against a simpler reading, namely that the pipeline only needs more compute: a three-sample vote that spends more tokens than re-grounding stays 0.090 to 0.380 behind.

Thank you for your detailed response and engagement with all the reviewers. I'm sure your paper will have improved through this process.

Reviewer_RszTOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces the decomposition tax—the accuracy degradationcaused by restricting context between sequential calls in multi-stage LLM pipelines. Through extensive evaluations of 21 open-weight models on GSM-Hard and MATH-500, the study reveals that the tax is driven by missing problem context and interface design rather than mere token length or decoding artifacts. Key findings show that preserving relational clauses alleviates losses, and downstream re-grounding after a lossy interface is essential for effective repair.

Critical strengths

S1. The paper transforms a broad architectural question—how much performance multi-stage LLM systems lose when communicating through messages—into a measurable phenomenon: the decomposition tax. It identifies several key determinants of success: content completeness, relationship continuity, interface form, and repair stage. This gives researchers and engineers empirical guidance for building multi-stage LLM architectures.

Critical weaknesses

W1. The writing requires substantial polish.

W2. The core quantitative findings focus on mathematical reasoning tasks and open-source models up to 24B parameters, leaving generalization across domains and to larger or proprietary architectures remains unexplored.

Questions for authors

Q1. Since the strict arm removes both the original problem and earlier-stage history, what portion of tax is attributable to losing prior history rather than losing the original problem?

4 comments
AuthorsOpenReview ↗

Reply to Reviewer RszT

Thank you for the review.

Q1. Since the strict arm removes both the original problem and earlier-stage history, what portion of tax is attributable to losing prior history rather than losing the original problem?

The submission could not say, because its message-only condition removes both. We have now separated them: every later stage gets the outputs of all earlier stages, once without the problem and once with it.

share of the tax recovered by the earlier outputs alonewhat showing the problem still adds
Qwen2.5-7B, MATH-50012%0.365 [0.290, 0.435]
Qwen2.5-7B, GSM-Hard6%0.230 [0.160, 0.300]
Gemma-4-12B, MATH-50054%0.170 [0.105, 0.240]
Gemma-4-12B, GSM-Hard0%0.100 [0.060, 0.145]

So on Qwen2.5-7B, and on Gemma's GSM-Hard, nearly all of the loss is the missing problem, while on Gemma-4-12B's MATH-500 a little over half of it is history. Because of that case, and because an exploratory run on four more models is mixed (on Qwen3-8B the earlier outputs alone recover 79% of the MATH-500 tax), we will restrict the paper's effect sizes to pipelines that forward one message, and the appendix will list every configuration.

W2. The core quantitative findings focus on mathematical reasoning tasks and open-source models up to 24B parameters

On domains, we have added HotpotQA in the distractor setting. With a condensing pipeline the tax there is 0.430 on Qwen2.5-7B and 0.435 on Gemma-4-12B, both with intervals above zero. Unlike in mathematics, a preserving pipeline does not pay less on HotpotQA, and we will say so in the paper.

On size, Qwen2.5-32B-Instruct (4-bit AWQ) still pays a tax: 0.190 on MATH-500 and 0.115 on GSM-Hard, with every condition passing our checks. With the earlier outputs passed, showing the problem still adds 0.165 and 0.110, so most of that tax is again the missing problem. The tax is smaller than Qwen2.5-7B's 0.380 and 0.265 in our new runs, but the 32B weights are quantised and the 7B ones are not, so we do not read this as an effect of size. Every model we ran is open-weight; we ran no proprietary model, and we will list that as a limitation.

W1. The writing requires substantial polish.

We agree. We will rewrite the abstract and introduction to state the setting, the definition and the findings without test counts and p-values. The conditions will be named in plain words in the main text and the figures, and the registration history will sit in one appendix.

Thanks for the additional experiments. I'll keep the score.

AuthorsOpenReview ↗

Closed and newer models, and HotpotQA

Thank you for reading our replies. Since then we have worked on the weaknesses in your review, and we hope you will look at the outcome before the discussion closes.

On W2, proprietary and larger models and tasks beyond mathematics: we ran the same pipelines on four Claude models (Haiku 5.5, Sonnet 5.5, Opus 5.5, Fable 5.2) and on five further models through a third-party API gateway (deepseek-v4-pro, qwen3.8-max, kimi-k3, gpt-6.1-terra, gemini-4-argon). Every one of them loses accuracy on MATH-500 when later stages see only the previous message, from 6.5 points on gpt-6.1-terra to 25.0 on Haiku 5.5 and deepseek-v4-pro. Outside mathematics, the four Claude models lose 4.5 to 11.0 points on HotpotQA, and showing the problem again brings each of them back to the accuracy of a single call. These runs are exploratory, 200 items each. They replace the sentence in our first reply that said we ran no proprietary model.

On W1, we are improving the writing throughout the paper.

The revised PDF will be uploaded within the next two days; the new models are in Section 3.4 and Appendix J.

Thank you for your efforts in running the additional experiments and updating the manuscript. Although the new results help address some technical points, the presentation of the revised paper still falls short of the ICLR standard, particularly given the pervasive AI-generated phrasing noted by multiple reviewers. Consequently, I will maintain my original score.

Comments on the paper

3 comments
AuthorsOpenReview ↗

New experiments and changes

We thank the reviewers. Two questions came up in several reviews: how much of the tax comes from the hidden problem and how much from the dropped history (RszT, qL78), and whether the tax exists outside mathematics (RszT, TcVn, qL78) and beyond small open models (RszT). We ran five new tests with Qwen2.5-7B and Gemma-4-12B, four on the paper's four-stage pipeline and one on HotpotQA, and repeated the first on Qwen2.5-32B. We fixed their pass criteria before the first run but did not post them. The numbers are in our comment "Results of the five new tests".

A tax remains when every earlier stage's output is passed along, when the sentence that tells a stage it lacks the problem is deleted, against a placebo with the problem's word count, and against a three-sample vote that spends more tokens than re-grounding. It also appears on HotpotQA. Of the 20 new primary tests (18 on the two models, two on Qwen2.5-32B), 19 persist and survive Holm correction; the last one, on HotpotQA, gets no verdict because one condition misses our parse check by one item.

We narrow two claims. Passing the history recovers 54% of the tax on Gemma-4-12B's MATH-500, 0% to 12% in the other three configurations of the two models, and 32% and 9% on Qwen2.5-32B; in an exploratory run on four more models it recovers 79% on Qwen3-8B's MATH-500. We will therefore restrict the paper's effect sizes to pipelines that forward a single message. And on HotpotQA a pipeline whose first stage preserves the problem pays about as much as one whose first stage condenses it, so the finding that condensing costs more will be stated for the mathematical pipelines only.

We will also move into the main text what Appendix H said: with every stage re-grounded, our pipelines are still below a single call on average on the two mathematical benchmarks. The abstract will lead with the median tax on the four-stage MATH-500 pipeline after our output checks (24.0 points over the 13 models that pass them) and say that the 40.5 points behind the title come from a configuration that fails one of those checks; among the configurations that pass, the largest tax is 38.0 points.

AuthorsOpenReview ↗

Results of the five new tests

Setup: the paper's four-stage pipeline with its frozen prompts, the first 200 items of MATH-500 and of GSM-Hard, greedy decoding, 4096 tokens per call. The models are Qwen2.5-7B-Instruct and Gemma-4-12B-it, whose taxes are at or above the roster median, so these runs ask whether a large tax survives each control. A contrast persists when the lower end of its paired bootstrap 95% interval is above zero and both conditions pass the paper's checks (parse rate at least 0.99, no empty stage outputs, no failed calls). As a correctness check, Qwen2.5-7B on MATH-500 re-measures at 0.380, the paper's value, and Gemma-4-12B at 0.355 (submitted: 0.370).

Test, and the contrast that must persistQwen MATH-500Qwen GSM-HardGemma MATH-500Gemma GSM-Hard
0. Tax in these runs (re-grounded minus message-only)0.3800.2650.3550.095
1. History passed: with the problem minus without0.365 [0.290, 0.435]0.230 [0.160, 0.300]0.170 [0.105, 0.240]0.100 [0.060, 0.145]
2. Tax after deleting the sentence that tells a stage it lacks the problem0.4700.2500.3900.100
3. Re-grounded minus placebo (MATH-500 only)0.4050.375
4. Re-grounded minus message-only with a three-sample vote0.3800.2600.3250.090

Test 1. A tax remains in all four configurations when every earlier stage's output is passed. We also report how much of the tax the history alone recovers, i.e. the accuracy gained by passing the history without the problem, divided by the tax: 12% and 6% on Qwen, and on Gemma 54% on MATH-500 (0.190 of 0.355) and 0% on GSM-Hard. Our plan said we would narrow the claim if history recovered most of the tax somewhere, so we narrow it. The same contrast on Qwen2.5-32B-Instruct in 4-bit is in our reply to RszT.

Test 2. The sentence does not drive the tax. Deleting it shrank the tax reliably in no configuration, and on Qwen's MATH-500 the tax grew by 0.090 [0.030, 0.150]. On Gemma's GSM-Hard the change, 0.005 [-0.025, 0.035], lies inside the band of plus or minus 0.05 that we set in advance for calling the two taxes equivalent. In the other two configurations it is indistinguishable from zero (-0.015 and 0.035) but not shown equivalent.

Test 3. The placebo has the problem's word count with about 1% of its words shown, and it scores below the message-only pipeline (0.085 vs 0.110 on Qwen, 0.435 vs 0.455 on Gemma). It is not a length control here, because it carries only 19% and 8% of re-grounding's extra tokens; test 4 does that job.

Test 4. More compute barely moves the gap (row 4 against row 0). The message-only pipeline gets three samples at temperature 0.7 and a majority vote, and it spends more tokens than the re-grounded pipeline: 2738 vs 1609, 2507 vs 1060, 5020 vs 2253 and 3205 vs 1365 per item.

Test 5, HotpotQA (distractor setting, 200 validation items drawn at random with a fixed seed, exact match, one condensing and one preserving four-stage pipeline). The tax is 0.430 [0.360, 0.500] for Qwen and 0.435 [0.365, 0.505] for Gemma on the condensing pipeline, and 0.465 [0.390, 0.540] for Qwen on the preserving one. Gemma's preserving pipeline shows 0.430 [0.355, 0.505], but its message-only condition parses 0.985 of its answers, one item short of our check, so by our rule this test gets no verdict. We had expected the preserving pipeline to pay less, as it does in mathematics. It does not: the condensing-minus-preserving difference is -0.035 [-0.105, 0.035] for Qwen and 0.005 [-0.055, 0.065] for Gemma.

AuthorsOpenReview ↗

Revised PDF uploaded

One example: gemma-4-12B solves 86.5% of 200 MATH-500 problems in a single call, 45.0% in a four-stage message-only pipeline, and 82.0% once its later stages are re-grounded. Most of what splitting costs is paid at the interfaces: where a single call does markedly better, re-grounding alone closes a median of 60% of the gap (Section 3.1). The 13 open-weight models with valid runs pay a median tax of 24.0 points on this pipeline, and two of them pay 43.0 to 46.5 points on HotpotQA. All nine API models pay a MATH-500 tax of 6.5 to 25.0 points (Section 3.4, Appendix G). Re-grounding the receiver beats re-grounding the sender on all seven models and both math benchmarks (Section 3.3).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.