acceptodds
Under review as a conference paper at ICLR 2027

When Tasks Compete for Tokens: Implicit Budgeting in Language Models

Abstract

Modern language models are increasingly asked to handle several tasks within a single prompt. Doing so requires solving each task correctly while integrating the resulting answers into a coherent response. In this paper, we identify a fundamental limitation in this process, where models make substantially more errors when tasks are composed, even when they can solve every constituent task reliably in isolation. This degradation cannot be fully explained by the accumulation of independent errors. Across both open source and frontier models such as Claude Opus 5, our findings instead point to an implicit thinking budget underlying this effect. In particular, under composition, models use substantially fewer tokens per task than in isolation, even when the full sequence remains well within the context window. Tasks that require longer solution traces in isolation are consequently more fragile under composition, i.e., models make more errors on prompts containing these tasks. This fragility can also be exploited for efficiency. By appropriately grouping fragile tasks, we obtain 40% token savings across a range of math and linguistic tasks while also preserving accuracy.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Multi-task reasoning, compositional generalization, task interference

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_Nbh4OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies a phenomena where performance drops when batching tasks together into a single prompt where the tasks are solvable on their own. They show that the proportion of tokens allotted to each subtask decreases according to a power law in the number of tasks included in the context, which they call an “implicit token budget”. This informs their design of a task batching procedure that tries to minimize the number of total tokens used while still allowing each task to have the right token budget.

Critical strengths

The paper introduces the problem well and situates itself nicely with respect to related work. The results claimed in the paper are appropriate for the evidence provided, and the findings are interesting. The idea that LLMs have an implicit token budget per task and that as more tasks are included that budget decreases proportionally is neat and the fragility-aware batching shows that this can be overcome in a simple way.

Critical weaknesses

My main concern is that several of the results presented in this paper have already been shown in prior work, (e.g. that the number of tokens shrinks when more tasks are added, compositional degradation, etc.), and the authors do note these appropriately. This lessens the overall impact of the contribution of this paper in my opinion, but I do think the identification of the implicit token budget/power law and the fragility-aware batching are still nice contributions.

Another weakness relates to the clarity of presentation: there were a few times where I had to go back and reread something or go ahead to look for a definition of something I did not understand. For example, it is not clear what a "fragile task" is early on (lines 53, 74-75), as it is not defined until line 256. I think these issues could be improved with some reorganization of the existing text.

Questions for authors
  • In Lines 134-137, you mention that the reasoning models you study have been "post-trained with reinforcement learning", or undergone "supervised fine-tuning". Does this refer to additional training you have done on top of standard huggingface checkpoints? Or are you just trying to clarify how these models differ from their base model variants?
2 comments

We would like to thank you for your constructive feedback.

As you noted, we have discussed the work that is directly related to our findings. However, we would like to emphasize that our paper goes beyond established results from the literature. This leads to a fundamentally more complete picture on the limitations of models when it comes to dealing with compositions and new insights on the nature of those limitations. Below we highlight some of the key ways in which our contributions provide clarity and offer a unifying perspective.

  1. Why do harder task degrade more? Previous literature observed that task difficulty is potentially related to degradation. The original Batch Prompting paper (Cheng et al., 2023) noted that “the complexity of tasks affect its performance”. Another line of work shows that tasks requiring more reasoning steps degrade more when additional context is added to the prompt (Yang et al., 2025b; Rodionov et al. 2026). However, neither explains why these effects occur. We identify implicit thinking budget misallocation as a primary mechanism explaining both observations. From this key concept we have derived a power law, relating the size of a composition to the thinking tokens spent on each constituent task, and introduced an intervention to reduce degradation leading to a method which outperforms best constant-k batching (see responses to Reviewer qXTH and Section 4.1). We support our claims with evidence from 8 LLMs, including a frontier model, across 8 benchmark datasets that span different task complexities and domains.

  2. Does compression based on batching damage accuracy? This was studied in ConPress (Deng et al. 2026). Our main contribution here lies in identifying each task's reasoning length as an important proxy: the compression itself may not cause accuracy drop if the questions are not fragile. Our concept of fragility provides a structured reason why certain tasks fail under composition and the power law we discovered demonstrates that this reasoning length behaves consistently under composition.

  3. What makes a good batching strategy? Prior papers on batching have identified that optimal batching is specific to the task domain and the model (Cheng et al., 2023; Xu et al., 2026). Our proposed mechanism captures these design considerations under the concept of task fragility, which is highly model-specific and also varies between individual tasks. By identifying the key mechanisms that underpin compositional degradation, we believe our work can lead to a more principled understanding and development of batching strategies.

We believe we have provided thorough evidence and key missing pieces of this story that lead to a more unified perspective on the findings from the literature on batch prompting and compositional degradation. By postulating the implicit budgeting effect and identifying task fragility as its key concept, we also opened up interesting directions for future research toward a better mechanistic understanding of compositional degradation.


Thank you for pointing out the unclear definition. In the revision, we have clearly defined the most important concepts of implicit thinking budget and fragility in section 2, which we referenced in the later sections of the paper.

Q: Does “post-trained with reinforcement learning” or “supervised fine-tuning” (lines 134–137) refer to additional training on our side?

No, we use the publicly released checkpoints without any further training. The sentence only describes how these reasoning models differ from their base variants. We have rephrased the lines to make things more clear.

If there are additional issues or concrete related work you would like us to address, please let us know and we will do so as soon as possible.

References

  • Cheng et al. (2023). Batch Prompting: Efficient Inference with Large Language Model APIs. EMNLP 2023.
  • Deng et al. (2026). ConPress: Learning Efficient Reasoning from Multi-Question Contextual Pressure. ICML 2026.
  • Rodionov et al. (2026). Reasoning Shift: How Context Silently Shortens LLM Reasoning. arXiv:2604.01161.
  • Xu et al. (2026). Towards Cost-effective LLMs Routing with Batch Prompting. arXiv:2605.28268.
  • Yang et al. (2025b). How Is LLM Reasoning Distracted by Irrelevant Context? An Analysis Using a Controlled Benchmark. EMNLP 2025.

Thank you for the reply. I appreciate the more detailed response about how this paper tries to unify and expand prior work's findings, and have updated my score to weak accept based on the updates.

Reviewer_qXTHOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary
  • This paper studies accuracy loss when independent questions are combined in one prompt. It focuses on questions a model can solve in isolation, and measures how their accuracy and reasoning-token use change under batching.
  • The paper finds that longer isolated reasoning traces are associated with greater fragility, and that fragile questions are relatively stable across batch sizes within a model. It proposes fragility-aware batching and a predictor for unseen questions.
  • The efficiency experiments report about 41.7% average output-token savings with measured fragility and 29.7% with held-out predictions, with small changes in average accuracy. A separate Claude Opus 5 experiment on GPQA reports about 20% fewer output tokens with a 0.2-point accuracy decrease.
Critical strengths
  • The studied problem is very well-motivated and practical in LLM world.
  • I like the unchanged-question setup as a useful control. Rather than making the underlying reasoning task more complicated, the paper places independently solvable questions together. This gives cleaner evidence of a batching problem than a benchmark that combines dependent reasoning steps. The per-question token measurements also go beyond reporting a drop in aggregate accuracy.
  • The analysis leads to a concrete/actionable intervention. Fragility is stable enough within a model to support a batching policy, and the held-out predictor tests whether some of the benefit transfers to new questions. This is a useful extension of prior batching work, rather than only another observation that large batches can hurt.
Critical weaknesses
  • The paper does not yet isolate budget misallocation as the cause. Batching changes attention, ordering, and the surrounding context at the same time that it shortens reasoning. BatchPrompt already shows that order and position matter, and Reasoning Under Constraint already reports shorter reasoning under batching. The current results connect shorter traces with errors, but do not show that restoring reasoning effort while keeping the batch fixed would restore accuracy.
  • The practical savings need more qualification. Estimating fragility requires repeated labeled trials, and the reported savings do not include that calibration cost or the embedding model used by the predictor. Generalization also varies: the held-out savings in Table 1 average 10.1% on GPQA, 6.4% on MMLU-Redux, and 4.3% on MATH-500 across models. Appendix A.1.5 excludes AIME from the efficiency average because the policy falls back to isolated solving. These details matter for the claim that the approach is useful on difficult new workloads.
Questions for authors
  • If the questions, order, and batch size stay fixed, does giving the model more reasoning effort recover the lost accuracy? This would help distinguish budget misallocation from other forms of interference.
  • What are the net savings on new workloads after including calibration, embedding, input tokens, and output tokens? How many questions are needed to recover the setup cost?
  • How does fragility-aware batching compare with the best tuned fixed batch size at the same accuracy target, rather than only random batching at size six? Would a simpler length-based policy recover much of the gain?
5 comments

Thank you for the review!

Is budget misallocation the sole cause?

Q: "If the questions, order, and batch size stay fixed, does giving the model more reasoning effort recover the lost accuracy? This would help distinguish budget misallocation from other forms of interference."

We agree that batching changes more than reasoning length. Attention, ordering and context are affected too. Thus, we do not claim that budget misallocation is the only isolated effect. We claim that it is a primary mechanism, since it unifies various observations from the batch-prompting literature. Our results suggest the other effects are comparatively small, however they can change the magnitude of accuracy degradation (i.e. Section 5 for post-training, Appendix 1.4 and 1.5 for neighbor and positioning effects).

Restoring reasoning effort within a fixed batch would test whether the implicit thinking budget is the sole cause of degradation, which is a stronger claim than ours. Extending the reasoning process beyond the budget a model naturally asigns, without invasively steering the reasoning itself, is an open and interesting question on its own, which we see as a promising path for future work.

Nevertheless, we did run a similar experiment (see Section 3.1 and A.1.3 of the revised manuscript) where we interrupted the reasoning when prompted with a single question in isolation, at the exact thinking token budget it assigns to that question in a composition. The accuracy drop showed the same pattern, with fragile tasks degrading more strongly, however also with the overall degradation being larger than under composition (see Figure 4). Thus, reducing the thinking length by interrupting the thinking process when solving an isolated question is sufficient to reproduce the degradation, without tasks interfering, showing that the amount of thinking tokens and task fragility have a strong effect alone.

Compared to previous work such as BatchPrompt or Reasoning Under Constraint our paper proposes an explanation that, according to our experiments, appears to have a strong effect. Consistent with literature, both papers observe that batched reasoning is shorter and position-dependent, nonetheless they leave unclear which tasks fail how strongly and at what batch size. Our mechanism explains why tasks from different datasets degrade differently, since tasks that need long reasoning in isolation are fragile and fail first (Section 3.1). It also predicts how the implicit budget changes with batch size, following a power law (Section 3.1), and this consistency supports a principled intervention that outperforms the best fixed batch size at the same accuracy bound (see below).

Overall, we believe our work opens a path toward a more fundamental understanding of compositional generalization and of the reasoning process, by proposing a well-evidenced mechanism.

Q: “What are the net savings on new workloads after including calibration, embedding, input tokens, and output tokens? How many questions are needed to recover the setup cost?”

We agree that efficiency metrics have to be taken into account. The question is not straightforward, though, since it is again a tradeoff: estimating fragility from more samples is more costly but may yield more precise batching and therefore larger savings.

In deployment, fragility-aware batching relies on a predictor for fragility estimation, whose inference cost is negligible compared to that of a LLM. Collecting training data and training the predictor is an initial cost per model, which amortizes in large-scale deployments. The table below shows how many question would need to be asked to amortize the cost of setup, for both the fragility-aware batching with measured fragility and with the predictor:

DatasetSetup (M tok)Predictor break-evenMeasured break-even
GPQA-Diamond23.242k12k
GSM8K30.7198k58k
MATH-50051.4324k160k
Integrals55.159k189k
OneWordQA12.948k48k
StrategyQA7.623k22k
MMLU-Redux36.3589k60k
Average31.287k48k

According to our experiments, fragility-aware batching would amortize its cost after 48k question when using measured fragility and 87k questions when using predicted fragility (with reasoning intensive datasets usually requiring more questions until break-even and simple question answering requiring less).

A further important metric for deployment is the number of input tokens. Batching reduces input tokens substantially, since the system prompt and other setup context are shared across the tasks. Our measured savings count only output tokens, so one can expect larger savings once input tokens are included.

Fragility-aware batching has a second property that is important when deploying it on new tasks since it composes questions such that the accuracy drop stays within a set bound. This avoids batching taks when it would cause degradation. On AIME, for example, it estimated the problems to be too fragile to batch at all and thus chose isolated solving. (batching AIME tasks with k=3 leads to an accuracy drop of 16.5pp on average)

Finally, we would like to clarify the role of fragility-aware batching in the paper. Its main purpose is to provide further evidence that fragility is a key variable that can be predicted from the question text and strongly modulates compositional degradation. By postulating the implicit budget misallocation and identifying task fragility, our work opens up new research toward a better mechanistic understanding of compositional degradation, including a closer study of the properties of fragility.

Of course we acknowledge that this may also help in the development of batching strategies that are deployed in the real world, even though the main goal of the paper is to establish a fundamental understanding of compositional degradation.

Q: “How does fragility-aware batching compare with the best tuned fixed batch size at the same accuracy target, rather than only random batching at size six? Would a simpler length-based policy recover much of the gain?”

Part 1: best tunded fixed batch size

We compare our fragility-aware batching against the best fixed-k baseline, obtained when choosing the largest k we tested whose accuracy degradation still fulfills the same accurracy target of 0.5pp. The percentages in the parentheses are relative to the best fixed k baseline. The following two tables shows the results on GPQA-Diamond and MMLU-Redux (more results in Table 3 of the revised manuscript):

GPQA-Diamond

ModelBest fixed kAcc.Tok.Fragility-aware Acc.Fragility-aware Tok.
Qwen3 4B T.166.7%766366.8% (+0.2)5200 (−32.1%)
Qwen3 8B158.7%597360.0% (+1.3)3520 (−41.1%)
Qwen3 14B161.5%459757.9% (−3.7)2111 (−54.1%)
Qwen3.8 27B183.2%430384.0% (+0.8)4102 (−4.7%)
R1 Qwen 7B149.1%668649.6% (+0.5)4037 (−39.6%)
R1 Llama 8B147.7%742449.6% (+1.9)4526 (−39.0%)
R1 Qwen3 8B160.8%908461.9% (+1.2)6595 (−27.4%)
Claude Opus 5189.5%68989.2% (−0.4)521 (−24.3%)
Average64.7%580264.9% (+0.2)3827 (−34.1%)

MMLU-Redux

ModelBest fixed kAcc.Tok.Fragility-aware Acc.Fragility-aware Tok.
Qwen3 4B T.384.3%102485.3% (+1.0)527 (−48.5%)
Qwen3 8B187.2%92186.1% (−1.1)285 (−69.0%)
Qwen3 14B189.0%71888.2% (−0.9)319 (−55.6%)
Qwen3.8 27B2091.9%18891.8% (−0.1)196 (+4.3%)
R1 Qwen 7B170.1%107969.7% (−0.3)601 (−44.3%)
R1 Llama 8B175.5%145875.9% (+0.3)896 (−38.5%)
R1 Qwen3 8B181.6%171182.1% (+0.5)1273 (−25.6%)
Average82.8%101482.7% (−0.1)585 (−42.3%)

Compared to this baseline, fragility-aware batching saves substantially more tokens on average at a similar accuracy drop. On the reasoning-intensive datasets GPQA-Diamond and MMLU-Redux, it uses 34.1% and 42.3% fewer thinking tokens respectively than the best fixed-k baseline. This shows that constant-k batching is less effective on these datasets, because it ignores task-specific fragility, more fragile tasks degrade, so the constant batch size must be kept low or batching must be avoided altogether. By contrast, batching tasks according to their fragility into compositions of different sizes avoids compositional degradation in most cases, allowing a substantially larger token reduction under the same accuracy drop budget.

Part 2: simple length-based policy. A batching policy based on length would need to know in advance how many thinking tokens a task requires in order to decide how to batch it. Since running the LLM on the question defeats the purpose, such a policy would also need a predictor. However, our experiments suggest that trace length is only a proxy for accuracy drop under composition (Figure 4: trace length correlates with fragility), so predicting the drop directly, as fragility-aware batching does, is more precise.

Reviewer_QCUYOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies the reasoning degradation effect of composing multiple tasks into a single request, thereby inferring hypotheses over LLM's reasoning effort allocation in multitasking.

Critical strengths

Findings such as a power law between performance decrease and composition size, implicit thinking budget, and task fragility are significant, informative, and well-supported by experiments. The paper is well-written and a breeze to read.

Critical weaknesses

The paper is not organized with high technical rigor, especially around the core findings. A more scientific approach would separate hypotheses from phenomena: hypotheses as main findings, experiments as motivation and evidence.

Questions for authors

Questions

My overall assessment is positive (borderline accept). The questions below fall into two tiers. Part A is the main limitation; addressing it would lead me to raise my score to clear accept. Part B lists minor issues I would like the authors to resolve to retain my current score.

Part A — Would raise my score

Precise framing of the central concepts. The results are scientifically significant, but the way they are presented, in terms of terminology and hypotheses, lacks rigor. To be clear, I am not asking for a formal theory built on rigid mathematics. I am asking for something closer to how a physical law is proposed: center the main findings on a few key concepts, state the explanatory claim as an explicit hypothesis, and ground each concept in a precise definition in terms of measured quantities. Three concepts carry the paper, and I would like each to be treated this way.

A1. The implicit thinking budget (ITB) as an explicit hypothesis. The ITB is the paper's central explanation for compositional degradation, but it is never defined. In the technical sections it first appears as a label (l.194; §3.1 title). It is first used in the argument at the end of §3.1, which says models "estimate" a budget that "does not generalize" to composed tasks; the reader is left to supply the meaning. Specifically:

  • What is the ITB as a quantity? Contribution 2 states that the ITB "shrinks as a power law," and the only quantity fit with a power law is ri(k)=Ti(k)/Ti(1)r_i(k)=T_i(k)/T_i(1) (Eq. 1). Operationally, then, the ITB appears to be Ti(k)T_i(k) itself. If it is something more, that should be defined.
  • What does it depend on? Is the ITB a property of the task alone, or does it depend on the model and on kk?
  • Hypothesis vs. finding. The ITB is something the authors "propose" (l.194), but elsewhere something the results "reveal" (§1; end of §3.1; §6) or "identify" (Contribution 2). The Limitations paragraph acknowledges that the driver of the reduction is unknown, which fits a hypothesis.

I would appreciate a dedicated, clearly marked statement of the ITB hypothesis early in the technical sections. It should define the ITB in terms of measured quantities, state what it depends on (task, model, dataset, kk), and either make the "estimation"/"generalization" language precise or replace it with a description consistent with the uniform-shrinkage finding. The framing (a hypothesis supported by evidence) should be consistent throughout.

A2. Fragility as a defined, highlighted concept. Fragility carries much of the paper's explanatory and practical weight, but it has no formal definition. §3.1 gives it an italicized description ("tasks that break under composition in this way," l.255). §3.2 uses it in its title and conclusion ("a stable property of a task for a given model"). It is only estimated in §4.1.

Intuitively, fragility is a property of a task's accuracy-versus-reasoning-budget curve: how much compression can the model tolerate before it can no longer solve the task? In the paper, compression is controlled solely by composing kk tasks, and fragility is estimated through that single mechanism.

I would appreciate a highlighted definition of fragility. It could sit in §4, where fragility is estimated. The definition should state the estimand and how it is estimated via kk, be forward-referenced from the earlier uses (abstract, §1, §3.1, §3.2), and explain how the binary usage relates to the graded estimates.

A3. Consistent naming for the power law (encouraged, not required). The power-law scaling is a significant finding, but the quantity that follows it goes by many names, such as "reasoning per task" (§1), "implicit per-task thinking budget" (Contribution 2), "relative token usage" (Fig. 3).

The exponent is only called the "fitted slope" (l.265). I would appreciate A single name for the scaling relation and its exponent, used consistently, ideally tied to the ITB definition in A1.

Part B — Needed to retain my score

B1. Accuracy drop and token ratio as expectations over compositions. Δim(k)\Delta_i^m(k) and ri(k)r_i(k) are written as functions of (task, kk). However, they are measured inside sampled compositions, and their dependence on the co-composed tasks is suppressed in the notation (l.104–105; l.208–209). Do the authors intend these as properties of a task within a particular composition, or as properties of the task at size kk, averaged over compositions? The latter seems to be the intent, since fragility is used as a task-level property (§3.2). It is also the better definition if the values are meant to describe the task rather than a full suite of tasks.

Can the authors state which interpretation is intended? If it is the expectation, define both quantities as expectations over an explicit composition distribution, and state which distribution each experiment uses.

B2. Does compression depend on the co-composed tasks? (ablation) The paper finds that each task's token ratio ri(k)r_i(k) shrinks roughly as kmk^m without systematic dependence on the task's own isolated length Ti(1)T_i(1) (l.266–268; Fig. 3). I could not find a test of whether ri(k)r_i(k) is likewise independent of the other tasks in the composition. For example, at fixed kk, is a task compressed more strongly when all of its co-tasks are long-reasoning or fragile than when they are short?

Can the authors perform a controlled ablation on at least one model and dataset? At fixed kk, compose the same target tasks with co-tasks drawn only from the High isolated-length bin versus only from the Low bin, and compare their ri(k)r_i(k). If compression turns out to depend on the co-tasks, add a discussion of the implications for the ITB hypothesis and for fragility-aware batching.

B3. Definition of std(Δˉ(λ))\mathrm{std}(\bar\Delta(\lambda)) in Eq. 3. Eq. 3 uses std(Δˉ(λ))\mathrm{std}(\bar\Delta(\lambda)) without defining it or pointing to its definition. A.2.4 (Eq. 7) defines it as 1n sd\frac{1}{\sqrt{n}}\,\mathrm{sd} of the per-task expected drops. That is a standard error of the mean, not a standard deviation. Can the authors define the term where Eq. 3 introduces it (or reference Eq. 7), and rename it to reflect that it is a standard error?

B4. Reference for costk\mathrm{cost}_k. costk\mathrm{cost}_k is described only in prose in §4.1; its formal definition is Eq. 8 in A.2.4. Please reference Eq. 8 where costk\mathrm{cost}_k is introduced.

Suggestion (not a condition)

Fig. 3 shows one marker per (length bin, kk), which hides the spread of ri(k)r_i(k) within each bin. Box plots would show this, but with three tiers per panel they may overlap enough to defeat the purpose of the figure. If the authors find a clean way to show the dispersion, it would make the power-law claim more informative.

4 comments

Thanks for the review!

Part A We have taken your constructive feedback into account. We strengthened the definitions of the main concepts and have clearly separated our hypothesis from the evidence that supports it.

A1. The implicit thinking budget as an explicit hypothesis

We agree that this concept needed more clarification. Our changes therefore have two parts: a formal definition of the implicit thinking budget, and an explicit hypothesis of the mechanism.

First, Section 2 now formally defines the implicit thinking budget as the number of thinking tokens Ti(k)T_i(k) that a model devotes to a task tit_i within a composition of size kk. This quantity depends on the model, the task, the dataset from which the constituent tasks come, and the composition size. It is defined as an expectation over uniformly sampled compositions, which is estimated by measuring the thinking tokens of each task (Section 3.1, Appendix A.2.1).

Our proposed power law then describes a relationship between the implicit thinking budget and the composition size.

Second, we state our proposed mechanism as the implicit thinking budget misallocation hypothesis, which is introduced at the beginning of Section 3.1 and referenced whenever we discuss the hypothesis. In particular, the wording of "reveal" and "identify" in Section 1, the contribution list, Section 3.1 and Section 7 has been adjusted to "support the hypothesis", "hypothesize" or "consistent with this hypothesis" to clearly separate hypothesis from evidence.

Finally, we replaced the ambiguous terms "estimation" and "generalization", because we agree that these terms could be misinterpreted. Originally, we meant how many thinking tokens a model devotes to a given task, not that the model estimates an explicit value. We therefore removed "estimation" throughout the paper and now refer to the hypothesis instead (Sections 3.1 and 4). Where the word "generalize" remains in Section 1, it is phrased as a possibility.

A2. Fragility as a defined, highlighted concept

We have taken your considerations into account and clarified the definition. Section 2 now also defines accuracy drop. We define fragility as a model-specific continuous property of a task. A task is more fragile than another if its expected accuracy drop is larger at the same composition size. Fragility is therefore ordered by, and quantified through, the accuracy drop. Since fragility is the concept quantified through accuracy drop, its estimand and estimator are those of the accuracy drop, which we have stated explicitly. We also added forward references in Section 1, Section 3.1 and Section 4.1.

A3. Consistent naming for the power law

We have renamed the power law to the implicit thinking budget scaling law and its exponent to the scaling exponent. We also renamed the exponent from mm to α\alpha to avoid confusion with the symbol for the model. The name is introduced in Section 1 and defined with Equation 1. We have updated the references to the power law throughout the paper.

Part B

B1. Accuracy drop and token ratio as expectations over compositions

In every experiment, the accuracy drop is a property of the task, averaged over uniformly sampled compositions of size kk from a set of tasks, and not specific to particular compositions. We clarified this in the definition of the accuracy drop in Section 2.

B2. Does compression depend on the co-composed tasks?

Thank you for pointing out this aspect. We ran an ablation on the Qwen3 series, a representative family of open reasoning models, using Qwen3 4B Thinking and Qwen3.8 27B on MATH-500 and MMLU-Redux. For each dataset, we first took a random test subset of 20% of the tasks. We sorted the remaining 80% by their number of thinking tokens in isolation and chose the 33% with the most tokens as the long group and the 33% with the fewest tokens as the short group. For each task in the test subset, we then measured its thinking length in 5 compositions whose neighbors are uniformly sampled from the long group (long compositions) and in 5 compositions whose neighbors come from the short group (short compositions). All compositions have size k=6k = 6.

DatasetModelShort comp.Long comp.Long − short
MATH-500Qwen3 4B T.0.520.59+0.07 ± 0.04
MATH-500Qwen3.8 27B0.920.96+0.04 ± 0.07
MMLU-ReduxQwen3 4B T.0.340.24−0.10 ± 0.01
MMLU-ReduxQwen3.8 27B0.610.65+0.04 ± 0.02

The table reports the average token ratio of the test-subset tasks in short and in long compositions, with ± being the half-width of a 95% confidence interval. The difference in the thinking trace length stays within ±0.10 and changes its sign across settings. This is small compared to the effect of the composition size itself, which can reduce the token ratio by roughly 0.4–0.66 when increasing kk to 6. Neighbor trace lengths therefore have an effect, but it is substantially weaker than the effect of varying the composition size. For practitioners, this means that choosing the composition size matters much more than choosing which tasks share a prompt. We added this ablation as Appendix A.1.4 (Table 2).

B3. Definition of std in Equation 3

Equation 3 uses the standard error (SE), not the standard deviation. We corrected this where the term is introduced in Section 4.1 and refer to its definition in Appendix A.2.4.

B4. Reference for cost_k

At the first appearance of costk\mathrm{cost}_k in Section 4.1, we added an explicit reference to Appendix A.2.4, where it is defined in Equation 5.

Dear Reviewer, we kindly ask you to consider our comments and edits and would be happy to address any further questions you may have. If we addressed your concerns please reconsider your assessment.

Reviewer_QCUYOpenReview ↗

Final Assessment

The authors have responded adequately to all my questions and thereby addressed by concerns. I have raised my score to clear acceptance.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Contributions, Revisions and Rebuttal Outcomes

Dear Area Chair and Reviewers,

as the discussion phase nears its end, we want to thank the reviewers for their time and constructive feedback, which has significantly improved the manuscript. We would like to summarize our key contributions and the consensus reached during rebuttal.

Motivation

  • We address the problem of compositional generalization, asking how well LLMs can solve a composition of tasks.
  • As LLMs are increasingly employed in critical workflows and complex tasks, which naturally are compositions of smaller subtasks, it becomes important to paint a fundamentally more complete picture of the limitations of LLMs when dealing with compositions.

Core contributions

  1. We propose the implicit thinking budget misallocation as the primary mechanism, well-founded by broad empirical evidence and unifying different observations and perspectives from the literature (Section 3.1).

  2. Our implicit thinking budget scaling law uncovers how the implicit thinking budget behaves under composition, relating the growth in the number of tasks to the reduction in thinking tokens spent on each task (Section 3.1, Equation 1).

  3. Our experiments further reveal that specific tasks, even from the same dataset, require different implicit thinking budgets and thus degrade very differently, captured by the concept of task fragility (Sections 2 and 3.1). This explains why tasks degrade unevenly, which prior work observed but could not explain.

  4. We turn our mechanism into a useful algorithm by batching tasks according to their fragility, leading to a dynamic batching that saves substantially more tokens than constant batch sizes (Section 4.1).

We believe that by postulating the implicit budget misallocation effect and identifying task fragility as its key concept, we opened up interesting directions for future research toward a better mechanistic understanding of compositional degradation.

Revisions

Based on the constructive feedback of the reviewers, we improved the manuscript in the following points:

  • Revised core definitions, hypothesis separated from evidence: We now introduce the "implicit thinking budget misallocation" as an explicit hypothesis (Section 3.1), added more detailed definitions of the implicit thinking budget, accuracy drop and fragility (Section 2), and fixed the wording throughout the paper to clearly separate hypothesis from evidence.

  • We added an ablation on the influence of neighboring tasks on the implicit thinking budget, which shows that these effects are small compared to the effect of composition size (Appendix A.1.4, Table 2).

  • We added the best fixed batch size as a baseline, confirming that the gains of fragility-aware batching cannot be reproduced with a constant batch size (Section 4.1, Table 1; Appendix A.1.7, Table 3).

Rebuttal outcomes and consensus

We are pleased that the reviewers found the problem well motivated, the findings significant and well supported, and fragility-aware batching a concrete, actionable extension of prior batching work.

Reviewer Nbh4 (2 → 3). After we clarified our contributions, the reviewer raised their score and noted our paper as a unifying work.

Reviewer QCUY (3 → 4). The reviewer raised their score after we highlighted the core definitions and the hypothesis, and added the requested ablation on neighbor effects.

Reviewer qXTH. While Reviewer qXTH has not yet responded to our rebuttal, we have addressed their concerns:

  • New baseline and net savings: We added the best fixed batch size as a baseline in our results (Section 4.1, Table 1; Appendix A.1.7, Table 3), which confirmed that the gains of fragility-aware batching cannot be reproduced with a constant batch size, and also conducted a break-even analysis of the method's setup cost (see our response to Reviewer qXTH).

  • Isolating the mechanism: in response to qXTH, we clarified our mechanistic claim, since isolating our mechanism as the only one is a stronger claim than ours (see also Limitations, Section 7). Providing precise empirical evidence on this question is an open and interesting problem in its own right, which we see as a promising path for future work.

  • New experiment on confounding factors in batching: To strengthen the claim that our mechanism is a primary one, we truncated the thinking process when solving a single task to mimic the implicit thinking budget reduction under composition (Section 3.1, Figure 4; Appendix A.1.3). This suffices to reproduce the degradation without other tasks interfering, showing that the amount of thinking tokens and task fragility alone have a strong effect.

We hope that the Area Chair will consider our strengths the reviewers identified and our significant contributions toward a more complete picture of compositional degradation.

Best regards,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.