acceptodds
Under review as a conference paper at ICLR 2027

One Model, Multiple Compute Budgets: Multi-Ratio ConceptMoE for Flexible Compute Across Turns

Abstract

Large language models typically support different inference budgets through separate model sizes or fixed compute configurations. In multi-turn interactions, however, compute demand can vary substantially across turns. Using a high-compute model for every turn wastes computation, while routing among independent models may require rebuilding model-specific history states after switches. We therefore ask whether a single shared model can support multiple compute budgets that can be flexibly assigned across turns. We introduce Multi-Ratio ConceptMoE, which jointly supports token-to-concept compression ratios of 2, 4, and 8 (CR2/CR4/CR8) within a shared backbone. Lightweight full-token modules reduce compression-independent cost, ratio-specific boundary routers support distinct compression levels, and cross-ratio training enables mixed-ratio processing within the same context. Across PTBench and SFTBench, the three fixed modes outperform their compute-matched MoE baselines in all 6 comparisons while remaining competitive with independently trained fixed-ratio specialists. On CodeChat and CoSQL, mixed-ratio inference improves the accuracy–compute trade-off, outperforming the Large MoE by 5.87 and 2.68 percentage points while reducing FLOPs by 41.1% and 12.3%, respectively. Under the same turn-wise budget schedule, Multi-Ratio ConceptMoE further reduces FLOPs by 78.4% and 46.9% relative to routing among independent models. These results demonstrate that a single shared model can provide multiple compute budgets for flexible compute allocation across turns.

Primary area:
foundation or frontier models, including LLMs
Keywords:
large language models, token-to-concept compression, multi-turn inference

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_9K8kOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper addresses the varying computational demands across turns in multi-turn agentic interactions. It extends ConceptMoE to Multi-Ratio ConceptMoE, which supports multiple concept compression ratios within a single backbone and uses an external LLM judgement to assess the compute budget for each turn. The method explicitly introduces joint training with mixed compression ratios, allowing the shared backbone to process contexts containing multiple compression ratios. On downstream tasks such as CodeChat and CoSQL, it offers a more flexible accuracy–compute trade-off, e.g., achieving CR2-level performance with FLOPs comparable to CR4.

Critical strengths
  1. The simple yet effective training scheme randomly mixes segments with different compression ratios, enabling a single backbone to accommodate multiple ratios within the same context without re-prefill. The ablation results in Table 5 confirm the necessity of mixed-ratio training for downstream mixed-ratio inference.
  2. Mixed-ratio training does not significantly influence the concept model's underlying capabilities and can even improve them in the two coding tasks (Table 3).
Critical weaknesses
  1. Although mixed-ratio training enables the backbone to support multiple compression ratios almost as a free lunch, mixed-ratio inference still operates at the turn level and depends on an external LLM judge (gpt-5.6-terra in this paper). This dependence significantly limits the practical usability of Multi-Ratio ConceptMoE. The authors should also evaluate zero-overhead heuristic policies to determine whether the proposed method retains its benefits under these policies.
  2. Although the backbone follows ConceptMoE, the appendix should provide a self-contained description of the model architecture, training hyperparameters, and related settings to ensure reproducibility and fair comparisons. As presented, understanding the results depends heavily on first reading the ConceptMoE paper.
Questions for authors
  1. Appendix B.4 states that different checkpoints may require different prompt designs. Please provide concrete examples of these differences and explain their effects. If the checkpoints are this sensitive to specific inference settings, does this also suggest fragility in the training and inference design? Training requires specific mixtures of compression ratios, inference prompts must be designed separately, and turn-level interactions even require assessment by an external model. I believe that the current design and experimental results weaken the practical value of Multi-Ratio ConceptMoE to some extent. The authors should conduct further targeted evaluations of the design's robustness.
  2. The paper does not provide detailed formulas for calculating FLOPs. Excluding assessment by the external model, it does not explain how the computational costs of the core components in the encoder → chunk → main concept backbone → dechunk → decoder pipeline are calculated. It only briefly states that "The reported FLOPs exclude the cost of the external routing policy and characterize model execution under the frozen turn-wise schedules."
4 comments
AuthorsOpenReview ↗

Part 1/2: Heuristic Routing, Architecture, and Training Details

W1

Thank you for raising this practical concern about the additional cost of using an external LLM router.

We added deterministic heuristic routing experiments on the complete CodeChat and CoSQL evaluation sets, using the same Multi-Ratio checkpoint, generation prompts, and decoding settings as the main experiments. Rules use only current/prior user instructions and static task information such as the database schema: complex or high-risk requests use CR2, ordinary follow-ups CR4, and clearly local modifications CR8. The rules are frozen before generation and access no future turns, model outputs, or correctness feedback. They require only CPU rule matching, with no additional model calls or LLM input/output tokens.

DatasetLarge MoE accuracyOriginal LLM routing accuracyHeuristic routing accuracyMain-model FLOPs reduction vs. Large MoE
CodeChat37.91%43.78%39.81%41.97%
CoSQL49.16%51.84%50.94%13.47%

CodeChat uses Combined Joint accuracy and CoSQL uses SQL execution accuracy. FLOPs use a common full-history workload across configurations within each dataset. Heuristic routing achieves higher accuracy than Large MoE with fewer theoretical FLOPs on both tasks, showing that an external LLM router is not required to obtain Multi-Ratio's benefits. We will include the results and explicit routing rules in the revision.

W2

Thank you for suggesting that we make the architecture and training details clearer in the paper.

We will consolidate the following architecture and training details in the appendix so the experimental setup can be understood directly from this paper.

  1. Architecture and cross-ratio training. The pipeline is encoder → Chunk → concept backbone → DeChunk → decoder. The encoder processes the full token sequence; Chunk groups tokens into concepts using predicted boundaries; the backbone processes these concepts; and DeChunk restores token resolution for the decoder. CR2/CR4/CR8 have separate lightweight boundary routers and share all other modules. Encoder and decoder hidden widths are approximately 2/32/3 of the backbone width.

    Each input is encoded once, then expanded into three independently sampled cross-ratio views. Within each view, interior segment lengths are sampled uniformly from 64–960 tokens (expected length 512); a residual suffix shorter than 64 tokens is merged into the preceding segment. Each segment independently selects CR2/CR4/CR8 with equal probability. A forced boundary starts each segment, and its router predicts the remaining boundaries. The backbone and decoder process the views with separate causal sequence boundaries.

  2. Training settings and comparison budgets. The model has 30B total / 0.5B activated parameters, with 500B pretraining / 5B SFT tokens, cosine learning-rate decay, and training on Blackwell GPUs. The three views share target text and supervision positions. Each view's language-modeling loss is averaged over valid positions, then the three losses are averaged. Compression control follows ConceptMoE's auxiliary objective: the three routers' compression losses are summed and weighted by λ=0.03\lambda=0.03, with boundary-noise parameter τ=6\tau=6. Training uses noisy boundary sampling; inference uses deterministic boundary decisions.

    SingleCR and Multi-Ratio both use 500B original pretraining tokens, counted before view expansion. Multi-Ratio reuses the same targets across its three views and averages their losses, preserving each original target's aggregate loss weight. Its additional computation comes from backbone/decoder processing of all three views after the shared encoder pass. For inference, Large/Medium/Small match the target activated FLOPs of CR2/CR4/CR8, excluding attention-map computation, at relative budgets 1:35:251:\frac{3}{5}:\frac{2}{5}. Reported task FLOPs use realized compression ratios and input/output lengths, including attention-map computation.

AuthorsOpenReview ↗

Part 2/2: SingleCR8 Prompt Adjustments and FLOPs Accounting

Q1

Thank you for asking us to clarify the prompt adjustments and whether they apply across compression ratios.

  1. The checkpoint-specific configuration concerns SingleCR8. Appendix B.4 refers to settings used to mitigate <think> loops in the independently trained SingleCR8 baseline. For each task, our Multi-Ratio checkpoint uses the same generation prompt and decoding settings across CR2, CR4, CR8, and Mixed; changing the compute budget does not require a new prompt. The heuristic experiments in W1 also evaluate its benefits without external LLM routing.

  2. Concrete changes and their effects. SingleCR8 sometimes reopened lengthy <think> blocks during multi-turn code editing, causing truncated outputs. We added instructions such as "return only the complete executable code" and "preserve the original return-container type," adjusted the initial-turn input format and history handling, and increased the output cap from 1,024 to 2,048 tokens. These changes mitigated the failures and improved Combined Joint accuracy from 7.75% to 18.02%.

    Similar thinking loops are documented in the official Qwen3.5-0.8B and Qwen3.5-2B model cards, including under recommended thinking-mode sampling settings. We will revise Appendix B.4 to identify the affected checkpoint, configuration changes, and measured effects explicitly.

Q2

Thank you for requesting a more detailed FLOPs breakdown to make our compute estimates easier to assess.

We count one multiply-add as two FLOPs and sum component costs using their matrix dimensions, sequence lengths, and active experts. Let NN be the original token count and MM the compressed concept count:

Fprefill=Fencoder(N)+Fchunk(N,M)+Fbackbone(M)+Fdechunk(M,N)+Fdecoder(N)+FIO.F_{\mathrm{prefill}} = F_{\mathrm{encoder}}(N) + F_{\mathrm{chunk}}(N,M) + F_{\mathrm{backbone}}(M) + F_{\mathrm{dechunk}}(M,N) + F_{\mathrm{decoder}}(N) + F_{\mathrm{IO}}.

  1. Transformer computation in the encoder, backbone, and decoder. A linear projection applied at nn positions costs

    Flinear=2ndindout.F_{\mathrm{linear}}=2n d_{\mathrm{in}}d_{\mathrm{out}}.

    Q, K, V, and attention output projections are counted separately using their actual dimensions. For an attention branch with hh query heads and head dimension dhd_h, the combined cost of QK⊤QK^\top and attention–value multiplication is

    Fattention=4hdh∣A∣.F_{\mathrm{attention}}=4h d_h|\mathcal A|.

    Here, ∣A∣|\mathcal A| counts query–key pairs allowed by the mask: n(n+1)/2n(n+1)/2 for global causal prefill, or the union of the local window and global prefix for sliding-window attention. Both attention branches are counted separately. For a MoE layer with EE experts, kk active experts per position, hidden width dd, and expert intermediate width ff,

    FMoE=2ndE+6ndfk.F_{\mathrm{MoE}}=2ndE+6ndfk.

    The terms represent expert-routing projection and the three projections of each active expert. These costs are summed over layers, using token lengths for the encoder/decoder and concept lengths for the backbone.

  2. Chunk, DeChunk, and input/output projections. Let ded_e, dcd_c, and ddd_d denote encoder, backbone, and decoder widths. Chunk includes boundary-prediction projections, adjacent-representation dot products, and the concept up-projection:

    =4(N-1)d_e^2+2(N-1)d_e+2Md_ed_c.$$ DeChunk includes an estimated five operations per concept dimension for its linear scan and the down-projection after restoring token resolution: $$F_{\mathrm{dechunk}}=5Md_c+2Nd_cd_d.$$ $F_{\mathrm{IO}}$ includes the input projection, down-projection before the encoder, up-projection after the decoder, and LM head, using the linear-projection formula above. Prefill computes logits only at the final position; KV-only history reconstruction does not execute the LM head.
  3. Realized compression, decoding, and a numerical example. We use measured prefill/decode compression ratios rprefillr_{\mathrm{prefill}} and rdecoder_{\mathrm{decode}}, with M≈N/rprefillM\approx N/r_{\mathrm{prefill}}. During decoding, encoder and decoder computation runs for every token; expected backbone and associated concept-operation costs are weighted by the concept-generation rate 1/rdecode1/r_{\mathrm{decode}}. Attention uses the corresponding history lengths.

    For the paper's CodeChat Mixed result under the standardized full-history workload,

    F_{\mathrm{decode}}=0.0877\ \mathrm{PF},$$ giving **1.0999 PF** in total. This theoretical accounting includes attention matrix multiplications, MoE routing and active experts, boundary projections, and the DeChunk scan. It excludes the external routing model, normalization, activations, softmax, and memory-transfer overhead. We will add these formulas and counting conventions to the appendix.

Thanks for the author's response. The new explanations and experiments address several details missing from the initial submission, particularly the comparison between heuristic routing and the external LLM. While the heuristic policy performs somewhat worse than the external LLM on tasks like CodeChat, it still retains an overall advantage. I encourage the authors to incorporate these additions into the paper and further discuss alternatives to external LLM judgment. The current results suggest that a more complex external policy does not completely outperform simple heuristic switching, making the choice of a balanced heuristic policy an additional important consideration for this method's application.

Based on these responses, I will maintain my positive score.

AuthorsOpenReview ↗

Thank You for Your Feedback

Thank you for your positive assessment and helpful suggestions. We have incorporated the additional experiments and clarifications into the revised appendix, including the heuristic-routing comparison in Appendix G.1.

Reviewer_xfLJOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper aims to enable the same model to flexibly adjust its computational load in multiple rounds of conversation. The authors control the computational cost using different compression levels and can continue to use the previous historical state when switching between levels. To make the use of multiple compression levels more effective, the model is exposed to content with different compression levels during training. The paper compares the performance of fixed and mixed compression levels and analyzes how much additional computation caused by switching models can be saved by reusing the previous state.

Critical strengths

S1. Switching compression levels within the same model while continuing to use the previous state is a straightforward design that is suitable for multiple rounds of conversation.

S2. The cross-ratio training experiments are quite persuasive. The model may use several compression levels separately, but this does not necessarily mean that it can perform well when mixed. The author's control experiments show that having the model encounter such mixed situations in advance is indeed helpful.

S3. The overall performance of the shared model across the fixed compression levels is also close to that of the separately trained models. This indicates that supporting multiple levels does not significantly sacrifice overall capability.

Critical weaknesses

W1. The paper has not clearly explained in which scenarios changing the compression ratio is more advantageous compared to directly adjusting the CoT length of the same model. It is hoped to compare under similar answer quality or computational budget to illustrate what additional benefits this compression mechanism brings.

W2. The model size and training investment have not been clearly explained. Especially, this paper uses three compressed versions of the same input for training. It is difficult to determine whether the performance improvement comes from the method itself or the different training investments.

W3. The cost accounting combines the base cost of the last round and the history reconstruction costs across the entire conversation. It is not clear what the relationship is between this and the total cost of completing the entire conversation. It is hoped to list it by round to clarify what the savings ratio reported measures.

W4. It is not clear how much increasing computing power can help when encountering problems that rely on previous conversation content compressed at a higher ratio. It is hoped to compare the situation where the current round uses high computing power but the previous history uses different compression ratios to see if the previous choice limits the subsequent performance.

Questions for authors

Q1. Can the same model be used on tasks suitable for CoT, with similar computational budgets, to compare the quality of answers between adjusting the compression ratio and adjusting the length of CoT?

Q2. Can specific model configurations and training budgets be provided? How is the loss combined when using three compressed views, and is the training computation and supervision amount comparable to the independent baseline trained separately?

Q3. In problems that rely on earlier details, how will fixing the current round to high computing power and changing the historical compression ratio affect the quality of the answers?

7 comments
AuthorsOpenReview ↗

Part 1/2: CoT Length and Historical Compression Experiments

W1 + Q1

Thank you for asking how compression ratio control compares with adjusting CoT length.

  1. Compression ratio and CoT length provide complementary controls over computation. Adjusting CoT length changes the number of tokens generated, while adjusting the compression ratio changes the number of concepts representing those tokens in the concept backbone, and thus the cost of processing a sequence of a given length. When compute is limited and a longer solution is desirable, generation length and compression ratio can be chosen jointly. Compression provides an additional option by reducing the backbone cost of processing a longer generated sequence.

  2. A full GSM8K evaluation shows that jointly choosing these controls can retain longer solutions at lower compute and similar accuracy. Using the same Multi-Ratio checkpoint, we report results on all 1,319 GSM8K test questions for two generation prompts, each evaluated at CR2/CR4/CR8, giving six configurations. We call the two strategies Compact reasoning and Expanded reasoning, reflecting their observed output lengths. The former requests only key equations or essential deductions, omitting repetition and explanations of obvious calculations; the latter requests key calculations and the final answer. We use actual generated lengths to quantify the difference between these strategies.

    All configurations use the same assistant prefix, temperature 0, a single generation call, and a common 1,024-token output cap. Accuracy checks numerical equality between the final answer and the reference; missing or unparseable answers count as incorrect. Theoretical FLOPs use actual input and output lengths and measured compression statistics, including all prefill and decode computation.

    Generation promptCompression ratioMean output tokensAccuracyMean TFLOPs/question
    Compact reasoningCR2112.673.62%0.5097
    Compact reasoningCR4132.470.36%0.4008
    Compact reasoningCR8147.362.70%0.3285
    Expanded reasoningCR2199.181.50%0.6419
    Expanded reasoningCR4206.474.45%0.4653
    Expanded reasoningCR8213.667.17%0.3820

    Compared with CR2 + Compact reasoning, CR4 + Expanded reasoning increases mean output length from 112.6 to 206.4 tokens while reducing theoretical FLOPs from 0.5097 to 0.4653 TFLOPs per question, a reduction of 8.71%. Accuracy is 73.62% for the former and 74.45% for the latter.

    This comparison demonstrates a compute benefit at similar observed answer quality: on multistep mathematical problems, jointly choosing compression ratio and generation strategy can retain a longer solution at lower compute, providing an additional option for allocating inference budgets.

W4 + Q3

Thank you for raising the question of how historical compression affects later turns.

We added a controlled experiment on 690 follow-up turns from 280 CoSQL dialogues, including a 168-question subset with lexical cues to earlier context, such as "those," "their," or "previous." Both intermediate and final follow-ups are evaluated as separate examples with frozen preceding histories; newly generated answers do not enter other examples' histories.

All comparisons hold the checkpoint, historical user instructions and SQL responses, current question, prompts, decoding settings, and output cap fixed. System/schema messages always use CR2. We vary historical compression across CR2/CR4/CR8 while keeping the current question and answer generation at CR2, and include a CR8-history/CR8-current-turn comparison. Scoring uses the same SQL execution accuracy metric as the main experiments.

History CRCurrent-turn CRAccuracy: all follow-ups (690)Accuracy: reference-cue subset (168)
CR2CR252.32%30.95%
CR4CR252.03%31.55%
CR8CR251.16%30.36%
CR8CR843.77%26.19%
  1. Effect of historical compression with the current turn fixed at CR2. Accuracy is 52.32%, 52.03%, and 51.16% for history at CR2, CR4, and CR8, respectively. Relative to CR2 history, the reductions are 0.29 and 1.16 percentage points for CR4 and CR8. On the reference-cue subset, changing history from CR2 to CR8 reduces accuracy by 0.60 points. These differences quantify the effect of historical compression on subsequent high-compute answers in this evaluation.

  2. Increasing current-turn compute remains effective with CR8 history. Changing the current turn from CR8 to CR2 improves accuracy from 43.77% to 51.16% overall (+7.39 points) and from 26.19% to 30.36% on the reference-cue subset (+4.17 points). Thus, even with history processed at CR8, allocating more compute to the current turn improves answer quality on these CoSQL examples.

AuthorsOpenReview ↗

Part 2/2: Training Objective, Training Budget, and FLOPs Accounting

W2 + Q2

Thank you for asking us to clarify the training objective and how the training budgets are compared.

  1. Model settings and the three-view training objective. Our model has 30B total / 0.5B activated parameters, with 500B pretraining / 5B SFT tokens, cosine learning-rate decay, and training on Blackwell GPUs. Three independently sampled compression views share the same target text and supervision mask. Each view's language-modeling loss is averaged over valid supervision positions, and the final loss averages the three views:

    =\frac{1}{3}\sum_{v=1}^{3} \left(\frac{1}{|\mathcal S|}\sum_{t\in\mathcal S}\ell_{v,t}\right).$$ Here, $\mathcal S$ contains valid supervision positions and $\ell_{v,t}$ is cross-entropy at position $t$ in view $v$. SingleCR uses the same mean over valid positions; Multi-Ratio additionally weights each view by $1/3$.
  2. Training investment and evidence for the method's contribution. SingleCR and Multi-Ratio use the same original-input token budget: 500B pretraining tokens, counted before view expansion. Multi-Ratio's three compression views reuse the same target text and supervision mask. Each target is predicted under all three views, and the losses are averaged, so each original target has the same aggregate weight in the total language-modeling loss as in SingleCR. The additional training computation comes from processing all three views through the backbone and decoder after one shared encoder pass.

    To examine the contribution of cross-ratio exposure with the view count held fixed, our ablation compares three single-ratio views, one single-ratio plus two cross-ratio views, and three cross-ratio views. All configurations use three views per input and the same number of supervised predictions. Single-ratio views use CR2/CR4/CR8 respectively; cross-ratio views sample the three ratios uniformly per segment, balancing token allocation in expectation. Fully cross-ratio training improves mixed-mode accuracy from 37.65% to 43.78% on CodeChat and 43.00% to 51.84% on CoSQL. This comparison supports the contribution of the training pattern itself to cross-turn budget composition. We will add the configuration, loss normalization, and training-budget definitions to the revision.

W3

Thank you for pointing out that our description of the FLOPs accounting needs clarification.

  1. What the full-history workload includes. The paper's “last round” refers to a full-history workload containing all user instructions up to that point and all preceding model responses. Earlier turns' inputs and outputs are processed as historical tokens during prefill, together with the current user input; the new response is counted under decode. All configurations use the same reference input/output lengths. We will consistently use “full-history workload” in the revision.

  2. Switching costs and the meaning of the savings ratio. For independent-model routing, each switching turn additionally incurs the cost of rebuilding historical KV on the selected model; a turn without a switch adds no reconstruction cost. These costs are accumulated across switches. Multi-Ratio reuses its shared KV state across ratio changes. The reported 78.4% / 46.9% reductions compare Multi-Ratio's full-history workload cost with the baseline's full-history workload plus cumulative switching-related reconstruction. This metric uses a standardized reference workload to compare model-processing costs together with switching-related reconstruction: earlier turns' content is counted in the full-history prefill, and reconstruction costs accumulate at switching turns. It does not sum the FLOPs of separately generating every turn. We will clarify this in the revised paper.

AuthorsOpenReview ↗

Kind Reminder to Reviewer xfLJ

Dear Reviewer xfLJ,

Thank you again for your careful review and helpful feedback. We would like to follow up on our responses to your comments.

In our rebuttal, we added experiments on jointly controlling CoT length and compression ratio, as well as the effect of historical compression on subsequent turns. We also clarified the three-view training objective, training-budget comparisons, and FLOPs accounting.

We would greatly appreciate your feedback on whether these clarifications and additional results address your concerns. If any questions remain, we would be happy to discuss them further and provide additional clarification.

Best regards,
The Authors

Reviewer_2c8hOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper asks whether one shared model can expose several compute budgets so multi-turn conversations spend more on hard turns and less on easy ones without switching models and rebuilding KV state. Multi-Ratio ConceptMoE extends token-to-concept compression with ratio-specific boundary routers over a shared encoder, a narrowed full-token encoder/decoder to lower the compression-independent FLOPs floor, and cross-ratio training that packs segments compressed at different ratios into one sequence (forced chunk boundaries at segment starts) so the backbone handles mixed-granularity contexts and KV state stays reusable across ratio changes. Validation: fixed points on PTBench/SFTBench aggregates mixed-ratio inference on CodeChat and CoSQL under frozen prefix-isolated LLM-judge turn schedules with realized-ratio FLOPs accounting, a switching-overhead comparison that charges the routed baseline for history re-prefill, and a cross-ratio exposure ablation.

Critical strengths
  • The question is practically well-posed and the mechanism coherent: forced-boundary segments make mixed-ratio contexts trainable while keeping one shared KV state valid across ratio changes, and the exposure ablation (Table 5: +6.14/+8.84 points at comparable or lower FLOPs) cleanly shows mixed-ratio capability must be trained in, not composed at inference.
  • The multi-turn experiments are methodologically careful: the same frozen, prefix-isolated judge schedule drives both arms, FLOPs use workload-realized rather than nominal compression ratios, and the routed baseline is explicitly charged for history re-prefill instead of being silently strawmanned.
Critical weaknesses
  • The headline 78.4%/46.9% switching savings rest on an accounting asymmetry: the routed baseline is forced to fully re-prefill history at every switch while warm-KV or KV-compatible alternatives are never costed, and the gpt-5.6-terra judge that decides every turn's ratio is excluded from the FLOPs bill entirely — its per-turn prompt (problem plus all visible instructions) is not free, so the practical saving is unbounded from above by anything the paper reports.
  • Auditability and generality are thin: no model sizes, training tokens, or hardware appear anywhere, so the compute-matched baseline claim cannot be checked; the shared model loses to SingleCR specialists on PTBench at CR2/CR4 (32.1/29.3 vs 33.2/30.0); and the mixed-ratio payoff — the paper's central promise — is demonstrated on only two multi-turn benchmarks, with no wall-clock latency anywhere.
Questions for authors
  1. what is the routing judge's per-turn token cost relative to the re-prefill it replaces, and does the mixed-ratio advantage survive a cheap small-model or feature-based router?
  2. please state model sizes, pretraining/SFT token counts and hardware; as submitted the compute-matched comparison cannot be audited.
2 comments
AuthorsOpenReview ↗

Part 1/2: Heuristic Routing and Warm-KV Compute Comparison

W1 + Q1

Thank you for highlighting these practical concerns about routing overhead and cache reuse.

  1. Routing overhead and heuristic routing. The FLOPs reported in the paper exclude the external LLM router. Its internal model configuration and execution details are unavailable, making this cost difficult to estimate accurately. To examine whether Multi-Ratio's benefits depend on an external LLM, we added deterministic heuristic routing experiments. The rules select compression ratios using only the current and preceding user instructions and static task information. They require simple rule matching on the CPU, with no additional model forward passes or LLM input/output tokens.

    Using the same Multi-Ratio checkpoint, generation prompts, and decoding settings, we obtain:

    DatasetLarge MoE accuracyHeuristic routing accuracyTheoretical FLOPs reduction vs. Large MoE
    CodeChat37.91%39.81%41.97%
    CoSQL49.16%50.94%13.47%

    On both tasks, heuristic routing achieves higher accuracy with fewer theoretical FLOPs. These results show that an external LLM router is not required for Multi-Ratio to deliver benefits: simple input-feature rules can also support effective compute-budget allocation across turns.

  2. Multi-Ratio retains compute savings when independent models preserve warm KV. To directly test whether the savings persist with warm KV, we added a theoretical FLOPs comparison that allows independent models to retain their own KV caches. CR2, CR4, and CR8 map to Large, Medium, and Small, respectively. All three models may retain their caches without eviction, and model-loading and cache-transfer costs are excluded.

    When a model is selected again, it reuses its valid cached prefix and processes only missing history and the current input; its first use requires full prefill. For example, under CR2 → CR4 → CR2, Large reuses its first-turn cache on the third turn and processes only the dialogue content added during the second turn and the new third-turn input.

    Using the same heuristic routes and reference input/output workload, we estimate theoretical FLOPs under the cache policies described above, following the paper's FLOPs accounting for Multi-Ratio:

    DatasetIndependent models: active-only KV (PF)Independent models: warm KV (PF)Multi-Ratio (PF)Reduction vs. warm-KV
    CodeChat6.14223.77741.082971.33%
    CoSQL0.64830.54180.312942.25%

    Retaining warm KV already reduces the independent-model baseline's computation by 38.50% on CodeChat and 16.42% on CoSQL. Against this stronger caching baseline, Multi-Ratio further reduces theoretical FLOPs by 71.33% and 42.25%, respectively.

    Independent models retain separate historical states: inactive models do not update their caches and must process the missing dialogue content when selected again. Multi-Ratio shares one backbone and valid historical states across compression ratios, allowing it to continually reuse previously processed history. Thus, Multi-Ratio's compute savings persist when independent models are allowed to retain and reuse warm KV.

AuthorsOpenReview ↗

Part 2/2: Model and Training Details, Specialist Comparisons, and Additional Experiments

W2 + Q2

Thank you for these helpful suggestions on clarifying the experimental setup and broadening the evaluation.

  1. Model and training settings. Our model has 30B total / 0.5B activated parameters, with 500B pretraining / 5B SFT tokens, cosine learning-rate decay, and training on Blackwell GPUs. SingleCR and Multi-Ratio both use 500B original pretraining tokens, counted before view expansion. We will add these details to the revision.

  2. Compute matching. Large/Medium/Small match the target inference FLOPs of CR2/CR4/CR8. At CR2, the encoder and decoder account for approximately 1/51/5 of computation and the concept backbone 4/54/5, giving

    \approx \frac{1}{5}+\frac{4}{5}\cdot\frac{2}{r}.$$ The matched budgets are therefore $1:3/5:2/5$, excluding attention-map computation. Reported task FLOPs use **realized compression ratios and input/output lengths, including attention**. Latency evaluation requires further engineering alignment of serving implementations, so we currently report theoretical FLOPs.
  3. Comparison with SingleCR specialists (Table 2). On PTBench, Multi-Ratio trails SingleCR by 1.1/0.7 points at CR2/CR4 but improves CR8 from 25.8 to 27.6; both average 29.7. On SFTBench, Multi-Ratio leads by 0.4/0.3/0.8 points at CR2/CR4/CR8, raising the average from 18.8 to 19.3. One checkpoint thus matches the specialists' average PTBench score and exceeds all three SFTBench scores, while supporting cross-turn budget changes and history-state reuse.

  4. Additional multi-turn benchmark: API-Bank. We evaluated 282 API calls from 142 dialogues in a multi-turn subset of API-Bank Level 1, using common reference histories and official execution-based scoring. All Multi-Ratio modes share one checkpoint. Mixed selects CR2/CR4/CR8 from current/prior user instructions using modification keywords, length, and constraints, without additional model calls. FLOPs sum prefill and decode across all evaluated calls.

    ConfigurationAccuracyFLOPs (PF)
    Large MoE52.84%0.2764
    Medium MoE30.14%0.1895
    Small MoE9.22%0.1287
    Multi-Ratio CR262.06%0.2616
    Multi-Ratio CR454.26%0.1764
    Multi-Ratio CR830.85%0.1383
    Multi-Ratio Mixed58.16%0.2045

    Each fixed mode exceeds its corresponding MoE baseline in accuracy. Mixed lies on the Pareto frontier of the compared configurations, improving accuracy by 5.32 points and reducing theoretical FLOPs by 26.02% versus Large MoE, extending the evidence to multi-turn tool use.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.