- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper addresses the varying computational demands across turns in multi-turn agentic interactions. It extends ConceptMoE to Multi-Ratio ConceptMoE, which supports multiple concept compression ratios within a single backbone and uses an external LLM judgement to assess the compute budget for each turn. The method explicitly introduces joint training with mixed compression ratios, allowing the shared backbone to process contexts containing multiple compression ratios. On downstream tasks such as CodeChat and CoSQL, it offers a more flexible accuracy–compute trade-off, e.g., achieving CR2-level performance with FLOPs comparable to CR4.
Critical strengths
- The simple yet effective training scheme randomly mixes segments with different compression ratios, enabling a single backbone to accommodate multiple ratios within the same context without re-prefill. The ablation results in Table 5 confirm the necessity of mixed-ratio training for downstream mixed-ratio inference.
- Mixed-ratio training does not significantly influence the concept model's underlying capabilities and can even improve them in the two coding tasks (Table 3).
Critical weaknesses
- Although mixed-ratio training enables the backbone to support multiple compression ratios almost as a free lunch, mixed-ratio inference still operates at the turn level and depends on an external LLM judge (gpt-5.6-terra in this paper). This dependence significantly limits the practical usability of Multi-Ratio ConceptMoE. The authors should also evaluate zero-overhead heuristic policies to determine whether the proposed method retains its benefits under these policies.
- Although the backbone follows ConceptMoE, the appendix should provide a self-contained description of the model architecture, training hyperparameters, and related settings to ensure reproducibility and fair comparisons. As presented, understanding the results depends heavily on first reading the ConceptMoE paper.
Questions for authors
- Appendix B.4 states that different checkpoints may require different prompt designs. Please provide concrete examples of these differences and explain their effects. If the checkpoints are this sensitive to specific inference settings, does this also suggest fragility in the training and inference design? Training requires specific mixtures of compression ratios, inference prompts must be designed separately, and turn-level interactions even require assessment by an external model. I believe that the current design and experimental results weaken the practical value of Multi-Ratio ConceptMoE to some extent. The authors should conduct further targeted evaluations of the design's robustness.
- The paper does not provide detailed formulas for calculating FLOPs. Excluding assessment by the external model, it does not explain how the computational costs of the core components in the encoder → chunk → main concept backbone → dechunk → decoder pipeline are calculated. It only briefly states that "The reported FLOPs exclude the cost of the external routing policy and characterize model execution under the frozen turn-wise schedules."
4 comments
Part 2/2: SingleCR8 Prompt Adjustments and FLOPs Accounting
Q1
Thank you for asking us to clarify the prompt adjustments and whether they apply across compression ratios.
-
The checkpoint-specific configuration concerns SingleCR8. Appendix B.4 refers to settings used to mitigate
<think>loops in the independently trained SingleCR8 baseline. For each task, our Multi-Ratio checkpoint uses the same generation prompt and decoding settings across CR2, CR4, CR8, and Mixed; changing the compute budget does not require a new prompt. The heuristic experiments in W1 also evaluate its benefits without external LLM routing. -
Concrete changes and their effects. SingleCR8 sometimes reopened lengthy
<think>blocks during multi-turn code editing, causing truncated outputs. We added instructions such as "return only the complete executable code" and "preserve the original return-container type," adjusted the initial-turn input format and history handling, and increased the output cap from 1,024 to 2,048 tokens. These changes mitigated the failures and improved Combined Joint accuracy from 7.75% to 18.02%.Similar thinking loops are documented in the official Qwen3.5-0.8B and Qwen3.5-2B model cards, including under recommended thinking-mode sampling settings. We will revise Appendix B.4 to identify the affected checkpoint, configuration changes, and measured effects explicitly.
Q2
Thank you for requesting a more detailed FLOPs breakdown to make our compute estimates easier to assess.
We count one multiply-add as two FLOPs and sum component costs using their matrix dimensions, sequence lengths, and active experts. Let be the original token count and the compressed concept count:
-
Transformer computation in the encoder, backbone, and decoder. A linear projection applied at positions costs
Q, K, V, and attention output projections are counted separately using their actual dimensions. For an attention branch with query heads and head dimension , the combined cost of and attention–value multiplication is
Here, counts query–key pairs allowed by the mask: for global causal prefill, or the union of the local window and global prefix for sliding-window attention. Both attention branches are counted separately. For a MoE layer with experts, active experts per position, hidden width , and expert intermediate width ,
The terms represent expert-routing projection and the three projections of each active expert. These costs are summed over layers, using token lengths for the encoder/decoder and concept lengths for the backbone.
-
Chunk, DeChunk, and input/output projections. Let , , and denote encoder, backbone, and decoder widths. Chunk includes boundary-prediction projections, adjacent-representation dot products, and the concept up-projection:
=4(N-1)d_e^2+2(N-1)d_e+2Md_ed_c.$$ DeChunk includes an estimated five operations per concept dimension for its linear scan and the down-projection after restoring token resolution: $$F_{\mathrm{dechunk}}=5Md_c+2Nd_cd_d.$$ $F_{\mathrm{IO}}$ includes the input projection, down-projection before the encoder, up-projection after the decoder, and LM head, using the linear-projection formula above. Prefill computes logits only at the final position; KV-only history reconstruction does not execute the LM head. -
Realized compression, decoding, and a numerical example. We use measured prefill/decode compression ratios and , with . During decoding, encoder and decoder computation runs for every token; expected backbone and associated concept-operation costs are weighted by the concept-generation rate . Attention uses the corresponding history lengths.
For the paper's CodeChat Mixed result under the standardized full-history workload,
F_{\mathrm{decode}}=0.0877\ \mathrm{PF},$$ giving **1.0999 PF** in total. This theoretical accounting includes attention matrix multiplications, MoE routing and active experts, boundary projections, and the DeChunk scan. It excludes the external routing model, normalization, activations, softmax, and memory-transfer overhead. We will add these formulas and counting conventions to the appendix.
Thanks for the author's response. The new explanations and experiments address several details missing from the initial submission, particularly the comparison between heuristic routing and the external LLM. While the heuristic policy performs somewhat worse than the external LLM on tasks like CodeChat, it still retains an overall advantage. I encourage the authors to incorporate these additions into the paper and further discuss alternatives to external LLM judgment. The current results suggest that a more complex external policy does not completely outperform simple heuristic switching, making the choice of a balanced heuristic policy an additional important consideration for this method's application.
Based on these responses, I will maintain my positive score.
Thank You for Your Feedback
Thank you for your positive assessment and helpful suggestions. We have incorporated the additional experiments and clarifications into the revised appendix, including the heuristic-routing comparison in Appendix G.1.
Part 1/2: Heuristic Routing, Architecture, and Training Details
W1
Thank you for raising this practical concern about the additional cost of using an external LLM router.
We added deterministic heuristic routing experiments on the complete CodeChat and CoSQL evaluation sets, using the same Multi-Ratio checkpoint, generation prompts, and decoding settings as the main experiments. Rules use only current/prior user instructions and static task information such as the database schema: complex or high-risk requests use CR2, ordinary follow-ups CR4, and clearly local modifications CR8. The rules are frozen before generation and access no future turns, model outputs, or correctness feedback. They require only CPU rule matching, with no additional model calls or LLM input/output tokens.
CodeChat uses Combined Joint accuracy and CoSQL uses SQL execution accuracy. FLOPs use a common full-history workload across configurations within each dataset. Heuristic routing achieves higher accuracy than Large MoE with fewer theoretical FLOPs on both tasks, showing that an external LLM router is not required to obtain Multi-Ratio's benefits. We will include the results and explicit routing rules in the revision.
W2
Thank you for suggesting that we make the architecture and training details clearer in the paper.
We will consolidate the following architecture and training details in the appendix so the experimental setup can be understood directly from this paper.
Architecture and cross-ratio training. The pipeline is encoder → Chunk → concept backbone → DeChunk → decoder. The encoder processes the full token sequence; Chunk groups tokens into concepts using predicted boundaries; the backbone processes these concepts; and DeChunk restores token resolution for the decoder. CR2/CR4/CR8 have separate lightweight boundary routers and share all other modules. Encoder and decoder hidden widths are approximately 2/3 of the backbone width.
Each input is encoded once, then expanded into three independently sampled cross-ratio views. Within each view, interior segment lengths are sampled uniformly from 64–960 tokens (expected length 512); a residual suffix shorter than 64 tokens is merged into the preceding segment. Each segment independently selects CR2/CR4/CR8 with equal probability. A forced boundary starts each segment, and its router predicts the remaining boundaries. The backbone and decoder process the views with separate causal sequence boundaries.
Training settings and comparison budgets. The model has 30B total / 0.5B activated parameters, with 500B pretraining / 5B SFT tokens, cosine learning-rate decay, and training on Blackwell GPUs. The three views share target text and supervision positions. Each view's language-modeling loss is averaged over valid positions, then the three losses are averaged. Compression control follows ConceptMoE's auxiliary objective: the three routers' compression losses are summed and weighted by λ=0.03, with boundary-noise parameter τ=6. Training uses noisy boundary sampling; inference uses deterministic boundary decisions.
SingleCR and Multi-Ratio both use 500B original pretraining tokens, counted before view expansion. Multi-Ratio reuses the same targets across its three views and averages their losses, preserving each original target's aggregate loss weight. Its additional computation comes from backbone/decoder processing of all three views after the shared encoder pass. For inference, Large/Medium/Small match the target activated FLOPs of CR2/CR4/CR8, excluding attention-map computation, at relative budgets 1:53:52. Reported task FLOPs use realized compression ratios and input/output lengths, including attention-map computation.