acceptodds
Under review as a conference paper at ICLR 2027

When Less is Enough: Efficient Inference via Collaborative Reasoning

Abstract

In this work, we introduce DUET (Dual-model Efficient Two-stage inference), a collaborative inference framework in which a capable model and a lightweight model work together to solve a task. Relying on a single large model to perform end-to-end reasoning and prediction often incurs substantial inference cost. In contrast, DUET decomposes inference into two stages: the capable model produces a reasoning signal, and the lightweight model interprets this signal to generate the final answer, allowing reasoning-intensive computation to be handled by the capable model while non–reasoning-intensive components are delegated to the lightweight model without sacrificing task performance. To achieve this objective we propose a length-penalized joint training objective that encourages the capable model to transmit only the information that is sufficient for the lightweight model to solve the task. As a result, DUET maintains strong reasoning performance with substantially lower inference cost than end-to-end inference using a large model alone, saving up to 60% of the large model’s output tokens on challenging reasoning benchmarks, including AIME and GPQA.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large language models, Reinforcement learning, Efficient reasoning, Model collaboration

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_8mL5OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes DUET, a two-stage inference framework in which a capable model generates a reasoning signal and a lightweight model uses that signal to produce the final answer. Both models are trained with GRPO. The capable model receives a reward based on the improvement over the lightweight model's standalone performance, together with an adaptive length penalty; the lightweight model receives the task reward. Experiments use Qwen3-4B and Qwen3-0.6B, several mathematical training datasets, and five reasoning benchmarks. The authors report substantial reductions in large-model output tokens, with ablations examining joint training and the reward components.

Critical strengths

1. A practical approach to reducing expensive-model generation.

Delegating final-answer generation to a lightweight model offers a useful route to reducing inference cost. Table 1 reports a reduction from 6,948 to 1,800 large-model tokens on AMC23, with accuracy changing from 0.90 to 0.88. This is a potentially valuable efficiency trade-off, even though it does not establish unchanged accuracy or an equivalent reduction in end-to-end latency.

2. Evaluation across multiple training distributions and reasoning tasks.

The paper evaluates several mathematical training datasets and five reasoning benchmarks, and includes comparisons with concise-reasoning baselines and component ablations. This breadth is useful for assessing where the method works and where it encounters limitations, provided the reported results and experimental settings are reconciled.

Critical weaknesses

1. The distinctive contribution is not sufficiently isolated or validated.

The two-stage architecture closely resembles the general principle of model delegation: a capable model provides guidance and a smaller model completes the response. The more specific contribution lies in the training objective and its adaptive penalty. Although the paper includes a fixed-decoder comparison and reward ablations, these do not sufficiently establish the distinctive benefit over simpler delegation under matched accuracy, computational budgets, and clearly documented regularization settings. The reported token reductions also do not establish that the models have learned the proposed division between reasoning-intensive computation and lightweight response generation. Without stronger controlled comparisons or analysis of the intermediate signals, the methodological contribution and its explanatory value remain limited.

2. Inconsistent numerical reporting undermines confidence in the experimental results.

For MATH-LightEval training, Tables 9 and 11 report 52.3% accuracy and 2,690 tokens on GPQA, implying an IPT of approximately 0.194. However, Table 5 reports 0.162 at 100 training steps. The entire GPQA triplet in Table 5 is also identical for MATH-LightEval and DAPO despite their different results in Table 2. These discrepancies cannot be explained by ordinary rounding. Additionally, the DeepScaleR GPQA result is reported as 0.48 accuracy in Tables 1, 3, and 4, but as 0.47 in Table 2 and 47.2% in Table 8; the run/checkpoint relationship is not explained.

The key ablation introduces another ambiguity. Table 6 reports 27% accuracy and zero output tokens for the -MU-LPS variant on AMC23, whereas Table 12 reports 77.5% accuracy and 1,394 tokens with KL regularization; the collapsed result corresponds closely to the no-KL condition. Since KL regularization is described as part of the default setup, it is unclear whether this ablation changes an additional factor. This affects the attribution of training collapse to the removal of marginal utility and the penalty schedule.

These inconsistencies affect evidence supporting both efficiency and component-level contributions. Until the authors reconcile them, I cannot confidently assess the empirical claims or support acceptance in the paper's current form.

Questions for authors
  1. What evidence distinguishes DUET's contribution from existing large-small model collaboration or delegation approaches? Could controlled comparisons establish the additional benefit of the proposed marginal-utility objective and adaptive length penalty beyond simpler alternatives?

  2. Could the authors provide accurate, internally consistent numerical results and reconcile the discrepancies across Tables 1-6 and 8-12? Please identify the corresponding runs, checkpoints, and KL configurations, and provide the underlying evaluation records needed to verify accuracy, token counts, and IPT.

Reviewer_ZYi2OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

To promote efficiency reasoning, the paper proposes a large-small collaboration framework, where a large model outputs a short chain-of-thought and a small model outputs the final answer based on the CoT. The paper employs a GRPO-style scheme to train both the large and the small model.

Critical strengths
  1. The motivation is reasonable, since the reasoning efficiency is a significant long-standing problem.
  2. The overall framework is easy to understand and follow. And the experiments show the promising efficiency improvement
Critical weaknesses
  1. It would be better to test more models and larger models. The experiments only use one group of large-small models, i.e., Qwen3-4B. Including more SoTA models (such as Qwen3.5/3.8) and larger models (8B/14B/32B) would enhance the paper.

  2. The baselines are not enough. There are some related efficient reasoning baselines that can be included. For example, early-exit methods (e.g., DEER [1]), adaptive reasoning methods (AutoThink [2]).

  3. Table 1 shows the proposed method can degrade the accuracy a lot on difficulty benchmarks, such as GPQA-Diamond.

  4. The paper states “We summarize our main results in Figure 4.” However, Figure 4 only compares with a naïve baseline called Naive Truncation, which seems to be unfair. The proposed method is trained, while Naive Truncation is zero-shot. Besides, could Figure 4 include more baselines for a better comparison.

  5. The paper does not provide a qualitative sample of the proposed dual-model collaboration decoding. Including a qualitative sample would make the decoding process more straightforward and easier to understand.


References

[1] Dynamic early exit in reasoning models

[2] Learning When to Think: Shaping Adaptive Reasoning in R1-Style Models via Multi-Stage RL

Questions for authors

The paper claims that when the small model is strong, directly optimizing M can result in an excessively short CoT. So it proposes to add an extra reward term “−R(x,ym)”. And the reward is expected to introduce random noise for the GRPO optimization. A question is that when the small model is strong enough, is there still such noise? Will the reward always be 1 in multiple sampling? In such cases, this reward also seems to be invalid. Could you provide an analysis of the actual reward curves during training.

Reviewer_ckUHOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes DUET (Dual-model Efficient Two-stage inference), a framework designed to reduce the inference cost of large language models through collaboration between large and small models. The core idea is to decouple complex reasoning from final answer generation: a large model (Qwen3-4B) generates a concise, compressed reasoning signal containing essential information, while a small model (Qwen3-0.6B) produces the final answer based on this signal. The main methodological contribution is the alternating joint training of both models using GRPO, together with a marginal-utility reward and an adaptive length-penalty mechanism. These designs encourage the large model to preserve the reasoning information most valuable for the small model to complete the task while reducing redundant tokens.

The experiments use MATH-LightEval, DeepScaleR, and DAPO-Math-17k for training, and evaluate the framework on MATH500, AMC23, AIME2024, AIME2025, and GPQA-Diamond. The results show that, compared with the original large model, DUET reduces large-model output tokens by approximately 50%–75% while maintaining comparable accuracy, and decreases actual inference time by approximately 34%. Compared with naive truncation, single-model compression, and other concise reasoning methods, DUET generally achieves a more competitive trade-off between accuracy and inference cost. Ablation studies further demonstrate that joint training, marginal-utility rewards, and adaptive length penalties contribute to inference efficiency and training stability. However, the compression benefits decrease significantly when the large model itself lacks sufficient problem-solving capability. Overall, the main innovation of this paper lies in extending traditional single-model chain-of-thought compression to task-relevant reasoning information compression and collaborative optimization for downstream small models.

Critical strengths
  1. Dual-model collaboration significantly reduces inference cost.DUET decouples complex reasoning from answer generation, assigning essential reasoning to the large model and final response generation to the small model. While maintaining accuracy comparable to the original model, it reduces large-model output tokens by approximately 50%–75% and shortens actual inference time by approximately 34%.

  2. Task-oriented reasoning information compression improves token efficiency.Unlike traditional chain-of-thought truncation or single-model compression, DUET uses joint training to encourage the large model to generate compressed reasoning signals containing the information most valuable for the small model to complete the task. This reduces redundant information and achieves a more competitive trade-off between accuracy and inference cost.

Critical weaknesses
  1. Limited model-scale evaluation and insufficient cross-scale generalization experiments (Page 5, Section 3.1; Page 9, Limitation).The paper primarily uses Qwen3-4B as the large model and Qwen3-0.6B as the small model. Although the appendix experiments with an alternative small model, the large model remains unchanged throughout all experiments, and no evaluations are conducted with larger models. Therefore, it remains unclear whether DUET can achieve comparable reasoning compression benefits across different large-model scales.

  2. Lack of direct validation of the actual role of the compressed reasoning signal (Page 3, Section 2; Page 8, Section 3.2.4). The central idea of the paper is that the large model generates a compressed signal z containing essential reasoning information, which the small model then uses to produce the final answer. However, the experiments mainly evaluate effectiveness through accuracy and token consumption, without presenting concrete examples of the compressed signals or including an "answer-only" baseline in which only the final answer is transmitted. Consequently, the current results cannot distinguish whether the small model genuinely utilizes compressed intermediate reasoning information or primarily generates its output based on an answer already obtained by the large model. This leaves the independent contribution of the dual-model reasoning decomposition insufficiently validated.

3.Ambiguity in the experimental configurations of key ablation studies (Page 20, Table 6; Page 24, Table 12).Table 6 shows that removing both marginal utility and the length-penalty schedule leads to training collapse, with the large model generating zero output tokens. However, Table 12 indicates that, when KL regularization is retained, removing the same two components still achieves 77.5% accuracy on AMC23. Since Table 6 does not explicitly specify its KL regularization configuration, it is difficult to clearly distinguish the independent contributions of these components to training stability.

Questions for authors
  1. Regarding model generalization: The paper only uses Qwen3-4B as the large model. Have the authors considered evaluating DUET with larger models or different model architectures? To what extent do the current experimental results support the effectiveness of DUET across different model combinations?

  2. Regarding the effectiveness of compressed reasoning signals: How can the authors demonstrate that the compressed signal z generated by the large model genuinely contains essential intermediate reasoning information rather than merely the final answer? Have the authors considered introducing an answer-only baseline, in which only the final answer is transmitted, to validate the independent contribution of the dual-model reasoning decomposition?

  3. Regarding the consistency of ablation experiments: Table 6 on Page 20 shows that removing both marginal utility and the length-penalty schedule leads to training collapse. However, Table 12 on Page 24 shows that the same ablation setting still achieves 77.5% accuracy when KL regularization is retained. Was KL regularization applied in the experiments reported in Table 6? How can the authors control for the influence of KL regularization on training stability to clarify the independent contributions of the two key components?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.