- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes CostDescent, a training framework that helps models solve problems with fewer generated tokens and tool calls. Instead of adding a cost penalty to the reward, it uses task rewards to evaluate answer quality and provides cost goals as separate inputs to the model. During training, CostDescent collects the model’s attempts, uses better-performing and lower-cost attempts to build task-specific cost targets, and updates the model through goal relabeling. The paper also identifies conditions under which these updates can reduce cost locally. Experiments on six mathematical reasoning and knowledge-based question-answering benchmarks show that CostDescent achieves the best average task performance and tool productivity among the compared methods.
Critical strengths
- A meaningful approach to controlling reasoning cost. CostDescent uses task-specific cost goals while keeping resource costs out of the task reward. The model can learn more efficient ways to solve problems while task rewards remain focused on solution quality.
- Convincing evidence of a better performance–cost balance. CostDescent achieves the highest average task performance and tool productivity among the compared methods. In the reported training comparison with a baseline using only task rewards, it uses about 34% fewer tool calls and 18% fewer tokens while achieving a similar task score.
Critical weaknesses
The method cannot directly learn efficiency from equally rewarded answers. If all sampled answers are correct and receive the same reward, their GRPO advantages are zero, even when some answers use many more tokens or tool calls. CostDescent therefore cannot use these cost differences to favor cheaper correct solutions, and dynamic sampling discards such groups.
Questions for authors
Can CostDescent reduce costs on tasks the initial model already solves reliably?
2 comments
Thank you for the response. I will keep my positive score.
Response to Reviewer ZCAW
Response to W1: We thank the reviewer for pointing out this property and for the opportunity to clarify the scope of cost descent.
(1) Scope of Cost Descent. As noted in App. A.4, groups with zero reward variance carry zero advantage and contribute no policy gradient. Cost descent therefore acts only on groups containing task-accepted trajectories with positive advantage, as stated in our Limitations (App. E). Dynamic sampling discards only such zero-gradient groups, so it removes no signal that the current objective would use.
(2) Without and With This Signal. We acknowledge that the cost differences within all-correct groups are useful supervision, which the current design does not use. Even without this signal, CostDescent reduces cost on test questions already solved by the RFT checkpoint (Response to Q1). It uses fewer tool calls than RFT on both families and attains the highest TP on both, including against ET-Agent, whose reward directly distinguishes costs within all-correct groups. Our framework can also use this signal without changing the reward, by moving the target toward the cheaper correct trajectories in all-correct groups and reinforcing them at their relabeled goals. We leave this as an extension.
In the revised manuscript, we make this scope more explicit in the Limitations (App. E) and describe the extension above.
Response to Q1: Thank you for this question. Yes. Initially solved prompts carry no direct cost signal in our training, so any cost reduction on them comes through generalization from the updates on other prompts. The natural reference is ET-Agent, whose reward also distinguishes costs within all-correct groups and thus receives this signal directly. We therefore compare against ET-Agent as in Sec. 5.3, where ET-Agent and CostDescent are trained from the same RFT checkpoint on ET-Agent's Stage 1 data, and include outcome-only ARPO from the same comparison as a reference without any cost signal. We evaluate these runs, together with the RFT checkpoint itself, on the test questions that the RFT checkpoint already solves.
As shown in Table A, relative to RFT, CostDescent lowers tool calls on MR from 1.19 to 0.40 at comparable P, and also lowers them on KI while raising P. Compared with ET-Agent, CostDescent uses fewer tool calls on MR and attains higher P on both families. ET-Agent uses fewer tokens on both families and fewer calls on KI, but its KI P falls below that of RFT (81.9 vs. 90.7). As a result, CostDescent attains the highest TP on both families. Relative to outcome-only ARPO, it also uses fewer calls and tokens on both families. These results indicate that CostDescent lowers tool use on already solved tasks without this direct signal, and that this reduction goes beyond the effect of outcome-only RL. In the revised manuscript, we have added this analysis to App. D.3 (Table 5).
Table A. The runs of Sec. 5.3, trained with an identical pipeline from the same RFT checkpoint, evaluated on test questions already solved by that checkpoint.