acceptodds
Under review as a conference paper at ICLR 2027

CostDescent: Decoupling Cost From Rewards for Efficient Tool-Integrated Reasoning

Abstract

Tool-integrated reasoning (TIR) expands the capabilities of language models but incurs substantial token and tool-use costs. Efficient TIR therefore seeks to retain task performance at reduced cost. Existing approaches often incorporate resource cost into the reward, so that lower cost can be rewarded at the expense of task success. Subsequent refinements temper this interaction but still keep both in one learning signal. Drawing on goal-conditioned RL (GCRL), we instead let task reward alone evaluate outcome, while resource cost enters policy optimization as a goal input. We introduce , a goal-conditioned training framework that expresses these goals relative to a task-dependent policy-induced reference. From on-policy performance-cost outcomes, constructs a task-aware target reference favoring lower-cost behavior, relabels achieved costs around it, and applies a goal-routed update. We establish sufficient conditions under which this update induces local cost descent. Across six mathematical-reasoning and knowledge-intensive benchmarks, achieves the strongest overall task performance and tool productivity. These results establish task-aware cost descent as an effective approach to Efficient TIR and suggest pursuing behavioral targets outside the task reward in broader LLM reinforcement learning.

Primary area:
reinforcement learning
Keywords:
reinforced learning, agentic RL, efficient reasoning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_ZCAWOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes CostDescent, a training framework that helps models solve problems with fewer generated tokens and tool calls. Instead of adding a cost penalty to the reward, it uses task rewards to evaluate answer quality and provides cost goals as separate inputs to the model. During training, CostDescent collects the model’s attempts, uses better-performing and lower-cost attempts to build task-specific cost targets, and updates the model through goal relabeling. The paper also identifies conditions under which these updates can reduce cost locally. Experiments on six mathematical reasoning and knowledge-based question-answering benchmarks show that CostDescent achieves the best average task performance and tool productivity among the compared methods.

Critical strengths
  1. A meaningful approach to controlling reasoning cost. CostDescent uses task-specific cost goals while keeping resource costs out of the task reward. The model can learn more efficient ways to solve problems while task rewards remain focused on solution quality.
  2. Convincing evidence of a better performance–cost balance. CostDescent achieves the highest average task performance and tool productivity among the compared methods. In the reported training comparison with a baseline using only task rewards, it uses about 34% fewer tool calls and 18% fewer tokens while achieving a similar task score.
Critical weaknesses

The method cannot directly learn efficiency from equally rewarded answers. If all sampled answers are correct and receive the same reward, their GRPO advantages are zero, even when some answers use many more tokens or tool calls. CostDescent therefore cannot use these cost differences to favor cheaper correct solutions, and dynamic sampling discards such groups.

Questions for authors

Can CostDescent reduce costs on tasks the initial model already solves reliably?

2 comments
AuthorsOpenReview ↗

Response to Reviewer ZCAW

W1. The method cannot directly learn efficiency from equally rewarded answers. If all sampled answers are correct and receive the same reward, their GRPO advantages are zero, even when some answers use many more tokens or tool calls. CostDescent therefore cannot use these cost differences to favor cheaper correct solutions, and dynamic sampling discards such groups.

Response to W1: We thank the reviewer for pointing out this property and for the opportunity to clarify the scope of cost descent.

(1) Scope of Cost Descent. As noted in App. A.4, groups with zero reward variance carry zero advantage and contribute no policy gradient. Cost descent therefore acts only on groups containing task-accepted trajectories with positive advantage, as stated in our Limitations (App. E). Dynamic sampling discards only such zero-gradient groups, so it removes no signal that the current objective would use.

(2) Without and With This Signal. We acknowledge that the cost differences within all-correct groups are useful supervision, which the current design does not use. Even without this signal, CostDescent reduces cost on test questions already solved by the RFT checkpoint (Response to Q1). It uses fewer tool calls than RFT on both families and attains the highest TP on both, including against ET-Agent, whose reward directly distinguishes costs within all-correct groups. Our framework can also use this signal without changing the reward, by moving the target toward the cheaper correct trajectories in all-correct groups and reinforcing them at their relabeled goals. We leave this as an extension.

In the revised manuscript, we make this scope more explicit in the Limitations (App. E) and describe the extension above.

Q1. Can CostDescent reduce costs on tasks the initial model already solves reliably?

Response to Q1: Thank you for this question. Yes. Initially solved prompts carry no direct cost signal in our training, so any cost reduction on them comes through generalization from the updates on other prompts. The natural reference is ET-Agent, whose reward also distinguishes costs within all-correct groups and thus receives this signal directly. We therefore compare against ET-Agent as in Sec. 5.3, where ET-Agent and CostDescent are trained from the same RFT checkpoint on ET-Agent's Stage 1 data, and include outcome-only ARPO from the same comparison as a reference without any cost signal. We evaluate these runs, together with the RFT checkpoint itself, on the test questions that the RFT checkpoint already solves.

As shown in Table A, relative to RFT, CostDescent lowers tool calls on MR from 1.19 to 0.40 at comparable P, and also lowers them on KI while raising P. Compared with ET-Agent, CostDescent uses fewer tool calls on MR and attains higher P on both families. ET-Agent uses fewer tokens on both families and fewer calls on KI, but its KI P falls below that of RFT (81.9 vs. 90.7). As a result, CostDescent attains the highest TP on both families. Relative to outcome-only ARPO, it also uses fewer calls and tokens on both families. These results indicate that CostDescent lowers tool use on already solved tasks without this direct signal, and that this reduction goes beyond the effect of outcome-only RL. In the revised manuscript, we have added this analysis to App. D.3 (Table 5).

Table A. The runs of Sec. 5.3, trained with an identical pipeline from the same RFT checkpoint, evaluated on test questions already solved by that checkpoint.

MethodMR PMR callsMR tokensMR TPKI PKI callsKI tokensKI TP
RFT96.41.1932588.090.71.6815062.7
Outcome-only ARPO98.90.9836693.291.81.7116961.3
ET-Agent95.30.6013594.281.91.137166.8
CostDescent96.20.4034394.993.81.5114570.8

Thank you for the response. I will keep my positive score.

Reviewer_nkbFOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper targets Efficient Tool-Integrated Reasoning (TIR). Existing methods fold token and tool cost into the RL reward (or into a shared advantage), so reducing cost can trade off against task success. Drawing on goal-conditioned RL, the authors instead let the task reward alone evaluate outcome and pass resource cost to the policy as a goal input. Three pieces make this work: (i) Relative Cost Conditioning, where an absolute cost goal is expressed as a policy-induced reference b(q) = E[c] at g = 0 plus a signed offset, keeping goals aligned as the policy drifts; (ii) Cost-Aware Rollout, which splits each group between g = 0 and exploratory goals; and (iii) Task-Aware Cost Descent, which builds a lower-cost target reference ^b(q) from on-policy performance-cost outcomes, relabels achieved costs around it, and routes positive-advantage trajectories to the relabeled goal (with a "hindsight credit inversion" reweighting tokens by the inverted relabeled-to-original likelihood ratio). A local analysis (Prop. 2, informal) gives conditions under which the positive-advantage update is a descent direction on the reference gap. Empirically, on six benchmarks (AIME24, AMC23, MATH500, 2WikiMultiHopQA, Bamboogle, MuSiQue) with Qwen2.5-7B, CostDescent reports the best average accuracy (56.9) and tool productivity (48.6), plus five-run stability, a Llama-3.1-8B transfer, and no inference-time overhead since g = 0 is merged into the backbone.

Critical strengths

1. A distinct mechanism, with a rare mechanistic check. Cost enters as a conditioning input rather than a reward term, so the task advantage stays cost-free, and the target reference is derived from on-policy evidence rather than a fixed router. What makes this more than a reweighting is Fig. 5: placing the target on opposite sides of the reinforced trajectories produces opposite drift in the default-policy cost (-0.088 vs. +0.065 tool calls per 100 steps), and the target-free control drifts the other way. Few papers in this area test their update direction this directly.

2. Strong, well-documented efficiency result. CostDescent reaches 56.9 accuracy and 48.6 TP, beating the best baseline accuracy (AutoTIR, 55.2) and the best baseline TP (ET-Agent, 39.9) simultaneously, and the five-run spread (Table 4) preserves the ranking with small variance. The Llama-3.1-8B transfer (Table 5) and the merge-at-inference property matter here: the gain does not come at deployment cost, which is exactly the paper's promise.

Critical weaknesses

1. The central "decoupling beats cost-in-reward" claim is not isolated. The only cost-reward baseline is ET-Agent, which differs in curriculum, data flywheel, and reward design, while the ablation "w/o Cost Conditioning" removes cost entirely rather than moving it into the reward. A matched variant that puts cost into the reward (e.g. R - λc at comparable pressure) inside the identical pipeline is the direct test of the thesis. This is consequential because the paper's headline contribution rests on a cross-pipeline comparison.

2. The argument for why reinforcement transfers across goals looks incomplete to me. Section 4.3 attributes transfer to shared parameters, and Appendix B.2 bounds the transfer error, but the bound depends on Lipschitz constants that are never estimated, and the descent condition is stated in terms of the realized same-goal progress rather than a property of the data or the design. Fig. 4 shows a small inference-time goal response while Fig. 5 shows substantial training drift, and I could not see how the two are reconciled. I may be misreading the notation here, and would welcome clarification.

Questions for authors

Q1. If I understand correctly, the "w/o Cost Conditioning" ablation removes cost rather than relocating it into the reward. Could the authors report a cost-in-reward variant trained in the identical pipeline? Also, does that ablation retain the SVD-LoRA parameterization, so that it isolates the goal input rather than added capacity?

Q2. Could the authors estimate or bound the Lipschitz constants in Eq. 22, and decompose the observed effects (the drift in Fig. 5, and the token-cost reduction) into the goal-conditioned pathway versus the shared parameters? Relatedly, Fig. 4 shows a small inference-time response on the think axis, so I would be interested in how that squares with the training-time reduction.

Q3. Section 3.2 motivates the cost reference by adapting resource use at inference, but at deployment g = 0 is merged and the goal input is unused. Could the authors clarify the role of that motivation?

5 comments
AuthorsOpenReview ↗

Response to Reviewer nkbF (Part 1 of 4)

W1. The central "decoupling beats cost-in-reward" claim is not isolated. The only cost-reward baseline is ET-Agent, which differs in curriculum, data flywheel, and reward design, while the ablation "w/o Cost Conditioning" removes cost entirely rather than moving it into the reward. A matched variant that puts cost into the reward (e.g. R - λc at comparable pressure) inside the identical pipeline is the direct test of the thesis. This is consequential because the paper's headline contribution rests on a cross-pipeline comparison.

Q1. If I understand correctly, the "w/o Cost Conditioning" ablation removes cost rather than relocating it into the reward. Could the authors report a cost-in-reward variant trained in the identical pipeline? Also, does that ablation retain the SVD-LoRA parameterization, so that it isolates the goal input rather than added capacity?

Response to W1 and Q1: Thank you for this insightful suggestion. To test the thesis directly, we train three cost-in-reward variants inside the identical pipeline, which removes the differences in curriculum and data flywheel. All variants start from the same RFT checkpoint, training data and sampling procedure as CostDescent$_\text{static}$. Each retains SVD-LoRA with g=0g=0 for all rollouts and updates, so it differs from CostDescent$_\text{static}$ only in moving cost from the goal input into the reward. We acknowledge that matching the pressure of a cost reward is difficult, since its effective pressure changes during training. We therefore cover both annealed and constant pressure.

(1) ET-Agent's reward [1]. R=Rcorr⋅ftool⋅flenR=R_{\mathrm{corr}}\cdot f_{\mathrm{tool}}\cdot f_{\mathrm{len}}, where ftoolf_{\mathrm{tool}} and flenf_{\mathrm{len}} are group-relative cost factors. We retain its annealing schedule, since ET-Agent's own ablation [1] shows that fixed pressure lowers task performance. Because the factors multiply RcorrR_{\mathrm{corr}}, cost affects only rewarded trajectories, in proportion to their score.

(2) Gated additive reward. R−λcI[success]R-\lambda c\mathbb{I}[\text{success}], where the constant λ\lambda is taken from prior additive-cost work (IKEA [2] for tool calls and Acoer [3] for length).

(3) Ungated additive reward. R−λcR-\lambda c. Since this baseline collapses late in training, we report its best checkpoint before the collapse.

For the gated variant, a response counts as successful if R=1R=1 on MR and R>=0.7R>=0.7 on KI. We compare them at comparable cost through productivity, i.e., task score normalized by cost: TP for tool calls, and Log Token Efficiency (LTE) for model-authored tokens, which normalizes the task score by the log token count (App. D.4).

Table A. Cost in the reward versus cost as a goal input. The best result in each column is in bold.

MethodSVD-LoRACost in rewardMR LJMR TPMR LTEKI F1KI TPKI LTEAvg PAvg TPAvg LTE
CostDescent$_\text{static}$✓–60.359.49.950.529.99.755.444.79.8
w/o Cost Conditioning––54.249.48.851.029.09.852.639.29.3
Fixed Goal✓–53.444.98.549.026.89.651.235.99.1
ET-Agent reward✓semi-gated51.250.49.245.432.29.848.341.39.5
R−λcI[success]R-\lambda c\mathbb{I}[\text{success}]✓gated49.348.78.937.132.48.843.240.68.9
R−λcR-\lambda c✓ungated44.844.38.435.532.39.240.238.38.8

As shown in Table A, CostDescent$_\text{static}$ achieves the highest average task performance (55.4), TP and LTE, outperforming the strongest cost-in-reward variant, ET-Agent's reward (48.3), by 7.1 points. While the cost-in-reward variants reach slightly higher TP on KI, and ET-Agent's reward a slightly higher KI LTE, their KI F1 is 5 to 15 points lower. These results indicate that, within the identical pipeline, moving cost from the reward into the goal input improves task performance without coming at the expense of tool or token efficiency.

[1] Chen et al., ET-Agent: Incentivizing Effective Tool-Integrated Reasoning Agent via Behavior Calibration, ACL 2026.

[2] Huang et al., Reinforced Internal-External Knowledge Synergistic Reasoning for Efficient Adaptive Search Agent, arXiv 2025.

[3] Lee et al., Beyond Penalizing Mistakes: Stabilizing Efficiency Training in Large Reasoning Models via Adaptive Correct-Only Rewards, arXiv 2026.

AuthorsOpenReview ↗

Response to Reviewer nkbF (Part 2 of 4)

Q1. ... Also, does that ablation retain the SVD-LoRA parameterization, so that it isolates the goal input rather than added capacity?

Response to Q1 (continued): We clarify that the w/o Cost Conditioning ablation does not retain SVD-LoRA and trains the backbone alone. Yet we would like to point out that SVD-LoRA adds no representational capacity, since for any fixed goal gg it reduces to a normal LoRA that can be merged into the backbone. It changes only the training dynamics, through additional parameters with their own learning rate (2e-5 vs. 1e-6 for the backbone). To isolate the goal input from this effect, we train a Fixed Goal variant that keeps SVD-LoRA but places all rollouts and updates at g=0g=0. As shown in Table A (Part 1), CostDescent$_\text{static}$ outperforms this variant by 4.2 points in average task performance. These results indicate that the gains are not explained by the SVD-LoRA parameterization. In the revised manuscript, we have added these variants and this control to App. D.2 (Table 4).

W2. The argument for why reinforcement transfers across goals looks incomplete to me. Section 4.3 attributes transfer to shared parameters, and Appendix B.2 bounds the transfer error, but the bound depends on Lipschitz constants that are never estimated, and the descent condition is stated in terms of the realized same-goal progress rather than a property of the data or the design. Fig. 4 shows a small inference-time goal response while Fig. 5 shows substantial training drift, and I could not see how the two are reconciled. I may be misreading the notation here, and would welcome clarification.

Q2. Could the authors estimate or bound the Lipschitz constants in Eq. 22, and decompose the observed effects (the drift in Fig. 5, and the token-cost reduction) into the goal-conditioned pathway versus the shared parameters? Relatedly, Fig. 4 shows a small inference-time response on the think axis, so I would be interested in how that squares with the training-time reduction.

Response to W2 and Q2: Thank you for the detailed reading of our analysis. Your reading of the notation is correct. Each term of the descent condition corresponds to a part of the design, and we test its consequence through controlled comparisons in the revised Fig. 5 and App. B.3, which now include the think axis and a Fixed Goal control from Table A. We address each point below.

(1) Transfer across goals. SVD-LoRA generates each weight Wℓ(g)W_\ell(g) affinely in the goal (App. A.2): a goal-independent term, shared by all goals together with the backbone, plus goal-dependent terms scaled by gg. An update at a relabeled goal therefore also changes the shared parameters that act at g=0g=0. Furthermore, under hindsight relabeling, a positive-advantage trajectory is reinforced at its relabeled goal girel=ci−b^(q)g_i^{\mathrm{rel}}=c_i-\hat b(q) where b^(q)\hat{b}(q) is the target reference (Eqs. 6 and 8), which is rarely close to g=0g=0: in the training rollouts of CostDescent$_\text{static}$, only 4.4% of positive-advantage trajectories are reinforced within ±0.01\pm 0.01 of g=0g=0. Positive reinforcement therefore reaches g=0g=0 mainly through transfer, which App. B.2 bounds through the affine goal path.

AuthorsOpenReview ↗

Response to Reviewer nkbF (Part 3 of 4)

Response to W2 and Q2 (continued):

(2) Decomposition. We acknowledge that the original Fig. 5 used outcome-only ARPO, which does not form a valid control for this decomposition due to the absence of SVD-LoRA. The revised Fig. 5 replaces it with Fixed Goal from Table A and adds the think axis, and the Support-Mean Target and Median-Elite Target runs are unchanged from the original figure. With this figure, we decompose the drift into the shared and the goal-dependent parameters through two controlled comparisons.

First, we compare Support-Mean Target with Fixed Goal. Support-Mean Target fixes the target reference b^\hat{b} at the support mean μsupport\mu_{\mathrm{support}}. Fixed Goal keeps the SVD-LoRA parameterization of Support-Mean Target but places all rollouts and updates at g=0g=0, so every update acts on the shared parameters alone. As shown in Fig. 5(d), after an initial decline in tool calls, the cost at g=0g=0 rises under both runs but stays lower under Support-Mean Target for most of training, consistent with the goal-dependent parameters taking up part of the update. Later, it levels off under Fixed Goal but keeps rising under Support-Mean Target, consistent with its target lying slightly above the reinforced correct trajectories. This changes one conclusion of the original appendix, which attributed the entire rise relative to outcome-only ARPO to the target. The reversal between Support-Mean Target and Median-Elite Target does not depend on this reference and is unchanged.

Second, we change only the target estimator. Median-Elite Target builds b^\hat{b} from the accepted trajectories at or below the median accepted cost, which places it below the support mean. In contrast to the rise of Support-Mean Target, the cost at g=0g=0 then falls on both axes (Fig. 5(d)). These results indicate that the parameterization both shares and separates the update: the shared parameters carry updates at relabeled goals to g=0g=0, with the direction set by b^\hat{b}, while the goal-dependent parameters take up part of them.

(3) Lipschitz Constants. App. B.2 derives each constant in Eq. 22 from the affine goal path of SVD-LoRA and the softmax total-variation bound, so the constants are finite for our architecture. We acknowledge that the paper does not estimate their values. To estimate them, we measure how the logits of the trained policy change with the goal input. Within the range we tested, the logits change by at most 0.075 (RMS) per unit goal over [−2,2][-2,2]. Instead of relying on these constants, we test the prediction of Prop. 2 directly in (4).

(4) Descent Condition. The terms of the descent condition correspond to parts of the design.

First, same-goal progress is a natural property of the policy gradient: reinforcing a trajectory at a goal moves the expected cost at that goal toward the trajectory's cost. This property motivates hindsight relabeling, and App. B.2 formalizes it for categorical policies.

Second, since relabeled goals are centered on b^\hat{b}, this progress splits into a part along the gap b^−bθ\hat{b}-b_\theta and a goal-response residual, so the target placement of our design sets the side toward which the progress points.

Third, the progress reaches g=0g=0 because relabeled goals and g=0g=0 share parameters. The affine form of SVD-LoRA makes the generated weights Lipschitz in the goal by design. Within the range we tested, the change per unit goal is at most 0.94% of the backbone weight norm in every projection. This bounds how much the weights differ between girelg_i^{\mathrm{rel}} and g=0g=0. We acknowledge that whether the descent condition holds depends on the current policy, and the design alone does not guarantee this. That is why we mainly validate our proposition via experiments following TRPO and PPO, whose practical algorithms also approximate a theoretically justified procedure [4–6]. As shown in Fig. 5(d), changing only the placement of b^\hat{b} reverses the drift of the cost at g=0g=0 (see (2)). Furthermore, the gap proxy b^−μsupport\hat{b}-\mu_{\mathrm{support}} contracts during the training of CostDescent (Sec. 5.3; see (5)). These results indicate that, under the practical update, the drift follows the placement of b^\hat{b} and the gap to b^\hat{b} contracts, consistent with Prop. 2.

[4] Schulman et al., Trust Region Policy Optimization, ICML 2015.

[5] Schulman et al., Proximal Policy Optimization Algorithms, arXiv 2017.

[6] Cai et al., Beyond Euclidean Clipping: Overcoming Exploration Collapse in LLM RL via Riemannian Isometric Policy Optimization, ICML 2026.

Reviewer_F3esOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes CostDescent, a framework for reducing token and tool costs without including cost in the task reward. It conditions the policy on relative cost goals, constructs task-dependent targets from rollout outcomes, and uses hindsight relabeling with asymmetric goal routing and token-level weighting. Experiments on six mathematical reasoning and knowledge-intensive QA benchmarks show improvements in average task performance and tool productivity.

Critical strengths
  1. Separating cost goals from task rewards is a useful idea for efficient tool-integrated reasoning.
  2. The experiments show a useful performance-efficiency trade-off, supported by component ablations and repeated evaluation.
Critical weaknesses
  1. The benefit of separating cost from reward is not fully established, without a matched success-gated reward baseline, the gains may come from selective cost pressure rather than the separate goal input.
  2. The credit-inversion ablation compares inverse weighting with forward weighting, but not uniform weights. It therefore does not establish whether inversion is better than simply removing token reweighting.
  3. The experiments are limited to Qwen2.5-7B and Llama-3.1-8B, both older, relatively small backbones. It remains unclear whether the efficiency gains generalize to newer or larger reasoning models.
Questions for authors
  1. How does CostDescent compare with a success-gated cost-reward baseline using the same checkpoint, training data, and sampling procedure? Does the advantage remain at comparable token and tool costs?
  2. What happens when the positive-advantage branch uses uniform token weights, while keeping goal relabeling and the other components unchanged?
4 comments
AuthorsOpenReview ↗

Response to Reviewer F3es (Part 1 of 2)

W1. The benefit of separating cost from reward is not fully established, without a matched success-gated reward baseline, the gains may come from selective cost pressure rather than the separate goal input.

Q1. How does CostDescent compare with a success-gated cost-reward baseline using the same checkpoint, training data, and sampling procedure? Does the advantage remain at comparable token and tool costs?

Response to W1 and Q1: Thank you for this important question. To isolate the effect of how cost enters training, we train three cost-reward baselines in our pipeline. All baselines start from the same RFT checkpoint, training data and sampling procedure. Each retains SVD-LoRA with g=0g=0 for all rollouts and updates, so it differs from CostDescent$_\text{static}$ only in how cost enters training.

(1) ET-Agent's reward [1]. R=Rcorr⋅ftool⋅flenR=R_{\mathrm{corr}}\cdot f_{\mathrm{tool}}\cdot f_{\mathrm{len}}, where ftoolf_{\mathrm{tool}} and flenf_{\mathrm{len}} are group-relative cost factors. We retain its annealing schedule, since ET-Agent's own ablation [1] shows that fixed pressure lowers task performance. Because the factors multiply RcorrR_{\mathrm{corr}}, cost affects only rewarded trajectories, in proportion to their score.

(2) Gated additive reward. R−λcI[success]R-\lambda c\mathbb{I}[\text{success}], where the constant λ\lambda is taken from prior additive-cost work (IKEA [2] for tool calls and Acoer [3] for length).

(3) Ungated additive reward. R−λcR-\lambda c. Since this baseline collapses late in training, we report its best checkpoint before the collapse.

For the gated baseline, a response counts as successful if R=1R=1 on MR and R>=0.7R>=0.7 on KI. Since the methods operate at different costs, we compare them at comparable cost through productivity, i.e., task score normalized by cost: TP for tool calls, and Log Token Efficiency (LTE) for model-authored tokens, which normalizes the task score by the log token count (App. D.4).

Table A. Cost in the reward versus cost as a goal input. The best result in each column is in bold.

MethodCost in rewardMR LJMR TPMR LTEKI F1KI TPKI LTEAvg PAvg TPAvg LTE
CostDescent$_\text{static}$–60.359.49.950.529.99.755.444.79.8
ET-Agent rewardsemi-gated51.250.49.245.432.29.848.341.39.5
R−λcI[success]R-\lambda c\mathbb{I}[\text{success}]gated49.348.78.937.132.48.843.240.68.9
R−λcR-\lambda cungated44.844.38.435.532.39.240.238.38.8

As shown in Table A, CostDescent$_\text{static}$ achieves the highest average task performance (55.4), TP and LTE, outperforming the strongest cost-reward baseline, ET-Agent's reward (48.3), by 7.1 points. Success gating improves the additive reward, yet a gap of 12.2 points remains, indicating that selective cost pressure alone does not account for the gains. While the cost-reward baselines reach slightly higher TP on KI, and ET-Agent's reward a slightly higher KI LTE, their KI F1 is 5 to 15 points lower. These results indicate that the higher task performance of CostDescent$_\text{static}$ does not come at the expense of tool or token efficiency. In the revised manuscript, we have added these baselines to App. D.2 (Table 4).

[1] Chen et al., ET-Agent: Incentivizing Effective Tool-Integrated Reasoning Agent via Behavior Calibration, ACL 2026.

[2] Huang et al., Reinforced Internal-External Knowledge Synergistic Reasoning for Efficient Adaptive Search Agent, arXiv 2025.

[3] Lee et al., Beyond Penalizing Mistakes: Stabilizing Efficiency Training in Large Reasoning Models via Adaptive Correct-Only Rewards, arXiv 2026.

AuthorsOpenReview ↗

Response to Reviewer F3es (Part 2 of 2)

W2. The credit-inversion ablation compares inverse weighting with forward weighting, but not uniform weights. It therefore does not establish whether inversion is better than simply removing token reweighting.

Q2. What happens when the positive-advantage branch uses uniform token weights, while keeping goal relabeling and the other components unchanged?

Response to W2 and Q2: Thank you for pointing out this missing control. To test whether inversion is better than simply removing token reweighting, we train a Uniform Credit variant, which sets every token weight in the positive-advantage branch to one, so each token receives its trajectory’s advantage. The variant uses the same initial checkpoint, training data, and pipeline as CostDescent$_\text{static}$, including goal relabeling and asymmetric routing. The results are summarized in the table below.

Table B. Token weighting in the positive-advantage branch.

MethodToken weightMR LJMR TPKI F1KI TPAvg PAvg TP
CostDescent$_\text{static}$inverse60.359.450.529.955.444.7
w/ Uniform Credituniform55.751.850.228.152.940.0
w/o Credit Inversionforward55.152.150.032.052.642.0

As shown in Table B, CostDescent$_\text{static}$ achieves the highest average task performance (55.4) and TP (44.7), outperforming both Uniform Credit and w/o Credit Inversion, with the largest gain on MR. While w/o Credit Inversion reaches a higher TP on KI, its KI F1 is slightly lower. These results indicate that inverse weighting is better than simply removing token reweighting. In the revised manuscript, we have added this variant to the ablation study (Table 2).

W3. The experiments are limited to Qwen2.5-7B and Llama-3.1-8B, both older, relatively small backbones. It remains unclear whether the efficiency gains generalize to newer or larger reasoning models.

Response to W3: Thank you for this suggestion. We acknowledge that results on newer and larger reasoning models would strengthen the paper. We use Qwen2.5-7B-Instruct because the strongest baselines we reproduce, AutoTIR, Tool-Star, Tool-Light and ET-Agent, are all built on it, so a shared backbone lets the comparison reflect the training method alone. In addition, the Llama-3.1-8B-Instruct results (App. D.6) show that the gains transfer across model families. Extending to a new backbone requires rerunning the full pipeline, including data collection for that backbone. The full pipeline takes about 13 days with our computational resources (about 10 days for data collection and 3 days for RL training). Since this nearly exceeds the discussion period, we will add results on a newer backbone in the camera-ready version, following the protocol of the Llama experiment (App. D.6).

AuthorsOpenReview ↗

Follow-up on Our Responses

Dear Reviewer F3es,

Thank you again for your thoughtful review and constructive suggestions. As the discussion period ends on November 18 (AoE), we would like to kindly check whether our responses have addressed your concerns.

In brief, we trained cost-in-reward baselines, including ET-Agent's reward and a success-gated additive reward, from the same RFT checkpoint, training data and sampling procedure (App. D.2), added a Uniform Credit variant to the ablation study (Table 2), and clarified our backbone choice with the Llama-3.1-8B results, with a newer backbone to be added in the camera-ready version.

If any concerns remain, we would be glad to address them before the deadline. If our responses have resolved them, we would greatly appreciate it if you could consider updating your score.

Thank you for your time and consideration.

Best regards,

The Authors

Thank you for the additional experiments and clarifications. I will maintain my positive rating.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Revision Summary

We thank all reviewers for their constructive feedback and have uploaded a revised manuscript. We mark the main revisions in blue in the updated PDF. Below we summarize the revisions, grouped by the reviewer whose comments prompted them. All numbers of sections, figures and tables refer to the revised manuscript.

Reviewer F3es

  • App. D.2 (Table 4, new). Cost-in-reward baselines trained in the CostDescent$_\text{static}$ pipeline, including ET-Agent's reward and a success-gated additive reward (W1, Q1).
  • Table 2 and Sec. 5.4. A w/ Uniform Credit variant that sets the token weight to one (W2, Q2). Its repetition curve is added to App. D.7 (Fig. 9).

Reviewer nkbF

  • App. D.2 (Table 4, new). The same cost-in-reward baselines, and a Fixed Goal control that keeps SVD-LoRA but trains on the task reward alone at g=0g=0 (W1, Q1).
  • Fig. 5 and App. B.3. Fig. 5 now includes the think axis and the Fixed Goal control, and App. B.3 is revised accordingly (W2, Q2). Sec. 5.4 no longer compares the Support-Mean Target run with outcome-only ARPO, since SVD-LoRA confounds this comparison.
  • Sec. 4.1. We state that the merged model at g=0g=0 internalizes the implicit cost router (Q3).

Reviewer ZCAW

  • App. D.3 (Table 5, new). Efficiency on test questions that the initial RFT checkpoint already solves (W, Q).
  • App. E. The Limitations now state that groups in which every answer is correct carry zero advantage, and describe an extension that uses their cost differences (W, Q).

Other changes

  • Since App. D.2 and D.3 are new, the later subsections of App. D are renumbered (Overall Efficiency Profile D.4, Reliability of Reported Results D.5, Generalization to Llama D.6, Repetition without Credit Inversion D.7, Case Study D.8), and the former Tables 4 and 5 are now Tables 6 and 7. Figure numbers are unchanged.
  • Fig. 7. Log Token Efficiency is scaled by 100 and renamed, matching the LTE columns of Table 4. All values change only by this scale.
  • We also made minor wording edits throughout for clarity, which do not change any claim or result.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.