acceptodds
Withdrawn from ICLR 2027

When Does Action Credit Need Updating?

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Tool-using agents are continually updated with new interaction data. After each policy update, however, previously estimated action credits may become stale. Recomputing them from scratch can require many additional tool calls and environment interactions, making repeated updates increasingly expensive. We ask a simple question: when does historical action credit actually need to be updated? Our key observation is that a change in action value does not necessarily imply a change in the decision. Historical credit can still be useful as long as policy-induced drift is too small to overturn the existing action ranking. Building on this idea, we introduce pairwise branch sensitivity to capture how strongly a policy update affects the downstream regions that distinguish two candidate actions. We then derive a first-order anchored credit-transport estimator that updates historical credit using old interventional trajectories, and propose a Decision-Sufficient Credit Gate (DSC-Gate) that chooses whether to reuse, transport, or resample credit. Experiments show that branch sensitivity explains credit drift substantially better than global policy distance. With sufficient historical data, credit transport reduces estimation error, while its benefit to decision making is concentrated on updates that affect action-distinguishing branches. On a fully independent test set, DSC-Gate changes mean regret by only +0.00004 relative to a gap-based gate while reducing mean new tool steps from 472 to 286, a 39.4% reduction. We observe the same pattern after a real tool-agent parameter update. Overall, our results show that agents do not need to recompute action credit after every policy update: much of the historical evidence can be reused or cheaply corrected, reducing the additional interaction required to keep action decisions up to date.

Primary area:
reinforcement learning
Keywords:
Action Credit, Policy Updates, Credit Transport, Branch Sensitivity, Tool-Using Agents

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_JAbwOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies whether credit assigned to past actions remains reliable after the policy for later decisions is updated. It explains how the relative credit of two initial actions can change when the update affects states that those actions reach with different probabilities. The authors propose a first-order method to estimate these changes from previously collected trajectories. They also introduce DSC-Gate, which decides whether to reuse old credit estimates, adjust them using the proposed method, or collect new trajectories to estimate credit again. Controlled experiments separately test the explanation for credit changes, estimation accuracy with limited data, and the gate’s decisions. An additional experiment in a retail tool-use setting, using a language model updated with LoRA, evaluates whether the method works beyond the controlled environment.

Critical strengths
  1. The paper connects changes in action credit to a practical decision about whether fresh execution is needed. A policy update matters when it affects downstream states that candidate actions reach with different probabilities. Fresh execution is necessary only when the resulting uncertainty could change the action ranking. Comparisons with global and pairwise KL help clarify the scope of this finding.

  2. The evaluation separates development, calibration, and test tasks and uses shared historical data and execution streams for fair comparisons. The independent L4 test includes paired noninferiority and cost comparisons. It reports 286 versus 472 new steps with only a small difference in regret, which is more informative than reporting cost savings alone. The paper also explicitly reports substantial reference and calibration costs, making clear that the marginal savings do not represent total savings.

Critical weaknesses
  1. The reuse conditions in the main text do not guarantee that the action ranking is preserved. Equation (6) uses the signed condition Δc < cμ. For example, an initial gap of 0.1 and a signed drift of −0.2 satisfy this condition but yield an updated gap of −0.1, reversing the ranking. Equation (8) has the same problem even when the uncertainty radii are zero. In contrast, Appendices A.4 and D.3 use conditions based on absolute magnitudes that avoid this counterexample. The authors should clarify which rule was implemented, as this determines whether the issue is limited to the written equations or whether the gate results also require reassessment.

  2. The real tool experiment lacks the detail needed to assess its practical claims. Sections 4.3 and 5.4 report an actual LoRA update and fewer new execution steps, but several key details remain unclear: the number of test tasks, the reference rollout budget, the LoRA training protocol, and the procedures for calibration and computing trajectory likelihood ratios. The regret comparison of 0.0291 versus 0.0288 also lacks a paired confidence interval for the difference. Although Appendix G describes artifacts for the controlled experiments, I could not locate a supplementary attachment or artifact link on the reviewed submission page to verify the implementation.

Questions for authors
  1. Please reconcile Equations (6) to (9) with the appendix, particularly the ordering of action pairs and the use of absolute values. Which reuse and transport rules were used in each reported gate experiment? Does the frozen implementation follow the appendix’s reuse test based on absolute magnitudes and its transport test that accounts for signs?

  2. For the real tool experiment, please provide the sample sizes, training and calibration procedures, reference rollout protocol, and the method for computing trajectory likelihood ratios. Were calibration quantities fitted in this environment or transferred from another task distribution? Please also report paired confidence intervals for the regret and cost differences.

Reviewer_7QonOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper investigates when historical action credit needs updating after policy changes in tool-using agents. The authors propose pairwise branch sensitivity and a credit-transport estimator, alongside DSC-Gate, to adaptively reuse, update, or resample historical credit. Experiments show that DSC-Gate reduces new tool interactions by 39.4% while maintaining comparable decision quality.

Critical strengths

The motivation is well justified. The paper distinguishes between "credit drift" and "decision reversal" and identifies three key factors in determining whether historical credit needs to be refreshed: the existing action gap, the alignment between credit drift and action-distinguishing branches, and uncertainty arising from finite samples.

Critical weaknesses

The experiments are insufficient and not clearly described. L1–L4 are never formally defined in the main text, and they are essentially small tabular MDPs in which the LLM only supplies a frozen initial policy. The baselines used in L4 (Gap-, WIS- and DR-Gate) are also never defined. The method adds costs that the paper does not count: calibration requires about 15.7M target-policy steps (Table A9), and the paper never tests whether κ transfers across updates. The gains are modest. DR matches or beats transport in MSE, transport is worse than gap-based refresh in L3, and L4 reports a single operating point with no Pareto curve or ablations. The real-model check covers a single LoRA update with few details and no confidence interval, so practical value under repeated updates remains unclear.

Questions for authors

In Figure A1, matched-KL updates on random supports produce ~2× larger drift MSE than updates on differential supports, at every KL level. This seems to contradict the claim that branch-differential regions drive relative drift, and the confirmatory L1-A test only contrasts differential with common supports. Could you explain this result?

Calibration (Appendix C.2) needs 1,024 target-policy reference trajectories per update condition and route, about 15.7M tool steps in total (Table A9). Does a new policy update require recalibration? If not, can you show that κ_R, κ_Δ and κ_T transfer across updates of different size or type, such as branch-selective updates, while the radii still achieve their intended coverage?

Reviewer_a2cCOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors focus on the research question that whether estimated action credits should be recomputed from scratch after the tool-using AI agent updates the policy. They first observe that a change in action value does not necessarily imply a change in the decision. Based on this, the authors introduce pairwise branch sensitivity to capture how strongly a policy update affects the comparison of two candidate actions; propose the first-order anchored credit-transport estimator that updates historical credit using old interventional trajectories; and propose a Decision-Sufficient Credit Gate that chooses whether to reuse, transport, or resample credit. The experiments on a fully independent test set and a real tool-agent parameter update show that the proposed method can reduce mean new tool steps while maintaining the mean regret, demonstrating that agents do not need to recompute action credit after every policy update.

Critical strengths
  1. The research question focusing on whether action credits should be recomputed from scratch after every policy update is meaningful and practical.

  2. The experiments include not only the controlled simulated setting but also the real scene.

  3. An interesting perspective. This paper focuses on minimizing the new execution cost instead of estimating action credits under the new policy as accurately as possible.

Critical weaknesses
  1. Texts in Figure 1, 2, 4 are too small to see.

  2. Experiments on real settings are only conducted in the retail environment with Qwen3-4B-Instruct. Could try more real environments with various LLMs.

  3. Although the three components (the proposed pairwise branch sensitivity, the first-order anchored credit-transport estimator, and the Decision-Sufficient Credit Gate) are individually motivated, their operational connections could be better clarified. In particular, it is not entirely clear how pairwise branch sensitivity directly contributes to the DSC-Gate.

  4. As it is mentioned in discussion, this paper focuses on the single policy update instead of multiple consecutive policy updates. Additional experiments across multiple successive policy updates would help further demonstrate the strengths or the limitations of the proposed method.

  5. There appears to be an inconsistency between equations 8,9 of Section 3.4 and their corresponding formulations in Appendix D.3.

Questions for authors

Could the authors clarify how pairwise branch sensitivity directly contributes to the DSC-Gate?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.