- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies whether credit assigned to past actions remains reliable after the policy for later decisions is updated. It explains how the relative credit of two initial actions can change when the update affects states that those actions reach with different probabilities. The authors propose a first-order method to estimate these changes from previously collected trajectories. They also introduce DSC-Gate, which decides whether to reuse old credit estimates, adjust them using the proposed method, or collect new trajectories to estimate credit again. Controlled experiments separately test the explanation for credit changes, estimation accuracy with limited data, and the gate’s decisions. An additional experiment in a retail tool-use setting, using a language model updated with LoRA, evaluates whether the method works beyond the controlled environment.
Critical strengths
-
The paper connects changes in action credit to a practical decision about whether fresh execution is needed. A policy update matters when it affects downstream states that candidate actions reach with different probabilities. Fresh execution is necessary only when the resulting uncertainty could change the action ranking. Comparisons with global and pairwise KL help clarify the scope of this finding.
-
The evaluation separates development, calibration, and test tasks and uses shared historical data and execution streams for fair comparisons. The independent L4 test includes paired noninferiority and cost comparisons. It reports 286 versus 472 new steps with only a small difference in regret, which is more informative than reporting cost savings alone. The paper also explicitly reports substantial reference and calibration costs, making clear that the marginal savings do not represent total savings.
Critical weaknesses
-
The reuse conditions in the main text do not guarantee that the action ranking is preserved. Equation (6) uses the signed condition Δc < cμ. For example, an initial gap of 0.1 and a signed drift of −0.2 satisfy this condition but yield an updated gap of −0.1, reversing the ranking. Equation (8) has the same problem even when the uncertainty radii are zero. In contrast, Appendices A.4 and D.3 use conditions based on absolute magnitudes that avoid this counterexample. The authors should clarify which rule was implemented, as this determines whether the issue is limited to the written equations or whether the gate results also require reassessment.
-
The real tool experiment lacks the detail needed to assess its practical claims. Sections 4.3 and 5.4 report an actual LoRA update and fewer new execution steps, but several key details remain unclear: the number of test tasks, the reference rollout budget, the LoRA training protocol, and the procedures for calibration and computing trajectory likelihood ratios. The regret comparison of 0.0291 versus 0.0288 also lacks a paired confidence interval for the difference. Although Appendix G describes artifacts for the controlled experiments, I could not locate a supplementary attachment or artifact link on the reviewed submission page to verify the implementation.
Questions for authors
-
Please reconcile Equations (6) to (9) with the appendix, particularly the ordering of action pairs and the use of absolute values. Which reuse and transport rules were used in each reported gate experiment? Does the frozen implementation follow the appendix’s reuse test based on absolute magnitudes and its transport test that accounts for signs?
-
For the real tool experiment, please provide the sample sizes, training and calibration procedures, reference rollout protocol, and the method for computing trajectory likelihood ratios. Were calibration quantities fitted in this environment or transferred from another task distribution? Please also report paired confidence intervals for the regret and cost differences.