- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper distinguishes the value of a tool observation from the value of selecting a tool action. It decomposes total answer-belief gain into observation-residual and action-side terms, showing why subtracting an action-dependent null-observation score can alter the optimization direction. A local gradient analysis motivates audits of reward alignment with terminal utility. Retrieval and calculator experiments test this alignment over candidate actions and actor parameters, followed by small-scale RL studies. The proposed EMAC reward retains action-side credit while averaging over tool outputs to reduce observation noise.
Critical strengths
-
The distinction between observation attribution and action learning is clear and useful: a counterfactual reward can isolate an observation's contribution while removing information needed to choose an action. The local analysis supports this distinction without claiming a global policy-improvement guarantee.
-
Candidate-action audits, actor-parameter gradients, fixed-judge comparisons, and RL experiments test the proposed mechanism across retrieval and calculator tasks. Together, they provide evidence beyond correlations between reward and task success.
Critical weaknesses
-
The observation-variance calculation needs clarification. At a fixed state, action, and scorer, total gain and residual credit differ by a constant action-side term, so their variance over the same tool observations should be identical before any transformations. Table 3 instead reports normalized variances of 1.00 and 0.74. Pooling different actions, states, or policies, or applying clipping, could explain the difference, but such a statistic would not measure the same conditional observation variance. The calculation must be clarified before these values can be interpreted as evidence about observation noise.
-
The reliability of the RL gains is difficult to assess without uncertainty estimates. The held-out evaluations contain 60 or 120 questions, but the tables do not specify the number of independent training runs or report intervals. The 3B result of 0.583 versus 0.567 is particularly small at this evaluation scale. The alignment audits support the qualitative hypothesis, but they do not establish the robustness of the trained-policy gains. The procedure for selecting K is also unclear.
Questions for authors
-
What sampling unit and conditioning variables are used for the variance in Table 3? Please explain the values of 1.00 and 0.74 and, if the current statistic pools across conditions, report the variance of the untransformed rewards over the same observations at fixed states and actions.
-
How many independent training runs contribute to each RL result, and how are exact-match scores averaged? Please report uncertainty for paired evaluation differences and, where available, variation across training runs.
-
Was K selected on development data independently of the held-out evaluation? How were reward-scoring costs matched across methods?