acceptodds
Under review as a conference paper at ICLR 2027

Attribution Is Not Optimization: Counterfactual Rewards for Language Agents

Abstract

Holding a tool action fixed and replacing its output with a null observation isolates the output's contribution. It does not necessarily identify which action a policy should repeat. We formalize this attribution–optimization gap and introduce a finite-action audit that compares reward and terminal-utility update directions. Across three retrieval datasets, total belief gain is more aligned with terminal success than observation-residual credit (cosine 0.430 versus 0.114). The ordering persists in trainable-parameter gradients for retrieval (0.33 versus 0.09) and calculator use (0.51 versus 0.26). Yet residual credit closely tracks programmatic expression correctness (0.934). This contrast separates tool-output attribution from action learning. Restoring action-side credit improves local alignment throughout a controlled interpolation. We therefore introduce environment-marginalized action credit (EMAC), which preserves action-side credit while averaging tool-output variation. EMAC matches total belief gain when retrieval is deterministic. When the same query produces different relevant passage blocks, it raises held-out exact match from 0.263 to 0.312 and reduces normalized observation variance from 1.00 to 0.35. Under a matched reward-scoring budget, EMAC also improves exact match by 6.0 points in corrupted retrieval. On a 3B actor, the gain is 1.6 points.

Primary area:
reinforcement learning
Keywords:
Credit Assignment, Counterfactual Rewards, Language Agents, Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_YecwOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper distinguishes the value of a tool observation from the value of selecting a tool action. It decomposes total answer-belief gain into observation-residual and action-side terms, showing why subtracting an action-dependent null-observation score can alter the optimization direction. A local gradient analysis motivates audits of reward alignment with terminal utility. Retrieval and calculator experiments test this alignment over candidate actions and actor parameters, followed by small-scale RL studies. The proposed EMAC reward retains action-side credit while averaging over tool outputs to reduce observation noise.

Critical strengths
  1. The distinction between observation attribution and action learning is clear and useful: a counterfactual reward can isolate an observation's contribution while removing information needed to choose an action. The local analysis supports this distinction without claiming a global policy-improvement guarantee.

  2. Candidate-action audits, actor-parameter gradients, fixed-judge comparisons, and RL experiments test the proposed mechanism across retrieval and calculator tasks. Together, they provide evidence beyond correlations between reward and task success.

Critical weaknesses
  1. The observation-variance calculation needs clarification. At a fixed state, action, and scorer, total gain and residual credit differ by a constant action-side term, so their variance over the same tool observations should be identical before any transformations. Table 3 instead reports normalized variances of 1.00 and 0.74. Pooling different actions, states, or policies, or applying clipping, could explain the difference, but such a statistic would not measure the same conditional observation variance. The calculation must be clarified before these values can be interpreted as evidence about observation noise.

  2. The reliability of the RL gains is difficult to assess without uncertainty estimates. The held-out evaluations contain 60 or 120 questions, but the tables do not specify the number of independent training runs or report intervals. The 3B result of 0.583 versus 0.567 is particularly small at this evaluation scale. The alignment audits support the qualitative hypothesis, but they do not establish the robustness of the trained-policy gains. The procedure for selecting K is also unclear.

Questions for authors
  1. What sampling unit and conditioning variables are used for the variance in Table 3? Please explain the values of 1.00 and 0.74 and, if the current statistic pools across conditions, report the variance of the untransformed rewards over the same observations at fixed states and actions.

  2. How many independent training runs contribute to each RL result, and how are exact-match scores averaged? Please report uncertainty for paired evaluation differences and, where available, variation across training runs.

  3. Was K selected on development data independently of the held-out evaluation? How were reward-scoring costs matched across methods?

Reviewer_mSK7OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces environment-marginalized action credit (EMAC), which averages total belief gain across multiple tool outputs for a fixed action, preserving action-side information while reducing reward variance due to output randomness. Experiments show improved answer accuracy under stochastic or corrupted retrieval, while EMAC reduces to total belief gain when tool outputs are deterministic.

Critical strengths

Not much.

Critical weaknesses
  1. The proposed method improves on total belief gain only when tool outputs vary for a fixed state and action; with deterministic outputs, the two are equivalent. The paper does not establish how common or consequential such variability is in practical agent settings, and the gains under artificially randomized retrieval leave its broader applicability unclear. Averaging rewards over repeated tool outputs also appears to offer only an incremental methodological contribution.
  2. The empirical evidence is limited. The comparisons mainly involve reward variants within the same training setup, making it difficult to assess the proposed method's practical advantage over competitive baselines.
  3. The manuscript is difficult to follow, and the presentation does not clearly convey the core method or how it fits into the overall training procedure.
Questions for authors

See weaknesses above.

Reviewer_nZALOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper argues that observation attribution and action optimization are different objectives. Subtracting a null-observation score isolates the contribution of a tool output, but may remove useful signal about which action should be repeated. The authors propose a finite-action gradient audit and EMAC, which averages total belief gain over multiple tool outputs. Experiments cover retrieval and calculator agents.

Critical strengths
  1. The paper makes a useful distinction between explaining the value of an observation and training the action that produced it. The finite-action audit is simple and interpretable.

  2. The analysis connects candidate-level gradients, parameter gradients, and downstream RL. The calculator experiment is helpful since expression correctness can be checked programmatically.

Critical weaknesses
  1. The main algebra follows directly from the known fact that an action-dependent baseline changes a policy gradient. EMAC is essentially a multi-sample Monte Carlo estimate of total belief gain. In expectation, it optimizes the same quantity as total gain, with lower observation variance.

  2. The empirical evidence is too small and lacks uncertainty estimates. The main retrieval audit uses only 96 informative action sets, after excluding constant-utility sets. Several RL evaluations use only 60 or 120 held-out questions, with no confidence intervals or clearly reported training seeds. On the 3B model, the claimed 1.6-point improvement corresponds to only about two additional correct answers out of 120. This is not enough to establish a reliable policy-learning advantage.

  3. The paper does not show the findings with model scales. Policy learning is evaluated only on 1.5B and 3B actors, and the reported 3B gain is only 1.6 points, corresponding to roughly two additional correct examples out of 120.

Questions for authors
  1. How many independent RL seeds were used? Please report confidence intervals or per-seed results for all held-out EM comparisons.

  2. What fraction of all action sets is excluded as non-informative, and do conclusions change if these sets are retained using another alignment statistic?

  3. Since EMAC is an averaged estimator of total gain, what is its conceptual contribution beyond standard multi-sample variance reduction?

  4. Does EMAC still help when compared at equal total compute, including retrieval, continuation generation, verifier scoring, and optimization?

  5. How do the attribution gap and EMAC benefit change with actor scale?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.