acceptodds
Under review as a conference paper at ICLR 2027

Credit Without Ground Truth: Claim-Matched Evaluation of Step-Level Credit with Finite Executed Replay

Abstract

Step-level credit evaluation can conflate numerical association with ordering and branch recovery with position-selection value. We test these distinctions in ALFWorld with two independently seeded replay banks: 27,520 continuations over 172 fixed turns. At 16 continuations per arm per bank, cross-bank replay-contrast associations are Pearson 0.736 and pooled Spearman 0.330. On 30 historical failures, A-count-selected branches at confidence-/implicit-selected positions gain 8.54 percentage points over same-position factual reruns, but their 3.33-point advantage over frozen-random positions under the same alternative-selection rule remains unresolved. Numerical association alone does not establish reliable pooled ordering; branch recovery alone does not establish better score-informed position selection. Historical judge alignment is modestly positive with an unresolved archived-control increment. Same-cohort C/D information-source comparisons and a short, sparse-exposure training pilot provide scope checks; incremental hindsight training utility remains unestablished. Findings are conditional on the tested policies, intervention support and finite budgets.

Primary area:
reinforcement learning
Keywords:
reinforcement learning, LLM agents, credit assignment, executed replay, process supervision, counterfactual evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_aiyUOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies how to evaluate step-level credit without ground-truth labels. The authors use finite replay to estimate an action’s contribution relative to fixed alternatives, then test reference reproducibility, credit-score fidelity, and failure recovery. Two independent replay banks cover 43 ALFWorld trajectories and 172 positions under a frozen Qwen2.5-7B-Instruct policy, with 16 continuations per arm and bank, totaling 27,520. A/B reference estimates have pooled Pearson correlation 0.736 and Spearman correlation 0.330; the within-trajectory median Spearman is 0.600 on 29/43 computable trajectories. In recovery on 30 historical failures, selected branches improve on factual replay at the same position. Confidence/implicit-selected positions outperform random positions by 3.33 percentage points, with adjusted interval [−2.08, 13.33], leaving the advantage unresolved. Confidence and implicit selectors choose the same position in 23/30 cases. The paper also includes a six-update training pilot.

Critical strengths
  1. The recovery controls are useful. Comparisons against same-position factual replay and random-position recovery help identify which selection step produces the gain. The authors select actions using one replay bank and evaluate the frozen choices on an independently seeded bank. They also rerun factual actions instead of assigning them the outcome of one historical failure (pp.2–7; Appendix A3).
  2. The uncertainty reporting is careful. The authors report ties, undefined correlations, and intervals, and retain the positive judge-alignment result. They distinguish pooled from within-trajectory ordering and leave unresolved position gains as unresolved. The supplement supports checks of the main A/B counts and some statistics.
Critical weaknesses
  1. The implications for credit evaluation remain unclear. The pooled Pearson/Spearman gap (0.736/0.330) concerns agreement between replay banks. Credit fidelity, however, is measured within trajectories, where median A/B reference agreement is 0.600 (29/43 defined). The paper does not directly compare a fixed scorer’s fidelity under A16 and B16 on the same trajectories. I therefore cannot tell which conclusions about the scorer depend on the choice of reference bank.
  2. The training pilot leaves the value of added hindsight unresolved. Both F and N use micro-credit, reaching 70/128 and 72/128 successes at t6 (95% interval for F−N: [−10.16, 7.03] percentage points). With one continuation seed, six updates and nonzero credit on 9.10% of F turns, this comparison gives little guidance on when hindsight is useful. The paper also does not show how much withholding hindsight changes the final credit term on identical inputs.
Questions for authors
  1. For implicit credit and the judge separately, please hold scores fixed and report the paired change in within-trajectory fidelity between A16 and B16, with uncertainty, on trajectories where both correlations are defined. Which conclusions about each scorer, if any, change? What further conclusion about these scorers follows from the pooled-order result?
  2. Both F and N retain micro-credit. Holding the scorer and trajectory groups fixed, how often, and by how much, does withholding hindsight change the final normalized/clipped micro-credit term used in the policy update? The exposure counts in Appendix A1.2 come from different arm-specific trajectories and do not isolate this change. Can you provide this comparison using saved trajectories to help interpret the unresolved F–N training contrast?
  3. Appendix A3.14 reports 31/480 successes with full position support and 46/480 with four positions, under full hindsight on the same C/D banks. What is the descriptive paired interval for the difference between full support and the four-position bridge? The reproducibility statement notes that C/D currently has only documentary support. Can you provide anonymized per-trajectory cell and bridge outcomes, the resampling procedure, and records showing that choices were fixed before D evaluation? These would allow reconstruction of this comparison and the Table 1 intervals.
  4. Appendix A1.4 acknowledges that the observed results trigger both stopping rule B and extension rule C, with no specified precedence. What determined the decision to stop at t6, and was that decision documented at the time?
2 comments
AuthorsOpenReview ↗

Response 1 of 2 — scorer fidelity and hindsight during training

Q1. We held each scorer’s values fixed and changed only the replay bank used as the reference. For each of the 43 trajectories, we computed the Spearman correlation across its four positions separately against A16 and B16. For each scorer, we retained the trajectories on which both correlations were defined. Table 9 (p. 20, summarized on p. 5) reports the following results:

• Implicit credit, 29/43 trajectories: median correlation 0.056 on A16 and 0.200 on B16. The median within-trajectory change (B16 minus A16) was +0.180, with a descriptive 95% trajectory-bootstrap interval of [0.000, 0.400]. • Judge, 15/43 trajectories: median correlation 0.577 on A16 and 0.707 on B16. The median within-trajectory change was 0.000 [0.000, 0.447].

We took the difference within each trajectory before taking the median. This is why the judge’s two bank medians can differ while its median paired change is zero. The two scorers also have different sets of computable trajectories. These results quantify the observed change for each scorer; they do not establish that either scorer is invariant to the reference bank or that one is more sensitive than the other. The pooled A/B rank correlation of 0.330 measures agreement between the references and, by itself, gives no further conclusion about either scorer. The revised supplement includes the fixed scores for all 172 positions, the A/B counts, and the analysis code.

Q2. We cannot recover how often, or by how much, withholding hindsight changed the final micro-credit term on identical inputs during the six training updates. We reconstructed the terms actually used in each arm from its saved scores. The arms generated different trajectory groups, the turn texts were not saved, and the intermediate actor checkpoints for steps 141–144 are unavailable. The reconstruction therefore verifies the realized weights but cannot supply the paired prompt comparison you requested.

There is one earlier same-batch check, which we now disclose on p. 8 and in Appendix A1.7 (p. 17). On a batch of 256 trajectories from the base 1.5B policy, a contemporaneous status entry reports a row-weighted relative L2 difference of 0.00275 between the final optimizer coefficients under the two prompts, below the prespecified minimum of 0.05. We have not recovered the raw metric file to recompute that value. This was a prerequisite for a planned three-seed study, which did not run. The six-update pilot started from a learned checkpoint under a protocol frozen after that result. Its protocol explicitly excluded this threshold as a stopping rule. We found no record of a corresponding treatment-strength check at the learned checkpoint or of the reason for the switch. The incremental training value of hindsight remains unresolved.

AuthorsOpenReview ↗

Response 2 of 2 — paired recovery outcomes and the stopping decision

Q3. With full hindsight and the same C/D replay banks, selecting from all eligible positions gave 31 successes in 480 continuations. Restricting selection to the original four positions gave 46/480. The full-minus-four difference is therefore −3.125 percentage points. We cannot supply a valid paired interval from those totals. It requires the two outcomes for each of the 30 trajectories so that both conditions stay together when trajectories are resampled.

For the two prespecified C/D contrasts reported in Table 1, the documented procedure used 10,000 paired trajectory-bootstrap draws, with the 1.25th and 98.75th percentiles as endpoints for the two-comparison family. The full-versus-four comparison was specified as descriptive, without an interval. We have not recovered the per-trajectory C/D cell and four-position outcomes needed to reconstruct the former intervals or calculate the latter one.

We did recover the plan specifying that choices would be made on C and frozen before evaluation on D. The plan predates the first C run. We did not recover a timestamped record of the actual choices showing that they had been frozen before D was evaluated. Appendix A3.14 (p. 37) and the reproducibility statement (p. 11) now distinguish what the surviving records document from what a reader can independently reconstruct.

Q4. The execution record shows that the scheduled run ended at global step 146, after six updates. At that point both the stopping condition B and the extension condition C applied, and the written protocol gave neither precedence. We found no contemporaneous record explaining why the run ended there rather than continuing. We can document when it ended, but cannot attribute that decision to a particular rule. We have clarified this in Appendix A1.4 and retain the six-update, single-seed result as exploratory.

Reviewer_EckGOpenReview ↗
Rating
2
Confidence
1
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

Step-level credit methods score each step of an agent’s trajectory by how much it mattered, and those scores are then used to train the agent. This paper audits two such approaches: HCAPO-style implicit credit (the policy’s own likelihood of the action once it’s told the outcome) and a TARL-style LLM judge (another model scores each step as −1, 0 or +1). Since there is no ground truth, the authors build a reference by executed replay. They rewind ALFWorld to just before a step, then compare the original action against four alternatives, running the frozen policy forward many times to see how the success rate changes.

Critical strengths

I am not an expert in the area, but the premise of the work is reasonable. Step-credit methods are popular for agent RL, and whether they credit the right steps is worth studying. The work also show good experimental design: they measure the replay reference twice with independent seeds. The recovery decomposition is also clean. Separating “a better alternative exists” from “the score picked a better position” is correct, and it changes how a positive recovery result should be read.

Critical weaknesses

I found the paper to be poorly written and very hard to follow. The authors do little exposition and the writing ends up fairly jargon-heavy and assumes intimate familiarity. At a cumulative 80 pages, this is also not cleanly presented. The experimnetal evaluations also seem to be, inconclusive. If I understand correctly, the recovery experiment uses 30 trajectories where the agent originally failed the task, and tries to recover each one. Letting the credit score pick the retry position beats picking a random position by an estimated +3.33 percentage points. But the interval is [−2.08, +13.33]. We cannot conclude from this whether the treatment might be slightly worse than random, no better than random, or substantially better?

Questions for authors

See weakness

1 comment
AuthorsOpenReview ↗

What the recovery interval tells us

Yes, your reading of the interval is correct. On these 30 failed trajectories, choosing the retry position with the credit score gave an estimated 3.33 percentage-point advantage over a random position. The adjusted 95% interval [−2.08, 13.33] leaves open a small disadvantage, no advantage, and a substantial advantage. We therefore cannot recommend using the score to choose where to retry on the strength of this result.

The 8.54-point gain answers a different question. Suppose the agent goes back to the step selected by the scorer. Trying the alternative action chosen from the first set of replays improved success by 8.54 points over repeating the original action at that same step, when evaluated in a second, independent set of replays. That tells us the agent could recover by acting differently there. To find out whether the scorer deserves credit for picking that step, we must let the agent try an alternative at a random step too. That comparison gives the 3.33-point estimate above.

Thank you for flagging how much work the original presentation left to the reader. We rewrote the explanation on p. 7 around these two questions and revised the conclusion on p. 10 to state explicitly which one remains open.

Reviewer_WEB3OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates how to reliably evaluate step-level credit assignment in language-model agents when the true causal contribution of individual actions is unknown. The paper introduces a claim-matched evaluation perspective, emphasizing that numerical agreement, reliable action rankings, effective intervention selection, and downstream learning improvements are distinct properties that require separate and appropriately matched evaluations. Through experiments using Qwen2.5-7B-Instruct, the authors find that replay-based credit estimates show moderate numerical agreement but substantially weaker ranking consistency, highlighting the limitations of using replay as a reference. The contribution is primarily methodological and empirical rather than a new credit-assignment algorithm.

Critical strengths
  • The paper addresses the important challenge of evaluating step-level credit assignment without reliable ground-truth credit scores.
  • The paper makes a clear methodological contribution, distinguishing numerical agreement, ranking reliability, and practical usefulness as separate evaluation goals.
  • The paper uses independent replay banks and extensive experiments to reduce evaluation bias and improve reliability.
  • The paper uses multiple statistical tests, uncertainty estimates, and tie-aware analyses to avoid misleading conclusions.
  • The paper clearly reports experimental limitations, inconclusive findings, and reproducibility details.
Critical weaknesses
  • The experiments focus on a single environment and model, making it unclear whether the findings apply to other agent tasks.
  • Despite many replay continuations, the limited number of independent trajectories reduces statistical power and potentially significance.
  • Replay-based credit estimates remain noisy and may not accurately reflect the true contribution of individual actions.
  • Using only one random-position baseline and limited control comparisons makes the benefits of credit-guided selection difficult to establish.
  • The paper does not convincingly demonstrate whether its evaluation framework leads to better credit-assignment methods or improved policy learning.
  • The tested credit signals are adapted versions of existing approaches, and the training experiment is too small to support broader conclusions.
Questions for authors

Please see the weaknesses.

1 comment
AuthorsOpenReview ↗

Recovery controls and scope of the training claim

Thank you for raising the concern about the single random retry position. We checked how much that draw mattered by averaging over all four eligible positions on each of the same 30 failed trajectories. At every position, we used the same rule to choose an alternative action on replay bank A and evaluated it on independently seeded bank B. The implicit scorer’s selected position improved success by 2.34 percentage points over this uniform-position baseline (adjusted 95% interval [−2.80, 8.65]). The advantage of using the score to choose where to retry therefore remains uncertain (Appendix A3.9).

A second post-hoc comparison holds the retry position fixed and changes which action is tried there. At the implicit scorer’s chosen positions, selecting an alternative by its success count on bank A improved success by 5.83 points over choosing uniformly among the four alternatives (adjusted 95% interval [0.37, 13.28], Appendix A3.10). This estimate applies to the complete action-selection rule, including how it breaks ties. Both checks use the same 30 trajectories.

The practical implication is that a recovery experiment needs a position baseline before it can attribute a gain to a step scorer. We now explain this distinction on p. 7. We also agree about scope: repeated continuations improve measurement on the sampled trajectories, but they do not increase the number of independent tasks. Our recovery evidence is limited to ALFWorld under the tested policy, and the separate six-update pilot does not establish an improvement in policy learning. We have made those limits explicit on pp. 8 and 10.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.