- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper studies how to evaluate step-level credit without ground-truth labels. The authors use finite replay to estimate an action’s contribution relative to fixed alternatives, then test reference reproducibility, credit-score fidelity, and failure recovery. Two independent replay banks cover 43 ALFWorld trajectories and 172 positions under a frozen Qwen2.5-7B-Instruct policy, with 16 continuations per arm and bank, totaling 27,520. A/B reference estimates have pooled Pearson correlation 0.736 and Spearman correlation 0.330; the within-trajectory median Spearman is 0.600 on 29/43 computable trajectories. In recovery on 30 historical failures, selected branches improve on factual replay at the same position. Confidence/implicit-selected positions outperform random positions by 3.33 percentage points, with adjusted interval [−2.08, 13.33], leaving the advantage unresolved. Confidence and implicit selectors choose the same position in 23/30 cases. The paper also includes a six-update training pilot.
Critical strengths
- The recovery controls are useful. Comparisons against same-position factual replay and random-position recovery help identify which selection step produces the gain. The authors select actions using one replay bank and evaluate the frozen choices on an independently seeded bank. They also rerun factual actions instead of assigning them the outcome of one historical failure (pp.2–7; Appendix A3).
- The uncertainty reporting is careful. The authors report ties, undefined correlations, and intervals, and retain the positive judge-alignment result. They distinguish pooled from within-trajectory ordering and leave unresolved position gains as unresolved. The supplement supports checks of the main A/B counts and some statistics.
Critical weaknesses
- The implications for credit evaluation remain unclear. The pooled Pearson/Spearman gap (0.736/0.330) concerns agreement between replay banks. Credit fidelity, however, is measured within trajectories, where median A/B reference agreement is 0.600 (29/43 defined). The paper does not directly compare a fixed scorer’s fidelity under A16 and B16 on the same trajectories. I therefore cannot tell which conclusions about the scorer depend on the choice of reference bank.
- The training pilot leaves the value of added hindsight unresolved. Both F and N use micro-credit, reaching 70/128 and 72/128 successes at t6 (95% interval for F−N: [−10.16, 7.03] percentage points). With one continuation seed, six updates and nonzero credit on 9.10% of F turns, this comparison gives little guidance on when hindsight is useful. The paper also does not show how much withholding hindsight changes the final credit term on identical inputs.
Questions for authors
- For implicit credit and the judge separately, please hold scores fixed and report the paired change in within-trajectory fidelity between A16 and B16, with uncertainty, on trajectories where both correlations are defined. Which conclusions about each scorer, if any, change? What further conclusion about these scorers follows from the pooled-order result?
- Both F and N retain micro-credit. Holding the scorer and trajectory groups fixed, how often, and by how much, does withholding hindsight change the final normalized/clipped micro-credit term used in the policy update? The exposure counts in Appendix A1.2 come from different arm-specific trajectories and do not isolate this change. Can you provide this comparison using saved trajectories to help interpret the unresolved F–N training contrast?
- Appendix A3.14 reports 31/480 successes with full position support and 46/480 with four positions, under full hindsight on the same C/D banks. What is the descriptive paired interval for the difference between full support and the four-position bridge? The reproducibility statement notes that C/D currently has only documentary support. Can you provide anonymized per-trajectory cell and bridge outcomes, the resampling procedure, and records showing that choices were fixed before D evaluation? These would allow reconstruction of this comparison and the Table 1 intervals.
- Appendix A1.4 acknowledges that the observed results trigger both stopping rule B and extension rule C, with no specified precedence. What determined the decision to stop at t6, and was that decision documented at the time?
2 comments
Response 2 of 2 — paired recovery outcomes and the stopping decision
Q3. With full hindsight and the same C/D replay banks, selecting from all eligible positions gave 31 successes in 480 continuations. Restricting selection to the original four positions gave 46/480. The full-minus-four difference is therefore −3.125 percentage points. We cannot supply a valid paired interval from those totals. It requires the two outcomes for each of the 30 trajectories so that both conditions stay together when trajectories are resampled.
For the two prespecified C/D contrasts reported in Table 1, the documented procedure used 10,000 paired trajectory-bootstrap draws, with the 1.25th and 98.75th percentiles as endpoints for the two-comparison family. The full-versus-four comparison was specified as descriptive, without an interval. We have not recovered the per-trajectory C/D cell and four-position outcomes needed to reconstruct the former intervals or calculate the latter one.
We did recover the plan specifying that choices would be made on C and frozen before evaluation on D. The plan predates the first C run. We did not recover a timestamped record of the actual choices showing that they had been frozen before D was evaluated. Appendix A3.14 (p. 37) and the reproducibility statement (p. 11) now distinguish what the surviving records document from what a reader can independently reconstruct.
Q4. The execution record shows that the scheduled run ended at global step 146, after six updates. At that point both the stopping condition B and the extension condition C applied, and the written protocol gave neither precedence. We found no contemporaneous record explaining why the run ended there rather than continuing. We can document when it ended, but cannot attribute that decision to a particular rule. We have clarified this in Appendix A1.4 and retain the six-update, single-seed result as exploratory.
Response 1 of 2 — scorer fidelity and hindsight during training
Q1. We held each scorer’s values fixed and changed only the replay bank used as the reference. For each of the 43 trajectories, we computed the Spearman correlation across its four positions separately against A16 and B16. For each scorer, we retained the trajectories on which both correlations were defined. Table 9 (p. 20, summarized on p. 5) reports the following results:
• Implicit credit, 29/43 trajectories: median correlation 0.056 on A16 and 0.200 on B16. The median within-trajectory change (B16 minus A16) was +0.180, with a descriptive 95% trajectory-bootstrap interval of [0.000, 0.400]. • Judge, 15/43 trajectories: median correlation 0.577 on A16 and 0.707 on B16. The median within-trajectory change was 0.000 [0.000, 0.447].
We took the difference within each trajectory before taking the median. This is why the judge’s two bank medians can differ while its median paired change is zero. The two scorers also have different sets of computable trajectories. These results quantify the observed change for each scorer; they do not establish that either scorer is invariant to the reference bank or that one is more sensitive than the other. The pooled A/B rank correlation of 0.330 measures agreement between the references and, by itself, gives no further conclusion about either scorer. The revised supplement includes the fixed scores for all 172 positions, the A/B counts, and the analysis code.
Q2. We cannot recover how often, or by how much, withholding hindsight changed the final micro-credit term on identical inputs during the six training updates. We reconstructed the terms actually used in each arm from its saved scores. The arms generated different trajectory groups, the turn texts were not saved, and the intermediate actor checkpoints for steps 141–144 are unavailable. The reconstruction therefore verifies the realized weights but cannot supply the paired prompt comparison you requested.
There is one earlier same-batch check, which we now disclose on p. 8 and in Appendix A1.7 (p. 17). On a batch of 256 trajectories from the base 1.5B policy, a contemporaneous status entry reports a row-weighted relative L2 difference of 0.00275 between the final optimizer coefficients under the two prompts, below the prespecified minimum of 0.05. We have not recovered the raw metric file to recompute that value. This was a prerequisite for a planned three-seed study, which did not run. The six-update pilot started from a learned checkpoint under a protocol frozen after that result. Its protocol explicitly excluded this threshold as a stopping rule. We found no record of a corresponding treatment-strength check at the learned checkpoint or of the reason for the switch. The incremental training value of hindsight remains unresolved.