- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper proposes an approach to computing process rewards that involves running rollouts from counterfactual decisions within the trajectory. This basic counterfactual is introduced in equation (5); there are then several layers of statistical tricks (importance sampling, isotonic regression, conformal abstention, distillation), which are somewhat difficult to follow but which apparently reduce the cost. Evaluation compares the proposed approach with many baselines, but on evaluations that are either saturated or small.
Critical strengths
- The core idea of counterfactual comparisons makes sense, and the formal argumentation is largely persuasive through section 3.3
- The student-teacher architecture significantly reduces costs while retaining accuracy.
- There are several ablations to try to isolate the key methodological features.
Critical weaknesses
- Relationship to prior work on PRMs could be clearer. I understand that this approach specifically targets the counterfactual comparison, but it would help to say more about when that is expected to matter.
- Empirical evaluation is weak: the code_math evals are saturated and the remaining evals are too small.
- The approach has lots of tunable knobs and hyperparameters, and it's not clear whether these were tuned for success on this specific set of evals, or whether they are generally applicable.
Questions for authors
What specifically is the openqa eval? Why not use something more standard here?
Eq 11 has many hyperparameters, how are they set?
Why the Huber loss in (13)?
What is the role of conformal abstention?
6 comments
Response to Reviewer rbNW (2/3)
W3 / Q2. Many knobs: how are the Eq. 11 hyperparameters set (Eq. 12 in the revision)?
All weights, together with β_$, and , were selected once on validation data and shared across benchmarks. There was no test-set tuning and no per-benchmark selection (Sec. 4). To show the results are not knife-edge, we varied one reward coefficient at a time with the others fixed (App. A.10, Table 15; default validation average 86.4):
| Coefficient | ×0 | ×0.5 | ×2 |
|---|---|---|---|
| Shaping | 84.9 | 86.1 | 86.2 |
| grounding | 86.0 | 86.3 | 86.2 |
| redundancy | 86.2 | 86.4 | 86.0 |
| conflict | 86.3 | 86.4 | 86.3 |
λ_$ cost | 86.5 | 86.4 | 85.8 |
| stopping | 86.4 | 86.4 | 86.3 |
Halving or doubling any coefficient moves the average by at most 0.6 points. Only removing shaping entirely costs more (1.5). The stopping trade-off β_$ is similarly flat around the default (Table 16): β_$ ∈ {0, 1, 10} changes the score by at most 0.3 points, while β_$ = 1 uses 15.2% fewer tokens than β_$ = 0.
Q3. Why the Huber loss in Eq. 13 (Eq. 14 in the revision)?
Replay-derived targets are noisy, with occasional large residuals. Huber is quadratic for small residuals and linear beyond , so it fits the bulk smoothly without letting outliers dominate. It is a modeling choice, not a theoretical requirement. Under the same data, architecture and protocol (Table 17):
| Loss | Spearman with teacher ↑ | Sibling rank accuracy (%) ↑ | Avg. score ↑ |
|---|---|---|---|
| MSE | 0.74 | 82.1 | 86.0 |
| L1 | 0.76 | 84.0 | 86.1 |
| Huber () | 0.80 | 86.7 | 86.4 |
Response to Reviewer rbNW (3/3)
W2 / Q1. Saturated code/math, small remaining evaluations, and what OpenQA is.
OpenQA is a 100-question subset of Natural Questions Open, using the DPR release with retrieved passages and the dataset's answer aliases, split 70/15/15 (Sec. 4, App. A.4). The underlying benchmark is standard; our subset is small, and the submission did not name its source, which it now does. Because the same judge committee also produces the training rewards, we validated it against the aliases (Sec. 4.4, Table 9): the calibrated committee reaches a credit correlation of 0.66 with alias-based credit, against 0.58 for a single judge.
Saturation. We agree that code/math leaves little headroom for accuracy gains. It is still informative in one direction: a strong single agent shows whether reward training preserves capability, and eight of nine trained baselines fall well below the single agent on GSM8K. While checking split sizes for the other reviewers, however, we found that the EventLedger rows on GSM8K and MBPP were computed over the full datasets, including training partitions, whereas the trained baselines were evaluated on held-out partitions. The revision marks those cells, withdraws the end-task margin over HCAPO (Sec. 4.1), and we are re-evaluating all methods on identical held-out IDs with repeated runs; we will post the results here when complete.
For this reason, the revised paper rests its claims on comparisons that saturation and evaluation size do not affect:
- credit fidelity against full replay (Table 14);
- the frozen-orchestrator controller and stopping-reward analysis (Table 6), where tokens drop by 36.6% at unchanged score;
- the oracle's agreement with reference aliases (Table 9).
Thank you again. We hope the component table, the PRM comparison and the sensitivity results make the method easier to follow and show that it is not tuned to these evaluations. If so, we would appreciate it if you would consider raising your score.
Best regards,
Authors
Response to Reviewer rbNW (1/3)
Dear Reviewer rbNW,
Thank you for the review. Below we clarify the method's structure, relate it to PRMs, and report the sensitivity and evaluation details you asked for. All changes are in the revised PDF (in blue); numbers below refer to it.
A clarification first. The credit estimator does not use importance sampling. Its core is the paired replay estimator of Eq. 5, whose unbiasedness is Prop. 3. The other components each address one practical need, and the revised Section 3 now opens with a table saying which (Table 1):
Four of the five benchmarks are scored by programmatic verifiers, so the calibration machinery enters only on OpenQA.
W1. Relationship to PRMs: when do counterfactual comparisons matter?
PRMs such as those of Lightman et al., Math-Shepherd and MASPRM score an intermediate step by its correctness or by outcome-related labels. EventLedger instead estimates how much the outcome changes when that specific event is replaced. The two diverge when an event is reasonable but unnecessary, which is common in orchestration:
We measured this divergence. On 40 trajectories in which every event was replayed (1,312 events), a MASPRM-style PRM's event scores correlate with full-replay credit at Spearman 0.31. EventLedger-S reaches 0.79 on replayed events and 0.68 on the rest (App. A.9, Table 14). Section 2 now states this.
Q4. What is the role of conformal abstention?
It filters uncertain oracle readouts when no verifier exists: a continuation is rejected when committee disagreement exceeds the calibrated threshold, and the pair is then discarded (Sec. 3.4). Its intended role is cleaner training targets. Against reference answer aliases on OpenQA, it lowers the measured credit bias from 0.031 to 0.021 and raises accuracy on retained continuations from 75.3% to 77.8% at 91.0% retention. Removing it lowers the downstream average by 0.7 points (App. A.8). Its formal guarantee covers only the abstention rate, so the error reduction is an empirical finding; the revised Prop. 4 also makes the selection effect of abstention explicit.