acceptodds
Under review as a conference paper at ICLR 2027

EventLedger: Typed Counterfactual Credit for Multi-Agent LLM Orchestration

Abstract

Multi-agent LLM systems solve a task by orchestrating typed events: an orchestrator spawns roles, delegates subgoals, issues tool calls, requests critiques, aggregates candidates, and decides to stop. Supervision still arrives once at the end, so the outcome reward is broadcast to every event and a decisive tool call is credited the same as a wasted search or a premature stop. Recent counterfactual credit methods sharpen this signal, but they assume a single homogeneous unit (a message or an agent), are evaluated only with a programmatic verifier, and do not score the stopping decision that governs cost. We present EventLedger, which treats one execution as an orchestration event graph and assigns each event an interventional, auditable process reward. Typed counterfactual operators define a type-compatible intervention for every event type, including a force-continue/force-stop pair that makes termination a scored decision. A perturb-rollout engine estimates the marginal effect of every replayed event without bias, pairs factual and counterfactual continuations with common random numbers, and spends its replay budget on the highest-leverage events. For tasks without a verifier, a cross-family judge committee is calibrated isotonically on verifiable anchors and abstains by a conformal rule, which bounds the credit bias under an explicit transfer assumption. The rewards are combined with potential-based shaping and distilled into a graph-aware student that makes no API calls at inference. Across coding, math, text-to-SQL and open-ended question answering, an orchestrator trained with EventLedger rewards reaches an average score of against for the strongest baseline; the budgeted estimator needs of the replay jobs of the full estimator; the oracle pipeline lowers a single judge's expected calibration error from to at retention; and the student reaches agreement with the teacher at its scoring cost while matching the end-task scores of the teacher-trained orchestrator.

Primary area:
reinforcement learning
Keywords:
Multi-Agent LLM Systems, Credit Assignment, Process Reward Models, Counterfactual Intervention, Conformal Calibration

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_rbNWOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes an approach to computing process rewards that involves running rollouts from counterfactual decisions within the trajectory. This basic counterfactual is introduced in equation (5); there are then several layers of statistical tricks (importance sampling, isotonic regression, conformal abstention, distillation), which are somewhat difficult to follow but which apparently reduce the cost. Evaluation compares the proposed approach with many baselines, but on evaluations that are either saturated or small.

Critical strengths
  • The core idea of counterfactual comparisons makes sense, and the formal argumentation is largely persuasive through section 3.3
  • The student-teacher architecture significantly reduces costs while retaining accuracy.
  • There are several ablations to try to isolate the key methodological features.
Critical weaknesses
  • Relationship to prior work on PRMs could be clearer. I understand that this approach specifically targets the counterfactual comparison, but it would help to say more about when that is expected to matter.
  • Empirical evaluation is weak: the code_math evals are saturated and the remaining evals are too small.
  • The approach has lots of tunable knobs and hyperparameters, and it's not clear whether these were tuned for success on this specific set of evals, or whether they are generally applicable.
Questions for authors

What specifically is the openqa eval? Why not use something more standard here?

Eq 11 has many hyperparameters, how are they set?

Why the Huber loss in (13)?

What is the role of conformal abstention?

6 comments
AuthorsOpenReview ↗

Response to Reviewer rbNW (1/3)

Dear Reviewer rbNW,

Thank you for the review. Below we clarify the method's structure, relate it to PRMs, and report the sensitivity and evaluation details you asked for. All changes are in the revised PDF (in blue); numbers below refer to it.

A clarification first. The credit estimator does not use importance sampling. Its core is the paired replay estimator of Eq. 5, whose unbiasedness is Prop. 3. The other components each address one practical need, and the revised Section 3 now opens with a table saying which (Table 1):

ComponentPurposeApplied to
Paired counterfactual replay (Eq. 5)estimate interventional event creditall tasks
CRN pairing; top-mm selectionreduce variance; reduce replay costall tasks
Isotonic calibration; conformal abstentioncalibrated readout; filter uncertain readoutstasks without a verifier (OpenQA only)
Student distillationscore events without judge or replay API callsall tasks

Four of the five benchmarks are scored by programmatic verifiers, so the calibration machinery enters only on OpenQA.

W1. Relationship to PRMs: when do counterfactual comparisons matter?

PRMs such as those of Lightman et al., Math-Shepherd and MASPRM score an intermediate step by its correctness or by outcome-related labels. EventLedger instead estimates how much the outcome changes when that specific event is replaced. The two diverge when an event is reasonable but unnecessary, which is common in orchestration:

  • a correct but redundant search or critique scores well as a step yet changes nothing;
  • a well-formed stop can still forgo a solution (the MBPP case in App. A.7).

We measured this divergence. On 40 trajectories in which every event was replayed (1,312 events), a MASPRM-style PRM's event scores correlate with full-replay credit at Spearman 0.31. EventLedger-S reaches 0.79 on replayed events and 0.68 on the rest (App. A.9, Table 14). Section 2 now states this.

Q4. What is the role of conformal abstention?

It filters uncertain oracle readouts when no verifier exists: a continuation is rejected when committee disagreement exceeds the calibrated threshold, and the pair is then discarded (Sec. 3.4). Its intended role is cleaner training targets. Against reference answer aliases on OpenQA, it lowers the measured credit bias from 0.031 to 0.021 and raises accuracy on retained continuations from 75.3% to 77.8% at 91.0% retention. Removing it lowers the downstream average by 0.7 points (App. A.8). Its formal guarantee covers only the abstention rate, so the error reduction is an empirical finding; the revised Prop. 4 also makes the selection effect of abstention explicit.

AuthorsOpenReview ↗

Response to Reviewer rbNW (2/3)

W3 / Q2. Many knobs: how are the Eq. 11 hyperparameters set (Eq. 12 in the revision)?

All weights, together with β_$, mm and α\alpha, were selected once on validation data and shared across benchmarks. There was no test-set tuning and no per-benchmark selection (Sec. 4). To show the results are not knife-edge, we varied one reward coefficient at a time with the others fixed (App. A.10, Table 15; default validation average 86.4):

Coefficient×0×0.5×2
Shaping84.986.186.2
λg\lambda_g grounding86.086.386.2
λr\lambda_r redundancy86.286.486.0
λc\lambda_c conflict86.386.486.3
λ_$ cost86.586.485.8
λs\lambda_s stopping86.486.486.3

Halving or doubling any coefficient moves the average by at most 0.6 points. Only removing shaping entirely costs more (1.5). The stopping trade-off β_$ is similarly flat around the default (Table 16): β_$ ∈ {0, 1, 10} changes the score by at most 0.3 points, while β_$ = 1 uses 15.2% fewer tokens than β_$ = 0.

Q3. Why the Huber loss in Eq. 13 (Eq. 14 in the revision)?

Replay-derived targets are noisy, with occasional large residuals. Huber is quadratic for small residuals and linear beyond δ\delta, so it fits the bulk smoothly without letting outliers dominate. It is a modeling choice, not a theoretical requirement. Under the same data, architecture and protocol (Table 17):

LossSpearman with teacher ↑Sibling rank accuracy (%) ↑Avg. score ↑
MSE0.7482.186.0
L10.7684.086.1
Huber (δ=1\delta=1)0.8086.786.4
AuthorsOpenReview ↗

Response to Reviewer rbNW (3/3)

W2 / Q1. Saturated code/math, small remaining evaluations, and what OpenQA is.

OpenQA is a 100-question subset of Natural Questions Open, using the DPR release with retrieved passages and the dataset's answer aliases, split 70/15/15 (Sec. 4, App. A.4). The underlying benchmark is standard; our subset is small, and the submission did not name its source, which it now does. Because the same judge committee also produces the training rewards, we validated it against the aliases (Sec. 4.4, Table 9): the calibrated committee reaches a credit correlation of 0.66 with alias-based credit, against 0.58 for a single judge.

Saturation. We agree that code/math leaves little headroom for accuracy gains. It is still informative in one direction: a strong single agent shows whether reward training preserves capability, and eight of nine trained baselines fall well below the single agent on GSM8K. While checking split sizes for the other reviewers, however, we found that the EventLedger rows on GSM8K and MBPP were computed over the full datasets, including training partitions, whereas the trained baselines were evaluated on held-out partitions. The revision marks those cells, withdraws the end-task margin over HCAPO (Sec. 4.1), and we are re-evaluating all methods on identical held-out IDs with repeated runs; we will post the results here when complete.

For this reason, the revised paper rests its claims on comparisons that saturation and evaluation size do not affect:

  • credit fidelity against full replay (Table 14);
  • the frozen-orchestrator controller and stopping-reward analysis (Table 6), where tokens drop by 36.6% at unchanged score;
  • the oracle's agreement with reference aliases (Table 9).

Thank you again. We hope the component table, the PRM comparison and the sensitivity results make the method easier to follow and show that it is not tuned to these evaluations. If so, we would appreciate it if you would consider raising your score.

Best regards,

Authors

Reviewer_ZPmwOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

EventLedger addresses fine-grained credit assignment in multi-agent LLM orchestration. It represents an execution as a graph of typed events and defines counterfactual interventions for decisions such as tool calls, critiques, aggregation, and stopping. Paired factual and counterfactual continuations estimate event-level contributions, while shared randomness and selective replay reduce estimation costs. For tasks without programmatic verification, a cross-family judge committee provides scores through isotonic calibration and an abstention mechanism. These rewards are distilled into a graph-aware student used for orchestrator training and inference-time control. Evaluation covers HumanEval, MBPP, GSM8K, Spider, and OpenQA, with component ablations, cost measurements, judge calibration, and transfer experiments. The paper reports an average score of 87.75 versus 85.57 for HCAPO, replay usage equal to 39.8% of the full estimator’s replay jobs, and a 52.21% reduction in deployment tokens for the student-based system relative to the untrained orchestrator. All reported results are single runs, and OpenQA is evaluated using the proposed CGO scorer.

Critical strengths
  1. A meaningful formulation of heterogeneous orchestration decisions. The method gives tool calls, critiques, and stopping distinct counterfactual interpretations. It also explicitly scores termination.
  2. A practical path from expensive attribution to inexpensive scoring. Selective replay and student distillation address the computational cost of counterfactual evaluation. The reported MBPP and GSM8K improvements provide evidence that the resulting rewards are useful for training.
Critical weaknesses
  1. The oracle’s reliability is not yet convincing. Proposition 4 does not clearly account for the selection bias introduced by abstention. Table 5 also needs explanation: with fixed predictions and ordinary per-query accuracy, retaining 93.33% of samples cannot raise accuracy from 76.67% to 83.33%.
  2. The experiments provide limited evidence for the claimed benefits. Results are based on single runs, and the entire gain over HCAPO comes from two benchmarks. The cost comparison also changes both the trained policy and the stopping controller, so it does not isolate the controller’s contribution.
Questions for authors

Q1: Does Proposition 4 bound error relative to Equation (2), or to a retention-conditioned credit? When one continuation abstains, is the pair discarded, resampled, or the entire event masked? Please specify the estimator and the assumption, bound, or correction controlling selection bias. Q2: Please provide train/dev/test/anchor sizes and evaluation IDs for each method, the OpenQA source and scoring protocol, and an explanation of the Spider evaluation resolution. Do the MBPP/GSM8K gains persist across repeated runs and paired uncertainty estimates? Is there independent human or evaluator-based validation of the OpenQA results? Q3: With the same frozen orchestrator, what are the quality and token/tool/latency costs when the stopping controller is enabled versus disabled? Alternatively, does a strictly matched stopping-reward ablation support the claimed cost attribution?

4 comments
AuthorsOpenReview ↗

Response to Reviewer ZPmw (1/3)

Dear Reviewer ZPmw,

Thank you for the precise questions. Each one identified something we have now corrected or measured. All changes are in the revised PDF (in blue); numbers below refer to it.

W1 / Q1. What does Prop. 4 bound, and how is selection by abstention handled?

You are right that the submitted Prop. 4 did not account for selection. The estimator is now stated explicitly (Sec. 3.4). When either the factual or the counterfactual continuation abstains, the whole pair is discarded without resampling; an event with no retained pair is masked. For event ete_t with jointly retained pairs KR\mathcal{K}_R:

Δ^R(et)=1∣KR∣∑k∈KR[g(Y^fac(k))−g(Y^cf(k))].\widehat{\Delta}_R(e_t)=\frac{1}{|\mathcal K_R|}\sum_{k\in\mathcal K_R}\Big[g(\hat Y_{\mathrm{fac}}^{(k)})-g(\hat Y_{\mathrm{cf}}^{(k)})\Big].

The revised Prop. 4 assumes the calibrated oracle is ϵ\epsilon-conditionally unbiased on jointly retained continuations. Under that assumption, Δ^R\widehat\Delta_R has bias at most 2ϵ2\epsilon relative to the retention-conditioned credit E[D∣R=1]\mathbb{E}[D\mid R{=}1]. Relative to the unconditional credit of Eq. 2, the bound adds the selection term

∣E[D∣R=1]−E[D]∣=Pr⁡(R=0) ∣E[D∣R=1]−E[D∣R=0]∣,\big|\mathbb{E}[D\mid R{=}1]-\mathbb{E}[D]\big| = \Pr(R{=}0)\,\big|\mathbb{E}[D\mid R{=}1]-\mathbb{E}[D\mid R{=}0]\big|,

where D=Vfac−VcfD=V_{\mathrm{fac}}-V_{\mathrm{cf}} (Eq. 11; proof in App. A.1). Discarding whole pairs preserves the CRN coupling but does not by itself remove this term, so we measured it on OpenQA against reference answer aliases (App. A.8, Table 13):

BranchAbstention rate aja_jMean outcome difference, retained vs. abstained djd_jShift ajdja_jd_j
Factual6.1%0.030.002
Counterfactual (all)9.8%0.050.005
Counterfactual: TOOL-corrupt14.2%0.090.013

The selection shift is small overall and largest on corrupted tool calls, whose continuations are hardest to judge.

W1 (Table 5, now Table 7). Your arithmetic is correct: with fixed predictions, declining 6.67% of queries cannot raise accuracy from 76.67% to 83.33%. The first three rows report the accuracy of the raw committee predictions. The conformal row thresholds the isotonic-calibrated score before abstaining, so both the predictions and the retained set change. The accuracy column is therefore not a cumulative pipeline, and Sec. 4.4 now says so explicitly. The per-example predictions and retained/correct counts are released with the code. Because this split consists of verifiable anchors, it also says nothing directly about OpenQA, so we evaluated OpenQA separately (Q2 below).

AuthorsOpenReview ↗

Response to Reviewer ZPmw (2/3)

W2 / Q3. The cost comparison changes both the policy and the controller.

Agreed, so we separated them. With each trained orchestrator frozen, we toggled only the controller (Sec. 4.3, Table 6):

Frozen orchestratorControllerAvg. scoreTokens (K)API callsTool callsLatency (s)
Teacher-trainedoff86.309.773.181.0313.3
Teacher-trainedon86.226.212.071.007.9
Student-trainedoff86.369.523.111.0212.9
Student-trainedon86.366.042.001.007.5

The controller cuts tokens by 36.4% and 36.6%, API calls by about a third and latency by over 40%, while the score moves by at most 0.08.

To check that the saving comes from the stopping reward and not just from having a controller, we retrained with only SstopS_{\text{stop}} removed from the reward, keeping the controller on. Tokens rise from 6.04K to 8.88K at a score of 86.29 versus 86.36. This is the strictly matched stopping-reward ablation you asked for.

W2. Single runs; the gain over HCAPO comes from two benchmarks.

While answering Q2 we found a problem that is more serious than single runs, and we report it plainly. The EventLedger rows on GSM8K and MBPP were computed over the full datasets, including the training partitions (95.44 = 859/900; 99.07 = 318/321), whereas the trained baselines were evaluated on held-out partitions. The MBPP/GSM8K margin over HCAPO, and therefore the +2.18 average, is not a valid comparison. The revision:

  • marks those cells in Table 3;
  • states the scope in Sec. 4.1;
  • removes the margin from the abstract and introduction.

We are re-evaluating every method on identical held-out IDs with repeated runs and paired intervals, and will post the results in this thread when they are complete. Until then, the paper's claims rest on the matched analyses above and in Q2, which do not depend on those cells.

AuthorsOpenReview ↗

Response to Reviewer ZPmw (3/3)

Q2. Splits, the OpenQA protocol, Spider resolution, and independent validation.

Splits. The partitions are now in Table 10 (App. A.4): HumanEval 120/20/24, MBPP 225/48/48, GSM8K 720/90/90, Spider 70/15/15 and OpenQA 70/15/15 (train/val/test). The oracle's anchor pool is the 1,065 code/math training problems, and Table 7 uses a held-out anchor partition. Task IDs for every method will be released with the re-evaluation.

OpenQA. It is 100 questions with retrieved passages from the Natural Questions Open development data released with DPR, with the dataset's answer aliases as references. The same committee produces training rewards and OpenQA scores.

Independent validation. We evaluated the oracle against the aliases, which never enter training (Sec. 4.4, Table 9). The calibrated committee tracks alias-based credit better than a single judge (Spearman 0.66 vs. 0.58), close to in-domain calibration (0.69); removing the worker family gives 0.68. Aliases can miss correct paraphrases, and we have not run a human evaluation.

Spider. Trained rows use the 30 validation and test problems, so one problem is 3.33 points (the submission's "one in fifteen" was wrong). Repeated runs measure generation variability but do not add unique problems, so we continue to treat small differences in that column as ties.

Repeated runs. They are part of the re-evaluation described under W2.


Thank you again for the review; Q1 and Q3 in particular led to concrete corrections. If the corrected Prop. 4, the matched controller and stopping-reward analyses, and the OpenQA validation address your concerns, we would be grateful if you would consider raising your score.

Best regards,

Authors

Thank you for the detailed response. The experiments addressed several of my concerns. However, with the claimed advantage withdrawn, I remain unconvinced of the method’s performance gains. I maintain my rating of 2.

Reviewer_4zVUOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper treats one multi-agent LLM run as a graph of typed events. It credits each event with its interventional effect: replay the run with that event swapped for a type-specific counterfactual and measure how much the outcome moves. The building blocks are a table of per-type operators, including a force-continue/force-stop pair so that stopping gets a reward; a paired-rollout estimator that uses common random numbers and replays only the top high-leverage events; and a two-model judge committee for tasks without a verifier, calibrated with isotonic regression on code/math anchors, that abstains via a conformal threshold. A distilled graph-aware student (EventLedger-S) then supplies the rewards for PPO training of a Qwen-3.5 orchestrator. Across HumanEval, MBPP, GSM8K, Spider and an unnamed OpenQA suite, the method reports an average of 87.75 vs. 85.57 for HCAPO. The student matches the teacher end to end and cuts deployment tokens by 52%.

Critical strengths
  1. The problem is real. Spreading one terminal reward across very different orchestration decisions is a known weak spot, and treating "when to stop" as a creditable decision is the paper's most original idea. It ties credit directly to cost.

  2. Typed operators are a sensible step past message deletion. Table 1 is clear, and pairing each operator with the question it asks makes the credit easier to interpret.

  3. The estimator section is careful about what it guarantees. It separates unbiasedness from variance reduction (Prop. 3). It points out that seed-exact replay isn't guaranteed by API providers. It also declines to force Shapley-style efficiency, monitoring the conservation gap (Eq. 8) instead of enforcing it. I found the argument for that choice convincing.

  4. Cost is reported, not just accuracy. The point that the outcome-reward orchestrator is "cheap for a bad reason" is a useful observation for the field.

Critical weaknesses

W1. The evaluation is too small for the claims being made, and some numbers don't fit the stated granularity.
The paper says Spider and OpenQA scores move in steps of 6.67 points, i.e., 15 problems each. All results are single runs with no significance tests. With +2.18 average points as the headline margin, that is thin to begin with. Also, the trained GSM8K entries (86.67, 90.00, 88.89, 84.44, 85.56) are all multiples of 1/90, which suggests a 90-problem split. But EventLedger's 95.44 and HCAPO's 89.89 are not (the nearest values are 95.56/94.44 and 90.00/88.89). Similarly, the trained MBPP entries (89.58, 93.75, 91.67, 95.83, 97.92) are multiples of 1/48, but EventLedger's 99.07 is not.

W2. The committee is small, includes the worker family, and is calibrated on the wrong domain.
The "cross-family committee" is two models, {GLM-5.2, GPT-4o}. GLM-5.2 is also every worker agent, so the self-preference the committee is meant to "dilute" is built in. The authors acknowledge this in A.7. Isotonic calibration and the conformal threshold are fit on HumanEval/MBPP/GSM8K anchors, then applied to open-ended QA continuations, which is exactly the transfer Prop. 4 assumes and the paper says can't be checked. Table 5 is also measured on a held-out verifiable anchor split, so it says nothing about reliability on OpenQA.

Questions for authors
  1. How many problems are in each evaluation split, and how do you get 95.44 on GSM8K and 99.07 on MBPP when the other rows sit on 1/90 and 1/48 grids?

  2. What is the OpenQA suite (source, size, how the references were built)? Is the evaluation judge the same committee that produced the training rewards?

  3. How does Prop. 4 handle different abstention rates for factual vs. counterfactual continuations?

  4. Where full-replay credit is available, how closely does the student's predicted credit match it for events outside the top-m?

  5. Was EventLedger-S retrained for each ablation row? Why are three ablation pairs identical to two decimals?

  6. How was β in Eq. 12 chosen, given that cost is in dollars and gain is in score units, and how sensitive is stopping behavior to it?

3 comments
AuthorsOpenReview ↗

Response to Reviewer 4zVU (1/3)

Dear Reviewer 4zVU,

Thank you for an unusually careful review. Your grid analysis found a real problem, and we address it first and directly. The revised PDF marks all changes in blue; table and section numbers below refer to it.

W1 / Q1. Split sizes, and how 95.44 and 99.07 arise.

The partitions are now in Appendix A.4 (Table 10):

BenchmarkTrainingValidationTestTotal
HumanEval1202024164
MBPP2254848321
GSM8K7209090900
Spider701515100
OpenQA (NQ-Open)701515100

You are right that 95.44 and 99.07 do not sit on the held-out grids. They are 859/900859/900 and 318/321318/321: the EventLedger rows on GSM8K and MBPP were computed over the full datasets, including the training partitions, whereas the trained baselines were evaluated on held-out partitions. Those cells therefore do not support a ranking, and neither do the averages that include them. In the revision we:

  • mark the affected cells in Table 3 and state the scope in its caption and in Sec. 4.1;
  • withdraw the claim of a +2.18 average margin over HCAPO;
  • remove the 87.75-vs-85.57 comparison from the abstract and introduction.

We are re-evaluating every method on identical held-out IDs, with repeated runs and paired intervals, and will post the results in this thread as soon as they are complete. The per-example outputs and task IDs will be released with the code.

On Spider and OpenQA, the "one problem in fifteen" sentence was wrong: trained rows use the 30 validation and test problems, so one problem is 3.33 points. We still treat differences of a few problems in those columns as ties.

We want to be clear about what this does and does not affect. The claims that rest on matched comparisons, where only one factor changes, do not depend on these cells: the stopping controller and stopping reward (Table 6), student-versus-full-replay credit (Table 14), the oracle on OpenQA (Table 9) and the sensitivity analyses (App. A.10). The revised abstract and introduction now lead with those.

Q2. What is the OpenQA suite, and is the evaluation judge the training committee?

OpenQA is 100 questions with retrieved passages from the Natural Questions Open development data released with DPR (Karpukhin et al., 2020). We took the first 100 valid examples with aligned question, answers and retrieval, split 70/15/15 (Sec. 4, App. A.4). The references are the dataset's answer aliases. And yes, the same committee (GLM-5.2 + GPT-4o) produces the training rewards and the OpenQA score. That is why we added an evaluation against the reference aliases, which never enter training (W2 below).

Q3. How does Prop. 4 handle different abstention rates for factual and counterfactual continuations?

The submitted version did not. Different abstention rates alone do not bias credit; what matters is whether abstention selects on the outcome. We now specify the estimator: if either continuation of a pair abstains, the whole pair is discarded without resampling, and an event with no retained pair is masked. Prop. 4 is restated accordingly (Sec. 3.4, Eq. 11). It bounds the bias by 2ϵ2\epsilon relative to the retention-conditioned credit E[D∣R=1]\mathbb{E}[D\mid R{=}1]. Relative to the unconditional credit, it adds the selection term

E[D∣R=1]−E[D]=Pr⁡(R=0) (E[D∣R=1]−E[D∣R=0]),\mathbb{E}[D\mid R{=}1]-\mathbb{E}[D] = \Pr(R{=}0)\,\big(\mathbb{E}[D\mid R{=}1]-\mathbb{E}[D\mid R{=}0]\big),

where DD is the factual-minus-counterfactual difference and R=1R=1 means both continuations were retained. We measured the per-branch selection shift on OpenQA against alias labels (Table 13): 0.002 for the factual branch, 0.005 for all counterfactual branches, and 0.013 for corrupted tool calls, the hardest to judge.

AuthorsOpenReview ↗

Response to Reviewer 4zVU (2/3)

W2. Two-member committee that includes the worker family, calibrated on another domain.

We agree that the submission did not validate the oracle on OpenQA. We now do so directly, using the NQ-Open answer aliases as analysis-only labels on held-out questions (Sec. 4.4, Table 9). The in-domain reference uses a separate OpenQA calibration split, disjoint from the evaluation questions.

CommitteeCalibration dataECE ↓Acc. on retained (%) ↑Retained (%)ρ with alias credit ↑
GLM-5.2code/math anchors0.2474.1100.00.58
GLM-5.2 + GPT-4o (paper)code/math anchors0.1777.891.00.66
GLM-5.2 + GPT-4oseparate OpenQA split0.1178.990.30.69
GPT-4o + Claude (no worker family)code/math anchors0.1578.492.00.68

Two conclusions. First, calibration transfer is imperfect but mostly holds: code/math calibration reaches a credit correlation of 0.66, against 0.69 with in-domain calibration. Second, your self-preference concern is borne out in direction: removing the worker family raises the correlation to 0.68. Aliases can miss correct paraphrases, and we have not run a human evaluation; the limitations section says both.

Q4. Student credit outside the top-mm versus full replay.

We replayed every event of 40 held-out trajectories (1,312 events), with the student and the top-mm selection frozen beforehand (App. A.9, Table 14):

EventsSpearman ↑Sign agreement ↑MAE ↓
Within top-mm0.7988%0.05
Outside top-mm0.6881%0.07
Outside top-mm, ∣Δfull∣>0.2\lvert\Delta_{\text{full}}\rvert>0.20.7493%0.11

Only 3.1% of events outside the budget carry large credit, and the student gets their sign right 93% of the time. The approximation outside the budget is real but bounded.

Q5. Was EventLedger-S retrained per ablation row, and why are three pairs identical?

Yes, every row retrains the student and the orchestrator with the component removed. The tied pairs differ in event scores and decisions but not in the number of solved problems on the development split, so they round to the same aggregate. We do not claim to separate those pairs, and the unrounded per-example results are released with the code (Sec. 4.2).

Q6. How was β_$ chosen, and how sensitive is stopping to it?

β_$ converts dollars into outcome units (score units per USD); we use β_$ = 1, chosen once on validation data. With outcomes in [0,1][0,1] and typical savings of $0.01–$0.05, a stop is rewarded only when its predicted forgone gain is below that small amount, which is a conservative trade-off (Sec. 3.5). With the orchestrator frozen and only β_$ varied (App. A.10, Table 16):

β_$01 (paper)101001000
Avg. score86.486.486.184.079.2
Tokens/task (K)7.126.045.614.473.30

β_$ = 1 keeps the score of β_$ = 0 with 15.2% fewer tokens. Only much larger values start to trade score for cost.

AuthorsOpenReview ↗

Response to Reviewer 4zVU (3/3)

Statistical reliability. We agree that single runs are thin evidence for small end-task margins, which is why the revision bases its claims on matched, within-method comparisons. Two of these speak directly to your cost-related strength:

  • Stopping controller (Table 6). With each trained orchestrator frozen, enabling the controller cuts tokens by 36.4% (teacher-trained) and 36.6% (student-trained) while the average score moves by at most 0.08.
  • Stopping reward. Retraining with only SstopS_{\text{stop}} removed raises tokens from 6.04K to 8.88K at essentially the same score (86.29 vs. 86.36).

Together these tie the cost saving to the stopping signal, which you identified as the most original part of the paper. The repeated-run, identical-ID re-evaluation of the end-task table will follow in this thread.


Thank you again; the review led to concrete corrections. If the revised scope of the claims, the OpenQA validation and the matched analyses address your concerns, we would appreciate it if you would consider raising your score.

Best regards,

Authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response to All Reviewers and the AC

Dear Reviewers and Area Chair,

Thank you for the reviews. They converged on three issues: the size and resolution of the evaluation, the reliability of the oracle where no verifier exists (including selection by abstention), and whether the cost savings can be attributed to the stopping signal. We uploaded a revised PDF with all changes in blue; the main text stays within 10 pages, and no new references were needed.

A correction we want to state up front. Checking split sizes for Reviewer 4zVU, we found that the EventLedger rows on GSM8K and MBPP in the end-task table were computed over the full datasets, including training partitions (859/900 and 318/321), whereas the trained baselines were evaluated on held-out partitions. These cells do not support a ranking. The revision:

  • marks them in Table 3 and explains the scope in Sec. 4.1 and App. A.4;
  • withdraws the +2.18 average margin over HCAPO from the abstract and introduction.

We are re-evaluating every method on identical held-out IDs, with repeated runs and paired intervals, and will post the results in this thread when they are complete.

Matched analyses, each changing one factor

AnalysisResultWhereReviewers
Stopping controller on/off, orchestrator frozenTokens −36.4% (teacher-trained) and −36.6% (student-trained); score change ≤ 0.08Sec. 4.3, Table 6ZPmw, 4zVU
Stopping reward removed, controller still onTokens 6.04K → 8.88K; score 86.36 vs. 86.29Table 6ZPmw
Student credit vs. full replay (40 trajectories, 1,312 events)Spearman 0.79 within / 0.68 outside the replay budget; MASPRM-style PRM: 0.31App. A.9, Table 144zVU, rbNW
Oracle vs. reference aliases on OpenQAρ = 0.66 (paper committee) vs. 0.58 (single judge) and 0.69 (in-domain calibration); 0.68 without the worker familySec. 4.4, Table 94zVU, ZPmw, rbNW
Selection by abstentionProp. 4 restated for retention-conditioned credit plus an explicit selection term; measured shift ≤ 0.013 per branchSec. 3.4, Eq. 11; App. A.8ZPmw, 4zVU
Sensitivity of weights, β_$ and lossHalving or doubling any coefficient: ≤ 0.6 points; β_$ = 1 saves 15.2% tokens over β_$ = 0 at equal score; Huber beats MSE and L1App. A.10rbNW, 4zVU

Clarifications

  • Benchmark partitions are now listed (Table 10).
  • Spider and OpenQA trained rows use 30 problems each (3.33 points per problem).
  • OpenQA is a 100-question subset of NQ-Open (DPR release).
  • The accuracy column of the oracle table (now Table 7) is not a fixed-prediction pipeline, and the text now says so.
  • Every ablation row retrains the student.
  • A component table opens Section 3.

Detailed responses are posted under each review.

Best regards,

Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.