- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The authors propose ROLLCREDITS, a method for credit assignment through counterfactual advantage estimation. They use execution graphs to provide a unified representation of multi agents interaction pipelines. At the same interaction history, an LLM agent generates multiple candidate responses, and each branch is executed to completion. The resulting outcomes are then compared using the acting agent’s final reward. Experiments demonstrate performance improvements in math, coding, and games.
Critical strengths
The DAG formulation is clearly presented, the overall pipeline is well motivated, and the experimental results are strong. The generalisation to a game not seen during training is particularly interesting.
Critical weaknesses
The proposed idea and method appear quite similar to this prior work [1], which already introduces a counterfactual rollout estimator with an unbiasedness guarantee and branching from saved states to reduce computation. This case would diminish the paper’s contributions and novelty. The unified execution-graph formulation appears to be a new contribution, but its significance is not clearly demonstrated in the method design or experiments. All above raises concerns about the paper’s novelty and overall contributions.
[1] Exact Is Easier: Credit Assignment for Cooperative LLM Agents
Questions for authors
-
The authors need to explain the difference between Exact Is Easier and their paper. If possible, the authors need to add the Exact Is Easier as one baseline in the exprionments.
-
Could you include at least one parallel fan-out/fan-in topology?
-
Is the token budget kept consistent across all the other baselines?
5 comments
Response to Reviewer NwT8 (2/2)
R3.Q3: Token budgets
The optimization schedule and sequence budget are the same across methods: 220 updates for math/games and 162 for code, with 64 groups of 8 sequences per update. Training-token totals vary with sequence length, and generation budgets vary with rollout collection. RollCredits, C3 and VinePPO all estimate credit from sampled continuations, so generation is the cost each pays for its credit signal, and a matched generation budget compares how efficiently each estimator uses it. For each pair, the budget is the smaller of the two methods' total generation, and the method that generated more is evaluated at its checkpoint reaching that budget.
| Domain | Baseline | Generation budget (B) | Baseline MAS pass@1 / pass@8 | RollCredits MAS pass@1 / pass@8 |
|---|---|---|---|---|
| Math | C3 | 0.89 | 70.54 / 84.60 | 70.62 / 85.29 |
| Math | VinePPO | 1.33 | 69.11 / 84.42 | 71.87 / 86.41 |
| Code | C3 | 0.42 | 58.65 / 71.21 | 60.36 / 73.14 |
| Code | VinePPO | 0.42 | 57.67 / 72.01 | 60.36 / 73.14 |
RollCredits achieves higher MAS pass@1 and pass@8 than both C3 and VinePPO at matched generation budgets in math and code. Our response to Reviewer mdEg (Q4) also reports generated and input tokens, approximate model calls and training tokens for every method.
Dear Reviewer NwT8,
Thanks for your time and effort in reviewing this paper. Your comments are valuable to improve the quality of this paper. As the author-reviewer discussion period ends in 48 hours, we kindly request that you review the rebuttal and confirm whether your concerns have been addressed. Thanks a lot.
Regards from the authors.
Response to Reviewer NwT8 (1/2)
We thank the reviewer for recognizing the DAG formulation, the strength of the results, and the held-out game generalization.
R3.W1 / R3.Q1: Distinctions from C3 and baseline comparison
Relation to C3. The table in the General Response compares RollCredits with C3 in sampling, replay, saved state and reward target.
Effect of the execution-graph formulation. The reviewer recognizes this formulation as new and asks for evidence of its significance, which competitive games and the parallel workflow provide. In games, each player is credited with its own reward, which a shared team reward cannot express; RollCredits leads in three of four games and wins 85.16 ± 0.52% on held-out Breakthrough, against 70.18 ± 0.60% for the strongest baseline. In the parallel workflow (Q2), the graph shows that a solver's siblings do not depend on it, so replay keeps their outputs and regenerates only the aggregator, which reduces generated tokens per comparison group by 46.2% and 31.0% at the first two solver positions with similar credit MSE.
C3 as a baseline. Over three training seeds, C3 reaches MAS pass@8 of 84.67 ± 0.17 in math and 71.53 ± 0.77 in code, versus 87.09 ± 0.68 and 72.95 ± 0.62 for RollCredits. C3's game entries are n/a because our games assign opposing rewards to the two players.
R3.Q2: Parallel fan-out/fan-in workflow
We added a fan-out/fan-in math workflow: three solvers answer in parallel and an aggregator produces the final answer from their outputs. A solver's history contains only the task, so its siblings are not downstream of it; the aggregator is.
The parallel-workflow comparison uses one training run per method and the overlap-filtered math test set.
Both RollCredits replay rules reach 86.53% MAS pass@8, the highest among these methods. Serialized-trace replay regenerates calls recorded after the decision, including later sibling solvers. Dependency-preserving replay retains unaffected sibling outputs and regenerates the aggregator. Both rules preserve the same solver inputs; a solver never sees its siblings' answers.
At a fixed checkpoint, we compare both replay rules against an independent high-budget advantage reference with matched visible histories and candidate actions. Repeated trials resample sibling outputs from their conditional distribution; dependency-preserving replay shares those outputs across candidates within each trial. Each cell below reports dependency-preserving / serialized-trace replay. Generated tokens include seed cost, allocated over the groups it serves.
Dependency-preserving replay reduces generation by 46.2% at solver 1 and 31.0% at solver 2, with similar MSE. At solver 3, both rules regenerate only the candidate and aggregator, so their generation counts coincide. The empirical bias estimates use eight trials per candidate and include finite-trial and reference sampling noise. The independent reference uses 32 continuations per candidate, as in Figure 3. We run the diagnostic on the final RollCredits checkpoint of the parallel workflow, using 96 decision states (32 problems × 3 solver positions).