- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes ARCO, a framework for learning step-level rubric rewards for multi-step LLM agents. A single trainable rubric model (denoted phi below), sharing one open-source LM backbone, carries two heads: (i) a generation head that writes K natural-language criteria per action r_t, conditioned on the pre-action prefix and the action itself; and (ii) a score head that mean-pools the hidden states of the sequence (h_t, a_t, r_t) and predicts one score per criterion in the range [-1, 1]. The step reward s_t is the mean of those criterion scores (Eq. 2).
Training has two stages. First, a supervised warmup: a weak policy teacher (gpt-4o-mini) generates trajectories and a strong rubric teacher (gpt-5.4) annotates per-step criteria and scores. The teacher scores are L2-projected onto the intersection of the box [-1, 1] and the hyperplane requiring the summed criterion scores to equal K times the terminal reward R(tau), so the summed step means exactly match the terminal reward; the rubric model phi is then trained with a dual objective that combines a generation cross-entropy term and a score-regression term. Second, an on-policy co-evolution stage: on each shared rollout batch, the policy pi is updated with a position-bucketed reward-to-go advantage plus a token-level policy KL term, while phi is updated by a trajectory-decomposition loss, namely the squared difference between one over K times the summed criterion scores and the terminal reward R(tau), plus a rubric-token KL term that anchors the generated rubric text to the frozen warmup reference. Reward density switches from sparse (outcome only) to dense (rubric scores) after beta epochs.
Validation is on HotpotQA, 2WikiMultiHopQA and MuSiQue (2k train and 500 val examples each) with a constrained top-1 SimCSE retrieval environment, two policy backbones (Qwen3-4B-Instruct-2507 and Llama-3.2-3B-Instruct), LoRA adapters, and EM/F1 metrics. Baselines are outcome reward models (Search-R1, R1-Searcher), trajectory-level rubric ORMs using a frozen closed-source judge (RaR, CARMO, RLER), and a process reward model (AgentPRM). ARCO reports the best EM in all six dataset-by-backbone cells, by +0.8 to +2.8 EM over the strongest competitor, plus ablations, a four-way step-binding diagnostic judged by gpt-4o-mini (42.5 to 54.8 percent versus 25 percent chance; specificity about 4.0 out of 5), a rubric-width sweep over K in {1, 3, 5, 7, 9} (best at K = 3), a rubric-model scale and family sweep, and an exploratory frozen-phi comparison.
Critical strengths
-
A clean, well-motivated problem framing and an honest empirical narrative. The paper correctly identifies a real gap in the literature, namely that rubric-based rewards are mostly trajectory-level and judged by frozen closed-source judges, so they cannot supply step-level or adaptive guidance, and it decomposes that gap along three explicit axes (evaluator adaptation, scoring granularity, judge model). Nearly every design decision in Section 3 is then justified against one of these axes. This is consequential because it makes the paper easy to place in the literature and makes the ablations diagnostic rather than decorative. It also reports evidence against itself (Appendix B, and the mixed results in Appendix C), which raises my confidence that the positive numbers are not curated.
-
The trajectory-decomposition training with exact projection is a concrete and reusable mechanism. Forcing the summed step scores to equal the terminal reward by construction — an L2 projection onto the intersection of a hyperplane and a box, with a closed-form clamp-plus-bisection solution (Eq. 6) — is a neat piece of engineering that lets a step-level reward model be trained online from outcome-only labels without an external judge. Combined with the rubric-anchoring KL that prevents the generated rubric text from drifting, this addresses a genuine failure mode (reward-model drift and non-stationarity) rather than a synthetic one. It is the part of the paper most likely to be reused by others.
Critical weaknesses
-
The paper's own diagnostic shows the "dense step-level" reward is essentially uninformative on the intermediate (Search) steps that process rewards exist to fix. Appendix B and Table 7 report that the ARCO step score separates eventually-correct from eventually-wrong trajectories with AUROC 0.58-0.65 and Gap 0.03-0.08 on Search steps, versus AUROC 0.82-0.87 and Gap 0.33-0.44 on Finish steps, consistently across all six cells. Since the search budget is 3-6 steps, the majority of scored steps in every trajectory are Search steps whose scores are barely better than a coin flip. This matters directly for acceptance because the paper's core claim, repeated in the abstract and TL;DR, is that ARCO provides "dense rubric-conditioned rewards" and "step-level guidance". The only trajectory-level supervision on the scores is the sum constraint, which fixes the total but says nothing about allocation, and the authors' own explanation in Appendix B concedes this. The ablation evidence is also mixed in a way that is hard to reconcile: the variant without prefix information improves F1 on MuSiQue/Llama (29.81 vs 29.70) and reaches 41.60 EM on HotpotQA/Qwen (vs 42.80), and the variant without rubric text (39.80 EM) beats the fixed-criteria variant (38.00 EM), that is, the rubric text matters most where the criteria are least adaptive. The paper never shows that the Search-step rewards contribute to any measured gain, for example through an advantage-attribution or masking experiment. As written, a defensible reading of the paper's own evidence is that the gain comes almost entirely from a well-calibrated terminal or Finish-step evaluator, not from dense process credit assignment.
-
The empirical margins are too small and too thinly measured to support the paper's headline universality claim, and the baseline comparison carries an unexplained supervision confound. Table 1 reports +0.8 to +2.8 EM over the strongest baseline across six cells, using a single seed (42) with no variance, confidence interval, or paired significance test anywhere in Table 1 or Table 2, even though Table 3, for a much simpler judged quantity, reports 95% CI half-widths of plus or minus 3.99 to 10.21 EM, that is, intervals several times larger than the claimed gains. Every (method, dataset, backbone) cell additionally picks the best-validation-EM checkpoint over 8-10 RL epochs using only 500 validation examples, with no held-out test split, so epoch selection is a real selection channel. This matters because the central claim is dominance in all settings. Separately, the fairness statement that the same warmup is applied to all baselines does not resolve what the baselines actually receive: ARCO is warm-started with gpt-5.4-generated per-step criteria and scores for every warmup trajectory (Eq. 7), that is, problem-specific process supervision, whereas the rubric baselines are described as using a frozen closed-source judge. The paper is explicit that gpt-5.4 was chosen over a weaker teacher because it provides reliable supervision, so the design deliberately injects teacher quality into the ARCO initialization while the baselines are not given an equivalent budget. As a result the reported deltas cannot distinguish "co-evolutionary step-level scoring" from "better supervision during initialization", and the paper's central mechanism is not isolated by the main table.
Questions for authors
Reward attribution. Can you show that the Search-step rewards actually drive policy improvement? A direct experiment would be to zero or mask the dense-stage advantages on intermediate steps (or to train densely only on Finish-step rewards) while keeping everything else fixed. If EM is unchanged, the "dense step-level credit assignment" contribution is not empirically supported.
Significance. Please report multi-seed results (at least 3 seeds) with means and standard deviations for Table 1 and Table 2, plus a paired test across the shared evaluation questions. Given CI half-widths of plus or minus 4-10 EM on a simpler metric, how confident are you that the +0.8 EM margin (MuSiQue/Qwen) is real? We have seen how single-seed LLM-RL comparisons can mislead.
Checkpoint selection. What fraction of the headline gain survives if checkpoints are chosen by a fixed-epoch or smoothed-curve rule instead of best validation EM, and if you split a genuine test set out of the 500 validation examples used for selection?
Baseline parity. For each baseline, which judge model and scoring granularity were used, and were the published recipes re-tuned under this environment rather than re-implemented? Did any baseline receive a comparable teacher-generated process supervision budget? A control in which a baseline rubric ORM is fine-tuned on the same scaled decomposition target would isolate co-evolution from supervision strength.
Co-evolution necessity. Appendix C is candid that the frozen- ϕ ϕ run degrades on HotpotQA/Llama and 2Wiki/Llama but remains approximately flat on both Qwen3-4B settings, and it was not run on MuSiQue. Since co-evolution is the paper's title and primary novelty, can you characterize, even on one additional cell, the conditions under which it is necessary rather than merely helpful? "Exploratory" currently reads as insufficient for the central claim.
Where is the signal, mechanistically? Appendix B shows Finish scores are strongly diagnostic and Search scores are not. Is the gain in Table 1 therefore explained by a better terminal evaluator rather than by process credit assignment? Relatedly, why does the variant without rubric text remain within about 3 EM of full ARCO on four of six cells if rubric text is the source of semantic structure?
Warmup data dependence. The warmup annotations behind Table 1 come from gpt-5.4 for all six cells (Appendix A.6), while only the HotpotQA/Qwen warmup data is released. How sensitive are the results to the rubric teacher's identity and scale? Does ARCO still beat the rubric baselines if the warmup rubrics come from a model of the baseline judge's class?
Practical cost. Appendix A.5 shows dense-stage times from 1 h 22 m (HotpotQA/Qwen) to 15 h 29 m (MuSiQue/Qwen) for the dense stage only. What are the wall-clock and memory overheads relative to Search-R1 and AgentPRM under a matched compute budget, not only matched data?
Scope. All three datasets are 2-4-hop retrieval QA in a closed 3-6-step loop with a top-1 retriever. Which of ARCO's claims do you expect to transfer to longer-horizon, non-retrieval or tool-use environments where the Finish-step advantage cannot carry the signal?
10 comments
Q4 (baseline parity and supervision). A. Thank you for asking us to clarify the comparison. We reimplemented the baselines in a shared QA environment with a common policy SFT starting point, making environment-specific adaptations. The reward sources and scoring granularity used in the paper runs are:
| Method | Judge / reward source | Scoring granularity |
|---|---|---|
| Search-R1 / R1-Searcher | Environment outcome rewards; no LLM judge | Trajectory outcome |
| RaR | GPT-4o-mini judge; GPT-4o rubric generator | Question-conditioned rubrics, trajectory-level scoring |
| CARMO | GPT-4o-mini | Context-dependent criteria, trajectory-level scoring |
| RLER | GPT-4o-mini | Evolving rubric buffer, trajectory-level scoring |
| AgentPRM | Trained local PRM | Step scores, used for BoN@8 selection |
| ARCO | Local trainable rubric evaluator | Action-specific criterion scores, updated online from terminal outcomes |
In our RaR/CARMO/RLER implementations, judge parameters are not updated during training; RLER updates its rubric buffer. Shared warmup refers to policy SFT; ARCO additionally uses teacher-generated criteria and scores to initialize its evaluator.
To assess dependence on teacher strength, we used GPT-4o-mini, also used by other baselines, to reannotate ARCO's original 1,000 warmup questions / 3,000 trajectories, replacing GPT-5.4. On HotpotQA/Qwen, the checkpoint selected by validation EM among completed epochs gives the following results on valid500 and the same 400-question test subset used in Q3:
| ARCO warmup rubric teacher | Valid500 EM / F1 (%) | Test400 EM / F1 (%) |
|---|---|---|
| GPT-5.4 | 42.80 / 55.07 | 44.00 / 55.82 |
| GPT-4o-mini | 43.20 / 55.50 | 45.50 / 56.41 |
ARCO retains strong performance with GPT-4o-mini warmup annotations, exceeding RaR, CARMO, and RLER on both splits. This shows that its performance does not depend on the stronger GPT-5.4 warmup teacher; the continued co-evolution of rubric generation and evaluator scoring during RL helps sustain this performance. Our RaR/CARMO/RLER implementations continue to use external API judges to score training rollouts. ARCO requires teacher API calls only for one-time warmup data preparation; subsequent RL uses a local, trainable open-weight evaluator supervised by terminal outcomes, with no external API scoring.
Q5 (co-evolution necessity). A. Thank you for suggesting an additional setting to clarify when continued adaptation matters. The evaluator-freezing ablations in Appendix C show a backbone-dependent pattern: freezing leads to declines on HotpotQA/Llama and 2Wiki/Llama, while the Qwen settings remain approximately flat.
We added a MuSiQue/Qwen dense-stage frozen-evaluator continuation, starting from the same post-sparse policy, evaluator, and score-head weights as ARCO. In the frozen branch, the evaluator and score head remain fixed while policy training continues. The first five dense epochs on valid500 are:
Table 4. MuSiQue/Qwen validation EM / F1 (%) during dense-stage continuation.
| Dense epoch | ARCO co-evolution | Frozen evaluator |
|---|---|---|
| 0 | 25.60 / 35.07 | 25.60 / 35.07 |
| 1 | 25.80 / 34.44 | 26.20 / 35.01 |
| 2 | 25.40 / 34.33 | 25.80 / 34.50 |
| 3 | 25.00 / 33.55 | 25.20 / 33.82 |
| 4 | 26.00 / 34.82 | 25.40 / 33.77 |
| 5 | 27.40 / 35.55 | 26.40 / 35.62 |
The frozen evaluator is competitive early, while its scores fluctuate across later epochs. By epoch 5, co-evolving ARCO reaches 27.40 EM / 35.55 F1, compared with 26.40 / 35.62 for the frozen evaluator. Relative to the shared starting point, ARCO gains 1.80 EM points, versus 0.80 for the frozen evaluator, with nearly identical F1. Together with the Llama results in Appendix C, this suggests that the benefit of evaluator adaptation depends on the setting. One possibility is that on difficult MuSiQue, a capable post-sparse evaluator remains useful because policy behavior changes more slowly, while continued co-evolution can help as the policy evolves. This motivates future work on matching evaluator updates to policy change, including delayed or asynchronous updates.
Q6 (mechanistic role of the reward signal).
A. Appendix B indicates that both Search and Finish scores carry information about eventual correctness, with Finish scores being more predictive. Our ablations further examine their contributions to policy learning. Finish-only retains strong performance at 41.60 EM / 54.64 F1, since its terminal reward reaches earlier Search actions through reward-to-go. Full ARCO improves by 1.20 EM and 0.43 F1 to 42.80 / 55.07; masking Search-step advantages lowers performance to 40.80 / 53.79. These results suggest that Finish provides a strong signal, while explicit Search-step rewards provide an additional benefit in this setting.
The w/o-Rubric variant removes explicit criterion text but retains the pretrained evaluator, trajectory/action context, and outcome-supervised score learning. Full ARCO improves EM by 0.60–3.00 points over this variant across all six settings. This suggests that explicit rubric text contributes to performance, while the evaluator’s pretrained representations and outcome supervision also contribute.
Q7 (warmup teacher dependence). A. Thank you for raising teacher dependence. On HotpotQA/Qwen, we replaced GPT-5.4 with GPT-4o-mini, an API scoring model also used in our baseline comparisons, to generate ARCO's warmup rubrics and scores. We retained the policy trajectories and SFT initialization, retrained the evaluator, and completed sparse+dense training. The selected checkpoint results are:
| Method / warmup teacher | Valid500 EM / F1 (%) | Held-out test400 EM / F1 (%) |
|---|---|---|
| RaR | 38.00 / 50.75 | 38.25 / 50.29 |
| CARMO | 34.80 / 48.52 | 39.00 / 51.35 |
| RLER | 32.20 / 44.40 | 35.25 / 45.65 |
| ARCO / GPT-5.4 | 42.80 / 55.07 | 44.00 / 55.82 |
| ARCO / GPT-4o-mini | 43.20 / 55.50 | 45.50 / 56.41 |
With GPT-4o-mini warmup, ARCO remains ahead of the rubric baselines on both splits and performs comparably to the GPT-5.4-warmup run. After warmup, ARCO's RL updates use its local evaluator trained from task outcomes, without external API scoring. This result shows that ARCO's performance on HotpotQA/Qwen does not require GPT-5.4 as its warmup teacher.
Q8 (practical cost).
A. Thank you for asking about training cost. The approximate training GPU-hours are:
| Method | HotpotQA/Qwen (GPU-h) | MuSiQue/Qwen (GPU-h) |
|---|---|---|
| Search-R1, 3 GRPO epochs | ~13 | ~34 |
| AgentPRM, trajectory collection + reward-model fitting | ~7 | ~22 |
| ARCO, sparse + dense training | ~16 | ~47 |
ARCO reuses policy rollouts for evaluator updates and needs no separate evaluator rollout pass. Its training cost is in a similar range to Search-R1, with additional computation for rubric scoring and evaluator updates, while achieving the strongest overall held-out results in our comparison. At inference, ARCO uses one policy rollout with the policy and retriever; AgentPRM uses BoN@8 to sample and score eight candidates.
Q9 (scope and transfer). A. Thank you for highlighting the scope of our evidence. ARCO's transferable design is state/action-conditioned rubric generation combined with evaluator adaptation as policy behavior changes; it can be instantiated with action interfaces beyond Search/Finish. In tool-use settings, criteria could assess argument validity, satisfied preconditions, or progress toward a subgoal, using task outcomes and verifiable environment feedback. For longer horizons with delayed or weak terminal signals, verifiable intermediate outcomes—such as passed tests or completed subgoals—could complement terminal feedback and support temporal or hierarchical credit assignment. Our current experiments focus on short-horizon retrieval QA; we will clarify this scope and describe these extensions as future directions.
Thank you again for your careful reading and thoughtful questions. Your feedback has helped us clarify ARCO's contributions, strengthen our empirical analysis, and better define the scope of our claims. We hope our responses address your concerns and look forward to your further feedback.
Thank you for your careful reading and constructive suggestions. We respond to your questions below (Q = your question; A = our response).
Q1 (Search-step reward attribution).
A. We ran two HotpotQA/Qwen ablations starting from the same post-sparse policy and evaluator weights: Finish-only, which retains only Finish-step rewards while allowing Search actions to receive them through reward-to-go, and Search-masked, which directly masks policy advantages on Search actions. We compare these ablations with the original full-ARCO run on the same 500-question validation set.
Both EM and F1 decrease from full ARCO to Finish-only to Search-masked. This pattern supports the usefulness of delayed Finish rewards for training Search actions, with explicit Search-step rewards providing an additional benefit in this setting. Thank you for prompting this more direct examination of Search-step rewards.
Q2 (multi-seed robustness).
A. Thank you for highlighting the importance of seed robustness. We evaluated ARCO on HotpotQA/Qwen with three seeds (42, 43, 44) using the same 500-question validation set. Below we report the per-seed EM / F1 (%) and their mean ± sample standard deviation.
Table 1. HotpotQA/Qwen validation results.
ARCO's three-seed mean remains above all listed baselines in both EM and F1, with small variation across these runs.
Table 2. HotpotQA/Qwen ablations on validation500 (EM / F1, %).
Across the three seeds, full ARCO achieves the highest mean EM and F1, outperforming all three ablations.
Q3 (held-out evaluation and checkpoint selection).
A. Thank you for highlighting this evaluation concern. Table 1 uses the same 500 validation examples for checkpoint selection and reporting. All methods followed this protocol, but we agree that a separate evaluation is needed. We therefore retained best-validation-EM checkpoint selection and evaluated the selected checkpoints on 400 randomly sampled held-out questions per dataset, disjoint from training and validation, without test-based tuning or checkpoint selection.
Table 3. Independent held-out test results. Each entry is EM / F1 (%); Overall is the unweighted mean across the six settings.
Overall, ARCO achieves the highest average EM and F1 across the six settings (34.46 / 43.01). ARCO uses a single rollout at inference, whereas AgentPRM (BoN@8) generates and scores eight candidate trajectories, requiring additional inference computation.