acceptodds
Under review as a conference paper at ICLR 2027

ARCO: Adaptive Rubrics with Co-Evolution for Multi-Step LLM-Based Agents

Abstract

Reinforcement learning for multi-step LLM agents often relies on scalar rewards that indicate success but cannot explain why a trajectory is good or bad. Rubric-based rewards improve interpretability through natural-language criteria, but many existing approaches score complete trajectories with a frozen closed-source judge, limiting step-level guidance and adaptation during training. We propose ARCO (Adaptive Rubric CO-evolution), which generates interpretable, action-specific per-step rubrics and dense rubric-conditioned rewards, then continuously co-evolves the rubric model with the agent on shared on-policy rollouts. After warmup, trajectory decomposition fits step rewards to terminal outcomes without step labels or an external judge during online training. Across HotpotQA, 2WikiMultiHopQA, and MuSiQue with two open-source backbones, ARCO achieves the highest EM in all settings over outcome-, rubric-, and process-reward baselines. We further analyze ARCO along several dimensions—the action-specificity of its rubrics, the number of criteria per step, and the rubric model's scale and family—showing that it yields an interpretable step-level signal useful for diagnosing agent behavior. Code and data are available at https://anonymous.4open.science/r/arco-anon-2C61.

Primary area:
reinforcement learning
Keywords:
LLM Agents, Reinforcement Learning, Process Reward Modeling, Rubric-Based Rewards, Co-Evolution

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_2mEjOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes ARCO, a framework for learning step-level rubric rewards for multi-step LLM agents. A single trainable rubric model (denoted phi below), sharing one open-source LM backbone, carries two heads: (i) a generation head that writes K natural-language criteria per action r_t, conditioned on the pre-action prefix and the action itself; and (ii) a score head that mean-pools the hidden states of the sequence (h_t, a_t, r_t) and predicts one score per criterion in the range [-1, 1]. The step reward s_t is the mean of those criterion scores (Eq. 2).

Training has two stages. First, a supervised warmup: a weak policy teacher (gpt-4o-mini) generates trajectories and a strong rubric teacher (gpt-5.4) annotates per-step criteria and scores. The teacher scores are L2-projected onto the intersection of the box [-1, 1] and the hyperplane requiring the summed criterion scores to equal K times the terminal reward R(tau), so the summed step means exactly match the terminal reward; the rubric model phi is then trained with a dual objective that combines a generation cross-entropy term and a score-regression term. Second, an on-policy co-evolution stage: on each shared rollout batch, the policy pi is updated with a position-bucketed reward-to-go advantage plus a token-level policy KL term, while phi is updated by a trajectory-decomposition loss, namely the squared difference between one over K times the summed criterion scores and the terminal reward R(tau), plus a rubric-token KL term that anchors the generated rubric text to the frozen warmup reference. Reward density switches from sparse (outcome only) to dense (rubric scores) after beta epochs.

Validation is on HotpotQA, 2WikiMultiHopQA and MuSiQue (2k train and 500 val examples each) with a constrained top-1 SimCSE retrieval environment, two policy backbones (Qwen3-4B-Instruct-2507 and Llama-3.2-3B-Instruct), LoRA adapters, and EM/F1 metrics. Baselines are outcome reward models (Search-R1, R1-Searcher), trajectory-level rubric ORMs using a frozen closed-source judge (RaR, CARMO, RLER), and a process reward model (AgentPRM). ARCO reports the best EM in all six dataset-by-backbone cells, by +0.8 to +2.8 EM over the strongest competitor, plus ablations, a four-way step-binding diagnostic judged by gpt-4o-mini (42.5 to 54.8 percent versus 25 percent chance; specificity about 4.0 out of 5), a rubric-width sweep over K in {1, 3, 5, 7, 9} (best at K = 3), a rubric-model scale and family sweep, and an exploratory frozen-phi comparison.

Critical strengths
  1. A clean, well-motivated problem framing and an honest empirical narrative. The paper correctly identifies a real gap in the literature, namely that rubric-based rewards are mostly trajectory-level and judged by frozen closed-source judges, so they cannot supply step-level or adaptive guidance, and it decomposes that gap along three explicit axes (evaluator adaptation, scoring granularity, judge model). Nearly every design decision in Section 3 is then justified against one of these axes. This is consequential because it makes the paper easy to place in the literature and makes the ablations diagnostic rather than decorative. It also reports evidence against itself (Appendix B, and the mixed results in Appendix C), which raises my confidence that the positive numbers are not curated.

  2. The trajectory-decomposition training with exact projection is a concrete and reusable mechanism. Forcing the summed step scores to equal the terminal reward by construction — an L2 projection onto the intersection of a hyperplane and a box, with a closed-form clamp-plus-bisection solution (Eq. 6) — is a neat piece of engineering that lets a step-level reward model be trained online from outcome-only labels without an external judge. Combined with the rubric-anchoring KL that prevents the generated rubric text from drifting, this addresses a genuine failure mode (reward-model drift and non-stationarity) rather than a synthetic one. It is the part of the paper most likely to be reused by others.

Critical weaknesses
  1. The paper's own diagnostic shows the "dense step-level" reward is essentially uninformative on the intermediate (Search) steps that process rewards exist to fix. Appendix B and Table 7 report that the ARCO step score separates eventually-correct from eventually-wrong trajectories with AUROC 0.58-0.65 and Gap 0.03-0.08 on Search steps, versus AUROC 0.82-0.87 and Gap 0.33-0.44 on Finish steps, consistently across all six cells. Since the search budget is 3-6 steps, the majority of scored steps in every trajectory are Search steps whose scores are barely better than a coin flip. This matters directly for acceptance because the paper's core claim, repeated in the abstract and TL;DR, is that ARCO provides "dense rubric-conditioned rewards" and "step-level guidance". The only trajectory-level supervision on the scores is the sum constraint, which fixes the total but says nothing about allocation, and the authors' own explanation in Appendix B concedes this. The ablation evidence is also mixed in a way that is hard to reconcile: the variant without prefix information improves F1 on MuSiQue/Llama (29.81 vs 29.70) and reaches 41.60 EM on HotpotQA/Qwen (vs 42.80), and the variant without rubric text (39.80 EM) beats the fixed-criteria variant (38.00 EM), that is, the rubric text matters most where the criteria are least adaptive. The paper never shows that the Search-step rewards contribute to any measured gain, for example through an advantage-attribution or masking experiment. As written, a defensible reading of the paper's own evidence is that the gain comes almost entirely from a well-calibrated terminal or Finish-step evaluator, not from dense process credit assignment.

  2. The empirical margins are too small and too thinly measured to support the paper's headline universality claim, and the baseline comparison carries an unexplained supervision confound. Table 1 reports +0.8 to +2.8 EM over the strongest baseline across six cells, using a single seed (42) with no variance, confidence interval, or paired significance test anywhere in Table 1 or Table 2, even though Table 3, for a much simpler judged quantity, reports 95% CI half-widths of plus or minus 3.99 to 10.21 EM, that is, intervals several times larger than the claimed gains. Every (method, dataset, backbone) cell additionally picks the best-validation-EM checkpoint over 8-10 RL epochs using only 500 validation examples, with no held-out test split, so epoch selection is a real selection channel. This matters because the central claim is dominance in all settings. Separately, the fairness statement that the same warmup is applied to all baselines does not resolve what the baselines actually receive: ARCO is warm-started with gpt-5.4-generated per-step criteria and scores for every warmup trajectory (Eq. 7), that is, problem-specific process supervision, whereas the rubric baselines are described as using a frozen closed-source judge. The paper is explicit that gpt-5.4 was chosen over a weaker teacher because it provides reliable supervision, so the design deliberately injects teacher quality into the ARCO initialization while the baselines are not given an equivalent budget. As a result the reported deltas cannot distinguish "co-evolutionary step-level scoring" from "better supervision during initialization", and the paper's central mechanism is not isolated by the main table.

Questions for authors

Reward attribution. Can you show that the Search-step rewards actually drive policy improvement? A direct experiment would be to zero or mask the dense-stage advantages on intermediate steps (or to train densely only on Finish-step rewards) while keeping everything else fixed. If EM is unchanged, the "dense step-level credit assignment" contribution is not empirically supported.

Significance. Please report multi-seed results (at least 3 seeds) with means and standard deviations for Table 1 and Table 2, plus a paired test across the shared evaluation questions. Given CI half-widths of plus or minus 4-10 EM on a simpler metric, how confident are you that the +0.8 EM margin (MuSiQue/Qwen) is real? We have seen how single-seed LLM-RL comparisons can mislead.

Checkpoint selection. What fraction of the headline gain survives if checkpoints are chosen by a fixed-epoch or smoothed-curve rule instead of best validation EM, and if you split a genuine test set out of the 500 validation examples used for selection?

Baseline parity. For each baseline, which judge model and scoring granularity were used, and were the published recipes re-tuned under this environment rather than re-implemented? Did any baseline receive a comparable teacher-generated process supervision budget? A control in which a baseline rubric ORM is fine-tuned on the same scaled decomposition target would isolate co-evolution from supervision strength.

Co-evolution necessity. Appendix C is candid that the frozen- ϕ ϕ run degrades on HotpotQA/Llama and 2Wiki/Llama but remains approximately flat on both Qwen3-4B settings, and it was not run on MuSiQue. Since co-evolution is the paper's title and primary novelty, can you characterize, even on one additional cell, the conditions under which it is necessary rather than merely helpful? "Exploratory" currently reads as insufficient for the central claim.

Where is the signal, mechanistically? Appendix B shows Finish scores are strongly diagnostic and Search scores are not. Is the gain in Table 1 therefore explained by a better terminal evaluator rather than by process credit assignment? Relatedly, why does the variant without rubric text remain within about 3 EM of full ARCO on four of six cells if rubric text is the source of semantic structure?

Warmup data dependence. The warmup annotations behind Table 1 come from gpt-5.4 for all six cells (Appendix A.6), while only the HotpotQA/Qwen warmup data is released. How sensitive are the results to the rubric teacher's identity and scale? Does ARCO still beat the rubric baselines if the warmup rubrics come from a model of the baseline judge's class?

Practical cost. Appendix A.5 shows dense-stage times from 1 h 22 m (HotpotQA/Qwen) to 15 h 29 m (MuSiQue/Qwen) for the dense stage only. What are the wall-clock and memory overheads relative to Search-R1 and AgentPRM under a matched compute budget, not only matched data?

Scope. All three datasets are 2-4-hop retrieval QA in a closed 3-6-step loop with a top-1 retriever. Which of ARCO's claims do you expect to transfer to longer-horizon, non-retrieval or tool-use environments where the Finish-step advantage cannot carry the signal?

10 comments

Thank you for your careful reading and constructive suggestions. We respond to your questions below (Q = your question; A = our response).

Q1 (Search-step reward attribution).

A. We ran two HotpotQA/Qwen ablations starting from the same post-sparse policy and evaluator weights: Finish-only, which retains only Finish-step rewards while allowing Search actions to receive them through reward-to-go, and Search-masked, which directly masks policy advantages on Search actions. We compare these ablations with the original full-ARCO run on the same 500-question validation set.

MethodValidation EM / F1 (%)
Full ARCO (original paper run)42.80 / 55.07
Finish reward only41.60 / 54.64
Search-step advantages masked40.80 / 53.79

Both EM and F1 decrease from full ARCO to Finish-only to Search-masked. This pattern supports the usefulness of delayed Finish rewards for training Search actions, with explicit Search-step rewards providing an additional benefit in this setting. Thank you for prompting this more direct examination of Search-step rewards.

Q2 (multi-seed robustness).

A. Thank you for highlighting the importance of seed robustness. We evaluated ARCO on HotpotQA/Qwen with three seeds (42, 43, 44) using the same 500-question validation set. Below we report the per-seed EM / F1 (%) and their mean ± sample standard deviation.

Table 1. HotpotQA/Qwen validation results.

MethodValidation EM / F1 (%) by seedMean ± SD (EM / F1, %)
Base ModelSeed 42: 34.40 / 45.93
Search-R1Seed 42: 39.40 / 51.67
R1-SearcherSeed 42: 41.00 / 53.42
RaRSeed 42: 38.00 / 50.75
CARMOSeed 42: 34.80 / 48.52
RLERSeed 42: 32.20 / 44.40
AgentPRMSeed 42: 38.60 / 51.49
ARCOSeed 42: 42.80 / 55.07; Seed 43: 42.20 / 54.67; Seed 44: 43.20 / 55.7842.73 ± 0.50 / 55.17 ± 0.56

ARCO's three-seed mean remains above all listed baselines in both EM and F1, with small variation across these runs.

Table 2. HotpotQA/Qwen ablations on validation500 (EM / F1, %).

VariantSeed 42Seed 43Seed 44Mean ± SD
w/o Rubric39.80 / 52.6741.00 / 53.1740.80 / 53.1140.53 ± 0.64 / 52.98 ± 0.27
w/o Adaptive38.00 / 50.4139.00 / 51.3139.00 / 51.0538.67 ± 0.58 / 50.92 ± 0.46
w/o Prefix41.60 / 54.2840.20 / 53.1841.80 / 54.0241.20 ± 0.87 / 53.83 ± 0.57
ARCO (Full)42.80 / 55.0742.20 / 54.6743.20 / 55.7842.73 ± 0.50 / 55.17 ± 0.56

Across the three seeds, full ARCO achieves the highest mean EM and F1, outperforming all three ablations.

Q3 (held-out evaluation and checkpoint selection).

A. Thank you for highlighting this evaluation concern. Table 1 uses the same 500 validation examples for checkpoint selection and reporting. All methods followed this protocol, but we agree that a separate evaluation is needed. We therefore retained best-validation-EM checkpoint selection and evaluated the selected checkpoints on 400 randomly sampled held-out questions per dataset, disjoint from training and validation, without test-based tuning or checkpoint selection.

Table 3. Independent held-out test results. Each entry is EM / F1 (%); Overall is the unweighted mean across the six settings.

MethodHotpotQA / QwenHotpotQA / Llama2Wiki / Qwen2Wiki / LlamaMuSiQue / QwenMuSiQue / LlamaOverall
Base Model37.00 / 45.1718.50 / 29.3324.50 / 34.6510.50 / 19.8614.75 / 22.676.50 / 12.2018.63 / 27.31
SFT36.50 / 47.9431.00 / 42.6939.75 / 48.0727.00 / 35.0817.00 / 26.239.50 / 17.3526.79 / 36.23
Search-R143.00 / 54.5037.25 / 48.7147.75 / 52.9438.75 / 44.8819.25 / 28.3912.50 / 20.8533.08 / 41.71
R1-Searcher41.50 / 53.0436.25 / 47.4245.25 / 52.2739.00 / 44.9618.25 / 28.5110.75 / 18.6931.83 / 40.82
RaR38.25 / 50.2930.50 / 43.9442.50 / 50.1729.25 / 38.0819.75 / 29.8211.50 / 20.6428.63 / 38.82
CARMO39.00 / 51.3533.75 / 46.3038.50 / 48.3026.25 / 37.0716.00 / 25.549.50 / 18.6027.17 / 37.86
RLER35.25 / 45.6529.50 / 43.2741.00 / 48.6825.00 / 34.7917.25 / 26.429.50 / 18.9326.25 / 36.29
AgentPRM (BoN@8)39.00 / 51.1936.50 / 48.4246.50 / 53.3826.75 / 35.0420.75 / 31.9512.25 / 19.8030.29 / 39.96
ARCO (ours)44.00 / 55.8238.25 / 48.8248.25 / 53.4941.00 / 47.5820.75 / 29.9214.50 / 22.4434.46 / 43.01

Overall, ARCO achieves the highest average EM and F1 across the six settings (34.46 / 43.01). ARCO uses a single rollout at inference, whereas AgentPRM (BoN@8) generates and scores eight candidate trajectories, requiring additional inference computation.

Q4 (baseline parity and supervision). A. Thank you for asking us to clarify the comparison. We reimplemented the baselines in a shared QA environment with a common policy SFT starting point, making environment-specific adaptations. The reward sources and scoring granularity used in the paper runs are:

MethodJudge / reward sourceScoring granularity
Search-R1 / R1-SearcherEnvironment outcome rewards; no LLM judgeTrajectory outcome
RaRGPT-4o-mini judge; GPT-4o rubric generatorQuestion-conditioned rubrics, trajectory-level scoring
CARMOGPT-4o-miniContext-dependent criteria, trajectory-level scoring
RLERGPT-4o-miniEvolving rubric buffer, trajectory-level scoring
AgentPRMTrained local PRMStep scores, used for BoN@8 selection
ARCOLocal trainable rubric evaluatorAction-specific criterion scores, updated online from terminal outcomes

In our RaR/CARMO/RLER implementations, judge parameters are not updated during training; RLER updates its rubric buffer. Shared warmup refers to policy SFT; ARCO additionally uses teacher-generated criteria and scores to initialize its evaluator.

To assess dependence on teacher strength, we used GPT-4o-mini, also used by other baselines, to reannotate ARCO's original 1,000 warmup questions / 3,000 trajectories, replacing GPT-5.4. On HotpotQA/Qwen, the checkpoint selected by validation EM among completed epochs gives the following results on valid500 and the same 400-question test subset used in Q3:

ARCO warmup rubric teacherValid500 EM / F1 (%)Test400 EM / F1 (%)
GPT-5.442.80 / 55.0744.00 / 55.82
GPT-4o-mini43.20 / 55.5045.50 / 56.41

ARCO retains strong performance with GPT-4o-mini warmup annotations, exceeding RaR, CARMO, and RLER on both splits. This shows that its performance does not depend on the stronger GPT-5.4 warmup teacher; the continued co-evolution of rubric generation and evaluator scoring during RL helps sustain this performance. Our RaR/CARMO/RLER implementations continue to use external API judges to score training rollouts. ARCO requires teacher API calls only for one-time warmup data preparation; subsequent RL uses a local, trainable open-weight evaluator supervised by terminal outcomes, with no external API scoring.

Q5 (co-evolution necessity). A. Thank you for suggesting an additional setting to clarify when continued adaptation matters. The evaluator-freezing ablations in Appendix C show a backbone-dependent pattern: freezing leads to declines on HotpotQA/Llama and 2Wiki/Llama, while the Qwen settings remain approximately flat.

We added a MuSiQue/Qwen dense-stage frozen-evaluator continuation, starting from the same post-sparse policy, evaluator, and score-head weights as ARCO. In the frozen branch, the evaluator and score head remain fixed while policy training continues. The first five dense epochs on valid500 are:

Table 4. MuSiQue/Qwen validation EM / F1 (%) during dense-stage continuation.

Dense epochARCO co-evolutionFrozen evaluator
025.60 / 35.0725.60 / 35.07
125.80 / 34.4426.20 / 35.01
225.40 / 34.3325.80 / 34.50
325.00 / 33.5525.20 / 33.82
426.00 / 34.8225.40 / 33.77
527.40 / 35.5526.40 / 35.62

The frozen evaluator is competitive early, while its scores fluctuate across later epochs. By epoch 5, co-evolving ARCO reaches 27.40 EM / 35.55 F1, compared with 26.40 / 35.62 for the frozen evaluator. Relative to the shared starting point, ARCO gains 1.80 EM points, versus 0.80 for the frozen evaluator, with nearly identical F1. Together with the Llama results in Appendix C, this suggests that the benefit of evaluator adaptation depends on the setting. One possibility is that on difficult MuSiQue, a capable post-sparse evaluator remains useful because policy behavior changes more slowly, while continued co-evolution can help as the policy evolves. This motivates future work on matching evaluator updates to policy change, including delayed or asynchronous updates.

Q6 (mechanistic role of the reward signal).

A. Appendix B indicates that both Search and Finish scores carry information about eventual correctness, with Finish scores being more predictive. Our ablations further examine their contributions to policy learning. Finish-only retains strong performance at 41.60 EM / 54.64 F1, since its terminal reward reaches earlier Search actions through reward-to-go. Full ARCO improves by 1.20 EM and 0.43 F1 to 42.80 / 55.07; masking Search-step advantages lowers performance to 40.80 / 53.79. These results suggest that Finish provides a strong signal, while explicit Search-step rewards provide an additional benefit in this setting.

The w/o-Rubric variant removes explicit criterion text but retains the pretrained evaluator, trajectory/action context, and outcome-supervised score learning. Full ARCO improves EM by 0.60–3.00 points over this variant across all six settings. This suggests that explicit rubric text contributes to performance, while the evaluator’s pretrained representations and outcome supervision also contribute.

Q7 (warmup teacher dependence). A. Thank you for raising teacher dependence. On HotpotQA/Qwen, we replaced GPT-5.4 with GPT-4o-mini, an API scoring model also used in our baseline comparisons, to generate ARCO's warmup rubrics and scores. We retained the policy trajectories and SFT initialization, retrained the evaluator, and completed sparse+dense training. The selected checkpoint results are:

Method / warmup teacherValid500 EM / F1 (%)Held-out test400 EM / F1 (%)
RaR38.00 / 50.7538.25 / 50.29
CARMO34.80 / 48.5239.00 / 51.35
RLER32.20 / 44.4035.25 / 45.65
ARCO / GPT-5.442.80 / 55.0744.00 / 55.82
ARCO / GPT-4o-mini43.20 / 55.5045.50 / 56.41

With GPT-4o-mini warmup, ARCO remains ahead of the rubric baselines on both splits and performs comparably to the GPT-5.4-warmup run. After warmup, ARCO's RL updates use its local evaluator trained from task outcomes, without external API scoring. This result shows that ARCO's performance on HotpotQA/Qwen does not require GPT-5.4 as its warmup teacher.

Q8 (practical cost).

A. Thank you for asking about training cost. The approximate training GPU-hours are:

MethodHotpotQA/Qwen (GPU-h)MuSiQue/Qwen (GPU-h)
Search-R1, 3 GRPO epochs~13~34
AgentPRM, trajectory collection + reward-model fitting~7~22
ARCO, sparse + dense training~16~47

ARCO reuses policy rollouts for evaluator updates and needs no separate evaluator rollout pass. Its training cost is in a similar range to Search-R1, with additional computation for rubric scoring and evaluator updates, while achieving the strongest overall held-out results in our comparison. At inference, ARCO uses one policy rollout with the policy and retriever; AgentPRM uses BoN@8 to sample and score eight candidates.

Q9 (scope and transfer). A. Thank you for highlighting the scope of our evidence. ARCO's transferable design is state/action-conditioned rubric generation combined with evaluator adaptation as policy behavior changes; it can be instantiated with action interfaces beyond Search/Finish. In tool-use settings, criteria could assess argument validity, satisfied preconditions, or progress toward a subgoal, using task outcomes and verifiable environment feedback. For longer horizons with delayed or weak terminal signals, verifiable intermediate outcomes—such as passed tests or completed subgoals—could complement terminal feedback and support temporal or hierarchical credit assignment. Our current experiments focus on short-horizon retrieval QA; we will clarify this scope and describe these extensions as future directions.


Thank you again for your careful reading and thoughtful questions. Your feedback has helped us clarify ARCO's contributions, strengthen our empirical analysis, and better define the scope of our claims. We hope our responses address your concerns and look forward to your further feedback.

Reviewer_UooNOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes ARCO, an adaptive rubric co-evolution framework for training multi-step LLM agents. The paper aims to address two limitations of conventional reward models: outcome rewards provide only sparse trajectory-level supervision, while existing rubric-based methods often use trajectory-level criteria and a fixed external judge.

ARCO introduces a trainable rubric model with a shared open-source language-model backbone and two heads. Given the trajectory prefix and current action, the generation head produces an action-specific natural-language rubric, while the score head assigns criterion-level scores whose mean becomes the step reward. Following supervised warmup, the policy and rubric model are updated on the same on-policy trajectories. The policy is optimized using rubric-conditioned reward-to-go with a position-bucketed baseline, while the rubric model is trained through a trajectory-decomposition objective that makes the sum of step rewards match the terminal outcome, together with a KL term intended to stabilize rubric language.

The method is evaluated on HotpotQA, 2WikiMultiHopQA, and MuSiQue using Qwen3-4B and Llama-3.2-3B backbones. ARCO reports the highest exact-match score in all six dataset-backbone settings relative to the included outcome-reward, rubric-reward, and process-reward baselines. The paper also analyzes component ablations, action specificity, the number of rubric criteria, and the architecture and scale of the rubric model.

Critical strengths
  1. The paper presents a coherent framework for combining readable action-level criteria with dense process rewards. The use of a shared backbone with a rubric-generation head and a criterion-scoring head is conceptually simple, and the method directly connects each scalar step reward to natural-language evaluation dimensions.

  2. The paper includes several useful diagnostic analyses beyond aggregate task accuracy. In particular, the action-binding evaluation, rubric-width analysis, cross-family rubric-model study, and comparison between Search and Finish score discriminativeness expose meaningful properties and limitations of the learned evaluator. These analyses are valuable even though they do not resolve the novelty and evaluation concerns below.

Critical weaknesses
  1. The comparison set omits several of the closest methods, making the novelty and empirical advantage difficult to establish. Most components of ARCO have clear precedents: CARMO dynamically generates context-dependent evaluation criteria; RLER evolves rubrics together with an on-policy model; Rubric-ARM jointly optimizes a rubric generator and a judge; Web-Shepherd and DG-PRM provide interpretable or dynamic step-level process evaluation; and Self-Guide co-evolves a policy with a step-level internal reward. In particular, the paper states that existing rubric methods use frozen scoring models, but Rubric-ARM already jointly trains the rubric generator and judge. This framing is therefore inaccurate. Without controlled comparisons against these methods, it is unclear whether ARCO contributes a substantially new learning principle or mainly combines existing ideas in a new implementation.

  2. The main evaluation is affected by selection bias and provides insufficient evidence that the reported gains generalize. The same 500-example validation set is used both to select the best checkpoint and to report the final results. There is no independent test set, and all runs use a single random seed. This is especially consequential because ARCO’s advantage over the strongest baseline is only 0.8–2.8 EM in several settings. The reported improvements may therefore reflect validation-set overfitting, checkpoint-selection noise, or RL variance rather than a reliable advantage of the proposed method. Addressing this issue requires new held-out evaluation and repeated training runs, not only a clarification in the paper.

Questions for authors
  1. Can the authors compare ARCO against the closest adaptive-rubric and step-level reward methods under the same environment and training budget? At minimum, this should include Rubric-ARM, a dynamic step-level PRM such as DG-PRM or Web-Shepherd, and a co-evolving internal-reward method such as Self-Guide. Please also explain precisely how ARCO differs from RLER and Rubric-ARM beyond applying the rubric at the action level.

  2. How do the authors justify the statement that existing rubric-based methods use frozen scorers? Rubric-ARM jointly optimizes a rubric generator and a judge, which appears to contradict this characterization. Please revise the related-work framing and clarify the narrower combination that is genuinely new in ARCO.

  3. Can the authors report results on an independent held-out test set? The current protocol selects the best checkpoint using the 500 validation examples and reports the main results on the same examples. Please reserve a separate test set that is never used for checkpoint or hyperparameter selection, and report the test performance of checkpoints chosen only on validation data.

11 comments

Thank you for your careful reading and constructive suggestions. We respond to your questions below (Q = your question; A = our response).

Q3 (independent held-out evaluation and seed robustness).

A. Thank you for this suggestion. We fixed the checkpoints selected on the original validation set and evaluated them on 400 randomly sampled held-out questions per dataset, disjoint from training and validation, without test-based checkpoint or hyperparameter selection.

Table 1. Independent held-out test results. Each entry is EM / F1 (%); Overall is the unweighted mean across the six settings.

MethodHotpotQA / QwenHotpotQA / Llama2Wiki / Qwen2Wiki / LlamaMuSiQue / QwenMuSiQue / LlamaOverall
Base Model37.00 / 45.1718.50 / 29.3324.50 / 34.6510.50 / 19.8614.75 / 22.676.50 / 12.2018.63 / 27.31
SFT36.50 / 47.9431.00 / 42.6939.75 / 48.0727.00 / 35.0817.00 / 26.239.50 / 17.3526.79 / 36.23
Search-R143.00 / 54.5037.25 / 48.7147.75 / 52.9438.75 / 44.8819.25 / 28.3912.50 / 20.8533.08 / 41.71
R1-Searcher41.50 / 53.0436.25 / 47.4245.25 / 52.2739.00 / 44.9618.25 / 28.5110.75 / 18.6931.83 / 40.82
RaR38.25 / 50.2930.50 / 43.9442.50 / 50.1729.25 / 38.0819.75 / 29.8211.50 / 20.6428.63 / 38.82
CARMO39.00 / 51.3533.75 / 46.3038.50 / 48.3026.25 / 37.0716.00 / 25.549.50 / 18.6027.17 / 37.86
RLER35.25 / 45.6529.50 / 43.2741.00 / 48.6825.00 / 34.7917.25 / 26.429.50 / 18.9326.25 / 36.29
AgentPRM (BoN@8)39.00 / 51.1936.50 / 48.4246.50 / 53.3826.75 / 35.0420.75 / 31.9512.25 / 19.8030.29 / 39.96
ARCO (ours)44.00 / 55.8238.25 / 48.8248.25 / 53.4941.00 / 47.5820.75 / 29.9214.50 / 22.4434.46 / 43.01

Overall, ARCO achieves the highest average EM and F1 across the six settings (34.46 / 43.01). ARCO uses a single rollout at inference, whereas AgentPRM (BoN@8) generates and scores eight candidate trajectories, requiring additional inference computation.

We additionally assessed seed robustness on HotpotQA/Qwen over three seeds. Each checkpoint was selected only on validation and evaluated on the same 400-question test set. The table reports ARCO's three-seed mean ± standard deviation.

Table 2. HotpotQA/Qwen held-out test results (%).

MethodEMF1
Base Model37.0045.17
SFT36.5047.94
Search-R143.0054.50
R1-Searcher41.5053.04
RaR38.2550.29
CARMO39.0051.35
RLER35.2545.65
AgentPRM (BoN@8)39.0051.19
ARCO (seeds 42, 43, 44)44.17 ± 0.2955.80 ± 0.37

ARCO maintains the highest EM and F1, with small standard deviations across the three seeds. These results indicate consistent performance across seeds in this setting, supporting the robustness of our method.

Q1 / Q2 (closest-method comparisons, ARCO's distinction, and related-work framing).

A. Thank you for highlighting these closely related methods. We agree that joint rubric–judge training and policy–internal-reward co-evolution have important precedents, including Rubric-ARM's trainable generator and judge. We will revise the frozen-scorer characterization and clarify ARCO's specific contribution.

ARCO's contribution is adaptive step-level rubric generation and rubric-conditioned scoring that co-evolve with the policy in multi-step agent tasks. For each action, a shared evaluator backbone generates action-specific criteria and predicts criterion-level scores. Both its score head and backbone LoRA parameters are continually updated by fitting summed step rewards to terminal outcomes on current-policy rollouts. These updates also adapt the representation used for rubric generation, while per-action reward-to-go guides policy learning. After one-time supervised warmup, co-evolution requires no further teacher API calls or external step-level annotations.

The distinctions are:

  • Search-R1 / R1-Searcher and RaR / CARMO: the former primarily use outcome rewards; the latter provide response- or trajectory-level rubric evaluation. ARCO constructs rubric-conditioned rewards for individual actions and continually adapts the evaluator with the policy.
  • RLER: updates a rubric buffer to reflect current policy behavior while keeping judge parameters fixed. ARCO additionally updates the evaluator's parameters from interaction outcomes, learning how to distribute terminal supervision across steps.
  • Rubric-ARM: alternately trains a rubric generator and a judge using preference feedback, then uses the trained components to supervise downstream policy optimization. Its described downstream procedure does not continue updating these components alongside the policy. ARCO continually updates its shared rubric-and-score backbone from current agent rollouts through terminal-outcome decomposition.
  • Web-Shepherd / DG-PRM and AgentPRM: Web-Shepherd provides checklist-based process judgments, DG-PRM dynamically selects criteria from a reward tree, and AgentPRM estimates step values. ARCO combines explicit action-specific criteria with continuing, outcome-supervised evaluator updates on the same rollouts used for policy learning.
  • Self-Guide: the same policy first generates a verbal assessment of trajectory progress, then takes an action conditioned on that guidance. Its positive/neutral/negative assessments are mapped to fixed scalar rewards and combined with the environment reward to jointly optimize guidance and actions. ARCO trains a separate evaluator to generate action-specific rubrics and continuous criterion scores, explicitly fitting summed step rewards to terminal outcomes, while the policy learns through per-action reward-to-go. At deployment, ARCO uses only the policy and retriever, without an additional guidance-generation step.

We added Rubric-ARM, Web-Shepherd, and Self-Guide comparisons on HotpotQA/Qwen, sharing the Qwen3-4B policy initialization, training questions, Search/Finish environment, SimCSE top-1 retriever, and three-search budget. All three are adapted to this QA setting: Web-Shepherd uses question-level checklists and a fixed GPT-4o-mini step judge for reward-to-go policy RL; Rubric-ARM alternately trains its generator and judge before freezing both for policy RL; Self-Guide retains guidance generation, polarity rewards, and its trust schedule.

Following the same evaluation protocol described in Q3, checkpoints are selected only on the original validation set, then evaluated on the same independent 400-question HotpotQA test set:

MethodEM (%)F1 (%)
Web-Shepherd34.0043.85
Self-Guide38.7550.71
Rubric-ARM36.2548.17
ARCO (ours)44.0055.82

ARCO achieves the highest test EM and F1 among these comparisons. Together with the distinctions above, these results support its combination of action-specific rubrics, trainable scoring, and outcome-supervised co-evolution in multi-step agent learning.

We will rewrite the introduction and related work to explicitly acknowledge trainable rubric generators and judges, interpretable process rewards, and internal-reward co-evolution, and distinguish these precedents from ARCO's specific contribution.


Thank you again for your careful reading and thoughtful questions. They have helped us clarify ARCO's contribution and strengthen its evaluation. We will further improve the manuscript accordingly and look forward to your feedback.

AuthorsOpenReview ↗

Follow-up and Additional Analysis

Dear Reviewer UooN,

Thank you again for your careful reading and constructive feedback. We wanted to briefly follow up on our responses to your concerns.

In response to your questions, we clarified ARCO’s relationship to closely related methods and revised our related-work framing. We also added comparisons with Rubric-ARM, Web-Shepherd, and Self-Guide under a shared QA setup, as well as independent held-out evaluation and three-seed results on HotpotQA/Qwen.

To further examine the interpretability of ARCO’s rubric-conditioned evaluator, we also conducted controlled criterion edits on 100 HotpotQA/Qwen search states. We compared meaning-preserving paraphrases, meaning-changing edits, cross-question swaps, and identical-input repeats, while keeping the remaining inputs and evaluator fixed. A separate blinded check confirmed both intended edit types in 36 cases: the paraphrase preserved the original meaning, and the semantic edit changed it.

We report normalized score deviation, calculated as 100 × Σ|edited score − original score| / Σ|original score|. A larger percentage indicates a larger aggregate score change relative to the magnitude of the original scores. The length-matched subset contains 9 verified cases in which both edited criteria were within ±20% of the original criterion’s token length.

SamplenIdentical repeatParaphraseSemantic editCross-question swap
All cases1000.00%7.09%11.45%24.74%
Verified cases360.00%8.38%20.20%30.46%
Verified, length-matched cases90.00%4.30%26.86%34.26%

Across all cases, semantic edits produced 1.61× the score deviation of paraphrases; among verified cases, this increased to 2.41×. Paraphrase scores remained highly correlated with the original scores (Pearson r = 0.986). Identical repeats produced zero deviation, while cross-question swaps produced the largest deviation in each group.

For example, consider the action Search[Battle of Dunkirk opposing forces]. Each criterion below has 13 tokens:

VersionCriterionScore
OriginalFollow-up query directly checks the battle context identified by the prior result.0.136
ParaphraseSubsequent query explicitly verifies the battle context established by the previous finding.0.104
Semantic editFollow-up query indirectly checks the battle context identified by the prior result.0.033
Cross-question swapTargets the second person’s background directly.0.007

The paraphrase receives a score close to the original, while changing “directly” to “indirectly” produces a larger score change. The unrelated cross-question criterion produces the largest change. These results show that, in this analysis, the evaluator responds more strongly to changes in criterion meaning and relevance than to paraphrasing.

We hope our responses and additional results address your concerns. We would be grateful to hear whether any questions remain, and we would be happy to provide further clarification during the discussion period. If the additional evidence has changed your assessment, we would sincerely appreciate your reconsideration.

Best regards,
The Authors

Reviewer_WGKjOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

ARCO trains multi-step LLM agents with action-specific rubrics that provide step-level rewards. After supervised warmup, the policy and rubric model co-evolve on shared rollouts, using terminal outcomes to supervise the reward model. Experiments on three multi-hop QA benchmarks with Qwen and Llama backbones report improved exact-match performance over the compared baselines, alongside ablations and analyses of the learned rewards.

Critical strengths

S1. The method gives each action its own rubric and step score, and updates the evaluator along with the policy. This is a useful way to make process rewards easier to inspect without calling an external judge during online training.

S2. The evaluation covers three datasets and two backbone families, with component ablations, rubric-width and model-scale studies, and both successful and failed rubric-binding examples. These analyses examine several parts of the proposed mechanism, beyond the main performance table.

Critical weaknesses

W1. As I read Section 4.1, the same 500 validation examples are used to select checkpoints and report results. This makes it harder to assess performance independently of checkpoint selection.

W2. The frozen-evaluator study changes both sparse-stage adaptation and dense-stage updates, so it does not cleanly isolate continued co-evolution. Also, the binding diagnostic shows that rubrics are action-specific, but does not establish whether their criterion scores track actual action quality.

Questions for authors

Q1. Could you clarify whether the EM/F1 results in Table 1 are reported on the same 500 validation examples used for checkpoint selection? If so, are results available on a separate held-out set after fixing the checkpoint? This would address the evaluation concern directly.

Q2. For me, the key distinction is adaptation during the sparse stage versus continued co-evolution. If both runs start from the same post-sparse policy and evaluator checkpoints, how does freezing the evaluator only in the dense stage compare with the full method under matched updates? A Qwen setting and MuSiQue would be especially helpful.

Q3. Do criterion scores track actual changes in action quality when rubric wording or actions are controlled, for example under rubric swaps or independently assessed search-step comparisons? The binding diagnostic and final-outcome AUROC do not by themselves answer this question. Existing evidence on this point could materially change my assessment of the interpretability claim.

Q4. Would a representative strong API model under the same Search/Finish interface, dataset-context retriever, and search budget provide a useful accuracy–cost reference? This would complement the controlled 3B/4B training comparisons. Please separate warmup teacher and training costs from deployment inference costs, and clarify the intended practical advantage of the trained agent.

10 comments

Thank you for your careful reading and constructive suggestions. We respond to your questions below (Q = your question; A = our response).

Q1 (separate held-out evaluation).

A. Yes, Table 1 uses the same 500 validation examples for checkpoint selection and reporting. All methods followed this protocol, but we agree that a separate evaluation is needed. We therefore fixed the validation-selected checkpoints and evaluated them on 400 randomly sampled held-out questions per dataset, disjoint from training and validation, without test-based tuning or checkpoint selection.

Table 1. Independent held-out test results. Each entry is EM / F1 (%); Overall is the unweighted mean across the six settings.

MethodHotpotQA / QwenHotpotQA / Llama2Wiki / Qwen2Wiki / LlamaMuSiQue / QwenMuSiQue / LlamaOverall
Base Model37.00 / 45.1718.50 / 29.3324.50 / 34.6510.50 / 19.8614.75 / 22.676.50 / 12.2018.63 / 27.31
SFT36.50 / 47.9431.00 / 42.6939.75 / 48.0727.00 / 35.0817.00 / 26.239.50 / 17.3526.79 / 36.23
Search-R143.00 / 54.5037.25 / 48.7147.75 / 52.9438.75 / 44.8819.25 / 28.3912.50 / 20.8533.08 / 41.71
R1-Searcher41.50 / 53.0436.25 / 47.4245.25 / 52.2739.00 / 44.9618.25 / 28.5110.75 / 18.6931.83 / 40.82
RaR38.25 / 50.2930.50 / 43.9442.50 / 50.1729.25 / 38.0819.75 / 29.8211.50 / 20.6428.63 / 38.82
CARMO39.00 / 51.3533.75 / 46.3038.50 / 48.3026.25 / 37.0716.00 / 25.549.50 / 18.6027.17 / 37.86
RLER35.25 / 45.6529.50 / 43.2741.00 / 48.6825.00 / 34.7917.25 / 26.429.50 / 18.9326.25 / 36.29
AgentPRM (BoN@8)39.00 / 51.1936.50 / 48.4246.50 / 53.3826.75 / 35.0420.75 / 31.9512.25 / 19.8030.29 / 39.96
ARCO (ours)44.00 / 55.8238.25 / 48.8248.25 / 53.4941.00 / 47.5820.75 / 29.9214.50 / 22.4434.46 / 43.01

Overall, ARCO achieves the highest average EM and F1 across the six settings (34.46 / 43.01). ARCO uses a single rollout at inference, whereas AgentPRM (BoN@8) generates and scores eight candidate trajectories, requiring additional inference computation.

Q2 (freezing the evaluator only during the dense stage).

A. Thank you for this suggestion. We added a dense-stage frozen-evaluator continuation on MuSiQue/Qwen, starting from the same post-sparse policy, evaluator, and score-head weights as the original ARCO run. We froze the evaluator and score head while continuing policy training.

Validation EM / F1 (%) on the original 500 questions:

Dense epochARCO (original run) EM / F1 (%)Frozen evaluator EM / F1 (%)
025.60 / 35.0725.60 / 35.07
125.80 / 34.4426.20 / 35.01
225.40 / 34.3325.80 / 34.50
325.00 / 33.5525.20 / 33.82
426.00 / 34.8225.40 / 33.77
527.40 / 35.5526.40 / 35.62

The variants perform similarly early on and both fluctuate; ARCO improves over the later epochs and finishes with higher EM and nearly identical F1. A strong frozen evaluator also remains effective. We hypothesize that slower policy learning on difficult tasks allows the post-sparse evaluator to remain useful for longer. This motivates future work on matching policy and evaluator update schedules across tasks, including asynchronous updates.

Q3 (criterion scores under controlled rubric interventions).

A. Thank you for this suggestion. We added controlled rubric interventions on 100 HotpotQA/Qwen search states (50 first-search, 50 later-search). We changed one criterion while holding the question, history, action, budget, other criteria, and evaluator fixed.

GPT-4o-mini generated paraphrases and meaning-changing edits. A separate request to the same model randomized edit order, hid edit types and scores, and labeled each same / different / unclear against the original. 36/100 cases passed both checks (paraphrase: same; semantic edit: different). All edits and labels were fixed before scoring. We retain all 100 cases in the primary analysis and report the 36 accepted cases, including the nested 9 whose edited token lengths both fall within ±20% of the original. Cross-question swaps and identical-input repeats serve as additional controls.

The table reports absolute score deviation relative to the original score magnitude: R = 100 × Σ|s_edited − s_original| / Σ|s_original|, with scores in [-1, 1].

SamplenIdentical repeat RParaphrase RIntended semantic edit RCross-question swap R
All cases1000.00%7.09%11.45%24.74%
Semantic-check accepted360.00%8.38%20.20%30.46%
Accepted and length matched90.00%4.30%26.86%34.26%

Across all cases, semantic edits produce 1.61× the deviation of paraphrases, whose scores remain highly correlated with the originals (Pearson r = 0.986). The ratio rises to 2.41× among accepted cases. Length-matched results show the same ordering; identical repeats produce zero deviation.

Example. In the first accepted case, the action is Search[Battle of Dunkirk opposing forces]. Each version below has 13 tokens:

VersionCriterion textScore
OriginalFollow-up query directly checks the battle context identified by the prior result0.136
ParaphraseSubsequent query explicitly verifies the battle context established by the previous finding0.104
Semantic editFollow-up query indirectly checks the battle context identified by the prior result0.033
Cross-question swapTargets the second person's background directly0.007

Changing “directly” to “indirectly” shifts the score more than paraphrasing, despite equal token lengths.

These interventions provide evidence that the evaluator responds more to meaning changes than wording changes in this sample.

Q4 (strong API references and deployment costs).

A. Thank you for this suggestion. We evaluated three API agents on the same 400-question HotpotQA held-out test set as Q1, using the validation-selected ARCO checkpoint. All agents share the Search/Finish interface, prompts, SimCSE top-1 dataset-context retriever, and budget of at most three searches.

AgentEM (%)F1 (%)
Qwen3-4B SFT36.5047.94
ARCO Qwen3-4B44.0055.82
GPT-4o-mini36.2551.20
DeepSeek-V3.248.7562.96
Gemini-2.5-Pro58.7574.80

ARCO improves over SFT by 7.50 EM / 7.88 F1 points and over GPT-4o-mini by 7.75 EM / 4.62 F1 points, using a compact 4B policy.

We separate three costs:

  • One-time teacher warmup: 1,000 training questions, 3,000 retained trajectories, and 9,456 actions. Reconstructed visible-text costs are about $0.61 for GPT-4o-mini trajectories and $19.13 for GPT-5.4 annotations (~$20 total), assuming input/output rates of $0.15/$0.60 and $2.50/$15 per million tokens.
  • One-time local training: SFT and RL on 2,000 training questions, with 26,000 rollouts through the selected checkpoint (34,000 in the full eight-epoch run).
  • Deployment: one rollout uses only the trained 4B policy and shared retriever, with no teacher API calls, training-time rubric evaluator, or candidate scoring.

ARCO’s practical advantage is learning stronger task capability into a compact, locally deployable policy: it improves over SFT and GPT-4o-mini while retaining SFT’s policy-and-retriever deployment structure, without ongoing teacher API dependence.

Thank you for your careful reading. Your feedback helped us clarify ARCO’s contribution and strengthen its evaluation. We will revise the manuscript accordingly and welcome further feedback.

AuthorsOpenReview ↗

Follow-up on Our Responses

Dear Reviewer WGKj,

Thank you again for your careful reading and constructive suggestions. We wanted to briefly follow up on our responses to your questions.

Following your suggestions, we added an independent held-out evaluation using checkpoints selected on the original validation set. We also compared continued co-evolution with a frozen evaluator on MuSiQue/Qwen, starting from the same post-sparse checkpoints. To further examine the criterion scores, we conducted controlled rubric interventions, including paraphrases, meaning-changing edits, and cross-question swaps. We also evaluated strong API models under the same Search/Finish interface and retriever, and clarified the separate costs of teacher warmup, local training, and deployment.

We hope these responses and additional results address your concerns. We would be grateful to hear whether any questions remain, and we would be happy to provide further clarification during the discussion period.

Best regards,
The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.