- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes ARRCE, a framework for open-domain RL that learns adaptive process rubrics under fixed static supervision. Firstly ARRCE synthesizes and freezes task-specific and evidence-backed factual criteria. During RL training, the policy proposes process rubrics from sampled trajectories using relative quality information derived from the static rubrics. These candidates are evaluated on disjoint trajectories, and accepted criteria are stored in query-specific memory to guide subsequent response learning. Response generation and rubric proposal share model parameters but use separate training objectives.
The authors evaluate ARRCE on multiple benchmarks. The results show consistent improvements over response SFT and competitive performance against static and dynamic rubric-based RL baselines. Ablation studies further examine the contributions of stage decoupling, static supervision, trajectory comparisons, held-out candidate evaluation, etc.
Critical strengths
The paper addresses an important challenge in evolving rubric-based RL: adapting evaluation criteria to changing policy behavior without losing the original task requirements. ARRCE separates frozen task and fact rubrics from dynamically generated process rubrics, providing a stable reference for evaluating new criteria. The use of disjoint trajectories for rubric proposal and evaluation is also a useful design choice that helps reduce direct overfitting to proposal examples.
The paper evaluates ARRCE across four benchmarks covering scientific question answering, healthcare, writing, and instruction following. The ablation studies examine several important design choices, including static supervision, score-based trajectory comparisons, stage decoupling, and held-out candidate evaluation. These results provide useful evidence for the contributions of different components.
Critical weaknesses
ARRCE relies on automatically synthesized static rubrics to supervise trajectory comparisons and process-rubric learning. However, the validation procedure mainly checks structural consistency and citation validity, which do not necessarily guarantee factual correctness or alignment with actual response quality. Errors in static rubrics may propagate into the learned process criteria, potentially affecting the reliability of the proposed framework.
Although the ablation studies show that several components contribute to the final performance, the paper provides limited direct analysis of the learned process rubrics themselves. It remains unclear how frequently the generated criteria are accepted, whether they provide useful supervision over time, and how much additional information they capture beyond the static rubrics. A systematic analysis would better support the central contribution of learning adaptive process rubrics.
ARRCE introduces additional computation through process-rubric generation, held-out candidate evaluation, and joint optimization. However, the paper does not systematically compare the total training costs with the baselines. Since the improvements over strong RL baselines are relatively modest on some benchmarks, compute-matched comparisons would help establish the practical value of the proposed framework.
Questions for authors
Q1. How reliable are the automatically synthesized static rubrics? Have the authors evaluated their correctness or agreement with human annotations?
Q2. Could the authors provide statistics on the learned process rubrics, such as their acceptance rates, retention rates, and contribution to downstream RL performance?
Q3. Could the authors compare the total training costs of ARRCE and the baselines? Does ARRCE retain its advantage under a matched computational budget?
2 comments
Focused case: what information the dynamic rubric adds
Case query
What evidence do hospital-based case-control studies in China provide regarding the association between alcohol use and sexually transmitted infections (STIs)?
Static rubric
The frozen static rubric specifies the answer content that must be present.
| Criterion | Static rubric content | Full ARRCE | Response SFT |
|---|---|---|---|
| S1 | Provide evidence of a significant positive association between alcohol use and STIs from multiple studies conducted in different regions of China. | 5 | 5 |
| S2 | Mention odds ratios or another measure quantifying the increased STI risk linked to alcohol use. | 5 | 1 |
| S3 | Address potential confounding factors considered in the studies, including sexual behavior and demographic variables. | 5 | 1 |
| S4 | Analyze the strength and consistency of the evidence linking alcohol use to STIs. | 5 | 5 |
| Normalized coverage score | 1.00 | 0.50 |
Dynamic process rubric
Given search results for Chinese hospital-based case-control studies, organize the final synthesis around study population/region, at least one quantitative association, behavioral or demographic confounders, and study-design limitations; preserve the distinction between association and causation, and omit mechanisms that are not supported by the retrieved evidence.
The dynamic rubric is evaluated through four observable checks: D1, targeted retrieval from the broad association to populations, regions, effect sizes, and confounders; D2, retention of study-level details rather than a generic consensus statement; D3, linking quantitative and behavioral evidence while distinguishing adjustment from mediation; and D4, explicit study limitations without unsupported causal or mechanistic escalation.
| Trajectory | Static verdicts and score | Trajectory evidence | Dynamic verdict |
|---|---|---|---|
| ; | Broad association summary with no staged study-level retrieval path | 0 | |
| ; | Population, estimate, adjustment, and limitation queries before synthesis | 1 |
The static rubric is the factual and task-level anchor: S1--S4 check determinate, auditable answer content. The dynamic rubric is trajectory-derived and judgment-oriented: D1--D4 assess how the policy finds, organizes, connects, and qualifies evidence through observable process behavior. The tied pair therefore demonstrates the information increment directly: identical static coverage receives different dynamic verdicts because the evidence-gathering processes differ.
Static-rubric reliability, lifecycle, and matched cost
Thank you for highlighting the reliability of static supervision, the lifecycle of learned process rubrics, and the computational cost of rubric learning, we will answer the questions in the following.
Reliability of automatically synthesized static rubrics
We stratified 150 RL tasks (50 each from HealthBench, ResearchQA, and LongWriter-Zero), covering 556 frozen criteria, and 300 blinded trajectory--criterion pairs.
Fact criteria use the exact contiguous-substring check in Appendix A.1. Equation (1) aggregates criteria; ϵS=0.05 handles static ties, and Appendix A.4 removes a rule after two consecutive gaps below −0.05.
Process-rubric lifecycle and accepted-rubric quality
Acceptance requires schema, observable, boundary, principle, non-proxy, alignment, and direction gates. Activation, retention@k, and reuse are defined by later same-query visits and later response rewards. The ledger is the per-seed mean over seeds 42--44 across 6,165 visits; the 4/4 split produces six candidates per visit.
Static S1--S4 score answer content; dynamic D1--D4 score trajectory quality. The (e1,e3) tie has identical static scores but dynamic verdicts 0 and 1; details are in Comment 2.
Total training cost and matched computational value
All in-house rows share initialization, mock search, Judge, retries, optimization/checkpoint rules, and 8-GPU runtime. The ledger covers 6,165 visits per seed; ARRCE's 4/4 split yields six proposer outputs and evaluates each candidate on four trajectories.
At the Judge-token/wall-clock matched Static G=12 setting, Full ARRCE improves all four benchmarks and exceeds the listed controls.