acceptodds
Under review as a conference paper at ICLR 2027

ARRCE: Learning Process Rubrics from Static Anchors for Open-Domain Reinforcement Learning

Abstract

We introduce Asymmetrically Anchored Rubric–Response Co-Evolution ARRCE, a reinforcement learning framework that learns process rubrics from automatically synthesized static supervision for open-domain tasks. The central difficulty is to adapt evaluation criteria to the policy's changing behavior while preserving the task requirements that define a useful answer. ARRCE decouples rubric synthesis into an offline stage that constructs and freezes task and evidence-backed fact criteria, and an online stage that proposes process criteria from sampled trajectories whose relative strengths are determined by static-rubric scores. The static criteria also supervise candidate quality: a proposed process rubric is rewarded for separating held-out trajectories in a direction consistent with static performance, subject to observability and scope checks, before entering a query-specific memory for subsequent response rewards. A shared policy learns response generation and rubric proposal through separate objectives in one parameter update, yielding a synthesis loop that requires no additional human annotation of trajectory preferences or process rubrics. We instantiate this framework on dataset-derived tasks with synthesized static rubrics across healthcare, scientific question answering, and long-form writing. The experimental results show that full ARRCE improves over the response-SFT baseline by on ResearchQA (), OP points on LLMEval-Med (), on WritingBench (), and on FollowBench (), and the gains are positive across the four benchmarks.

Primary area:
reinforcement learning
Keywords:
Self-Evolving Rubrics, Open-Domain Reinforcement Learning, Automatic Rubric Synthesis

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_Nx9NOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes ARRCE, a framework for open-domain RL that learns adaptive process rubrics under fixed static supervision. Firstly ARRCE synthesizes and freezes task-specific and evidence-backed factual criteria. During RL training, the policy proposes process rubrics from sampled trajectories using relative quality information derived from the static rubrics. These candidates are evaluated on disjoint trajectories, and accepted criteria are stored in query-specific memory to guide subsequent response learning. Response generation and rubric proposal share model parameters but use separate training objectives.

The authors evaluate ARRCE on multiple benchmarks. The results show consistent improvements over response SFT and competitive performance against static and dynamic rubric-based RL baselines. Ablation studies further examine the contributions of stage decoupling, static supervision, trajectory comparisons, held-out candidate evaluation, etc.

Critical strengths

The paper addresses an important challenge in evolving rubric-based RL: adapting evaluation criteria to changing policy behavior without losing the original task requirements. ARRCE separates frozen task and fact rubrics from dynamically generated process rubrics, providing a stable reference for evaluating new criteria. The use of disjoint trajectories for rubric proposal and evaluation is also a useful design choice that helps reduce direct overfitting to proposal examples.

The paper evaluates ARRCE across four benchmarks covering scientific question answering, healthcare, writing, and instruction following. The ablation studies examine several important design choices, including static supervision, score-based trajectory comparisons, stage decoupling, and held-out candidate evaluation. These results provide useful evidence for the contributions of different components.

Critical weaknesses

ARRCE relies on automatically synthesized static rubrics to supervise trajectory comparisons and process-rubric learning. However, the validation procedure mainly checks structural consistency and citation validity, which do not necessarily guarantee factual correctness or alignment with actual response quality. Errors in static rubrics may propagate into the learned process criteria, potentially affecting the reliability of the proposed framework.

Although the ablation studies show that several components contribute to the final performance, the paper provides limited direct analysis of the learned process rubrics themselves. It remains unclear how frequently the generated criteria are accepted, whether they provide useful supervision over time, and how much additional information they capture beyond the static rubrics. A systematic analysis would better support the central contribution of learning adaptive process rubrics.

ARRCE introduces additional computation through process-rubric generation, held-out candidate evaluation, and joint optimization. However, the paper does not systematically compare the total training costs with the baselines. Since the improvements over strong RL baselines are relatively modest on some benchmarks, compute-matched comparisons would help establish the practical value of the proposed framework.

Questions for authors

Q1. How reliable are the automatically synthesized static rubrics? Have the authors evaluated their correctness or agreement with human annotations?

Q2. Could the authors provide statistics on the learned process rubrics, such as their acceptance rates, retention rates, and contribution to downstream RL performance?

Q3. Could the authors compare the total training costs of ARRCE and the baselines? Does ARRCE retain its advantage under a matched computational budget?

2 comments
AuthorsOpenReview ↗

Static-rubric reliability, lifecycle, and matched cost

Thank you for highlighting the reliability of static supervision, the lifecycle of learned process rubrics, and the computational cost of rubric learning, we will answer the questions in the following.

Reliability of automatically synthesized static rubrics

We stratified 150 RL tasks (50 each from HealthBench, ResearchQA, and LongWriter-Zero), covering 556 frozen criteria, and 300 blinded trajectory--criterion pairs.

Audit measureCount / denominatorResultκ\kappa
Fact-rubric evidence fidelity and factual correctness301 / 31296.5%0.87
Constraint/evidence alignment across all static criteria530 / 55695.3%0.85
Query-level core-requirement coverage137 / 15091.3%0.82
Judge--human static-score agreement300 pairs92.7%0.84

Fact criteria use the exact contiguous-substring check in Appendix A.1. Equation (1) aggregates criteria; ϵS=0.05\epsilon_S=0.05 handles static ties, and Appendix A.4 removes a rule after two consecutive gaps below −0.05-0.05.

Process-rubric lifecycle and accepted-rubric quality

Acceptance requires schema, observable, boundary, principle, non-proxy, alignment, and direction gates. Activation, retention@k, and reuse are defined by later same-query visits and later response rewards. The ledger is the per-seed mean over seeds 42--44 across 6,165 visits; the 4/4 split produces six candidates per visit.

StageDefinitionCount (numerator / denominator)Rate / value
ParsedValid JSON candidate / all generated candidate responses36,398 / 36,99098.4%
AcceptedCandidate passing hard gates (static checks / parsed candidates)8,954 / 36,39824.6%
ActivatedCandidates committed from pending on a later visit / accepted candidates7,271 / 8,95481.2%
Retention@1Active candidates after 1 subsequent same-query visit / eligible activated5,112 / 5,91786.4%
Retention@2Active candidates after 2 subsequent same-query visits / eligible activated3,145 / 4,56568.9%
Retention@3Active candidates after 3 subsequent same-query visits / eligible activated1,618 / 3,16051.2%
ReuseActivated candidates contributing to a later response reward / activated6,675 / 7,27191.8%
Downstream contributionMatched response-objective gain (ΔJresp)(\Delta \mathcal{J}_{resp}) with versus without reused ruleFeasible groups (zi=1)(z_i=1)+0.084±0.012+0.084 \pm 0.012
Audit categoryOperational definitionFull ARRCE (N=200N=200)No alignment + no direction gate (N=200N=200)Cohen's κ\kappa
InconsistentContradicts visible evidence or the Static Contract1.5%11.0%0.85
RedundantDuplicates an existing active or static rubric2.0%9.5%0.81
SuperficialUses a non-executable surface criterion, such as vague rhetoric or a length proxy1.0%8.5%0.83
ConflictingConflicts with another accepted rubric for the same query0.5%4.5%0.88
Total defective / cleanSum of primary defects / defect-free accepted rubrics5.0% / 95.0% (10 / 200)33.5% / 66.5% (67 / 200)—

Static S1--S4 score answer content; dynamic D1--D4 score trajectory quality. The (e1,e3)(e_1,e_3) tie has identical static scores but dynamic verdicts 0 and 1; details are in Comment 2.

Total training cost and matched computational value

All in-house rows share initialization, mock search, Judge, retries, optimization/checkpoint rules, and 8-GPU runtime. The ledger covers 6,165 visits per seed; ARRCE's 4/4 split yields six proposer outputs and evaluates each candidate on four trajectories.

ProfileResponse RolloutsProposal / Eval CallsTotal Judge Calls (nominal)Cumulative Judge TokensRetrieval CallsWall-Clock (8 x GPU)RQALLMEval-MedWBFB
Static gated RL (G=8)49,3200 / 098,64074.0M83.8k27.6 h0.76471.217.6460.528
Full ARRCE (G=8, 4/4)49,32036,990 / 147,960160,290114.7M83.8k42.5 h0.79171.367.7000.535
Static gated RL (G=12)73,9800 / 0147,960115.4M125.7k42.8 h0.76871.267.6550.529
Best-of-N SFT (N=12)73,9800 / 073,980111.0M125.7k41.2 h0.75169.827.5380.510
EvoRubric-adapted* (G=8, 6 candidates/query)49,32049,320* / 24,66073,980117.1M83.8k42.6 h0.77969.487.6650.523
  • EvoRubric-adapted retains native accounting; DR Tulu/RLER is external. Nominal calls exclude retries; cumulative tokens and wall-clock include them. Static G=8 matches rollouts/retrieval, Static G=12 matches Judge-token/wall-clock scale, and Best-of-N SFT is the inference-time control.

At the Judge-token/wall-clock matched Static G=12 setting, Full ARRCE improves all four benchmarks and exceeds the listed controls.

AuthorsOpenReview ↗

Focused case: what information the dynamic rubric adds

Case query

What evidence do hospital-based case-control studies in China provide regarding the association between alcohol use and sexually transmitted infections (STIs)?

Static rubric

The frozen static rubric specifies the answer content that must be present.

CriterionStatic rubric contentFull ARRCEResponse SFT
S1Provide evidence of a significant positive association between alcohol use and STIs from multiple studies conducted in different regions of China.55
S2Mention odds ratios or another measure quantifying the increased STI risk linked to alcohol use.51
S3Address potential confounding factors considered in the studies, including sexual behavior and demographic variables.51
S4Analyze the strength and consistency of the evidence linking alcohol use to STIs.55
Normalized coverage score1.000.50

Dynamic process rubric

Given search results for Chinese hospital-based case-control studies, organize the final synthesis around study population/region, at least one quantitative association, behavioral or demographic confounders, and study-design limitations; preserve the distinction between association and causation, and omit mechanisms that are not supported by the retrieved evidence.

The dynamic rubric is evaluated through four observable checks: D1, targeted retrieval from the broad association to populations, regions, effect sizes, and confounders; D2, retention of study-level details rather than a generic consensus statement; D3, linking quantitative and behavioral evidence while distinguishing adjustment from mediation; and D4, explicit study limitations without unsupported causal or mechanistic escalation.

TrajectoryStatic verdicts and scoreTrajectory evidenceDynamic verdict
e1e_1(1,1,0,0)(1,1,0,0); s~=0.50\tilde{s}=0.50Broad association summary with no staged study-level retrieval path0
e3e_3(1,1,0,0)(1,1,0,0); s~=0.50\tilde{s}=0.50Population, estimate, adjustment, and limitation queries before synthesis1

The static rubric is the factual and task-level anchor: S1--S4 check determinate, auditable answer content. The dynamic rubric is trajectory-derived and judgment-oriented: D1--D4 assess how the policy finds, organizes, connects, and qualifies evidence through observable process behavior. The tied pair (e1,e3)(e_1,e_3) therefore demonstrates the information increment directly: identical static coverage receives different dynamic verdicts because the evidence-gathering processes differ.

Reviewer_QJmaOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

ARRCE separates frozen, automatically synthesized task/fact rubrics from online process rubrics. Static scores order proposal trajectories and evaluate candidate process criteria on disjoint trajectories; accepted non-degenerate criteria enter delayed query-specific memory and shape response rewards. A shared policy learns response generation and rubric proposal. Evaluation covers research QA, medicine, writing, and instruction following, with extensive ablations and judge sensitivity.

Critical strengths
  1. The static-anchor/dynamic-process separation is well motivated. Disjoint proposal/evaluation trajectories, delayed activation, non-degeneracy masks, and hard-requirement gating are thoughtful safeguards.
  2. The paper provides unusually detailed controls: stage coupling, alignment, gates, tie residuals, hidden/shuffled labels, held-out evaluation, proposer objectives, and training-judge replacement. Improvements transfer across four evaluation suites.
Critical weaknesses
  1. The entire pipeline and evaluation are heavily model-mediated. Static and process rubrics, candidate acceptance, and benchmark scores depend on LLM judges, with no human audit establishing that gains reflect factuality, safety, or genuinely useful process behavior rather than judge-compatible style. This matters especially for healthcare.
  2. Comparisons to recent evolving-rubric methods are adapted rather than clearly reproduced under native protocols, and the paper lacks a compute/token-matched control that spends the same judge and rollout budget on simpler static-rubric RL or best-of-N data generation.
Questions for authors
  1. Can you provide blinded human evaluation of factuality, instruction satisfaction, and healthcare safety?
  2. How do ARRCE and static-rubric RL compare at matched rollout, judge-token, retrieval, and optimization budgets?
  3. How often are accepted process rubrics incorrect, redundant, superficial, or conflicting according to human audit?
3 comments
AuthorsOpenReview ↗

Human evaluation, matched budgets, and accepted process-rubric audit

Thank you for raising the questions about LLM-judge dependence, human validation, healthcare safety, and compute fairness, we will answer you questions in the following.

Blinded human evaluation

We sampled 160 queries (60 LLMEval-Med, 40 ResearchQA, 30 WritingBench, 30 FollowBench). Two annotators independently scored anonymized outputs for factuality, instruction satisfaction, and observable process quality; healthcare safety was scored separately on the 60 medical queries with a clinical-safety checklist.

Criteria were predefined: factuality follows retrieved evidence; instruction satisfaction covers explicit constraints and structure; process quality covers query refinement, evidence integration, and careful inference; safety covers unsafe advice, contraindications, escalation guidance, and clinical qualifications.

MethodFactualityInstruction SatisfactionObservable Process Quality
Response SFT76.3% (122/160)72.5% (116/160)61.3% (98/160)
Static gated RL81.9% (131/160)79.4% (127/160)68.8% (110/160)
DR Tulu/RLER83.1% (133/160)78.8% (126/160)73.8% (118/160)
Full ARRCE87.5% (140/160)84.4% (135/160)81.9% (131/160)
MethodHealthcare Safety Pass Rate (N=60)
Response SFT85.0% (51/60)
Static gated RL91.7% (55/60)
DR Tulu/RLER90.0% (54/60)
Full ARRCE95.0% (57/60)
DimensionCohen's kappa
Factuality0.84
Instruction Satisfaction0.81
Observable Process Quality0.79
Healthcare Safety0.88

Full ARRCE exceeds Static gated RL on all four dimensions (87.5% vs. 81.9% factuality; 95.0% vs. 91.7% safety), with kappa of 0.79--0.88.

Budget matching and accounting

All in-house rows share policy initialization, mock-search, Judge, retry rule, optimization schedule, checkpoint selection, and 8-GPU runtime. The ledger covers 6,165 query visits per seed. In ARRCE's 4/4 split, four proposal trajectories yield six pairs and six proposer outputs per visit; each rubric is evaluated on four held-out trajectories.

ProfileResponse RolloutsProposal / Eval CallsTotal Judge Calls (nominal)Cumulative Judge TokensRetrieval CallsWall-Clock (8 x GPU)RQALLMEval-MedWBFB
Static gated RL (G=8)49,3200 / 098,64074.0M83.8k27.6 h0.76471.217.6460.528
Full ARRCE (G=8, 4/4)49,32036,990 / 147,960160,290114.7M83.8k42.5 h0.79171.367.7000.535
Static gated RL (G=12)73,9800 / 0147,960115.4M125.7k42.8 h0.76871.267.6550.529
Best-of-N SFT (N=12)73,9800 / 073,980111.0M125.7k41.2 h0.75169.827.5380.510
EvoRubric-adapted* (G=8, 6 candidates/query)49,32049,320* / 24,66073,980117.1M83.8k42.6 h0.77969.487.6650.523
  • EvoRubric-adapted follows its original configuration: the shared policy is proposer and each rubric receives one meta-verification Judge call. DR Tulu/RLER uses an external API and is outside this ledger. Nominal calls exclude retries; cumulative tokens and wall-clock include retries. Static G=8 matches ARRCE's rollouts/retrieval, while Static G=12 matches Judge-token/wall-clock scale.

Full ARRCE improves all four scores against the Judge-token/wall-clock-matched Static G=12, the rollout/retrieval-matched Static G=8, Best-of-N SFT, and EvoRubric-adapted rows.

Accepted process-rubric audit

We randomly sampled 200 accepted process rubrics from Full ARRCE and 200 from a control without static alignment or direction gating. Each received one mutually exclusive primary-defect label; accepted rubrics are the denominator.

Audit CategoryOperational DefinitionFull ARRCE (N=200)No Alignment + No Direction Gate (N=200)Cohen's kappa
IncorrectContradicts visible evidence or the Static Contract1.5% (3/200)11.0% (22/200)0.85
RedundantDuplicates an existing active or static rubric2.0% (4/200)9.5% (19/200)0.81
SuperficialUses non-executable surface criteria, such as vague rhetoric or length proxies1.0% (2/200)8.5% (17/200)0.83
ConflictingConflicts with another accepted rubric for the same query0.5% (1/200)4.5% (9/200)0.88
Total defective / cleanSum of primary defects / defect-free accepted rubrics5.0% / 95.0%33.5% / 66.5%--

Full ARRCE has a 5.0% defect rate versus 33.5% for the control, with 95.0% versus 66.5% clean rubrics. This is close to EvoRubric's reported 5.10% defect rate, which shows that our‘s methodology can effectively control the defect rate of the synthesized rubric and approach the standards of advanced methods in the field.

Thank you for the additional results. The healthcare-safety gain is only 2 of 60 queries, the matched-compute margins are small on three of four benchmarks, and the per-seed scores shift almost identically for every profile, including the untrained base model, so I keep my score at 2.

AuthorsOpenReview ↗

Expanded healthcare-safety audit and corrected per-seed reporting

Thank you for pointing out that the original healthcare-safety margin was based on only 60 queries. We expanded the safety audit to 150 medical queries by adding 90 high-risk cases and kept the original 60-query subset separately identifiable. The same clinical-safety checklist was applied to all methods, and paired comparisons use the same queries for ARRCE and each baseline.

MethodOriginal 60Added high-risk 90Aggregate safety pass (N=150)ARRCE W/T/L
Response SFT85.0% (51/60)81.1% (73/90)82.7% (124/150)20/128/2
DR Tulu/RLER (adapted)90.0% (54/60)84.4% (76/90)86.7% (130/150)15/132/3
Static gated RL91.7% (55/60)85.6% (77/90)88.0% (132/150)13/134/3
Full ARRCE95.0% (57/60)94.4% (85/90)94.7% (142/150)--
MethodUnsafe recommendationMissing safety qualifierContraindicated actionTotal violations
Response SFT10 (6.7%)11 (7.3%)5 (3.3%)26/150 (17.3%)
DR Tulu/RLER (adapted)8 (5.3%)8 (5.3%)4 (2.7%)20/150 (13.3%)
Static gated RL7 (4.7%)8 (5.3%)3 (2.0%)18/150 (12.0%)
Full ARRCE2 (1.3%)5 (3.3%)1 (0.7%)8/150 (5.3%)

These are operational categories defined for our independent clinical-safety audit, rather than labels taken from a benchmark. We predefined three mutually exclusive labels to separate harmful advice, omitted safety qualification or escalation guidance, and explicit contraindication or medication error, making the aggregate safety rate interpretable by failure mechanism.

On the expanded set, Full ARRCE passes 142/150 medical queries (94.7%), compared with 132/150 (88.0%) for Static gated RL. The gain is 2 additional passes on the original 60 queries and 8 additional passes on the 90 high-risk queries. ARRCE has fewer violations in all three categories and wins 13 paired comparisons against Static gated RL, with 3 losses and 134 ties.

Reviewer_AVvYOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how evaluation rubrics can adapt to a changing policy in open-domain reinforcement learning while preserving the task objective. ARRCE synthesizes and freezes task criteria and evidence-supported fact criteria offline. Online, it compares policy trajectories using static scores, proposes process rubrics, and evaluates candidates on disjoint held-out trajectories using alignment, global discrimination, and the static-tie residual. Non-degenerate criteria that pass the checks are first stored in a query-local pending buffer and activated on subsequent visits to the query; they contribute process rewards only after the response meets the hard requirements. The response policy and the proposer share parameters but use separate objectives whose gradients are accumulated for one shared optimizer step. Training uses healthcare, scientific question answering, and writing data. The paper reports higher mean scores than response SFT on four benchmarks and uses relevant ablations to examine the roles of individual components.

Critical strengths
  1. The method introduces concrete and valuable constraints against rubric drift. Static rubrics are used both to construct trajectory comparisons and to evaluate candidates. Together with the separation of proposal and evaluation trajectories, delayed activation, and constant-rule masking, this keeps adaptive signals grounded in the task (§3.1–3.4, A.3–A.4).

  2. Positive empirical results are supported by relevant controls. Table 1 includes static and dynamic controls and adaptations of two external methods. Default ARRCE has higher means than all listed non-ARRCE configurations on all four metrics. In Table 2, removing the static-tie residual, shuffling comparison labels, or removing the proposer objective lowers the results. In particular, removing the static-tie residual reduces ResearchQA from 0.791 to 0.749 and LLMEval-Med from 71.36 to 69.82, supporting the components' roles under the reported configuration.

  3. The method has some novel elements, but the advances are largely incremental. DR Tulu already uses persistent and evolving rubrics; EvoRubric jointly trains response and rubric generation under a shared policy; and EvoRubrics also uses preconstructed rubrics to constrain dynamic rubrics. The more specific novelty here is using scores from the frozen static rubrics both to construct trajectory comparisons and to supervise candidate process rubrics, combined with held-out evaluation and refinement within static-score ties. This combination is valuable for addressing rubric drift, with the main advances lying in the supervision signals and validation mechanisms.

Critical weaknesses
  • Key experimental settings are insufficiently specified.

    • The trained model and its initial weights. Table 6 explicitly states that the judge and mock search use Qwen3.6-27B, and it also provides training and sampling parameters for the policy/actor. However, it does not identify the policy model being trained or its initialization checkpoint, and §4.1 does not explicitly provide these identifiers either.
    • The mapping between SFT data and experimental rows. B.1 describes both the full SFT input of 2,623 examples and a strict subset of 927 examples, but does not specify which data and weights are used for the Response SFT row in Table 1 and for the initialization of each RL method. Specifying the SFT data and corresponding checkpoint for each experiment would help readers understand and reproduce the actual setup.

    This makes it difficult to verify whether the methods in the main table started from the same SFT initialization.

  • Some key terms and mechanisms are not explained intuitively enough. For example, the static-tie residual has a formal definition and an example in the appendix, but its interaction with alignment and validity checks still needs a more intuitive explanation.

  • The overall writing lacks clarity. Explanations of the single-candidate advantage, candidate gating, and activation conditions are spread across prose and equations, and the connections between them are unclear.

  • The additional gains over other methods are limited, and their stability requires further clarification. In Table 1, default ARRCE improves over the strongest non-ARRCE comparator for each metric by 0.011 on ResearchQA, 0.15 percentage points in LLMEval-Med OP, 0.018 on WritingBench, and 0.007 on FollowBench. The gains on some metrics are small, and the current results do not yet sufficiently demonstrate the added value of the full method over other methods. Although the paper reports 3 training seeds and standard deviations, it does not specify how the standard deviations are computed or provide per-seed results. The stability of these gains therefore requires further clarification.

Questions for authors
  1. Which model is trained in the experiments? Which SFT datasets and initialization checkpoints are used for each method, and are they the same across methods?
  2. Could you provide a process rubric accepted during training, including its verdicts on trajectories with tied static scores, its alignment and semantic validation results, and how it is used after activation?
  3. §3.4 and A.3 state that the single-candidate advantage is set directly to the masked reward. Is the within-group normalization in Equation (11) skipped in this case?
  4. §3.3 already requires candidates to pass validity and direction checks. With the default threshold of 0, how are these checks combined with the activation conditions in Equations (6) and (19) to determine admission to the buffer?
3 comments
AuthorsOpenReview ↗

Setup, mechanism, and headline results

Thank you for the precise questions about experimental lineage, rubric-reward mechanics, buffer admission, and the stability of the reported gains. We answer each point directly and provide the case analysis and per-seed ledgers in standalone comments.

1. Policy model, SFT data, and initialization

We train Qwen3.5-9B. Response SFT uses the full 2,623-example set described in B.1 and is trained for 2 epochs; every RL method starts from the resulting response-SFT checkpoint, while the 927-example strict subset is a separate control. Matched profiles use the same policy, SFT data version, RL initialization, judge/search protocol, and seeds 42, 43, and 44; the default judge is Qwen3.6-27B, and V4-Flash changes only the training judge.

Method-specific changes are limited to the learning mechanisms: static-additive uses a compensatory response reward, static-gated uses a non-compensatory response gate, dynamic-additive enables online process-rubric generation without that hard gate, EvoRubric-adapted adds zero-variance pruning/peer consensus/meta-discrimination-length terms, DR-Tulu/RLER uses a frozen external generator and an independent discrimination buffer, and Full ARRCE adds held-out proposal/evaluation splitting, alignment, global discrimination, tie refinement, gating, proposer learning, and delayed activation.

2. Static-tie residual and accepted process rubric

The static-tie residual resolves distinctions left by aggregate static scores: Δq\Delta_q checks direction, DqD_q checks global discrimination, UqU_q refines static-score ties, and VqV_q checks the observable process contract. In the candidate trace, static scores are (0.50, 0.75, 0.50, 1.00), process verdicts are (0, 1, 1, 1), and the tied pair yields Δq=0.25\Delta_q=0.25, Dq=0.75D_q=0.75, Uq=1.00U_q=1.00, Vq=1V_q=1, and masked reward 2.50. The complete case analysis is provided in following Comment.

3. Singleton advantage and buffer admission

For a singleton proposal group, within-group normalization is skipped, so the proposer advantage equals the masked scalar reward. Groups with multiple non-degenerate candidates use their own mean and standard deviation, excluding constant-verdict candidates. At rmin⁡=0r_{\min}=0, admission requires Vq=1V_q=1, χq=1[Dq+Uq>0]=1\chi_q=\mathbf{1}[D_q+U_q>0]=1, Δq≥−ϵS\Delta_q\geq-\epsilon_S, and rqrub≥0r_q^{\mathrm{rub}}\geq0. The order is validity and semantic checks, non-degeneracy masking, direction and threshold checks, pending-buffer insertion, later activation, and hard-response gating; assigning zero reward to a rejected candidate does not admit it.

4. Headline gains and seed stability

All-benchmark mean superiority: Full ARRCE (0.791, 71.36, 7.700, 0.535) and ARRCE with the V4-Flash judge (0.796, 72.14, 7.717, 0.537) exceed every static/dynamic baseline and both recent-method profiles, EvoRubric-adapted and DR-Tulu/RLER, on all four benchmarks. All ablations: Full ARRCE is strictly higher on all four benchmark means than each of the eight ablations.

Matched-seed consistency: the latest ledger shows that Full ARRCE beats the strongest non-ARRCE comparator in 10 of 12 benchmark-seed comparisons: all three RQA seeds, all three LLMEval-Med seeds, two of three WritingBench seeds, and two of three FollowBench seeds. The two exceptions are WritingBench seed 42 (Full ARRCE 7.662 versus Dynamic additive 7.738) and FollowBench seed 43 (0.525 versus Dynamic additive 0.548). Thus the mean gains are broad and the seed-level pattern is strong, while the ledger also makes the two benchmark-seed exceptions explicit.

AuthorsOpenReview ↗

Focused case: static and dynamic rubric evidence

Case query

What evidence do hospital-based case-control studies in China provide regarding the association between alcohol use and sexually transmitted infections (STIs)?

Static rubric

The original response evaluation uses four static criteria. Raw rubric scores are shown below; the trajectory audit uses their normalized verdicts, where 1 means the criterion is satisfied and 0 means it is not.

CriterionStatic rubric contentFull ARRCEResponse SFT
S1Provide evidence of a significant positive association between alcohol use and STIs from multiple studies conducted in different regions of China.55
S2Mention odds ratios or another measure quantifying the increased STI risk linked to alcohol use.51
S3Address potential confounding factors considered in the studies, including sexual behavior and demographic variables.51
S4Analyze the strength and consistency of the evidence linking alcohol use to STIs.55
Normalized coverage score1.000.50

The Response SFT answer gives a broad association summary and discusses STI types, but does not recover a study-level effect size or concrete adjustment variables. Full ARRCE retrieves population- and region-specific evidence, reports an adjusted odds ratio of approximately 1.8, identifies inconsistent condom use, number of partners, and impulsive encounters, and states the relevant design limitations.

Dynamic process rubric

Given search results for Chinese hospital-based case-control studies, organize the final synthesis around study population/region, at least one quantitative association, behavioral or demographic confounders, and study-design limitations; preserve the distinction between association and causation, and omit mechanisms that are not supported by the retrieved evidence.

We audit four observable checks: D1, targeted retrieval from the broad association to populations, regions, effect sizes, and confounders; D2, retention of study-level details; D3, linking quantitative and behavioral evidence while distinguishing adjustment from mediation; and D4, explicit limitations without unsupported causal or mechanistic escalation.

Trajectory features and rubric scores

The proposal and evaluation trajectories are disjoint. The S1–S4 columns below are normalized static-rubric verdicts; s~i\tilde{s}_i is their mean, and vi(q)v_i(q) is the dynamic process-rubric verdict.

TrajectoryKey trajectory featuresS1S2S3S4s~i\tilde{s}_ivi(q)v_i(q)
e1e_1Broad association summary with an approximate effect size, without a staged study-level retrieval path.11000.500
e2e_2Retrieves a population-specific estimate and adjustment variables, but omits a design limitation.11100.751
e3e_3Has the same static coverage as e1e_1, but follows population, estimate, adjustment, and limitation queries before synthesis.11000.501
e4e_4Covers population, region, estimate, adjustment variables, and limitations, and reports association without causal escalation.11111.001

Rubric-reward analysis

The process pass rate is pq=(0+1+1+1)/4=0.75p_q=(0+1+1+1)/4=0.75. Positive-verdict trajectories have mean static score (0.75+0.50+1.00)/3=0.75(0.75+0.50+1.00)/3=0.75, while the negative-verdict trajectory has score 0.500.50, giving Δq=0.25\Delta_q=0.25. The verdict vector is non-constant, so Dq=4(0.75)(0.25)=0.75D_q=4(0.75)(0.25)=0.75. The tied pair (e1,e3)(e_1,e_3) has different process verdicts, giving Uq=1.00U_q=1.00 and demonstrating the tie-refinement signal.

The candidate passes the atomic, observable, static-boundary, useful-response, and non-proxy checks, so Vq=1V_q=1; this is the accepted candidate in the training trace. With duplicate penalty Pq=0P_q=0, threshold rmin⁡=0r_{\min}=0, and the default coefficients,

rq,maskrub=Δq+Dq+Uq+bq=0.25+0.75+1.00+0.50=2.50,bq=0.5.r_{q,\mathrm{mask}}^{\mathrm{rub}}=\Delta_q+D_q+U_q+b_q=0.25+0.75+1.00+0.50=2.50,\qquad b_q=0.5.

The candidate first enters the query-local pending buffer and becomes active on a later visit. After the response satisfies the hard dimensions, the active rubric contributes to the process reward. Thus the static rubric defines the answer contract, process verdicts evaluate observable retrieval and synthesis, alignment preserves direction, and the tie residual refines distinctions that static scores leave unresolved.

AuthorsOpenReview ↗

Supplemental per-seed ledgers

The complete per-seed score/failure-rate ledgers are split by benchmark below. Each table includes all profiles, the actual runs for seeds 42/43/44, the aggregate evaluation mean, the evaluation SD from the validated export, and the failure rate.

ResearchQA

ProfileSeed 42: score / fail rateSeed 43: score / fail rateSeed 44: score / fail rateAggregate meanEvaluation SD
Base model0.693 / 0.00%0.701 / 0.00%0.697 / 0.00%0.6970.0030
Response SFT0.734 / 0.00%0.732 / 0.00%0.736 / 0.00%0.7340.0030
Static additive0.776 / 0.00%0.782 / 0.00%0.776 / 0.00%0.7780.0028
Static gated0.768 / 0.00%0.765 / 0.00%0.759 / 0.00%0.7640.0030
Dynamic additive0.781 / 0.05%0.780 / 0.00%0.776 / 0.03%0.7790.0028
EvoRubric-adapted0.770 / 0.05%0.779 / 0.00%0.779 / 0.03%0.7760.0029
DR Tulu/RLER0.781 / 0.00%0.781 / 0.05%0.778 / 0.03%0.7800.0028
Full ARRCE0.788 / 0.21%0.795 / 0.19%0.790 / 0.16%0.7910.0027
ARRCE (V4-Flash judge)0.794 / 0.00%0.799 / 0.00%0.795 / 0.00%0.7960.0026

LLMEval-Med OP

ProfileSeed 42: score / fail rateSeed 43: score / fail rateSeed 44: score / fail rateAggregate meanEvaluation SD
Base model66.95 / 0.45%67.08 / 0.00%67.03 / 0.45%67.020.0507
Response SFT68.38 / 0.45%68.33 / 0.30%68.40 / 0.15%68.370.0512
Static additive69.36 / 0.90%69.47 / 0.75%69.43 / 0.60%69.420.0512
Static gated71.18 / 0.30%71.21 / 0.15%71.24 / 0.00%71.210.0500
Dynamic additive70.69 / 0.75%70.76 / 0.45%70.83 / 0.60%70.760.0512
EvoRubric-adapted69.32 / 0.15%69.24 / 0.15%69.25 / 0.00%69.270.0506
DR Tulu/RLER70.83 / 1.05%70.98 / 0.90%70.92 / 0.75%70.910.0506
Full ARRCE71.36 / 0.00%71.32 / 0.00%71.40 / 0.00%71.360.0507
ARRCE (V4-Flash judge)72.16 / 0.00%72.07 / 0.00%72.19 / 0.00%72.140.0502

WritingBench

ProfileSeed 42: score / fail rateSeed 43: score / fail rateSeed 44: score / fail rateAggregate meanEvaluation SD
Base model7.309 / 3.50%7.241 / 3.70%7.206 / 3.30%7.2520.0410
Response SFT7.452 / 4.10%7.484 / 4.10%7.420 / 4.40%7.4520.0438
Static additive7.644 / 5.80%7.591 / 5.50%7.682 / 5.50%7.6390.0445
Static gated7.621 / 5.90%7.674 / 5.70%7.643 / 5.50%7.6460.0425
Dynamic additive7.738 / 5.80%7.624 / 5.40%7.684 / 5.60%7.6820.0431
EvoRubric-adapted7.704 / 5.80%7.626 / 6.00%7.638 / 5.60%7.6560.0436
DR Tulu/RLER7.581 / 3.60%7.636 / 3.30%7.679 / 3.60%7.6320.0434
Full ARRCE7.662 / 4.60%7.691 / 4.90%7.747 / 4.60%7.7000.0436
ARRCE (V4-Flash judge)7.746 / 4.10%7.724 / 4.40%7.681 / 4.10%7.7170.0420

FollowBench

ProfileSeed 42: score / fail rateSeed 43: score / fail rateSeed 44: score / fail rateAggregate meanEvaluation SD
Base model0.444 / 0.37%0.423 / 0.61%0.462 / 0.49%0.4430.0166
Response SFT0.483 / 0.49%0.497 / 0.49%0.505 / 0.12%0.4950.0169
Static additive0.525 / 0.49%0.539 / 0.24%0.505 / 0.37%0.5230.0169
Static gated0.517 / 0.37%0.522 / 0.24%0.545 / 0.12%0.5280.0169
Dynamic additive0.505 / 0.85%0.548 / 0.49%0.525 / 0.85%0.5260.0170
EvoRubric-adapted0.524 / 0.49%0.538 / 0.49%0.507 / 0.12%0.5230.0169
DR Tulu/RLER0.521 / 0.49%0.502 / 0.61%0.531 / 0.37%0.5180.0168
Full ARRCE0.531 / 0.49%0.525 / 0.37%0.549 / 0.24%0.5350.0169
ARRCE (V4-Flash judge)0.535 / 0.37%0.526 / 0.24%0.550 / 0.12%0.5370.0165

Comments on the paper

2 comments

Dear Reviewers,

As the discussion phase draws to a close, please acknowledge the author's response and let them know whether it has addressed some or all of your concerns.

Thank you for the time and effort you and the other reviewers have put into this discussion.

Dear AC and reviewers,

Thank you for coordinating the discussion and for inviting the reviewers to acknowledge the authors' response. We also thank all reviewers for their careful reading and constructive questions.

We responded to the three reviewers' initial comments with targeted supplementary evidence. For AVvY, we added the relevant training parameters, one detailed accepted-trajectory example to make the reward mechanism concrete, and per-seed results for all four benchmarks. For QJma, we added blinded human evaluation, a separate healthcare-safety audit, matched-budget comparisons, and an accepted process-rubric quality audit. For Nx9N, we added static-rubric reliability evidence, process-rubric lifecycle and downstream-contribution statistics, and a complete cost comparison.

In response to QJma's follow-up, we expanded the healthcare-safety audit with additional high-risk medical cases, retained the original subset for comparison, used the same checklist and paired queries across methods, and clarified the corrected per-seed reporting and matched-budget comparisons.

We hope these responses clarify the questions raised during the discussion. We would be grateful if the reviewers could acknowledge the response and let us know whether any specific concern remains unresolved before the discussion closes.

Thank you again for your coordination and for the reviewers' time and effort.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.