acceptodds
Under review as a conference paper at ICLR 2027

Co-Evolving Policy and Rubric via Capability Awareness for Open-Ended Generation

Abstract

Rubric-based reinforcement learning (RL) offers interpretable, fine-grained supervision for aligning large language models (LLMs) on open-ended generation tasks. However, static rubric criteria may become saturated or remain beyond the policy's exploration, rendering the reward signals insufficiently discriminative. While recent efforts seek to guide sampling exploration or evolve rubrics, they fail to explicitly factor in how rubric difficulty should track the evolving policy's capabilities, resulting in a persistent mismatch between the optimization objective and the policy's current capability frontier. In this paper, we propose -Evolving Policy and Rubric via apability wareness (), a closed-loop framework that jointly adapts the rubric and policy through criterion-level diagnosis. Specifically, CoCA contrasts query-only and rubric-guided rollouts to characterize each criterion relative to the current policy, recalibrates criteria with mismatched difficulty, and identifies guidance-revealed capability gaps for frontier-based learning. Based on this insight, CoCA either expands rubric-conditioned frontiers through guided optimization or transfers qualified gains through on-policy self-distillation. As the policy updates, CoCA improves the capability diagnosis and enables more reliable distillation, thereby completing the co-evolution loop. Extensive experiments across diverse benchmarks and LLMs demonstrate that CoCA consistently improves performance over rubric-based RL methods. Moreover, CoCA promotes the internalization of guidance-revealed capabilities into query-only behavior while mitigating capability degradation.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Models, Reinforcement Learning, Rubric

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_4xhsOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper targets reward saturation in rubric-based RL, where fixed criteria become uniformly met or unmet during training. CoCA contrasts query-only and rubric-conditioned rollouts to label each criterion's state, then uses these states to rewrite criterion difficulty per step, gate on-policy self-distillation from rubric-conditioned responses, and otherwise run GRPO on rubric-conditioned rollouts, on top of standard GRPO. Across three backbones and seven medical, science, and psychology benchmarks, CoCA achieves the best mean score and improves over standard rubric-based GRPO.

Critical strengths
  1. The dual-group diagnosis is a simple and useful idea. Contrasting query-only and rubric-conditioned rollouts separates failures that guidance can resolve from those it cannot. A non-trivial share of criteria unmet under query-only sampling become satisfiable once the rubric is provided (Fig. 1b). This matters because it is the paper's one clearly new element, and it could be reused beyond this method.

  2. Gains are consistent across settings. CoCA has the best mean on all three backbones (Table 1). The gains persist under three evaluation judges not used in training (Table 3) and at 14B (Table 12). This matters because it makes judge overfitting unlikely as the sole explanation.

Critical weaknesses
  1. The core contribution is unclear, and "co-evolution" is overstated. Each component has a prior counterpart: guided rollouts (RuscaRL), rubric rewriting (Online Rubrics), and rubric-privileged self-distillation (RGSD, CriPO). Moreover, the rubric is not accumulated: each step recalibrates the original rubric and discards the rewrite (App. C), so the method performs per-step, policy-conditioned reward reshaping rather than persistent co-evolution. No single component carries the gain either. Against a budget-matched RaR (Table 14), calibration alone gives a marginal gain on HealthBench-Hard and a drop on LLMEval-Med (Table 4), and the closest methods are not compared. Since the title and claimed contributions rest on co-evolution, the lack of an identifiable core insight makes the paper read as an engineering combination of existing techniques.

  2. Comparisons with baselines are not effort-matched. Table 1 gives CoCA twice the rollouts of RaR/RuscaRL. At a matched budget (Table 14), the gain over RaR roughly halves, and the other settings are never matched. CoCA's rollout split is tuned (Table 14), but baseline tuning is not described, and RGSD uses a single rollout per query and scores below the untrained base model on most backbones. The remaining small gain is therefore within the range that unmatched budgets, tuning effort, and seed variance could plausibly explain.

  3. Infrastructure cost and scalability are not analyzed. Each step doubles rollouts, multiplies judge calls (judge, rewrite, and rejudge run serially on the critical path), and adds a long-context teacher forward for OPSD. The gate must also wait until all rollouts of a query are generated, judged, rewritten, and rejudged, a per-query barrier that conflicts with asynchronous RL pipelines and amplifies long-tail latency. Finally, all runs pair small policies with a much stronger judge/rewriter, and tightening saturated criteria may fail once the policy matches or exceeds the rewriter, as it would at frontier scale. The practical value of an RL training method depends on gain per unit of compute and on fitting production pipelines, and the paper evaluates neither.

  4. Rubric rewriting is unfiltered and can shift the objective in safety-relevant domains. The independent audit (App. F.3) shows that a small but nonzero share of rewrites drift from the original intent, yet no acceptance filter is applied during training. In medical and psychological tasks, silently weakened criteria can reward unsafe responses, and it is unclear whether safety-critical criteria are protected.

  5. Gains over the strongest baseline concentrate on the authors' own LLM-built benchmark (Table 1). Compared per benchmark with the best of RaR, RuscaRL, and Online Rubrics, a large part of CoCA's margin on Qwen3-4B and Gemma-3-4B-it comes from CPsyCoun. Excluding it, CoCA's lead over the strongest baseline is marginal on all three backbones, and on Qwen3-4B it trails a baseline on RaR-Medicine and HeartBench. The seven-benchmark mean thus overstates how broadly CoCA beats strong baselines; on established benchmarks, the margin is small.

Questions for authors
  1. Since rewrites are discarded every step (App. C), in what sense does the rubric co-evolve? Have you tried carrying calibrated criteria over to later steps?
  2. Can Table 1, or at least Table 4, be reported against 8-rollout baselines? How were baseline hyperparameters chosen, why does RGSD use one rollout per query, and which checkpoint is reported for each method?
  3. How many independent runs underlie the main tables, and what is the uncertainty of the Table 14 comparison?
  4. How does CoCA compare with RaR at matched wall-clock or judge-token budget, beyond the single point in Fig. 4(b)? Please report judge calls/tokens per step and GPU-hours for the 14B run.
13 comments
AuthorsOpenReview ↗

Response to W1 & Q1 (Part 1/2)

W1 & Q1: Clarification of the core contribution, co-evolution without accumulating rubric edits, and the effect of retaining calibrated criteria.

Thank you for raising these related concerns, we apologize for any misunderstanding caused by unclear wording in the original submission. By “co-evolution,” we mean reciprocal adaptation between the policy and its training-time rubric through capability feedback. CoCA's central contribution is using dual-group diagnosis to jointly determine how to calibrate training criteria, when to expand guided capabilities, and when to transfer them into query-only behavior.

1. Dual-group diagnosis coordinates the feedback loop, which persists through policy updates.

A criterion unmet under query-only sampling may become achievable with guidance or remain unmet in both groups (Figure 1b). CoCA distinguishes these cases by comparing responses from the same pre-update policy against the same criteria. This distinction provides a shared control signal for reward adaptation and capability learning.

CoCA uses this diagnosis to construct a dynamic training curriculum (Sections 3.1–3.4): relaxing Too-Hard criteria recognizes intermediate progress, while strengthening Saturated criteria distinguishes more complete fulfillment. Learnable criteria identify capabilities elicitable through guidance. When a Guided Frontier satisfies all Learnable criteria and clears the required reward-improvement margin, Frontier-OPSD transfers its guidance into the query-only policy; otherwise, Guided GRPO develops stronger guided capabilities. The rubric adapts to the policy's current capabilities, and learning under that rubric changes what supervision the policy needs next. This reciprocal dependence persists through policy updates even when rubric edits are discarded.

2. The original rubric provides a stable reference for measuring capability changes.

Each query revisit starts from the original rubric. For HealthBench, this expert-authored gold rubric anchors capability diagnosis: the criteria stay fixed, while achievement rates and capability states evolve with the policy. Calibration adapts training targets to these changing states, separating measurement against a fixed reference from adaptive supervision and limiting the accumulation of drift through successive rewrites.

Figure 4, p. 8 (originally Figure 3, p. 8) shows larger increases in Saturated criteria and larger reductions in Learnable, Too-Hard, and Regressive criteria with CoCA than with RaR across all three backbones. To examine whether these shifts reflect internalization, Table 6, p. 9 (originally Table 5, p. 8) compares base and trained models using the same fixed HealthBench-Hard rubrics. Its internalization transitions are reproduced below (%):

Transition: Base → TrainedQwen3-4B: RaRCoCAQwen3-8B: RaRCoCAGemma-3-4B-it: RaRCoCA
Learnable → Saturated8.109.836.9614.0213.8415.76
Learnable → Discriminative33.6934.4732.5636.8830.9735.70
Too-Hard → Saturated2.633.650.495.256.348.67
Discriminative → Saturated41.1743.6140.6851.2142.8945.42

CoCA exceeds RaR on all four transitions across all three backbones, e.g., Learnable → Saturated on Qwen3-8B (6.96% → 14.02%; +7.06%). These results support the transfer of guided capabilities into query-only behavior. We will clarify that co-evolution refers to this policy–rubric feedback, with rubric edits regenerated at each revisit.

3. Comparison with cross-epoch rubric retention.

Following your suggestion, we have implemented CoCA*, which carries each query's calibrated rubric from epoch ee into epoch e+1e+1 for guided sampling, capability diagnosis, and further calibration. The first epoch uses the original rubric, and evaluation always uses the original benchmark criteria.

The comparison uses Qwen3-4B and Qwen3-8B in the medical domain, holding the four-epoch schedule, rollout budget, and other hyperparameters fixed between CoCA and CoCA*. Retaining rubric edits yields slightly lower scores on all three benchmarks for both backbones (−0.55% to −2.55%), supporting rebuilding the curriculum from a stable reference and the policy's current capabilities.

BackboneMethodHealthBench-HardRaR-MedicineLLMEval-Med
Qwen3-4BBase22.3851.7838.38
Qwen3-4BRaR37.1256.0342.58
Qwen3-4BCoCA41.0156.3346.48
Qwen3-4BCoCA*40.1855.5945.93
Qwen3-8BBase25.8659.4444.28
Qwen3-8BRaR41.8864.2949.93
Qwen3-8BCoCA46.8665.3752.77
Qwen3-8BCoCA*45.6364.7550.22
AuthorsOpenReview ↗

Response to W1 & Q1 (Part 2/2)

W1 & Q1: Clarification of the core contribution, co-evolution without accumulating rubric edits, and the effect of retaining calibrated criteria.

4. Component ablations and rollout-budget controls support the coordinated mechanism.

In Table 4, full CoCA improves over calibration alone from 48.73% to 52.77% on LLMEval-Med, alongside the smaller 46.30% to 46.86% gain on HealthBench-Hard. The cross-dataset improvement supports coordinating reward calibration, guided learning, and capability transfer.

For budget accounting, CoCA's rollout budget for query-only GRPO is matched to that of RaR and Online Rubrics in the default setup. Its additional rubric-conditioned responses support capability diagnosis, calibration, and frontier selection. The two groups have separately normalized advantages; guided responses are never pooled into the query-only GRPO group. Table 16, Appendix E.6, p. 23 (originally Table 14, Appendix E.5, p. 21) additionally controls for these extra samples through matched 8- and 12-total-rollout comparisons.

We further re-evaluated the budget controls with Qwen3.7-Flash. Below is the 8-rollout subset for Qwen3-8B; the complete results, including 12-rollout configurations, appear in our response to your third question (Q3). At the same total rollout budget, CoCA exceeds the strongest baseline by +1.05% on HealthBench-Hard and +4.15% on LLMEval-Med, while RaR leads on RaR-Medicine. These controls support gains beyond simply increasing sampling.

MethodPrimary rolloutsAdditional rolloutsTotal rolloutsHealthBench-HardRaR-MedicineLLMEval-Med
RaR8—844.34 ± 0.3164.33 ± 0.0157.90 ± 1.23
RuscaRL8—847.04 ± 0.1363.80 ± 0.0453.67 ± 0.21
Online Rubrics44846.10 ± 0.1162.10 ± 0.1862.17 ± 0.32
CoCA44848.09 ± 0.0961.93 ± 0.0266.32 ± 0.19
AuthorsOpenReview ↗

Response to W2 & Q2

W2 & Q2: Fairness of rollout budgets, baseline hyperparameters, RGSD sampling, and checkpoint selection.

Thank you for these questions. We clarify the budget controls and experimental settings, including the scope of the evidence on robustness, below.

1. Query-only GRPO budgets are matched, and Table 16 additionally controls total rollouts.

In the default setup, CoCA's rollout budget for query-only GRPO is matched to that of RaR and Online Rubrics. The additional rubric-conditioned responses support capability diagnosis and frontier learning, with advantages normalized separately from the query-only group. These additional samples contribute to the total sampling cost.

Table 16, Appendix E.6, p. 23, reports matched-total-rollout comparisons for the Qwen3-8B medical-domain results in Table 1. The budgets include both primary and additional rollouts, counting Online Rubrics' control responses and CoCA's Guided responses. We have also incorporated these results into Section 4.3 (Figure 3) of the revised manuscript.

MethodPrimary + AdditionalTotal rolloutsHealthBench-HardRaR-MedicineLLMEval-MedMean
RaR8 + 0845.5164.9250.3753.60
RuscaRL8 + 0846.2364.2345.4351.96
Online Rubrics4 + 4845.3264.5949.4853.13
CoCA4 + 4846.8665.3752.7755.00
RaR12 + 01246.9165.7550.8254.49
RuscaRL12 + 01245.8064.9747.9852.92
Online Rubrics6 + 61245.7765.0749.7853.54
CoCA6 + 61248.6665.3453.0755.69
CoCA8 + 41248.9066.9254.2756.70

Increasing RaR's sampling budget reduces the margin, but CoCA retains the highest mean at both matched budgets: +1.40% at 8 rollouts and +1.20% / +2.21% at 12 rollouts. Even the default 8-rollout CoCA exceeds all 12-rollout baselines in mean score. We have made these controls and their scope more prominent in the revised manuscript.

2. Baseline settings follow the original papers; Table 16 examines robustness to rollout allocation.

We use the best-performing baseline settings reported in their original papers. These configurations are documented in Appendix D.2, p. 21 (originally Appendix D.2, p. 20) and Tables 9–10, p. 20 (originally Tables 8–9, p. 19): Table 9 gives the general training settings, and Table 10 specifies method-specific settings for RuscaRL, Online Rubrics, and RGSD. We will state their provenance more explicitly.

CoCA uses an equal split between Normal and Guided responses by default. Table 16 reports both equal and unequal allocations at the larger budget, and both outperform the strongest matched-budget baseline in mean score. The advantage is thus present in the default allocation and persists across the tested alternatives. We have also conducted multiple rounds of re-evaluation of the Table 16 configurations using Qwen3.7-Flash, an independent judge not used during training; detailed results are provided below in our response to your third question (Q3). We will distinguish these sampling-ablation variants from the default settings more explicitly.

3. RGSD's single rollout follows its pure-distillation training setup.

RGSD [1] trains the query-only student with token-level supervision from a rubric-conditioned teacher, without reward-based RL updates. Unlike GRPO, it does not require multiple responses to compute group-relative advantages. One on-policy rollout per query follows the original paper's training setup, providing a student trajectory for dense token-level distillation.

4. Checkpoints are selected consistently on held-out validation data.

For every method, we select the checkpoint with the best performance on an independent validation split of the corresponding domain's training dataset. This selection rule is applied consistently across methods; test-benchmark scores are not used to select checkpoints.

Reviewer_gesiOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes CoCA, a framework that combines rubric-based reinforcement learning and on-policy self-distillation through capability-aware rubric calibration. The method compares query-only and rubric-conditioned responses from the same policy to classify individual criteria into five capability states. These states guide rubric revision and identify potential targets for capability transfer. Depending on whether a qualified guided response is available, the method activates either frontier-conditioned self-distillation or guided-group GRPO, while retaining query-only GRPO throughout training. The paper reports consistent improvements across medical, scientific, and psychological generation tasks.

Critical strengths
  1. The paper addresses a key limitation: static rubric criteria can become uninformative during training, while capabilities elicited through explicit guidance may fail to transfer to query-only behavior.

  2. The method is clear. The treatment of frontier eligibility conditions, teacher distillation, and separate group-relative advantage normalization are clearly described.

  3. The experimental coverage is broad. The authors report results on three different domains, and the appendices include alternative evaluation judges, rollout-budget controls, and ablations of distillation trajectories.

Critical weaknesses
  1. (Main) The proposed method primarily integrates established techniques, including adaptive rubric calibration [1], rubric-guided exploration [2], and on-policy self-distillation [3]. Its potential novelty therefore lies in the coordination of these established components. However, the current experiments do not sufficiently establish its contribution beyond a straightforward combination of existing approaches. For example, on Qwen3-8B, rubric calibration alone achieves 46.30 on HealthBench-Hard, compared with 46.86 for the CoCA method. Adding either distillation or guided optimization individually to calibration decreases performance on this benchmark. Besides,

  2. (Main) The selection and sensitivity of the loss coefficients are insufficiently analyzed. Although Section 4.1 and Table 9 specify α=0.001\alpha=0.001 and β=0.1\beta=0.1, the manuscript does not explain how these values were selected, including the search ranges, validation protocol, or selection criterion. Given that the method combines multiple objectives with potentially different loss scales and varying activation frequencies, these coefficients may materially affect the optimization balance.

  3. (Minor) Section 3.1 assigns capability states using empirical achievement rates, with only four responses per group by default. Such boundary-based classifications may be sensitive to sampling variability.


[1] Online Rubrics Elicitation from Pairwise Comparisons

[2] Breaking the Exploration Bottleneck: Rubric-Scaffolded Reinforcement Learning for Open-Ended LLM Reasoning

[3] Rubrics as Privileged Information for Open-Ended Generation

Questions for authors

I'm a bit confused about why the author strengthen/relax the rubric criteria. This is different from the hypothesis of Online Rubrics (i.e., static rubrics may be imperfect). For benchmarks such as HealthBench, the rubric specifies the desired properties of a response to a given prompt. A criterion being difficult for the current policy does not, by itself, justify relaxing that requirement; similarly, saturation does not necessarily justify introducing stricter requirements that do not specified in the task.

11 comments
AuthorsOpenReview ↗

Response to W1 (Part 1/2)

W1: Clarification needed on the contribution beyond a straightforward combination of existing techniques.

Thank you for raising this concern, which reflects your familiarity with the relevant work in this field. However, we would like to clarify that CoCA goes beyond a straightforward combination of existing techniques. CoCA's central contribution is a closed-loop learning mechanism in which probing the policy's empirical capability boundary jointly determines how to calibrate training criteria, when to expand guided capabilities, and when to transfer them into query-only behavior. We clarify this mechanism and its empirical support below.

1. Dual-group capability diagnosis supplies the shared control signal for the loop.

A failure under query-only sampling can have different causes: the behavior may be elicitable through guidance, or it may remain unattained even with guidance, as illustrated in Figure 1 in the Introduction. CoCA distinguishes these cases by comparing query-only and rubric-conditioned responses from the same pre-update policy against the same criteria (Section 3.1, Eqs. (3)–(5)). As Reviewer 4xhs also recognizes in Strength 1, this dual-group diagnosis is a useful new element because it separates failures that guidance can resolve from those it cannot resolve in the sampled responses.

This diagnosis controls subsequent learning through three connected decisions:

  • Calibrate the learning signal. Saturated and Too-Hard criteria are selectively tightened or relaxed to recover reward discrimination, while Learnable criteria remain unchanged and identify behaviors available under guidance but absent from query-only samples.
  • Expand or transfer capabilities. If a Guided Frontier satisfies every Learnable criterion and improves the full-rubric reward over the Current Frontier by the required margin, Frontier-OPSD transfers its guidance into the query-only policy. Otherwise, Guided GRPO seeks a stronger guided frontier. Normal GRPO remains active throughout.
  • Refresh the diagnosis after policy updates. Updated policies generate new responses, which change the empirical capability states, subsequent calibration, and the decision between expansion and transfer.

Thus, the contribution lies in using capability probing to coordinate reward adaptation and capability internalization within one feedback loop (Sections 3.1–3.4; Figure 2).

2. Component ablations and full-table evaluation with an independent judge support CoCA's contribution.

The ablation reveals complementary, task-dependent effects rather than uniformly additive gains. Its interpretation should therefore consider both HealthBench-Hard performance and whether guided capabilities are internalized into model parameters and generalize to unseen datasets without rubric guidance.

Component complementarity and cross-dataset transfer. The results from Table 4, p. 7, are:

ConfigurationHealthBench-HardLLMEval-Med
RaR41.8849.93
Calibration only46.3048.73
Calibration + Frontier-OPSD45.0151.12
Calibration + Guided GRPO44.4752.62
Frontier-OPSD + Guided GRPO43.1150.97
Full CoCA46.8652.77

With training on HealthBench, LLMEval-Med tests cross-dataset generalization. Full CoCA improves over calibration alone on both LLMEval-Med (48.73% → 52.77%; +4.04%) and HealthBench-Hard (+0.56%). Full CoCA performs best on both benchmarks among the ablated configurations, supporting the complementary roles of calibration, guided expansion, and capability transfer.

AuthorsOpenReview ↗

Response to W1 (Part 2/2)

W1: Clarification needed on the contribution beyond a straightforward combination of existing techniques.

Comparison with recent methods across all benchmarks under an independent judge. To assess whether CoCA's gains over methods focused on individual components are consistent and sustained, we additionally re-evaluate the full set of Table 1 models and baselines using Qwen3.7-Flash under 3 runs.

Best and second-best results within each backbone are shown in bold and underlined, respectively.

MethodHealthBenchRaR-MedicineLLMEval-MedResearchQARaR-ScienceCPsyCounHeartBenchMean
Qwen3-4B23.35 ± 0.4252.16 ± 1.1446.83 ± 0.3551.41 ± 0.0862.02 ± 0.296.71 ± 0.4641.93 ± 0.5740.63 ± 0.21
LLM-Rank30.65 ± 0.3952.10 ± 0.0950.62 ± 0.5570.01 ± 0.0966.82 ± 0.156.01 ± 0.3045.60 ± 0.7045.97 ± 0.15
RaR37.80 ± 0.4255.77 ± 0.1051.18 ± 0.5072.11 ± 0.0764.90 ± 0.1043.54 ± 0.7250.80 ± 1.2753.73 ± 0.23
RuscaRL38.48±0.29‾\underline{38.48 \pm 0.29}55.74 ± 0.1051.57 ± 0.1277.14±0.07‾\underline{77.14 \pm 0.07}66.94 ± 0.2447.06±0.40‾\underline{47.06 \pm 0.40}50.12 ± 0.8355.29±0.14‾\underline{55.29 \pm 0.14}
Online Rubrics38.38 ± 0.3356.35±0.16‾\underline{56.35 \pm 0.16}52.26 ± 0.9873.35 ± 0.0467.12±0.25‾\underline{67.12 \pm 0.25}44.79 ± 0.1550.86±0.78‾\underline{50.86 \pm 0.78}54.73 ± 0.19
RGSD20.05 ± 0.2650.86 ± 0.0743.78 ± 0.2455.90 ± 0.0960.72 ± 0.149.80 ± 0.2143.37 ± 0.6940.64 ± 0.12
CoCA (Ours)40.71 ± 0.0357.45 ± 0.1152.12±0.39‾\underline{52.12 \pm 0.39}78.41 ± 0.1668.84 ± 0.2449.66 ± 0.2752.45 ± 1.0857.09 ± 0.17
Qwen3-8B29.75 ± 0.2457.91 ± 0.1754.49 ± 0.5557.77 ± 0.0665.46 ± 1.6513.72 ± 0.4545.87 ± 0.1546.42 ± 0.26
LLM-Rank38.47 ± 0.1259.81 ± 0.0557.73 ± 0.1370.64 ± 0.2967.98 ± 0.2817.16 ± 0.4747.53 ± 0.3751.33 ± 0.11
RaR43.71 ± 0.4260.17 ± 0.0362.34±0.02‾\underline{62.34 \pm 0.02}77.40 ± 0.0470.32±0.20‾\underline{70.32 \pm 0.20}52.22 ± 0.5451.71 ± 0.6159.70 ± 0.13
RuscaRL46.95±0.30‾\underline{46.95 \pm 0.30}61.56 ± 0.1461.45 ± 0.7877.57±0.08‾\underline{77.57 \pm 0.08}69.92 ± 0.0555.88±0.51‾\underline{55.88 \pm 0.51}53.03 ± 0.7860.91±0.18‾\underline{60.91 \pm 0.18}
Online Rubrics46.10 ± 0.1162.10 ± 0.1862.17 ± 0.3276.53 ± 0.0769.40 ± 0.1954.45 ± 0.1454.75±0.61‾\underline{54.75 \pm 0.61}60.79 ± 0.11
RGSD25.99 ± 0.3757.20 ± 0.0450.02 ± 0.0761.17 ± 0.1966.39 ± 0.2518.87 ± 0.1349.40 ± 0.2747.01 ± 0.08
CoCA (Ours)48.09 ± 0.0961.93±0.02‾\underline{61.93 \pm 0.02}66.32 ± 0.1978.89 ± 0.1171.98 ± 0.1357.27 ± 0.2955.83 ± 0.2862.90 ± 0.07
Gemma-3-4B-it38.13 ± 0.5246.50 ± 0.0530.68±0.14‾\underline{30.68 \pm 0.14}55.91 ± 0.1442.99 ± 0.169.05 ± 0.2346.32 ± 0.3938.51 ± 0.11
LLM-Rank39.46 ± 0.5345.32 ± 0.0630.16 ± 0.3558.64 ± 0.1243.97 ± 0.1728.58 ± 0.1852.81 ± 0.9242.71 ± 0.16
RaR50.47 ± 0.1549.36±0.04‾\underline{49.36 \pm 0.04}26.99 ± 0.3269.12 ± 0.2244.15±0.14‾\underline{44.15 \pm 0.14}39.21 ± 0.7452.11±0.43‾\underline{52.11 \pm 0.43}47.34 ± 0.14
RuscaRL45.28 ± 0.3247.20 ± 0.0330.28 ± 0.1270.93±0.16‾\underline{70.93 \pm 0.16}37.60 ± 0.2247.88 ± 0.6133.75 ± 0.6344.70 ± 0.14
Online Rubrics50.94±0.36‾\underline{50.94 \pm 0.36}48.51 ± 0.0128.49 ± 0.3370.14 ± 0.3543.38 ± 0.1048.54±0.59‾\underline{48.54 \pm 0.59}50.75 ± 0.9548.68±0.18‾\underline{48.68 \pm 0.18}
RGSD30.96 ± 0.2644.87 ± 0.1028.59 ± 0.5159.00 ± 0.1342.12 ± 0.3015.86 ± 0.4544.46 ± 0.9637.98 ± 0.18
CoCA (Ours)52.07 ± 0.2451.39 ± 0.0231.78 ± 0.1273.94 ± 0.1445.64 ± 0.4152.98 ± 0.3149.24 ± 0.5251.01 ± 0.11

CoCA achieves the highest mean on all three backbones (+1.80%, +1.99%, and +2.33% over the strongest baseline) and ranks first in 18 of 21 backbone–benchmark combinations. Its consistent average-score advantage over Online Rubrics, RuscaRL, and RGSD under an independent judge further supports the effectiveness of its coordinated learning mechanism.

W2: Selection and sensitivity of the loss coefficients.

Thank you for raising this question. We understand and share your emphasis on hyperparameter sensitivity, particularly when multiple objectives have different loss scales and activation frequencies.

The manuscript examines robustness to evaluation judge choice and component complementarity in Section 4.3. Additional experiments cover distillation trajectories and privileged information (Appendix E.2, p. 21) (originally Appendix E.1, p. 20), dynamic gating (Appendix E.3, p. 22) (originally Appendix E.2, p. 20), compatibility with other GRPO variants (Appendix E.5, p. 22) (originally Appendix E.4, p. 21), and rollout budgets and allocations (Appendix E.6, p. 23) (originally Appendix E.5, p. 21).

To directly address coefficient sensitivity, we conduct ablations of the two loss coefficients using Qwen3-8B on HealthBench-Hard and LLMEval-Med. We vary one coefficient at a time while keeping the other at its default value and holding all other training settings fixed. All scores below are on a 0–100 scale. We have also incorporated the results below into Section 4.3 of the main text in the revised manuscript.

1. Frontier-OPSD loss coefficient α\alpha (with β=0.1\beta=0.1 fixed).

α\alphaHealthBench-HardLLMEval-Med
044.4752.62
0.001 (default)46.8652.77
0.00346.9352.32
0.00643.6149.78

Performance is relatively insensitive to moderate distillation weights, but excessive weighting degrades both benchmarks. Relative to α=0\alpha=0, the default α=0.001\alpha=0.001 improves HealthBench-Hard by +2.39%, while LLMEval-Med remains similar (52.62% → 52.77%). Increasing α\alpha to 0.0030.003 yields comparable performance, indicating limited sensitivity within the tested moderate range. At α=0.006\alpha=0.006, however, scores drop by 3.25% and 2.99%, respectively, relative to the default. This suggests that distillation benefits capability transfer at a moderate weight, while excessive emphasis on it can disrupt the balance with the query-only learning objective.

2. Guided GRPO loss coefficient β\beta (with α=0.001\alpha=0.001 fixed).

β\betaHealthBench-HardLLMEval-Med
045.0151.12
0.1 (default)46.8652.77
0.546.9052.32
1.043.5649.63

Guided optimization also benefits from a moderate weight, with limited sensitivity to the tested moderate increase. Compared with β=0\beta=0, the default β=0.1\beta=0.1 improves HealthBench-Hard and LLMEval-Med by +1.85% and +1.65%, respectively. Performance remains comparable at β=0.5\beta=0.5, whereas β=1.0\beta=1.0 reduces the scores by 3.30% and 3.14% relative to the default. This supports using guided optimization to expand rubric-conditioned capabilities while preserving a balanced contribution from query-only optimization.

Together, these ablations support α=0.001\alpha=0.001 and β=0.1\beta=0.1 as balanced defaults. The tested moderate alternatives change either benchmark by at most 0.45%, while larger weights degrade both scores. Thus, CoCA is relatively insensitive to moderate coefficient changes, but the auxiliary objectives should not be weighted excessively.

Reviewer_qMySOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper tries to improve rubric based reinforcement learning by simultaneously improving the rubric and the policy.

Critical strengths

The idea to evolve both policy and rubric looks interesting.

Critical weaknesses

I think this paper is not sound and has critical issues. The equation (3) tries to sample an output from a rubric which after careful reading of the paper couldn't find any place to explain how this is possible. A criterion within a rubric can be used to judge an answer. Passing a whole rubric to generate doesn't mean anything. This is what a critera in a rubric looks like[1]:

Q1– In terms of naturalness and tone of the assistant utterances, to what degree are they likely to be produced by an intelligent human in a conversation? Disregard whether they are grounded in the search results.

  1. Unlikely.
  2. Somewhat unlikely.
  3. Somewhat likely.
  4. Likely.

You can't pass this and do pi(.|x , Q1) , worse considering the whole rubric is passed.

[1] LM-RUBRIC: A Multidimensional, Calibrated Approach to Automated Evaluation of Natural Language Texts, Hashemi et al.

Questions for authors

Could you please explain how do you sample from the model?

Also could you make your figures visible for a printed version?

6 comments

W1 / Q1: Clarification of rubric-conditioned sampling in Eq. (3).

Thank you for raising this concern, and we apologize for any confusion our presentation may have caused.

In Eq. (3), we append the checklist-style rubric text to the query and sample the response token by token from the language model conditioned on this combined input.

Below, we clarify the rubric definition, our sampling procedure, and the supporting related work and manuscript passages.

1. Task formulation: what does “rubric” mean in our setting?

Following HealthBench [1] and Rubrics as Rewards (RaR) [2], a rubric is a query-specific checklist of natural-language criteria describing desirable or undesirable properties of a response, with a weight assigned to each criterion.

For example, HealthBench includes the query:

what daily percentage of carbs is recommended for a healthy adult who isn't diabetic

Its rubric includes the following items:

HealthBench Rubric CriteriaWeight
Specifies a carbohydrate intake range of 45–65% of daily calories, citing the dietary guidelines identified in the criterion.+7
Distinguishes carbohydrate quality, including complex versus simple/refined carbohydrates.+6
Gives a single fixed percentage instead of the specified range.−8

Why does open-ended RL need checklist-style rubrics? RL with verifiable rewards (RLVR) obtains rewards from verifiable checks, such as answer correctness or test outcomes. For open-ended tasks, where comparable deterministic checks are often unavailable, a rubric decomposes response quality into explicit, query-specific requirements. As defined in Section 2, p. 2, lines 103–107, we represent these as Rx=(cj,wj)j=1Jx\mathcal{R}_x=\\{(c_j,w_j)\\}_{j=1}^{J_x}: a fixed LLM judge evaluates whether each criterion is met, and Eq. (1) aggregates the binary verdicts with their signed weights into a scalar training reward. Positive-weight criteria reward desirable behaviors, while negative-weight criteria penalize undesirable ones. This provides an explicit reward definition for open-ended RL, while still relying on the judge's assessments.

2. Concrete use in our CoCA: inputs, sampling, and capability diagnosis.

Input construction. The Normal Group receives the original query or conversation. For the Guided Group, we append the full rubric—including criterion descriptions, signed weights, and available metadata—to the final user message. We provide the complete prompt for Guided sampling in Appendix I, p. 27 of the revised manuscript. The template is:

The following evaluation rubric specifies the requirements for an ideal response.
Use the complete rubric as guidance when responding to the user's request.
Do not mention, quote, enumerate, or discuss the rubric itself.

<rubric>
{complete serialized rubric}
</rubric>

Sampling. After applying the model's chat template and tokenizer, we sample answer tokens from the same pre-update policy. The Guided sampling in Eq. (3) means

ytG∼πθold(⋅∣Format⁡(x,Rx),y<tG),y_t^{\mathrm{G}}\sim\pi_{\theta_{\mathrm{old}}}\bigl(\cdot\mid\operatorname{Format}(x,\mathcal{R}_x),y_{\lt t}^{\mathrm{G}}\bigr),

where Format⁡\operatorname{Format} constructs the input above; Normal sampling uses Format⁡(x)\operatorname{Format}(x) without the rubric.

Capability diagnosis. We judge both groups against the original criteria and compare achievement rates (Eqs. (4)–(5)). For example, a positive criterion is Learnable if none of the Normal responses meets it but at least one Guided response does. This diagnosis informs rubric calibration and policy learning.

3. Related work and where this use is stated in the revised manuscript.

RuscaRL [3] supplies rubric criteria in the prompt to guide generation. RGSD [4] uses rubric text as privileged teacher context for token-level distillation into a student without that context. The revised manuscript retains the following descriptions of these uses and our sampling procedure from the original submission:

  • Section 6, p. 9, lines 484–485 (originally Section 6, p. 9, lines 454–455): “RuscaRL (Zhou et al., 2026) uses rubric criteria to scaffold policy exploration.”
  • Section 2, p. 3, lines 143–148: “On-Policy Self-Distillation (OPSD) converts privileged information ξ\xi (e.g., rubric (Rezaei et al., 2026a)) into token-level supervision on trajectories sampled by the student policy itself”.
  • Section 3.1, p. 3, lines 154–158: “we independently sample KNK_{\mathrm{N}} query-only responses and KGK_{\mathrm{G}} rubric-conditioned responses from the same pre-update policy”. Figure 2(a) labels the inputs as “Normal: [Query]” and “Guided: [Query][Rubric]”.

Q2: Figure readability in the printed version.

Thank you for pointing this out, and we apologize for any inconvenience this may have caused. We have improved the clarity of all figures in the revised manuscript to enhance readability in the printed version.

References

[1] Arora et al. HealthBench: Evaluating Large Language Models Towards Improved Human Health. 2025.

[2] Gunjal et al. Rubrics as Rewards: Reinforcement Learning Beyond Verifiable Domains. ICLR 2026.

[3] Zhou et al. Breaking the Exploration Bottleneck: Rubric-Scaffolded Reinforcement Learning for General LLM Reasoning. ICML 2026.

[4] Rezaei et al. Rubric-Guided Self-Distillation: Post-Training Without Rubric Verifiers. Arxiv 2026.

AuthorsOpenReview ↗

Follow-up on Rebuttal Responses

Dear Reviewer,

We have provided detailed responses to your concerns and would appreciate it if you could take a look when you have a chance.

As the discussion period is coming to an end, we would be grateful for any feedback on whether our responses have addressed your concerns. We would be happy to clarify any remaining questions.

Best regards,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.