- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper targets reward saturation in rubric-based RL, where fixed criteria become uniformly met or unmet during training. CoCA contrasts query-only and rubric-conditioned rollouts to label each criterion's state, then uses these states to rewrite criterion difficulty per step, gate on-policy self-distillation from rubric-conditioned responses, and otherwise run GRPO on rubric-conditioned rollouts, on top of standard GRPO. Across three backbones and seven medical, science, and psychology benchmarks, CoCA achieves the best mean score and improves over standard rubric-based GRPO.
Critical strengths
-
The dual-group diagnosis is a simple and useful idea. Contrasting query-only and rubric-conditioned rollouts separates failures that guidance can resolve from those it cannot. A non-trivial share of criteria unmet under query-only sampling become satisfiable once the rubric is provided (Fig. 1b). This matters because it is the paper's one clearly new element, and it could be reused beyond this method.
-
Gains are consistent across settings. CoCA has the best mean on all three backbones (Table 1). The gains persist under three evaluation judges not used in training (Table 3) and at 14B (Table 12). This matters because it makes judge overfitting unlikely as the sole explanation.
Critical weaknesses
-
The core contribution is unclear, and "co-evolution" is overstated. Each component has a prior counterpart: guided rollouts (RuscaRL), rubric rewriting (Online Rubrics), and rubric-privileged self-distillation (RGSD, CriPO). Moreover, the rubric is not accumulated: each step recalibrates the original rubric and discards the rewrite (App. C), so the method performs per-step, policy-conditioned reward reshaping rather than persistent co-evolution. No single component carries the gain either. Against a budget-matched RaR (Table 14), calibration alone gives a marginal gain on HealthBench-Hard and a drop on LLMEval-Med (Table 4), and the closest methods are not compared. Since the title and claimed contributions rest on co-evolution, the lack of an identifiable core insight makes the paper read as an engineering combination of existing techniques.
-
Comparisons with baselines are not effort-matched. Table 1 gives CoCA twice the rollouts of RaR/RuscaRL. At a matched budget (Table 14), the gain over RaR roughly halves, and the other settings are never matched. CoCA's rollout split is tuned (Table 14), but baseline tuning is not described, and RGSD uses a single rollout per query and scores below the untrained base model on most backbones. The remaining small gain is therefore within the range that unmatched budgets, tuning effort, and seed variance could plausibly explain.
-
Infrastructure cost and scalability are not analyzed. Each step doubles rollouts, multiplies judge calls (judge, rewrite, and rejudge run serially on the critical path), and adds a long-context teacher forward for OPSD. The gate must also wait until all rollouts of a query are generated, judged, rewritten, and rejudged, a per-query barrier that conflicts with asynchronous RL pipelines and amplifies long-tail latency. Finally, all runs pair small policies with a much stronger judge/rewriter, and tightening saturated criteria may fail once the policy matches or exceeds the rewriter, as it would at frontier scale. The practical value of an RL training method depends on gain per unit of compute and on fitting production pipelines, and the paper evaluates neither.
-
Rubric rewriting is unfiltered and can shift the objective in safety-relevant domains. The independent audit (App. F.3) shows that a small but nonzero share of rewrites drift from the original intent, yet no acceptance filter is applied during training. In medical and psychological tasks, silently weakened criteria can reward unsafe responses, and it is unclear whether safety-critical criteria are protected.
-
Gains over the strongest baseline concentrate on the authors' own LLM-built benchmark (Table 1). Compared per benchmark with the best of RaR, RuscaRL, and Online Rubrics, a large part of CoCA's margin on Qwen3-4B and Gemma-3-4B-it comes from CPsyCoun. Excluding it, CoCA's lead over the strongest baseline is marginal on all three backbones, and on Qwen3-4B it trails a baseline on RaR-Medicine and HeartBench. The seven-benchmark mean thus overstates how broadly CoCA beats strong baselines; on established benchmarks, the margin is small.
Questions for authors
- Since rewrites are discarded every step (App. C), in what sense does the rubric co-evolve? Have you tried carrying calibrated criteria over to later steps?
- Can Table 1, or at least Table 4, be reported against 8-rollout baselines? How were baseline hyperparameters chosen, why does RGSD use one rollout per query, and which checkpoint is reported for each method?
- How many independent runs underlie the main tables, and what is the uncertainty of the Table 14 comparison?
- How does CoCA compare with RaR at matched wall-clock or judge-token budget, beyond the single point in Fig. 4(b)? Please report judge calls/tokens per step and GPU-hours for the 14B run.
13 comments
Response to W1 & Q1 (Part 2/2)
W1 & Q1: Clarification of the core contribution, co-evolution without accumulating rubric edits, and the effect of retaining calibrated criteria.
4. Component ablations and rollout-budget controls support the coordinated mechanism.
In Table 4, full CoCA improves over calibration alone from 48.73% to 52.77% on LLMEval-Med, alongside the smaller 46.30% to 46.86% gain on HealthBench-Hard. The cross-dataset improvement supports coordinating reward calibration, guided learning, and capability transfer.
For budget accounting, CoCA's rollout budget for query-only GRPO is matched to that of RaR and Online Rubrics in the default setup. Its additional rubric-conditioned responses support capability diagnosis, calibration, and frontier selection. The two groups have separately normalized advantages; guided responses are never pooled into the query-only GRPO group. Table 16, Appendix E.6, p. 23 (originally Table 14, Appendix E.5, p. 21) additionally controls for these extra samples through matched 8- and 12-total-rollout comparisons.
We further re-evaluated the budget controls with Qwen3.7-Flash. Below is the 8-rollout subset for Qwen3-8B; the complete results, including 12-rollout configurations, appear in our response to your third question (Q3). At the same total rollout budget, CoCA exceeds the strongest baseline by +1.05% on HealthBench-Hard and +4.15% on LLMEval-Med, while RaR leads on RaR-Medicine. These controls support gains beyond simply increasing sampling.
| Method | Primary rollouts | Additional rollouts | Total rollouts | HealthBench-Hard | RaR-Medicine | LLMEval-Med |
|---|---|---|---|---|---|---|
| RaR | 8 | — | 8 | 44.34 ± 0.31 | 64.33 ± 0.01 | 57.90 ± 1.23 |
| RuscaRL | 8 | — | 8 | 47.04 ± 0.13 | 63.80 ± 0.04 | 53.67 ± 0.21 |
| Online Rubrics | 4 | 4 | 8 | 46.10 ± 0.11 | 62.10 ± 0.18 | 62.17 ± 0.32 |
| CoCA | 4 | 4 | 8 | 48.09 ± 0.09 | 61.93 ± 0.02 | 66.32 ± 0.19 |
Response to W2 & Q2
W2 & Q2: Fairness of rollout budgets, baseline hyperparameters, RGSD sampling, and checkpoint selection.
Thank you for these questions. We clarify the budget controls and experimental settings, including the scope of the evidence on robustness, below.
1. Query-only GRPO budgets are matched, and Table 16 additionally controls total rollouts.
In the default setup, CoCA's rollout budget for query-only GRPO is matched to that of RaR and Online Rubrics. The additional rubric-conditioned responses support capability diagnosis and frontier learning, with advantages normalized separately from the query-only group. These additional samples contribute to the total sampling cost.
Table 16, Appendix E.6, p. 23, reports matched-total-rollout comparisons for the Qwen3-8B medical-domain results in Table 1. The budgets include both primary and additional rollouts, counting Online Rubrics' control responses and CoCA's Guided responses. We have also incorporated these results into Section 4.3 (Figure 3) of the revised manuscript.
| Method | Primary + Additional | Total rollouts | HealthBench-Hard | RaR-Medicine | LLMEval-Med | Mean |
|---|---|---|---|---|---|---|
| RaR | 8 + 0 | 8 | 45.51 | 64.92 | 50.37 | 53.60 |
| RuscaRL | 8 + 0 | 8 | 46.23 | 64.23 | 45.43 | 51.96 |
| Online Rubrics | 4 + 4 | 8 | 45.32 | 64.59 | 49.48 | 53.13 |
| CoCA | 4 + 4 | 8 | 46.86 | 65.37 | 52.77 | 55.00 |
| RaR | 12 + 0 | 12 | 46.91 | 65.75 | 50.82 | 54.49 |
| RuscaRL | 12 + 0 | 12 | 45.80 | 64.97 | 47.98 | 52.92 |
| Online Rubrics | 6 + 6 | 12 | 45.77 | 65.07 | 49.78 | 53.54 |
| CoCA | 6 + 6 | 12 | 48.66 | 65.34 | 53.07 | 55.69 |
| CoCA | 8 + 4 | 12 | 48.90 | 66.92 | 54.27 | 56.70 |
Increasing RaR's sampling budget reduces the margin, but CoCA retains the highest mean at both matched budgets: +1.40% at 8 rollouts and +1.20% / +2.21% at 12 rollouts. Even the default 8-rollout CoCA exceeds all 12-rollout baselines in mean score. We have made these controls and their scope more prominent in the revised manuscript.
2. Baseline settings follow the original papers; Table 16 examines robustness to rollout allocation.
We use the best-performing baseline settings reported in their original papers. These configurations are documented in Appendix D.2, p. 21 (originally Appendix D.2, p. 20) and Tables 9–10, p. 20 (originally Tables 8–9, p. 19): Table 9 gives the general training settings, and Table 10 specifies method-specific settings for RuscaRL, Online Rubrics, and RGSD. We will state their provenance more explicitly.
CoCA uses an equal split between Normal and Guided responses by default. Table 16 reports both equal and unequal allocations at the larger budget, and both outperform the strongest matched-budget baseline in mean score. The advantage is thus present in the default allocation and persists across the tested alternatives. We have also conducted multiple rounds of re-evaluation of the Table 16 configurations using Qwen3.7-Flash, an independent judge not used during training; detailed results are provided below in our response to your third question (Q3). We will distinguish these sampling-ablation variants from the default settings more explicitly.
3. RGSD's single rollout follows its pure-distillation training setup.
RGSD [1] trains the query-only student with token-level supervision from a rubric-conditioned teacher, without reward-based RL updates. Unlike GRPO, it does not require multiple responses to compute group-relative advantages. One on-policy rollout per query follows the original paper's training setup, providing a student trajectory for dense token-level distillation.
4. Checkpoints are selected consistently on held-out validation data.
For every method, we select the checkpoint with the best performance on an independent validation split of the corresponding domain's training dataset. This selection rule is applied consistently across methods; test-benchmark scores are not used to select checkpoints.
Response to W1 & Q1 (Part 1/2)
Thank you for raising these related concerns, we apologize for any misunderstanding caused by unclear wording in the original submission. By “co-evolution,” we mean reciprocal adaptation between the policy and its training-time rubric through capability feedback. CoCA's central contribution is using dual-group diagnosis to jointly determine how to calibrate training criteria, when to expand guided capabilities, and when to transfer them into query-only behavior.
1. Dual-group diagnosis coordinates the feedback loop, which persists through policy updates.
A criterion unmet under query-only sampling may become achievable with guidance or remain unmet in both groups (Figure 1b). CoCA distinguishes these cases by comparing responses from the same pre-update policy against the same criteria. This distinction provides a shared control signal for reward adaptation and capability learning.
CoCA uses this diagnosis to construct a dynamic training curriculum (Sections 3.1–3.4): relaxing Too-Hard criteria recognizes intermediate progress, while strengthening Saturated criteria distinguishes more complete fulfillment. Learnable criteria identify capabilities elicitable through guidance. When a Guided Frontier satisfies all Learnable criteria and clears the required reward-improvement margin, Frontier-OPSD transfers its guidance into the query-only policy; otherwise, Guided GRPO develops stronger guided capabilities. The rubric adapts to the policy's current capabilities, and learning under that rubric changes what supervision the policy needs next. This reciprocal dependence persists through policy updates even when rubric edits are discarded.
2. The original rubric provides a stable reference for measuring capability changes.
Each query revisit starts from the original rubric. For HealthBench, this expert-authored gold rubric anchors capability diagnosis: the criteria stay fixed, while achievement rates and capability states evolve with the policy. Calibration adapts training targets to these changing states, separating measurement against a fixed reference from adaptive supervision and limiting the accumulation of drift through successive rewrites.
Figure 4, p. 8 (originally Figure 3, p. 8) shows larger increases in Saturated criteria and larger reductions in Learnable, Too-Hard, and Regressive criteria with CoCA than with RaR across all three backbones. To examine whether these shifts reflect internalization, Table 6, p. 9 (originally Table 5, p. 8) compares base and trained models using the same fixed HealthBench-Hard rubrics. Its internalization transitions are reproduced below (%):
CoCA exceeds RaR on all four transitions across all three backbones, e.g., Learnable → Saturated on Qwen3-8B (6.96% → 14.02%; +7.06%). These results support the transfer of guided capabilities into query-only behavior. We will clarify that co-evolution refers to this policy–rubric feedback, with rubric edits regenerated at each revisit.
3. Comparison with cross-epoch rubric retention.
Following your suggestion, we have implemented CoCA*, which carries each query's calibrated rubric from epoch e into epoch e+1 for guided sampling, capability diagnosis, and further calibration. The first epoch uses the original rubric, and evaluation always uses the original benchmark criteria.
The comparison uses Qwen3-4B and Qwen3-8B in the medical domain, holding the four-epoch schedule, rollout budget, and other hyperparameters fixed between CoCA and CoCA*. Retaining rubric edits yields slightly lower scores on all three benchmarks for both backbones (−0.55% to −2.55%), supporting rebuilding the curriculum from a stable reference and the policy's current capabilities.