- Rating
- 2(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
CoRT uses likelihood differences between criterion-conditioned and criteria-free replay to derive token weights for rubric-based policy optimization. The normalized positive weights modulate the original signed response advantage without requiring a separately trained relevance discriminator.
Critical strengths
The method is simple and preserves the existing response-level reward. Beyond the main comparison, Tables 2–3 provide encouraging results at 14B scale and with DAPO/GSPO. Table 4 supplies fixed-step results, while the appendix includes length-matched context controls, criterion-wise removal, and training-stability diagnostics.
Critical weaknesses
The primary comparative evidence remains insufficiently controlled. Table 1 imports baseline results from RTT while reporting new CoRT runs. Matching the stated protocol does not eliminate implementation or training differences. IFBench, IFEval, and MultiDimIF report best-within-500 validation scores; an independent selection/evaluation split is not established in the description. Fixed-step CoRT results partially address checkpoint sensitivity, but matched baseline checkpoints and training-seed uncertainty remain necessary to establish the main gain convincingly.
Sign-preserving weighting warrants a criterion-level diagnostic. Rubric-sensitive tokens in negative-advantage responses can receive stronger penalties, potentially including tokens implementing satisfied criteria. The Qwen2.5-7B AON IFBench regression of 2.31 points motivates this concern, but does not establish its cause.
The ablations reveal a metric tradeoff. At step 520, removing normalization improves IFBench by 5.39 points but reduces the three-benchmark average from 65.34 to 59.78. The paper acknowledges this behavior and provides stability evidence favoring normalization. The remaining question is why the benchmark-specific tradeoff occurs and whether it persists across seeds.
Novelty lies in the particular replay estimator and its practical tradeoff. RTT and Self-Distilled RLVR already establish related token-credit approaches and are acknowledged by the authors [1–2]. The reported 3.4% overhead is a preliminary short-run measurement rather than a full-training cost estimate.
The replay estimator is simple, and the additional model-scale, optimizer, and fixed-checkpoint results provide useful support. However, Table 1 combines imported baselines with new CoRT runs, and the evaluation description does not establish an independent checkpoint-selection/evaluation split or quantify training-seed uncertainty. These limitations prevent me from confidently attributing the headline improvement to CoRT. The AON regression additionally motivates a criterion-level diagnostic, without establishing a specific failure mechanism.
References
[1] Xu et al. Rubrics to Tokens: Bridging Response-level Rubrics and Token-level Rewards in Instruction Following Tasks. 2026.
[2] Yang et al. Self-Distilled RLVR. 2026.
Questions for authors
Do matched in-house baselines and multiple training seeds reproduce the main gains? Does a shuffled-token-weight training control retain the improvements? What explains the IFBench tradeoff and the behavior of satisfied-criterion tokens under negative advantages?
6 comments
W3&Q3. The IFBench trade-off in the ablations
Table 10 shows that the IFBench gains of the ablated variants come together with a broad degradation elsewhere:
- The gain appears only as other metrics collapse. At step 500,
w/o bothand full CoRT are essentially tied on IFBench (33.11 vs. 32.90) and on the three-benchmark average (64.38 vs. 64.61). The IFBench gap opens at steps 520–530, asw/o bothdrops on MultiDimIF (79.67 → 68.93) and IFEval (80.37 → 76.27). w/o normis unstable. From step 500 to 520 its IFEval falls from 76.19 to 71.46 and its MultiDimIF from 76.56 to 68.96, and its training reward drops to 0.674 (full CoRT: 0.878). Without normalization, the mean token weight drifts to ≈1.04 near the end of training, while length clipping, gradient norm and entropy also rise (Figures 3 and 5).w/o ramptrades IFBench for the other benchmarks. It is higher on IFBench throughout (35.09–36.38 vs. 32.90–33.52), but lower on IFEval (77.49–78.48 vs. 82.59–82.77) and MultiDimIF (72.59–74.21 vs. 78.04–79.72), so its three-benchmark average stays below full CoRT at every step (61.72–62.77 vs. 64.61–65.34). Its training reward is also lower (0.690–0.817 vs. 0.869–0.878), and removing the ramp increases late-training gradient norm and entropy (Figure 3).
We therefore select the full configuration for its average performance and stability, and we will discuss the trade-off as a limitation in the main text.
W4. Novelty relative to RTT and Self-Distilled RLVR, and overhead
Novelty.
- RTT trains a separate token-level relevance discriminator on ~63k LLM-annotated samples before RL. CoRT needs no annotation, no discriminator and no extra training stage.
- Self-Distilled RLVR (RLSD) is the closest work in form; we already cite and discuss it in Appendix B.3 (Relation to Self-OPD). It also keeps the advantage sign from the verifiable reward and uses a log-probability gap from the same model to set token-level magnitudes. The two methods differ in three ways:
- Pulling toward a teacher vs. pushing away from a baseline. RLSD's teacher is conditioned on privileged information that the student does not see (e.g., the ground-truth answer), and its update pulls the student toward this teacher, as in distillation. CoRT uses no privileged trace or target distribution: it keeps the sampled response fixed, removes the criteria from the prompt, and treats the criteria-free likelihood as a baseline. In form, its update pushes the policy away from this baseline, so CoRT is closer to signed counterfactual contrast shaping than to self-distillation. It also needs no reference answer, which open-ended rubric tasks usually lack.
- Sign handling. RLSD uses w = exp(sign(A)·Δ). On a failed response, tokens that the answer-aware teacher favours receive a smaller penalty, which is sensible when the teacher knows the answer. CoRT's contrast says only that a token was driven by the rubric, not that it is correct, so CoRT amplifies such tokens in both directions. In our preliminary experiments, a variant similar in spirit to RLSD's sign-dependent choice, which removed the extra penalty on rubric-dependent tokens in negative-advantage responses, led to training collapse.
- Weight construction and setting. CoRT scores both contexts with the current policy, so the contrast always reflects the model being updated. RLSD's teacher is synchronized with the student only every 10 steps. CoRT also uses a bounded sigmoid, a SmoothStep ramp, and response-level mean normalization, the last of which our ablations show matters for stability (Table 10). RLSD clips the ratio and does not normalize. RLSD is evaluated on multimodal reasoning with binary verifiable rewards, while CoRT targets multi-criterion rubric rewards (CSR/AON) for instruction following.
We will move this comparison from Appendix B.3 into the related-work section. We see the contribution as (i) showing that a privilege-free counterfactual contrast, which removes the rubric instead of adding an answer, is a usable token-credit signal for rubric-based RL, and (ii) a plug-in that also composes with DAPO and GSPO (Table 3).
Overhead. The 3.4% in the paper was measured on a short run. We have now measured the full 500-step training run: the counterfactual replay accounts for 3.0% of training time (rollout, judge, replay and update), i.e., a 3.1% increase over the same pipeline without replay, consistent with the short-run estimate. As a share of wall-clock time, it is 2.6% excluding periodic evaluation and 1.4% end to end. The replay is a single forward pass over the sampled response and requires no token-by-token generation. (Denominators: training-step time for 3.0%; training-step time without replay for 3.1%; wall-clock time excluding periodic evaluation for 2.6%; end-to-end wall-clock time for 1.4%.)
Thank you for the detailed response. The shuffled-weight experiment provides useful evidence that token placement matters, and the full-run timing addresses my concern about the preliminary overhead estimate. Moving to fixed step-500 checkpoints also reduces selection concerns.
However, my primary concern remains unresolved: the headline comparisons still use imported GRPO results rather than matched in-house baselines. The unavailable RTT discriminator explains the difficulty of reproducing RTT, but does not explain the absence of matched GRPO runs in the main configurations. I acknowledge the supporting in-house results at 14B and with DAPO/GSPO, but these do not establish the reproducibility of the headline gains. Furthermore, changing both the seed and reward judge in the additional run does not isolate training variability or support the conclusion that the original run was not a favourable outlier.
The criterion-level concern also remains open. Rubric dependence does not distinguish tokens implementing satisfied criteria from those contributing to violations, and the preliminary CSR collapse experiment does not directly diagnose this issue under AON. The normalization discussion supports the full configuration’s aggregate performance, but does not establish the seed robustness of the IFBench tradeoff.
These additions strengthen the submission, but I remain unconvinced that the main comparative gains are sufficiently controlled and reproducible. I would like clarification of the baseline checkpoint protocol and further matched-baseline evidence before revising my rating.
We thank the reviewer for the detailed and constructive review. We address each point below.
W1&Q1. Imported baselines, checkpoint selection, and training-seed uncertainty
We address the three parts in turn.
Checkpoint selection. Appendix Table 4 reports CoRT at the fixed step-500 checkpoint, with no selection. AdvancedIF in Table 1 was already evaluated only at step 500. Comparing these selection-free CoRT results with the GRPO numbers as reported by RTT, CoRT wins 14 of 16 benchmark-level comparisons (IFEval, IFBench, MultiDimIF and AdvancedIF across 4 model/reward settings, using strict prompt-level accuracy for IFEval and IFBench and the overall score for AdvancedIF; mean gain +3.59). The two losses are both IFBench under AON. We will replace the main table with the fixed step-500 results.
Imported baselines. We could not re-run RTT faithfully. Its token-level relevance discriminator is central to the method, but the discriminator checkpoint is listed as "To Be Added" in the official repository, and the 62,714 token-level annotated training samples (labelled by a strong LLM) are, to our knowledge, not released. We therefore keep RTT as a reported reference. Our conclusions do not rest on the imported GRPO numbers alone. The paper also compares CoRT with baselines trained in our own pipeline, at a larger scale (Qwen3-14B with GRPO under CSR and AON, Table 2) and with other policy optimizers (DAPO and GSPO, Table 3). CoRT improves 17 of these 20 metrics; the three exceptions are small (at most 0.46 points: IFEval for Qwen3-14B under AON, and MultiDimIF with GSPO). Together with the selection-free 14/16 result, this shows that the gain holds across model scales, optimizers and reward types.
Training-seed uncertainty. Multiple seeds per configuration are beyond our compute budget. We did train CoRT again on Qwen3-4B (CSR) with a different seed; this run had to use a newer reward judge (DeepSeek-V4.1-Flash), since the DeepSeek-V3 judge used in the paper is no longer available to us. It reaches 87.65 / 91.73 on IFEval, 37.55 / 40.30 on IFBench and 83.75 on MultiDimIF, above the step-500 results of the run reported in the paper on every metric (86.06 / 90.48, 34.49 / 36.24, 80.48). The reported run is therefore not a favourable outlier. AdvancedIF could not be evaluated for the new run because its judge (o3-mini) has been retired.
Q2. Does a shuffled-token-weight training control retain the improvements?
No. We trained Qwen3-4B under CSR with CoRT's token weights randomly permuted within each response. This keeps each response's weight distribution (range, mean and saturation) unchanged and only changes which token receives which weight. Both runs below share the same data, hyperparameters, seed and reward judge (DeepSeek-V4.1-Flash), and are evaluated at the final checkpoint (step 500):
All metrics drop after shuffling, most notably IFBench (−7.55 / −8.06) and MultiDimIF (−21.35), showing that the token placement given by the contrast is useful for training.
W2&Q3. Behaviour of satisfied-criterion tokens under negative advantages
We agree that under AON, CoRT increases the penalty on rubric-dependent tokens in a failed response, and that IFBench under AON is our weakest case. We address this in three points.
(i) Rubric dependence does not determine the direction of correction. The contrast shows that a token was driven by the rubric, not that it is correct. In a failed response, such a token may implement a satisfied criterion or cause a violated one, which call for opposite corrections. The direction must therefore come from the reward: CoRT keeps the sign of the advantage and only rescales its magnitude within (0.75, 1.25).
(ii) Under AON, the reward cannot supply this direction per criterion. It reduces all criteria to one binary outcome, so a failed response does not reveal which criteria it satisfied. CoRT is better suited to CSR, which preserves per-criterion structure; we include AON to follow the RTT protocol. Even under AON, CoRT improves IFEval, MultiDimIF and AdvancedIF for both models; IFBench is the exception, which we will discuss as a limitation.
(iii) Dropping the extra penalty causes collapse. In preliminary experiments under CSR, we removed CoRT's extra weight on rubric-dependent tokens in negative-advantage responses, so that these tokens received only the standard GRPO penalty, while positive-advantage responses were still up-weighted. This variant led to training collapse.