acceptodds
Under review as a conference paper at ICLR 2027

CoRT: Counterfactual Replay for Token-Level Rubric-Guided Policy Optimization

10 views

Abstract

Rubric-based reinforcement learning enriches language model training by evaluating model outputs against explicit criteria. Yet in GRPO-style pipelines, these structured judgments are reduced to a scalar response-level reward and converted into a response-level advantage, which is broadcast uniformly to all generated tokens. This leaves no explicit mechanism for allocating credit within a response, even when different criteria are grounded in different spans, formatting decisions, or semantic choices. We propose CoRT, a token-level credit weighting method for rubric-conditioned GRPO. Instead of training an auxiliary token scoring model, CoRT uses counterfactual replay to rescore the same sampled response under the original rubric-conditioned prompt and a matched criteria-free prompt. The resulting tokenwise log-likelihood contrasts serve as a proxy for dependence on the rubric context. CoRT maps these contrasts to bounded, response-normalized weights and uses them to redistribute the signed GRPO advantage across tokens, without introducing an auxiliary scorer or changing the response-level reward. Experiments across instruction-tuned models and reward granularities show that CoRT improves over matched response-level GRPO in the vast majority of comparisons, with an average gain of 4.4 percentage points. The method remains competitive with learned token-level credit baselines while avoiding a separate relevance-learning stage. These results suggest that policy-internal counterfactual likelihood contrasts provide an effective training signal for within-response credit allocation while retaining the simplicity and stability of GRPO.

Primary area:
foundation or frontier models, including LLMs
Keywords:
large language models, reinforcement learning, token-level credit assignment

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_UMW5OpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

CoRT uses likelihood differences between criterion-conditioned and criteria-free replay to derive token weights for rubric-based policy optimization. The normalized positive weights modulate the original signed response advantage without requiring a separately trained relevance discriminator.

Critical strengths

The method is simple and preserves the existing response-level reward. Beyond the main comparison, Tables 2–3 provide encouraging results at 14B scale and with DAPO/GSPO. Table 4 supplies fixed-step results, while the appendix includes length-matched context controls, criterion-wise removal, and training-stability diagnostics.

Critical weaknesses

The primary comparative evidence remains insufficiently controlled. Table 1 imports baseline results from RTT while reporting new CoRT runs. Matching the stated protocol does not eliminate implementation or training differences. IFBench, IFEval, and MultiDimIF report best-within-500 validation scores; an independent selection/evaluation split is not established in the description. Fixed-step CoRT results partially address checkpoint sensitivity, but matched baseline checkpoints and training-seed uncertainty remain necessary to establish the main gain convincingly.

Sign-preserving weighting warrants a criterion-level diagnostic. Rubric-sensitive tokens in negative-advantage responses can receive stronger penalties, potentially including tokens implementing satisfied criteria. The Qwen2.5-7B AON IFBench regression of 2.31 points motivates this concern, but does not establish its cause.

The ablations reveal a metric tradeoff. At step 520, removing normalization improves IFBench by 5.39 points but reduces the three-benchmark average from 65.34 to 59.78. The paper acknowledges this behavior and provides stability evidence favoring normalization. The remaining question is why the benchmark-specific tradeoff occurs and whether it persists across seeds.

Novelty lies in the particular replay estimator and its practical tradeoff. RTT and Self-Distilled RLVR already establish related token-credit approaches and are acknowledged by the authors [1–2]. The reported 3.4% overhead is a preliminary short-run measurement rather than a full-training cost estimate.

The replay estimator is simple, and the additional model-scale, optimizer, and fixed-checkpoint results provide useful support. However, Table 1 combines imported baselines with new CoRT runs, and the evaluation description does not establish an independent checkpoint-selection/evaluation split or quantify training-seed uncertainty. These limitations prevent me from confidently attributing the headline improvement to CoRT. The AON regression additionally motivates a criterion-level diagnostic, without establishing a specific failure mechanism.

References

[1] Xu et al. Rubrics to Tokens: Bridging Response-level Rubrics and Token-level Rewards in Instruction Following Tasks. 2026.

[2] Yang et al. Self-Distilled RLVR. 2026.

Questions for authors

Do matched in-house baselines and multiple training seeds reproduce the main gains? Does a shuffled-token-weight training control retain the improvements? What explains the IFBench tradeoff and the behavior of satisfied-criterion tokens under negative advantages?

6 comments

We thank the reviewer for the detailed and constructive review. We address each point below.

W1&Q1. Imported baselines, checkpoint selection, and training-seed uncertainty

We address the three parts in turn.

Checkpoint selection. Appendix Table 4 reports CoRT at the fixed step-500 checkpoint, with no selection. AdvancedIF in Table 1 was already evaluated only at step 500. Comparing these selection-free CoRT results with the GRPO numbers as reported by RTT, CoRT wins 14 of 16 benchmark-level comparisons (IFEval, IFBench, MultiDimIF and AdvancedIF across 4 model/reward settings, using strict prompt-level accuracy for IFEval and IFBench and the overall score for AdvancedIF; mean gain +3.59). The two losses are both IFBench under AON. We will replace the main table with the fixed step-500 results.

Imported baselines. We could not re-run RTT faithfully. Its token-level relevance discriminator is central to the method, but the discriminator checkpoint is listed as "To Be Added" in the official repository, and the 62,714 token-level annotated training samples (labelled by a strong LLM) are, to our knowledge, not released. We therefore keep RTT as a reported reference. Our conclusions do not rest on the imported GRPO numbers alone. The paper also compares CoRT with baselines trained in our own pipeline, at a larger scale (Qwen3-14B with GRPO under CSR and AON, Table 2) and with other policy optimizers (DAPO and GSPO, Table 3). CoRT improves 17 of these 20 metrics; the three exceptions are small (at most 0.46 points: IFEval for Qwen3-14B under AON, and MultiDimIF with GSPO). Together with the selection-free 14/16 result, this shows that the gain holds across model scales, optimizers and reward types.

Training-seed uncertainty. Multiple seeds per configuration are beyond our compute budget. We did train CoRT again on Qwen3-4B (CSR) with a different seed; this run had to use a newer reward judge (DeepSeek-V4.1-Flash), since the DeepSeek-V3 judge used in the paper is no longer available to us. It reaches 87.65 / 91.73 on IFEval, 37.55 / 40.30 on IFBench and 83.75 on MultiDimIF, above the step-500 results of the run reported in the paper on every metric (86.06 / 90.48, 34.49 / 36.24, 80.48). The reported run is therefore not a favourable outlier. AdvancedIF could not be evaluated for the new run because its judge (o3-mini) has been retired.

Q2. Does a shuffled-token-weight training control retain the improvements?

No. We trained Qwen3-4B under CSR with CoRT's token weights randomly permuted within each response. This keeps each response's weight distribution (range, mean and saturation) unchanged and only changes which token receives which weight. Both runs below share the same data, hyperparameters, seed and reward judge (DeepSeek-V4.1-Flash), and are evaluated at the final checkpoint (step 500):

MetricShuffled weightsCoRT
IFEval prompt / inst.84.10 / 89.1187.65 / 91.73
IFBench prompt / inst.30.00 / 32.2437.55 / 40.30
MultiDimIF62.4083.75

All metrics drop after shuffling, most notably IFBench (−7.55 / −8.06) and MultiDimIF (−21.35), showing that the token placement given by the contrast is useful for training.

W2&Q3. Behaviour of satisfied-criterion tokens under negative advantages

We agree that under AON, CoRT increases the penalty on rubric-dependent tokens in a failed response, and that IFBench under AON is our weakest case. We address this in three points.

(i) Rubric dependence does not determine the direction of correction. The contrast shows that a token was driven by the rubric, not that it is correct. In a failed response, such a token may implement a satisfied criterion or cause a violated one, which call for opposite corrections. The direction must therefore come from the reward: CoRT keeps the sign of the advantage and only rescales its magnitude within (0.75, 1.25).

(ii) Under AON, the reward cannot supply this direction per criterion. It reduces all criteria to one binary outcome, so a failed response does not reveal which criteria it satisfied. CoRT is better suited to CSR, which preserves per-criterion structure; we include AON to follow the RTT protocol. Even under AON, CoRT improves IFEval, MultiDimIF and AdvancedIF for both models; IFBench is the exception, which we will discuss as a limitation.

(iii) Dropping the extra penalty causes collapse. In preliminary experiments under CSR, we removed CoRT's extra weight on rubric-dependent tokens in negative-advantage responses, so that these tokens received only the standard GRPO penalty, while positive-advantage responses were still up-weighted. This variant led to training collapse.

W3&Q3. The IFBench trade-off in the ablations

Table 10 shows that the IFBench gains of the ablated variants come together with a broad degradation elsewhere:

  • The gain appears only as other metrics collapse. At step 500, w/o both and full CoRT are essentially tied on IFBench (33.11 vs. 32.90) and on the three-benchmark average (64.38 vs. 64.61). The IFBench gap opens at steps 520–530, as w/o both drops on MultiDimIF (79.67 → 68.93) and IFEval (80.37 → 76.27).
  • w/o norm is unstable. From step 500 to 520 its IFEval falls from 76.19 to 71.46 and its MultiDimIF from 76.56 to 68.96, and its training reward drops to 0.674 (full CoRT: 0.878). Without normalization, the mean token weight drifts to ≈1.04 near the end of training, while length clipping, gradient norm and entropy also rise (Figures 3 and 5).
  • w/o ramp trades IFBench for the other benchmarks. It is higher on IFBench throughout (35.09–36.38 vs. 32.90–33.52), but lower on IFEval (77.49–78.48 vs. 82.59–82.77) and MultiDimIF (72.59–74.21 vs. 78.04–79.72), so its three-benchmark average stays below full CoRT at every step (61.72–62.77 vs. 64.61–65.34). Its training reward is also lower (0.690–0.817 vs. 0.869–0.878), and removing the ramp increases late-training gradient norm and entropy (Figure 3).

We therefore select the full configuration for its average performance and stability, and we will discuss the trade-off as a limitation in the main text.

W4. Novelty relative to RTT and Self-Distilled RLVR, and overhead

Novelty.

  • RTT trains a separate token-level relevance discriminator on ~63k LLM-annotated samples before RL. CoRT needs no annotation, no discriminator and no extra training stage.
  • Self-Distilled RLVR (RLSD) is the closest work in form; we already cite and discuss it in Appendix B.3 (Relation to Self-OPD). It also keeps the advantage sign from the verifiable reward and uses a log-probability gap from the same model to set token-level magnitudes. The two methods differ in three ways:
    1. Pulling toward a teacher vs. pushing away from a baseline. RLSD's teacher is conditioned on privileged information that the student does not see (e.g., the ground-truth answer), and its update pulls the student toward this teacher, as in distillation. CoRT uses no privileged trace or target distribution: it keeps the sampled response fixed, removes the criteria from the prompt, and treats the criteria-free likelihood as a baseline. In form, its update pushes the policy away from this baseline, so CoRT is closer to signed counterfactual contrast shaping than to self-distillation. It also needs no reference answer, which open-ended rubric tasks usually lack.
    2. Sign handling. RLSD uses w = exp(sign(A)·Δ). On a failed response, tokens that the answer-aware teacher favours receive a smaller penalty, which is sensible when the teacher knows the answer. CoRT's contrast says only that a token was driven by the rubric, not that it is correct, so CoRT amplifies such tokens in both directions. In our preliminary experiments, a variant similar in spirit to RLSD's sign-dependent choice, which removed the extra penalty on rubric-dependent tokens in negative-advantage responses, led to training collapse.
    3. Weight construction and setting. CoRT scores both contexts with the current policy, so the contrast always reflects the model being updated. RLSD's teacher is synchronized with the student only every 10 steps. CoRT also uses a bounded sigmoid, a SmoothStep ramp, and response-level mean normalization, the last of which our ablations show matters for stability (Table 10). RLSD clips the ratio and does not normalize. RLSD is evaluated on multimodal reasoning with binary verifiable rewards, while CoRT targets multi-criterion rubric rewards (CSR/AON) for instruction following.

We will move this comparison from Appendix B.3 into the related-work section. We see the contribution as (i) showing that a privilege-free counterfactual contrast, which removes the rubric instead of adding an answer, is a usable token-credit signal for rubric-based RL, and (ii) a plug-in that also composes with DAPO and GSPO (Table 3).

Overhead. The 3.4% in the paper was measured on a short run. We have now measured the full 500-step training run: the counterfactual replay accounts for 3.0% of training time (rollout, judge, replay and update), i.e., a 3.1% increase over the same pipeline without replay, consistent with the short-run estimate. As a share of wall-clock time, it is 2.6% excluding periodic evaluation and 1.4% end to end. The replay is a single forward pass over the sampled response and requires no token-by-token generation. (Denominators: training-step time for 3.0%; training-step time without replay for 3.1%; wall-clock time excluding periodic evaluation for 2.6%; end-to-end wall-clock time for 1.4%.)

Thank you for the detailed response. The shuffled-weight experiment provides useful evidence that token placement matters, and the full-run timing addresses my concern about the preliminary overhead estimate. Moving to fixed step-500 checkpoints also reduces selection concerns.

However, my primary concern remains unresolved: the headline comparisons still use imported GRPO results rather than matched in-house baselines. The unavailable RTT discriminator explains the difficulty of reproducing RTT, but does not explain the absence of matched GRPO runs in the main configurations. I acknowledge the supporting in-house results at 14B and with DAPO/GSPO, but these do not establish the reproducibility of the headline gains. Furthermore, changing both the seed and reward judge in the additional run does not isolate training variability or support the conclusion that the original run was not a favourable outlier.

The criterion-level concern also remains open. Rubric dependence does not distinguish tokens implementing satisfied criteria from those contributing to violations, and the preliminary CSR collapse experiment does not directly diagnose this issue under AON. The normalization discussion supports the full configuration’s aggregate performance, but does not establish the seed robustness of the IFBench tradeoff.

These additions strengthen the submission, but I remain unconvinced that the main comparative gains are sufficiently controlled and reproducible. I would like clarification of the baseline checkpoint protocol and further matched-baseline evidence before revising my rating.

Reviewer_iSoUOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes CoRT, a token-level credit weighting method for rubric-based RL. It compares the token-wise log-likelihoods of a response under a full prompt and a criteria-free prompt to determine the importance of each token. Experiments show that CoRT improves training effectiveness.

Critical strengths
  1. Important Research Problem: Token-level credit assignment in rubric-based RL is an important problem, as broadcasting a single response-level advantage uniformly across all tokens ignores fine-grained rubric supervision.

  2. Clear and Efficient Design: CoRT is simple and efficient. It derives token weights directly from policy-internal likelihood contrasts via a single counterfactual forward pass, avoiding the extra training stage required by auxiliary token-scoring models.

Critical weaknesses
  1. Questionable Effectiveness under AON Rewards: Under the AON reward setting, when a response satisfies most rubrics but violates a few, it receives a negative advantage. Because CoRT scales AiA_i by rubric-dependence weights (A^i,t=sg(wi,t)Ai\hat{A}_{i,t} = \text{sg}(w_{i,t})A_i), it imposes heavier penalties on the tokens that correctly fulfilled the satisfied rubrics. This likely explains why CoRT underperforms GRPO on IFBench under the AON setting (e.g., Qwen2.5-7B in Table 1: 29.66/31.22 vs. 31.97/33.13).

  2. Unconvincing Ablation Results: In Table 10, almost all ablated variants (w/o norm, w/o ramp, and w/o both) outperform the full CoRT model on IFBench.

Questions for authors
  1. Under the AON setting, how can CoRT avoid imposing heavier penalties on tokens that correctly satisfy a subset of rubrics when the overall response fails?
1 comment

We thank the reviewer for the insightful questions. We address each point below.

W1&Q1. Under AON, CoRT may penalize tokens that correctly fulfill the satisfied rubrics

We agree that under AON, CoRT increases the penalty on rubric-dependent tokens in a failed response, and that IFBench under AON is our weakest case. We address this in three points.

(i) Rubric dependence does not determine the direction of correction. The contrast shows that a token was driven by the rubric, not that it is correct. In a failed response, such a token may implement a satisfied criterion or cause a violated one, which call for opposite corrections. The direction must therefore come from the reward: CoRT keeps the sign of the advantage and only rescales its magnitude within (0.75, 1.25).

(ii) Under AON, the reward cannot supply this direction per criterion. It reduces all criteria to one binary outcome, so a failed response does not reveal which criteria it satisfied. CoRT is better suited to CSR, which preserves per-criterion structure; we include AON to follow the RTT protocol. Even under AON, CoRT improves IFEval, MultiDimIF and AdvancedIF for both models; IFBench is the exception, which we will discuss as a limitation.

(iii) Dropping the extra penalty causes collapse. In preliminary experiments under CSR, we removed CoRT's extra weight on rubric-dependent tokens in negative-advantage responses, so that these tokens received only the standard GRPO penalty, while positive-advantage responses were still up-weighted. This variant led to training collapse.

W2. In Table 10, almost all ablated variants outperform full CoRT on IFBench

Table 10 shows that the IFBench gains of the ablated variants come together with a broad degradation elsewhere, so we do not read them as better configurations:

  • The gain appears only as other metrics collapse. At step 500, w/o both and full CoRT are essentially tied on IFBench (33.11 vs. 32.90) and on the three-benchmark average (64.38 vs. 64.61). The IFBench gap opens at steps 520–530, as w/o both drops on MultiDimIF (79.67 → 68.93) and IFEval (80.37 → 76.27).
  • w/o norm is unstable. From step 500 to 520 its IFEval falls from 76.19 to 71.46 and its MultiDimIF from 76.56 to 68.96, and its training reward drops to 0.674 (full CoRT: 0.878). Without normalization, the mean token weight drifts to ≈1.04 near the end of training, while length clipping, gradient norm and entropy also rise (Figures 3 and 5).
  • w/o ramp trades IFBench for the other benchmarks. It is higher on IFBench throughout (35.09–36.38 vs. 32.90–33.52), but lower on IFEval (77.49–78.48 vs. 82.59–82.77) and MultiDimIF (72.59–74.21 vs. 78.04–79.72), so its three-benchmark average stays below full CoRT at every step (61.72–62.77 vs. 64.61–65.34). Its training reward is also lower (0.690–0.817 vs. 0.869–0.878), and removing the ramp increases late-training gradient norm and entropy (Figure 3).

We select the full configuration for its average performance and its stability, and we will discuss this trade-off as a limitation in the main text.

Reviewer_AcMtOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes CoRT, a token-level credit weighting method for rubric-conditioned GRPO. The motivation is that GRPO turns structured rubric judgments into one scalar advantage and gives the same advantage to every token. CoRT rescores each sampled response a second time under a prompt with the rubric criteria removed, and uses the per-token log-probability difference between the two prompts as a proxy for how much a token depends on the rubric. These differences go through a bounded sigmoid, are scaled by a SmoothStep schedule, and are normalized within each response so the mean weight is one. The weights then multiply the original GRPO advantage, so the sign of the update stays the same and only its distribution across tokens changes. Experiments train Qwen3-4B and Qwen2.5-7B on HIR-16k with CSR and AON rewards and evaluate on IFEval, IFBench, MultiDimIF, and AdvancedIF. The paper reports an average gain of 4.4 points over GRPO, results competitive with the learned-discriminator method RTT, and gains when CoRT is added to DAPO and GSPO.

Critical strengths

The paper gets token-level credit from the policy’s own likelihood contrast on a fixed response, with no extra model, labels, or generation, and it uses this contrast only to set how big each token’s update is while the GRPO advantage still sets the sign.

Critical weaknesses

I think there is no control showing that the replay contrast is what drives the improvement. The weights are small (0.75 to 1.25) and look saturated in Tables 6 and 7, where contrasts from 3 to 26 all map to about the same weight, so CoRT behaves close to a binary upweighting rule. Also, the GRPO and RTT numbers are copied from the RTT paper, while only CoRT is run by the authors, which I believe with a single seed. It would be nice if the runs can be apple-to-apple.

Questions for authors

See weaknesses

1 comment

We thank the reviewer for the careful and constructive review. We address each point below.

W1. Evidence that the replay contrast drives the improvement

Shuffled-weight training control. We trained Qwen3-4B under CSR with CoRT's token weights randomly permuted within each response. This keeps each response's weight distribution (range, mean and saturation) unchanged and only changes which token receives which weight. Both runs below share the same data, hyperparameters, seed and reward judge (DeepSeek-V4.1-Flash, since the DeepSeek-V3 judge used in the paper is no longer available to us), and are evaluated at the final checkpoint (step 500):

MetricShuffled weightsCoRT
IFEval prompt / inst.84.10 / 89.1187.65 / 91.73
IFBench prompt / inst.30.00 / 32.2437.55 / 40.30
MultiDimIF62.4083.75

All metrics drop after shuffling, most notably IFBench (−7.55 / −8.06) and MultiDimIF (−21.35), showing that the token placement given by the contrast is useful for training.

W2. Weights are small and look saturated, so CoRT behaves like a binary rule

Tables 6–7 show hand-picked cue tokens, which by construction sit in the extreme tail of the distribution. To check saturation at scale, we analyzed all response tokens in full training batches saved every 100 steps of a standard CoRT run, using the final token weights w after normalization:

Stepw ≤ 0.8w ∈ [0.95, 1.05] (w ≈ 1)w ≥ 1.2Intermediate
1000.74%57.53%1.71%40.01%
2001.32%41.28%0.91%56.49%
3000.72%60.09%1.64%37.56%
4001.11%50.02%1.01%47.86%
5000.94%55.62%0.99%42.45%

(Intermediate = (0.8, 0.95) ∪ (1.05, 1.2).) About half of the tokens stay near GRPO-level credit (w ≈ 1), roughly 38–56% are graded continuously, and only about 2% reach the extremes (w ≤ 0.8 or w ≥ 1.2), so the weights are not effectively binary. The bounded range is deliberate: it keeps each update close to GRPO for stability, while the contrast decides which tokens move and by how much. We will add this diagnostic to the appendix and state explicitly that Tables 6–7 show tail examples.

W3. Imported baselines and a single seed

Baselines. We could not re-run RTT faithfully: its discriminator checkpoint is listed as "To Be Added" in the official repository, and its 62,714 LLM-annotated token-level training samples are, to our knowledge, not released. We therefore keep RTT as a reported reference. Our conclusions do not rest on the imported GRPO numbers alone. The paper also compares CoRT with baselines trained in our own pipeline, at a larger scale (Qwen3-14B with GRPO under CSR and AON, Table 2) and with other policy optimizers (DAPO and GSPO, Table 3). CoRT improves 17 of these 20 metrics; the three exceptions are small (at most 0.46 points: IFEval for Qwen3-14B under AON, and MultiDimIF with GSPO). Together with the selection-free 14/16 result, this shows that the gain holds across model scales, optimizers and reward types.

Seeds. Multiple seeds per configuration are beyond our compute budget. We did train CoRT again on Qwen3-4B (CSR) with a different seed; this run had to use a newer reward judge (DeepSeek-V4.1-Flash), since the DeepSeek-V3 judge used in the paper is no longer available to us. It reaches 87.65 / 91.73 on IFEval, 37.55 / 40.30 on IFBench and 83.75 on MultiDimIF, above the step-500 results of the run reported in the paper on every metric (86.06 / 90.48, 34.49 / 36.24, 80.48). The reported run is therefore not a favourable outlier.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Revisions

We sincerely thank all reviewers and the area chair for their time and constructive feedback. We have uploaded a revised version that incorporates the discussion. The main changes are:

  1. Fixed-checkpoint evaluation. Table 1 now reports all CoRT results at the fixed step-500 checkpoint. The abstract and results text are updated accordingly.

  2. Matched baseline and shuffled-weight control (Appendix). We add a matched in-house GRPO baseline on Qwen3-4B (CSR), trained with the same data, hyperparameters, seed and reward judge as CoRT, together with a control in which CoRT's token weights are shuffled within each response.

  3. Limitations (new Section 8). We discuss the behaviour of CoRT under all-or-nothing rewards, the single-run setting, and the IFBench trade-off in the integration ablations.

  4. Related work. The comparison with on-policy self-distillation, including Self-Distilled RLVR, is moved from the appendix into the Related Work section.

  5. Computation cost and weight diagnostics. We report the replay overhead measured over a full 500-step training run, and add a token-weight distribution analysis over all response tokens (Appendix Table 4 and Figure 6).

We are grateful for the reviewers' suggestions, which have helped us improve the paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.