acceptodds
Under review as a conference paper at ICLR 2027

Internalizing Curriculum Judgment for LLM Reinforcement Fine-Tuning

Abstract

In LLM Reinforcement Fine-Tuning (RFT), curriculum learning drives both efficiency and performance. Yet, current methods externalize curriculum judgment via handcrafted heuristics or auxiliary models, risking misalignment with the policy's training dynamics. In this paper, we introduce METIS (METacognitive Internalized Self-judgment), a novel framework that internalizes curriculum judgment as a native capability. Leveraging a critical observation that within-prompt reward variance effectively gauges prompt informativeness, METIS predicts this metric based on recent training outcomes as lightweight in-context learning examples. This intrinsic self-judgment then dynamically dictates the training allocation. Moreover, METIS closes the loop between judgment and optimization by jointly optimizing the standard RFT rewards and a self-judgment reward. This allows the policy to learn what to learn next, as a form of metacognition. Across mathematical reasoning, code generation, and agentic function-calling benchmarks, METIS delivers superior performance while achieving up to a 2.1 training speedup, with controlled ablations and in-depth analysis further validating the benefits of internalized curriculum judgment. By bypassing handcrafted heuristics and auxiliary models, our work establishes a simple, closed-loop, and highly efficient curriculum internalization paradigm for LLM reinforcement fine-tuning.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Reinforcement Fine-Tuning, Curriculum Learning, Large Language Models

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_Dg1wOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes METIS, a curriculum learning framework for LLM reinforcement fine-tuning (RFT) that internalizes curriculum judgment into the policy itself rather than relying on external heuristics or auxiliary selector models. The core mechanism has two parts: (1) before rollout, the policy uses in-context learning (ICL) on a calibration memory of recent (prompt, realized reward variance) pairs to predict the within-prompt reward variance v̂_θ(x) of each candidate prompt, selecting the top-B for rollout; (2) after rollout, the realized variance serves as a calibration target via a Brier-score reward, and the policy is jointly optimized on task rewards and a self-judgment reward (L_total = L_policy + λ·L_judge). The authors formalize within-prompt reward variance as a unified informativeness signal covering both binary and continuous rewards, and validate METIS on mathematical reasoning (DAPO-17k), code generation (CodeContests+), and agentic function-calling (BFCL v3) across four backbone models, reporting up to 2.1× wall-clock speedup with ~3.9% per-step overhead.

Critical strengths

Strength 1: Generalization of within-prompt reward variance to continuous rewards — the genuine contribution.

Beyond the binary pass-rate proxy used by all prior curriculum work (PCL, VCRL, SEC), METIS formally extends reward variance to continuous reward settings (Eq. 4–5), enabling curriculum selection for code generation (CodeContests) and agentic function-calling (BFCL v3) where rewards are not 0/1. This is a concrete and useful generalization: the variance signal v_θ(x) = Var_{y~πθ}[r(x,y)] is reward-agnostic by construction, and the extension to multi-turn agentic tasks (BFCL v3, Table 6) is, to my knowledge, the first curriculum RFT result on agentic settings. The Brier-score calibration reward is a reasonable design for turning variance prediction into a learnable signal.

Strength 2: Thorough and well-controlled ablation methodology.

The internal-vs-external × ICL-vs-value-head ablation (Table 2) is a well-designed 2×2 factorial that isolates the contribution of internalization from the choice of judgment interface. The component ablation (Table 3) and hyperparameter sweep (Table 8) cover the right design knobs. The failure-boundary analysis on Qwen3-1.7B (App. B.6) is a commendable honesty that strengthens credibility.

Critical weaknesses

Weakness 1: RFT should be judged on accuracy, not only wall-clock speedup — and on accuracy, METIS is marginal at best and dominated by a simpler method.

Reinforcement fine-tuning exists to improve model capability, not to accelerate training. Yet METIS's headline claims are framed almost entirely around wall-clock speedup ("up to 2.1×"), while the actual accuracy evidence is weak: (a) METIS loses to PCL on Llama-8B/DAPO (23.4 vs 24.2) and DS-R1-Distill/DAPO; (b) most other accuracy gaps (0.5–0.8 points) fall within reported standard deviations; (c) the 2.1× speedup is a single-point cherry-pick (Qwen3-8B/DAPO); the median across the other five settings is ~1.5×, with Qwen3-4B/DAPO at only 1.1×. A curriculum method that delivers comparable-or-worse accuracy at faster wall-clock is not an RFT improvement — it is a compute reduction at equal quality, closer to a data-ordering trick than a capability gain. Critically, the paper never reports final-step accuracy at matched gradient-step budget (same number of optimization steps, not same wall-clock), which is the comparison RFT actually cares about — without it, the reader cannot tell whether METIS improves the policy at all. This concern is compounded by VCRL (arXiv 2509.19803), which uses the same variance signal through a far simpler mechanism (post-rollout filtering + memory-bank replay, no ICL, no joint loss) and achieves +7.67 avg on the identical Qwen3-4B/DAPO-17k setting (GRPO 41.76→49.43 vs METIS's +1.2), with Theorem 1 (gradient-stability guarantee) that METIS lacks entirely. A simpler method gets larger accuracy gains and has theory — so what does the elaborate ICL+Ljudge internalization buy? The paper does not engage with this. (VCRL is binary-reward only, so METIS's continuous-reward extension remains a real differentiator — but that is a contribution about variance generalization, not about internalization, and the framing inverts the relative importance.)

Weakness 2: The "internalization" claim is confounded, nearly inert empirically, and sharply bounded by backbone ICL capacity.

Three pieces of evidence converge against the internalization narrative. (1) Confounded ablation (Table 2): the "internal" variant shares parameters with the task policy while the "external" variant uses a separately initialized model, so the benefit could come from parameter sharing (representation transfer) rather than the policy "knowing its own state." A cleaner test — an external ICL judge with parameters frozen at a periodic policy snapshot — is not provided. (2) Ljudge is nearly inert (Table 3): removing Ljudge (λ=0) costs only 0.8 accuracy points (28.9→28.1), while removing ICL causes 87.6% parse failure. The weight-level "internalization" signal contributes almost nothing; the method's functionality depends entirely on ICL, which is in-context (not weight-internalized) by definition — making the "internalized" framing nominal. (3) Sharp capacity boundary (App. B.6): the Qwen3-1.7B failure (93.5% parse failure) is not a peripheral limitation but a direct contradiction of the "policy self-assesses" narrative — if the method requires sufficient ICL capacity to even parse the judgment instruction, the claim holds only for models that are already strong instruction followers, which weakens the motivation. The paper positions internalization as the core contribution, but the evidence suggests the active ingredient is the variance signal (shared with VCRL), not the internalization machinery.

Questions for authors

Q1.** RFT is fundamentally about accuracy improvement, yet METIS's headline claims are framed around wall-clock speedup ("up to 2.1×") while actual accuracy gains are marginal (losing to PCL on Llama-8B/DAPO 23.4 vs 24.2; most gains 0.5–0.8 pts within std). Two requests: (a) Report final-step accuracy at matched gradient-step budget (same number of optimization steps, not same wall-clock) for METIS vs No-Curriculum vs PCL across all six main settings — this isolates whether METIS improves the policy at all or merely reaches a comparable endpoint faster via data ordering. (b) VCRL (arXiv 2509.19803) uses the same variance signal but through far simpler post-rollout filtering (no ICL, no joint loss) and achieves +7.67 avg on Qwen3-4B/DAPO-17k with Theorem 1. Could you run a VCRL-style filtering variant of METIS (no ICL, no Ljudge) as an ablation to isolate what internalization buys beyond variance filtering?

Q2. In Table 2, the internal variant shares parameters with the task policy while the external variant uses a separately initialized model. Could you provide a cleaner isolation: an external ICL judge that uses the same calibration memory and candidate input, but whose parameters are a frozen snapshot of the policy (updated every N steps, not jointly optimized)? This would separate the benefit of "internalization" from that of parameter sharing.

Q3. Table 3 shows removing Ljudge (λ=0) costs only 0.8 accuracy points (28.9→28.1), while removing ICL causes 87.6% parse failure. This suggests the "internalization" via Ljudge is nearly inert and the method depends entirely on ICL (which is in-context, not weight-internalized). Could you (a) report the Pearson/Spearman correlation between ICL-predicted v̂_θ(x) and realized v(x) with and without Ljudge, and (b) explain how the "internalization" framing is justified when the weight-level calibration signal contributes almost nothing?

Q4. Could you report the per-setting speedup distribution explicitly (as a table or figure) rather than only the maximum? Specifically, for each of the six main settings in Table 1, what is the speedup, and is the 2.1× representative or an outlier? Additionally, please include Dynamic Sampling (widely adopted, the DAPO paper's default) as a baseline under equal wall-clock budget, particularly on continuous-reward tasks.

Q5. VCRL provides Theorem 1 (gradient-norm bound: E_VCRL[‖∇log π‖] ≤ E_GRPO[‖∇log π‖]) as a formal guarantee. METIS has no analogous theoretical result. Could you provide any formal analysis — e.g., a bound on the bias of ICL-predicted v̂_θ(x), a convergence guarantee for the joint optimization, or a regret bound for the curriculum selection — that would elevate the method beyond empirical heuristics? Without theory, and given VCRL's stronger empirical results with theory, the contribution gap is clear.

8 comments
AuthorsOpenReview ↗

Response to Reviewer Dg1w (1/2)

We sincerely thank the reviwer for the careful reading and constructive suggestions. We respond to each point as following.

W1 / Q1(a): Accuracy at a matched gradient-step budget

Thank you for this suggestion. We have now evaluated all methods at the same final training step (step 150), which is the comparison you asked for. We also use more robust multi-sample evalution instead of pass@1 to reduce the evaluation noise:

  • Every method is evaluated at the final checkpoint given the training budget (step 150), identical for all methods.
  • AIME24/25 use avg@16; MATH500, Minerva, LCBv6, HumanEval, MBPP, and BCB use avg@4.
  • Decoding parameters: Temperature 0.6, top-p 0.95, top-k disabled, max response length 8192.

The values below are mean value over the four benchmarks in each suite (mean and std over 3 seeds):

SettingNo Curr.SECADCLPCLMETIS
Math, Qwen3-8B34.3 ± 0.934.2 ± 0.433.5 ± 0.634.8 ± 0.636.1 ± 0.8
Math, Qwen3-4B26.6 ± 0.625.3 ± 0.826.2 ± 0.227.3 ± 0.728.4 ± 0.8
Math, Llama-3.1-8B21.2 ± 0.622.0 ± 0.122.9 ± 0.322.1 ± 0.622.9 ± 0.8
Code, Qwen3-8B51.6 ± 0.352.5 ± 0.651.4 ± 0.251.9 ± 0.253.5 ± 0.4
Code, Llama-3.1-8B38.9 ± 0.539.3 ± 0.438.9 ± 0.440.0 ± 0.340.9 ± 0.4
Code, DS-R1-Distill48.5 ± 0.447.6 ± 0.147.5 ± 0.348.7 ± 0.249.7 ± 0.4

We see that METIS attains the highest average in all six settings with gains of 1.2–2.0 points over No Curriculum, including Llama-3.1-8B on Math where it was behind PCL under the original evaluation set-up. METIS therefore improves the policy at a matched number of updates, rather than only reaching a comparable endpoint faster. We have updated the abstract and Table 1 in Sec. 5.1 with analysis accordingly in the revised manuscript.

W1 / Q1(b): Comparison with VCRL

Following your suggestion we added VCRL [2] as a baseline. VCRL is currently an arXiv preprint without released code, so we re-implemented it from the paper with the variance threshold and memory-bank settings given in the paper. We evaluated it under the same protocol as above:

ModelMethodAIME24AIME25MATH500MinervaAvg.
Qwen3-8BVCRL21.8 ± 1.416.2 ± 1.476.4 ± 0.726.2 ± 0.935.1 ± 0.4
Qwen3-8BMETIS20.8 ± 1.318.5 ± 1.077.5 ± 0.427.5 ± 1.136.1 ± 0.8
Qwen3-4BVCRL18.5 ± 0.79.3 ± 1.063.0 ± 0.418.1 ± 0.427.2 ± 0.6
Qwen3-4BMETIS19.7 ± 1.68.6 ± 1.064.1 ± 0.321.0 ± 0.628.4 ± 0.8

We see that METIS outperforms VCRL on both model backbones. In our setup VCRL improves over No Curriculum but not following the exact number reported in the original VCRL paper. We believe the difference comes from the benchmark suite and differnet hyper-parameters for training and inference. We also note that the gains reported for VCRL are measured per step without normalizing for compute (which is a concern that was also raised in its public reviews on OpenReview). This indicates that VCRL step requires more rollouts than a METIS step because it filters prompts after rollout, while METIS instead predicts the variance before rollout and also covers continuous rewards. Also another advantage of METIS as suggested by the reviewer is that METIS works in both discrete and continuous reward settings while VCRL only solves discrete rewards. We have added the additional experiment result with VCRL in Appendix B.1 in the revised manuscript.

AuthorsOpenReview ↗

Response to Reviewer Dg1w (2/2)

W1 / Q4: Training speedup

Thank you for pointing this out. We now report the wall-clock time at which each method first reaches a common target τ\tau which is the 95% of the final-checkpoint average of No Curriculum. We also added the Dynamic Sampling from the DAPO[3] paper as you request. We report the training time in hours to reach the τ\tau with mean and std over 3 seeds:

Settingτ\tauNo Curr.Dynamic SamplingSECADCLPCLMETISvs. No Curr.
Math, Qwen3-8B32.68.6 ± 0.97.1 ± 0.67.5 ± 0.710.8 ± 1.16.7 ± 0.65.1 ± 0.31.69×
Math, Qwen3-4B25.34.8 ± 0.44.5 ± 0.55.3 ± 0.45.5 ± 0.54.1 ± 0.23.4 ± 0.31.41×
Math, Llama-3.1-8B20.16.8 ± 0.76.2 ± 0.95.0 ± 2.612.0 ± 1.010.5 ± 0.75.4 ± 1.11.26×
Code, Qwen3-8B49.010.5 ± 1.08.8 ± 0.78.5 ± 0.513.8 ± 1.09.0 ± 0.86.7 ± 0.51.57×
Code, Llama-3.1-8B37.07.8 ± 0.66.1 ± 0.78.6 ± 0.710.7 ± 0.75.5 ± 0.94.4 ± 0.81.77×
Code, DS-R1-Distill46.112.0 ± 1.010.0 ± 0.911.3 ± 0.916.0 ± 1.29.4 ± 0.77.6 ± 0.61.58×

METIS is the fastest method in five out of six settings with up to 1.77× faster than No Curriculum (and also outperforms the Dynamic Sampling baseline). We have updated the abstract, Table 1 in Sec. 5.1 and Table 8 in Appendix B.1 with analysis accordingly in the revised manuscript.

W2(1) / Q2: Parameter sharing vs. internalization

Thank you for this question. Table 2 already contains the experiment you suggest. An external ICL judge whose parameters are a snapshot of the policy is identical to row 4 (External, no judgment learning) when the snapshot is taken once (and frozen) at initialization, and in row 2 (Internal, no judgment learning) when it is refreshed at every step. These two reach 26.9 and 28.4 with the same exemplars, so the judge improves as its parameters stay closer to the current policy. This is what we mean by internalization: the judgment needs the state of the current policy and letting the policy produce the judgement is the simplest way to provide that state, without maintaining and syncing a second model. We have clarified this in Sec. 5.2 in the revised manuscript.

W2(2) / Q3: The role of Ljudge\mathcal{L}_{\text{judge}}

Thank you for this question. The effect of Ljudge\mathcal L_{\text{judge}} is shown in Figs. 9 and 10 (Appendix B.3 and B.4). With Ljudge\mathcal L_{\text{judge}}, the mean RjudgeR_{\text{judge}} (which can be interpreted as prediction accuracy) rises from 0.54 to 0.85 over training and the predictions stay differentiated across candidates, whereas with λ=0\lambda = 0 the mean prediction stays flat and its spread shrinks.

The exemplars alone does not make METIS effective, since the same exemplars read by a frozen external model give no gain (External, no judgment learning, row 4 of Table 2). Therefore we see the judgment does depend on the weights of the policy. We have clarified this in Sec. 5.2 in the revised manuscript.

W2(3): ICL Capacity boundary

We state this capacity boundary of ICL as a limitation in Sec. 6 and Appendix B.6. METIS requires the backbone has the ability to follow the judgment instruction, which holds for the 4B and 8B models but not for Qwen3-1.7B (93.5% parse-failure rates). In that case METIS falls back to uniform selection and matches No Curriculum rather than hurting it (Table 13). We also state that the parse-failure rate can be checked before training as a cheap diagnostic.

Q5: Theoretical contribution

Thank you for mentioning this. We clarify that theoretical guarantees are not claims of this paper. Our primary contribution is the prompt-informativeness formulation, the internalized curriculum judgment method METIS, and extensive empirical validation. This scope is consistent with the literature, e.g., PCL[1], SEC[4], and ADCL[5], which are also validated empirically. We will clarify this scope/limitation in the revision.

We hope these clarifications and additions address your concerns.


[1] Prompt curriculum learning for efficient LLM post-training (ICLR 2026).

[2] VCRL: Variance-based Curriculum Reinforcement Learning for Large Language Models (arXiv 2025)

[3] DAPO: An Open-Source LLM Reinforcement Learning System at Scale (arXiv 2025)

[4] Learning like humans: Advancing LLM reasoning capabilities via adaptive difficulty curriculum learning and expert-guided self-reformulation (EMNLP 2025).

[5] Self-evolving curriculum for LLM reasoning (arXiv 2025).

AuthorsOpenReview ↗

A Gentle Message to Reviewer Dg1w

Dear Reviewer Dg1w,

We sincerely appreciate the time and effort you have devoted to reviewing our manuscript.

As the discussion period will close in less than 48 hours, we would be very grateful if you could share any comments or questions you may have at your earliest convenience. Receiving your feedback before the discussion deadline would allow us to respond promptly and address any concerns thoroughly.

We understand that this is a particularly busy review period and that you are likely handling many submissions, so we are especially grateful for your valuable time and expertise!

Best regards, The Authors

Reviewer_pdNvOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces METIS, which uses the task policy to predict candidate prompts' reward variance from recent rollout examples and select prompts for reinforcement fine tuning. A calibration reward jointly trains curriculum judgment and task solving using subsequent rollout outcomes. The main experiments cover mathematical reasoning and code generation, with comparisons to alternative curricula, internal and external judges, and component ablations. The paper reports improvements in downstream performance and training efficiency.

Critical strengths
  1. The method integrates curriculum prediction and task solving within the same policy, obtaining judgment supervision from existing rollouts. This avoids a separate predictor and additional task rollouts for supervision, while accommodating both binary and continuous rewards. The design has practical appeal.

  2. Table 2 provides meaningful comparisons between internal and external judges, with and without judgment learning. Together with the results across multiple backbones and the explicit overhead measurements, these controls support the practical value of the proposed integration.

Critical weaknesses
  1. The novelty is limited. The variance criterion and learned curriculum selection build on prior approaches acknowledged in Sections 2 and 3. The main advance is integrating token based judgment with the task policy.

  2. Section 5.1 selects checkpoints using the benchmarks subsequently reported. This makes the accuracy gains vulnerable to checkpoint selection bias and weakens the evidence for improved generalization.

  3. Table 1 compares the time to each method's own best checkpoint. This does not establish the claimed acceleration at matched quality, although Figure 3 provides partial support in selected settings.

  4. Removing ICL examples in Table 3 also causes judgment generation failures. The ablation therefore does not separate the informational value of recent variance examples from their role in eliciting valid judgments.

  5. Figure 5 uses batch average rewards to support competence frontier tracking. The same averages can arise from mixing consistently solved and consistently failed prompts, so they do not establish the claimed behavior for individual prompts.

Questions for authors
  1. Were checkpoints selected using an independent validation set? If the reported benchmarks were used for selection, do the accuracy gains persist under independent checkpoint selection?

  2. Does the efficiency advantage persist when methods are compared at a common quality target or under matched training budgets?

  3. Can the contribution of recent variance labels be isolated while preserving the model's ability to generate valid judgments?

  4. What evidence directly establishes accurate variance rankings for individual candidate prompts, beyond the aggregate behavior shown in Figure 5?

  5. Since Algorithm 1 obtains judgment feedback only for selected prompts, how are underestimated candidates identified and given corrective feedback?

6 comments
AuthorsOpenReview ↗

Response to Reviewer pdNv (1/2)

We sincerely thank the reviwer for the careful reading and constructive suggestions. We respond to each point as following.

W1: Novelty

We agree that the using rollout vairance as curriculum indicator for binary rewards has appeared in prior work. As discussed in Secs. 2 and 3. our contributions are threefold: (i) conceptually, we formulate curriculum learning for RFT as policy-dependent competence-frontier tracking, with within-prompt reward variance as a general signal for both binary and continuous rewards; (ii) technically, METIS is the first RFT framework to internalize curriculum judgment, jointly optimizing task rewards and a self-judgment reward so that the curriculum becomes a learned capability of the policy rather than an external selector; (iii) empirically, METIS improves accuracy and training time on math, code, and agentic tasks, and the ablations show that internalization drives these gains.

W2 / Q1: Checkpoint selection

Thank you for pointing this out. We agree that best-checkpoint selection on the evaluation benchmarks introduces selection bias. We have now evaulated all methods without checkpoint selection on the test benchmark, with more robust multi-sample evalution instead of pass@1 to reduce the evaluation noise:

  • Every method is evaluated at the final checkpoint given the training budget (step 150), identical for all methods.
  • AIME24/25 use avg@16; MATH500, Minerva, LCBv6, HumanEval, MBPP, and BCB use avg@4.
  • Decoding parameters: Temperature 0.6, top-p 0.95, top-k disabled, max response length 8192.

The values below are mean value over the four benchmarks in each suite (mean and std over 3 seeds):

SettingNo Curr.SECADCLPCLMETIS
Math, Qwen3-8B34.3 ± 0.934.2 ± 0.433.5 ± 0.634.8 ± 0.636.1 ± 0.8
Math, Qwen3-4B26.6 ± 0.625.3 ± 0.826.2 ± 0.227.3 ± 0.728.4 ± 0.8
Math, Llama-3.1-8B21.2 ± 0.622.0 ± 0.122.9 ± 0.322.1 ± 0.622.9 ± 0.8
Code, Qwen3-8B51.6 ± 0.352.5 ± 0.651.4 ± 0.251.9 ± 0.253.5 ± 0.4
Code, Llama-3.1-8B38.9 ± 0.539.3 ± 0.438.9 ± 0.440.0 ± 0.340.9 ± 0.4
Code, DS-R1-Distill48.5 ± 0.447.6 ± 0.147.5 ± 0.348.7 ± 0.249.7 ± 0.4

We see that METIS attains the highest average in all six settings with gains of 1.2–2.0 points over No Curriculum. Selecting each method's best checkpoint on the test benchmarks was biased in our old set-up, as it inflated every method's score and blurred the differences between them. We thank the reviewer for pointing out this as under this evaluation setting which is a fairer compairsion and further demonstrates the effectiveness of METIS. We have updated the Table 1 in Sec. 5.1 accordingly in the revised manuscript.

W3 / Q2: Efficiency at a common performance target

Thanks for pointing this out. We agree that a fixed-target comparison is fairer. We now report the wall-clock time at which each method first reaches a common target τ\tau which is the 95% of the final-checkpoint average of No Curriculum. We report the training time in hours to reach the τ\tau with mean and std over 3 seeds:

Settingτ\tauNo Curr.SECADCLPCLMETISvs. No Curr.
Math, Qwen3-8B32.68.6 ± 0.97.5 ± 0.710.8 ± 1.16.7 ± 0.65.1 ± 0.31.69×
Math, Qwen3-4B25.34.8 ± 0.45.3 ± 0.45.5 ± 0.54.1 ± 0.23.4 ± 0.31.41×
Math, Llama-3.1-8B20.16.8 ± 0.75.0 ± 2.612.0 ± 1.010.5 ± 0.75.4 ± 1.11.26×
Code, Qwen3-8B49.010.5 ± 1.08.5 ± 0.513.8 ± 1.09.0 ± 0.86.7 ± 0.51.57×
Code, Llama-3.1-8B37.07.8 ± 0.68.6 ± 0.710.7 ± 0.75.5 ± 0.94.4 ± 0.81.77×
Code, DS-R1-Distill46.112.0 ± 1.011.3 ± 0.916.0 ± 1.29.4 ± 0.77.6 ± 0.61.58×

METIS is the fastest method in five out of six settings with up to 1.77× faster than No Curriculum. For matched training budgets on number of gradient updates, the table in W2 already evaluates all methods at the same step. We have updated the abstract and Table 1 in Sec. 5.1 with analysis accordingly in the revised manuscript.

AuthorsOpenReview ↗

Response to Reviewer pdNv (2/2)

W4 / Q3: ICL ablation

The ablation comparison in Table 2 answers your concern. In Table 2, the external ICL judge without judgment learning (row 4 in Table 2) is a frozen copy of the initial model that reads the same recent variance exemplars as METIS and outputs valid judgments, which means the exemplars are the only thing telling reflecting the current policy. This variant reaches 26.9, which is not better than No Curriculum (27.5 in Table 3), whereas the internal judge with the same exemplars and also no judgment learning reaches 28.4 (row 2 in Table 2).

In other words, the recent variance labels by themselves are not enough to produce a useful curriculum and the benefit shows up only when the model reading them is the current policy. This is consistent with Fig.10 (Appendix B.4) where the predictions barely change over training when the exemplars are kept but λ=0\lambda = 0 (without judgement learning). We will revise Sec. 4.1 to make clear that the exemplars mainly serve to elicit valid and anchored model judgments outputs, while the adaptivity and effectiveness comes from sharing the policy and from Ljudge\mathcal{L}_{\text{judge}}.

We have added the above analysis in Sec.5.2 in the revised manuscript.

W5 / Q4: Accurate prediction evidence

In our paper the judgment reward is defined as Rjudge=1−(4(v^θ−v))2R_{\text{judge}} = 1 - (4(\hat v_\theta - v))^2, which measures the prediction error on each selected prompt and gives RMSE=141−Rˉjudge\text{RMSE} = \frac{1}{4}\sqrt{1 - \bar R_{\text{judge}}}. In Fig. 9 (Appendix B.3), its mean increases from 0.54 in steps 0–20 to 0.77 at step 75 and 0.85 at step 150, which corresponds to the RMSE dropping from 0.170 to 0.120 and 0.097 on a variance range of [0, 0.25]. This indicates that the prediction accuracy improves as training goes on. We have added the above analysis in Sec.5.3 in the revised manuscript.

Q5: Underestimated candidates

Underestimated candidates receive no direct feedback as only the selected 1/8 of each candidate pool (m=8m = 8) is rolled out at every gradient step. However, they can still be corrected indirectly, as the judgment shares its parameters with the task policy and every candidate is judged again with fresh exemplars each time it is resampled. The unique prompt coverage rate in Table 4 suggests that METIS avoids overfitting into a fixed small subset of the entire dataset, since METIS rolls out 82–91% of the full dataset over a run while PCL[1] covers only 39–46%. We have added the above analysis in Sec.5.3 in the revised manuscript.

We hope these clarifications and additions address your concerns.

[1] Prompt curriculum learning for efficient LLM post-training (ICLR 2026).

AuthorsOpenReview ↗

A Gentle Reminder to Reviewer pdNv

Dear Reviewer pdNv,

We sincerely appreciate the time and effort you have devoted to reviewing our manuscript.

As the discussion period will close in less than 48 hours, we would be very grateful if you could share any comments or questions you may have at your earliest convenience. Receiving your feedback before the discussion deadline would allow us to respond promptly and address any concerns thoroughly.

We understand that this is a particularly busy review period and that you are likely handling many submissions, so we are especially grateful for your valuable time and expertise!

Best regards, The Authors

Reviewer_r1huOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces METIS, a reinforcement fine-tuning (RFT) framework that internalizes prompt-level curriculum judgment into the policy itself. Using within-prompt rollout reward variance as a measure of prompt informativeness, METIS prompts the current policy with a few recent prompt-variance examples via in-context learning (ICL) to predict candidate variances before rollout, selects the top-scoring prompts for group-relative RL updates, and jointly trains the policy with a calibration reward based on realized rollout variances. The authors evaluate METIS across mathematical reasoning, code generation, and agentic function-calling benchmarks over multiple model backbones and three random seeds, reporting higher peak accuracy and faster time-to-best-checkpoint than heuristic and auxiliary-model baselines.

Critical strengths
  • The core idea is novel and intuitive. Internalizing curriculum judgment via in-context self-assessment nicely addresses a key dilemma in existing RFT curricula, avoiding both predefined curriculum categories and expensive online estimation for problem difficulty.
  • The experimental study is comprehensive, with ablations covering the main design choices and main training runs repeated across three random seeds.
  • The authors honestly report the failure mode of their method in Appendix B.6, showing when and why self-judgment breaks down on smaller backbones (such as Qwen3-1.7B) with limited instruction-following ability.
Critical weaknesses
  • The evaluation protocol makes it difficult to separate genuine gains from noise. Math and code benchmarks only report pass@1, which is known to exhibit high variance for long-chain reasoning models. At the same time, the accuracy improvements over PCL and No Curriculum in Table 1 are quite marginal (mostly around 0.6 to 1.2 points), and checkpoints are selected directly by the highest average score on the test benchmarks themselves (Section 5.1). Under test-set checkpoint selection and high-variance pass@1 evaluation, these small margins may largely reflect evaluation noise.
  • The speedup metric in Table 1 is misleading because it measures the time for each method to reach its own peak score rather than the time required to reach a fixed target score. Under this definition, a baseline that continues improving later in training to reach a higher score is penalized as being slower (for example, PCL on Llama-3.1-8B math achieves a higher accuracy of 24.2 at 19.0 hours vs. METIS's 23.4 at 6.8 hours, yet PCL is reported as having a 0.4x speedup). A fair efficiency comparison should measure wall-clock time to reach the same target performance.
  • The effectiveness of the learned curriculum in early training and the actual mechanism of self-judgment remain questionable. In Figure 5 (DAPO-17k), the empirical rollout reward of selected prompts only approaches the 0.5 frontier fairly late in training, raising doubts about how useful the curriculum predictions are during earlier stages. Moreover, the qualitative examples in Appendix B.7 show the model equating 0.00 variance with simple problems and 0.25 variance with complex problems, suggesting it is estimating surface problem difficulty rather than true reward variance (since overly hard problems also have 0.00 variance).
Questions for authors
  • Could you clarify the exact decoding parameters and number of rollouts per problem used during math and code evaluation, and report results using a held-out validation split (or final checkpoints) with multi-sample evaluation (e.g., avg@8) to reduce evaluation variance?
  • How do the wall-clock speedups in Table 1 look when measured as the time required for each method to reach a fixed target accuracy (for instance, the peak accuracy of No Curriculum)?
  • Looking at Figure 5 and the qualitative traces in Appendix B.7, how accurate is the model's variance prediction during early training?
  • Although Section 3 states that METIS is agnostic to the underlying RL update rule, all experiments exclusively use an Dr.GRPO. Have you tested whether METIS provides similar gains under other RL algorithms?
5 comments
AuthorsOpenReview ↗

Response to Reviewer r1hu (1/2)

We sincerely thank the reviwer for the careful reading and constructive suggestions. We respond to each point as following.

W1 / Q1: Evaluation set-up

We agree that selecting the best checkpoint on the evaluation benchmarks with pass@1 makes small margins hard to interpret. We have now evaulated all methods under the protocol you proposed:

  • Every method is evaluated at the final checkpoint given the training budget (step 150), identical for all methods.
  • AIME24/25 use avg@16; MATH500, Minerva, LCBv6, HumanEval, MBPP, and BCB use avg@4.
  • Decoding parameters: Temperature 0.6, top-p 0.95, top-k disabled, max response length 8192.

The values below are mean value over the four benchmarks in each suite (mean and std over 3 seeds):

SettingNo Curr.SECADCLPCLMETIS
Math, Qwen3-8B34.3 ± 0.934.2 ± 0.433.5 ± 0.634.8 ± 0.636.1 ± 0.8
Math, Qwen3-4B26.6 ± 0.625.3 ± 0.826.2 ± 0.227.3 ± 0.728.4 ± 0.8
Math, Llama-3.1-8B21.2 ± 0.622.0 ± 0.122.9 ± 0.322.1 ± 0.622.9 ± 0.8
Code, Qwen3-8B51.6 ± 0.352.5 ± 0.651.4 ± 0.251.9 ± 0.253.5 ± 0.4
Code, Llama-3.1-8B38.9 ± 0.539.3 ± 0.438.9 ± 0.440.0 ± 0.340.9 ± 0.4
Code, DS-R1-Distill48.5 ± 0.447.6 ± 0.147.5 ± 0.348.7 ± 0.249.7 ± 0.4

We see that METIS attains the highest average in all six settings with gains of 1.2–2.0 points over No Curriculum. Selecting each method's best checkpoint on the test benchmarks was biased in our old set-up, as it inflated every method's score and blurred the differences between them. We thank the reviewer for pointing out this as under this evaluation setting which is a fairer comparison and further demonstrates the effectiveness of METIS. We have updated the Table 1 in Sec. 5.1 accordingly in the revised manuscript.

W2 / Q2: Speedup metric

Thanks for pointing this out. We agree that a fixed-target comparison is fairer. We now report the wall-clock time that each method first reaches a common target τ\tau, which is the 95% of the final-checkpoint average of No Curriculum. We report the training time in hours to reach the τ\tau with mean and std over 3 seeds:

Settingτ\tauNo Curr.SECADCLPCLMETISvs. No Curr.
Math, Qwen3-8B32.68.6 ± 0.97.5 ± 0.710.8 ± 1.16.7 ± 0.65.1 ± 0.31.69×
Math, Qwen3-4B25.34.8 ± 0.45.3 ± 0.45.5 ± 0.54.1 ± 0.23.4 ± 0.31.41×
Math, Llama-3.1-8B20.16.8 ± 0.75.0 ± 2.612.0 ± 1.010.5 ± 0.75.4 ± 1.11.26×
Code, Qwen3-8B49.010.5 ± 1.08.5 ± 0.513.8 ± 1.09.0 ± 0.86.7 ± 0.51.57×
Code, Llama-3.1-8B37.07.8 ± 0.68.6 ± 0.710.7 ± 0.75.5 ± 0.94.4 ± 0.81.77×
Code, DS-R1-Distill46.112.0 ± 1.011.3 ± 0.916.0 ± 1.29.4 ± 0.77.6 ± 0.61.58×

We see taht METIS has the fastest convergence rate in five out of six settings with up to 1.77× faster than No Curriculum. We have updated abstract and the Table 1 in Sec. 5.1 accordingly in the revised manuscript.

AuthorsOpenReview ↗

Response to Reviewer r1hu (2/2)

W3 / Q3: Early-training curriculum quality and what the model estimates

In Fig. 9 (Appendix B.3), we see the prediction accuracy gets better as training goes on. Since Rjudge=1−(4(v^θ−v))2R_{\text{judge}} = 1 - (4(\hat v_\theta - v))^2, its mean gives the prediction error directly: RMSE=141−Rˉjudge\text{RMSE} = \frac{1}{4}\sqrt{1 - \bar R_{\text{judge}}}. We see that the mean RjudgeR_{\text{judge}} is 0.54 in steps 0–20, 0.77 at step 75, and 0.85 at step 150. This means the RMSE drops from 0.170 to 0.120 to 0.097 (variance ranges over [0, 0.25]).

The curriculum therefore helps only a little at the start. We can also see this from Fig.3 (bottom chart) where the mean ∣A∣\lvert A \rvert of a prompt is propotional to its reward variance. In the first 20 steps METIS-selected prompts have ∣A∣=\lvert A \rvert = 0.047 against 0.042 for No Curriculum. The gap on ∣A∣\lvert A \rvert grows later and we clearly see METIS achieves larger ∣A∣\lvert A \rvert gradually, which is where most of the gain comes from.

On difficulty vs. variance, the initial policy treats hard problems as high-variance, so it overestimates on problems it never solves. But those problems have zero variance at early stage of training, so the judgment reward penalizes exactly this mistake and the rise of RjudgeR_{\text{judge}} shows the policy correcting it. Fig. 5 shows that the selected prompts move toward p=0.5p = 0.5 from below on DAPO-17k, and stay near 0.5 on MATH instead of drifting to 1.

We have added the above analysis to Sec.5 in the revised manuscript.

Q4: Other RL algorithms

METIS applies to other RFT method as well. To verify this empirically, we add additional experiments on Qwen3-8B Base using DAPO [1] under the same protocol as above (final checkpoint, AIME24/25 avg@16, other benchmarks avg@4, mean and std over 3 seeds):

TaskNo Curr.SECADCLPCLMETIS
Math35.0±0.335.0 \pm 0.335.8±0.635.8 \pm 0.632.8±0.732.8 \pm 0.734.9±0.334.9 \pm 0.337.3±0.5\mathbf{37.3} \pm 0.5
Code53.4±0.353.4 \pm 0.353.0±0.653.0 \pm 0.652.2±0.552.2 \pm 0.554.3±0.254.3 \pm 0.255.3±0.4\mathbf{55.3} \pm 0.4

METIS remains the best method on both tasks, similar to the gains under GRPO in the main experiments. We will revise Sec. 3 to state that METIS is compatible with group-relative RL updates and has been validated on GRPO and DAPO rather than claiming it is method agnostic in general. We have added these additional experiment result and analysis in Appendix B.1 in the revised manuscript.

We hope these clarifications and additions address your concerns.

[1] DAPO: An Open-Source LLM Reinforcement Learning System at Scale (arXiv 2025)

AuthorsOpenReview ↗

A Gentle Reminder to Reviewer r1hu

Dear Reviewer r1hu,

We sincerely appreciate the time and effort you have devoted to reviewing our manuscript.

As the discussion period will close in less than 48 hours, we would be very grateful if you could share any comments or questions you may have at your earliest convenience. Receiving your feedback before the discussion deadline would allow us to respond promptly and address any concerns thoroughly.

We understand that this is a particularly busy review period and that you are likely handling many submissions, so we are especially grateful for your valuable time and expertise!

Best regards, The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.