- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper proposes METIS, a curriculum learning framework for LLM reinforcement fine-tuning (RFT) that internalizes curriculum judgment into the policy itself rather than relying on external heuristics or auxiliary selector models. The core mechanism has two parts: (1) before rollout, the policy uses in-context learning (ICL) on a calibration memory of recent (prompt, realized reward variance) pairs to predict the within-prompt reward variance v̂_θ(x) of each candidate prompt, selecting the top-B for rollout; (2) after rollout, the realized variance serves as a calibration target via a Brier-score reward, and the policy is jointly optimized on task rewards and a self-judgment reward (L_total = L_policy + λ·L_judge). The authors formalize within-prompt reward variance as a unified informativeness signal covering both binary and continuous rewards, and validate METIS on mathematical reasoning (DAPO-17k), code generation (CodeContests+), and agentic function-calling (BFCL v3) across four backbone models, reporting up to 2.1× wall-clock speedup with ~3.9% per-step overhead.
Critical strengths
Strength 1: Generalization of within-prompt reward variance to continuous rewards — the genuine contribution.
Beyond the binary pass-rate proxy used by all prior curriculum work (PCL, VCRL, SEC), METIS formally extends reward variance to continuous reward settings (Eq. 4–5), enabling curriculum selection for code generation (CodeContests) and agentic function-calling (BFCL v3) where rewards are not 0/1. This is a concrete and useful generalization: the variance signal v_θ(x) = Var_{y~πθ}[r(x,y)] is reward-agnostic by construction, and the extension to multi-turn agentic tasks (BFCL v3, Table 6) is, to my knowledge, the first curriculum RFT result on agentic settings. The Brier-score calibration reward is a reasonable design for turning variance prediction into a learnable signal.
Strength 2: Thorough and well-controlled ablation methodology.
The internal-vs-external × ICL-vs-value-head ablation (Table 2) is a well-designed 2×2 factorial that isolates the contribution of internalization from the choice of judgment interface. The component ablation (Table 3) and hyperparameter sweep (Table 8) cover the right design knobs. The failure-boundary analysis on Qwen3-1.7B (App. B.6) is a commendable honesty that strengthens credibility.
Critical weaknesses
Weakness 1: RFT should be judged on accuracy, not only wall-clock speedup — and on accuracy, METIS is marginal at best and dominated by a simpler method.
Reinforcement fine-tuning exists to improve model capability, not to accelerate training. Yet METIS's headline claims are framed almost entirely around wall-clock speedup ("up to 2.1×"), while the actual accuracy evidence is weak: (a) METIS loses to PCL on Llama-8B/DAPO (23.4 vs 24.2) and DS-R1-Distill/DAPO; (b) most other accuracy gaps (0.5–0.8 points) fall within reported standard deviations; (c) the 2.1× speedup is a single-point cherry-pick (Qwen3-8B/DAPO); the median across the other five settings is ~1.5×, with Qwen3-4B/DAPO at only 1.1×. A curriculum method that delivers comparable-or-worse accuracy at faster wall-clock is not an RFT improvement — it is a compute reduction at equal quality, closer to a data-ordering trick than a capability gain. Critically, the paper never reports final-step accuracy at matched gradient-step budget (same number of optimization steps, not same wall-clock), which is the comparison RFT actually cares about — without it, the reader cannot tell whether METIS improves the policy at all. This concern is compounded by VCRL (arXiv 2509.19803), which uses the same variance signal through a far simpler mechanism (post-rollout filtering + memory-bank replay, no ICL, no joint loss) and achieves +7.67 avg on the identical Qwen3-4B/DAPO-17k setting (GRPO 41.76→49.43 vs METIS's +1.2), with Theorem 1 (gradient-stability guarantee) that METIS lacks entirely. A simpler method gets larger accuracy gains and has theory — so what does the elaborate ICL+Ljudge internalization buy? The paper does not engage with this. (VCRL is binary-reward only, so METIS's continuous-reward extension remains a real differentiator — but that is a contribution about variance generalization, not about internalization, and the framing inverts the relative importance.)
Weakness 2: The "internalization" claim is confounded, nearly inert empirically, and sharply bounded by backbone ICL capacity.
Three pieces of evidence converge against the internalization narrative. (1) Confounded ablation (Table 2): the "internal" variant shares parameters with the task policy while the "external" variant uses a separately initialized model, so the benefit could come from parameter sharing (representation transfer) rather than the policy "knowing its own state." A cleaner test — an external ICL judge with parameters frozen at a periodic policy snapshot — is not provided. (2) Ljudge is nearly inert (Table 3): removing Ljudge (λ=0) costs only 0.8 accuracy points (28.9→28.1), while removing ICL causes 87.6% parse failure. The weight-level "internalization" signal contributes almost nothing; the method's functionality depends entirely on ICL, which is in-context (not weight-internalized) by definition — making the "internalized" framing nominal. (3) Sharp capacity boundary (App. B.6): the Qwen3-1.7B failure (93.5% parse failure) is not a peripheral limitation but a direct contradiction of the "policy self-assesses" narrative — if the method requires sufficient ICL capacity to even parse the judgment instruction, the claim holds only for models that are already strong instruction followers, which weakens the motivation. The paper positions internalization as the core contribution, but the evidence suggests the active ingredient is the variance signal (shared with VCRL), not the internalization machinery.
Questions for authors
Q1.** RFT is fundamentally about accuracy improvement, yet METIS's headline claims are framed around wall-clock speedup ("up to 2.1×") while actual accuracy gains are marginal (losing to PCL on Llama-8B/DAPO 23.4 vs 24.2; most gains 0.5–0.8 pts within std). Two requests: (a) Report final-step accuracy at matched gradient-step budget (same number of optimization steps, not same wall-clock) for METIS vs No-Curriculum vs PCL across all six main settings — this isolates whether METIS improves the policy at all or merely reaches a comparable endpoint faster via data ordering. (b) VCRL (arXiv 2509.19803) uses the same variance signal but through far simpler post-rollout filtering (no ICL, no joint loss) and achieves +7.67 avg on Qwen3-4B/DAPO-17k with Theorem 1. Could you run a VCRL-style filtering variant of METIS (no ICL, no Ljudge) as an ablation to isolate what internalization buys beyond variance filtering?
Q2. In Table 2, the internal variant shares parameters with the task policy while the external variant uses a separately initialized model. Could you provide a cleaner isolation: an external ICL judge that uses the same calibration memory and candidate input, but whose parameters are a frozen snapshot of the policy (updated every N steps, not jointly optimized)? This would separate the benefit of "internalization" from that of parameter sharing.
Q3. Table 3 shows removing Ljudge (λ=0) costs only 0.8 accuracy points (28.9→28.1), while removing ICL causes 87.6% parse failure. This suggests the "internalization" via Ljudge is nearly inert and the method depends entirely on ICL (which is in-context, not weight-internalized). Could you (a) report the Pearson/Spearman correlation between ICL-predicted v̂_θ(x) and realized v(x) with and without Ljudge, and (b) explain how the "internalization" framing is justified when the weight-level calibration signal contributes almost nothing?
Q4. Could you report the per-setting speedup distribution explicitly (as a table or figure) rather than only the maximum? Specifically, for each of the six main settings in Table 1, what is the speedup, and is the 2.1× representative or an outlier? Additionally, please include Dynamic Sampling (widely adopted, the DAPO paper's default) as a baseline under equal wall-clock budget, particularly on continuous-reward tasks.
Q5. VCRL provides Theorem 1 (gradient-norm bound: E_VCRL[‖∇log π‖] ≤ E_GRPO[‖∇log π‖]) as a formal guarantee. METIS has no analogous theoretical result. Could you provide any formal analysis — e.g., a bound on the bias of ICL-predicted v̂_θ(x), a convergence guarantee for the joint optimization, or a regret bound for the curriculum selection — that would elevate the method beyond empirical heuristics? Without theory, and given VCRL's stronger empirical results with theory, the contribution gap is clear.
8 comments
Response to Reviewer Dg1w (2/2)
W1 / Q4: Training speedup
Thank you for pointing this out. We now report the wall-clock time at which each method first reaches a common target which is the 95% of the final-checkpoint average of No Curriculum. We also added the Dynamic Sampling from the DAPO[3] paper as you request. We report the training time in hours to reach the with mean and std over 3 seeds:
| Setting | No Curr. | Dynamic Sampling | SEC | ADCL | PCL | METIS | vs. No Curr. | |
|---|---|---|---|---|---|---|---|---|
| Math, Qwen3-8B | 32.6 | 8.6 ± 0.9 | 7.1 ± 0.6 | 7.5 ± 0.7 | 10.8 ± 1.1 | 6.7 ± 0.6 | 5.1 ± 0.3 | 1.69× |
| Math, Qwen3-4B | 25.3 | 4.8 ± 0.4 | 4.5 ± 0.5 | 5.3 ± 0.4 | 5.5 ± 0.5 | 4.1 ± 0.2 | 3.4 ± 0.3 | 1.41× |
| Math, Llama-3.1-8B | 20.1 | 6.8 ± 0.7 | 6.2 ± 0.9 | 5.0 ± 2.6 | 12.0 ± 1.0 | 10.5 ± 0.7 | 5.4 ± 1.1 | 1.26× |
| Code, Qwen3-8B | 49.0 | 10.5 ± 1.0 | 8.8 ± 0.7 | 8.5 ± 0.5 | 13.8 ± 1.0 | 9.0 ± 0.8 | 6.7 ± 0.5 | 1.57× |
| Code, Llama-3.1-8B | 37.0 | 7.8 ± 0.6 | 6.1 ± 0.7 | 8.6 ± 0.7 | 10.7 ± 0.7 | 5.5 ± 0.9 | 4.4 ± 0.8 | 1.77× |
| Code, DS-R1-Distill | 46.1 | 12.0 ± 1.0 | 10.0 ± 0.9 | 11.3 ± 0.9 | 16.0 ± 1.2 | 9.4 ± 0.7 | 7.6 ± 0.6 | 1.58× |
METIS is the fastest method in five out of six settings with up to 1.77× faster than No Curriculum (and also outperforms the Dynamic Sampling baseline). We have updated the abstract, Table 1 in Sec. 5.1 and Table 8 in Appendix B.1 with analysis accordingly in the revised manuscript.
W2(1) / Q2: Parameter sharing vs. internalization
Thank you for this question. Table 2 already contains the experiment you suggest. An external ICL judge whose parameters are a snapshot of the policy is identical to row 4 (External, no judgment learning) when the snapshot is taken once (and frozen) at initialization, and in row 2 (Internal, no judgment learning) when it is refreshed at every step. These two reach 26.9 and 28.4 with the same exemplars, so the judge improves as its parameters stay closer to the current policy. This is what we mean by internalization: the judgment needs the state of the current policy and letting the policy produce the judgement is the simplest way to provide that state, without maintaining and syncing a second model. We have clarified this in Sec. 5.2 in the revised manuscript.
W2(2) / Q3: The role of
Thank you for this question. The effect of is shown in Figs. 9 and 10 (Appendix B.3 and B.4). With , the mean (which can be interpreted as prediction accuracy) rises from 0.54 to 0.85 over training and the predictions stay differentiated across candidates, whereas with the mean prediction stays flat and its spread shrinks.
The exemplars alone does not make METIS effective, since the same exemplars read by a frozen external model give no gain (External, no judgment learning, row 4 of Table 2). Therefore we see the judgment does depend on the weights of the policy. We have clarified this in Sec. 5.2 in the revised manuscript.
W2(3): ICL Capacity boundary
We state this capacity boundary of ICL as a limitation in Sec. 6 and Appendix B.6. METIS requires the backbone has the ability to follow the judgment instruction, which holds for the 4B and 8B models but not for Qwen3-1.7B (93.5% parse-failure rates). In that case METIS falls back to uniform selection and matches No Curriculum rather than hurting it (Table 13). We also state that the parse-failure rate can be checked before training as a cheap diagnostic.
Q5: Theoretical contribution
Thank you for mentioning this. We clarify that theoretical guarantees are not claims of this paper. Our primary contribution is the prompt-informativeness formulation, the internalized curriculum judgment method METIS, and extensive empirical validation. This scope is consistent with the literature, e.g., PCL[1], SEC[4], and ADCL[5], which are also validated empirically. We will clarify this scope/limitation in the revision.
We hope these clarifications and additions address your concerns.
[1] Prompt curriculum learning for efficient LLM post-training (ICLR 2026).
[2] VCRL: Variance-based Curriculum Reinforcement Learning for Large Language Models (arXiv 2025)
[3] DAPO: An Open-Source LLM Reinforcement Learning System at Scale (arXiv 2025)
[4] Learning like humans: Advancing LLM reasoning capabilities via adaptive difficulty curriculum learning and expert-guided self-reformulation (EMNLP 2025).
[5] Self-evolving curriculum for LLM reasoning (arXiv 2025).
A Gentle Message to Reviewer Dg1w
Dear Reviewer Dg1w,
We sincerely appreciate the time and effort you have devoted to reviewing our manuscript.
As the discussion period will close in less than 48 hours, we would be very grateful if you could share any comments or questions you may have at your earliest convenience. Receiving your feedback before the discussion deadline would allow us to respond promptly and address any concerns thoroughly.
We understand that this is a particularly busy review period and that you are likely handling many submissions, so we are especially grateful for your valuable time and expertise!
Best regards, The Authors
Response to Reviewer Dg1w (1/2)
We sincerely thank the reviwer for the careful reading and constructive suggestions. We respond to each point as following.
W1 / Q1(a): Accuracy at a matched gradient-step budget
Thank you for this suggestion. We have now evaluated all methods at the same final training step (step 150), which is the comparison you asked for. We also use more robust multi-sample evalution instead of pass@1 to reduce the evaluation noise:
The values below are mean value over the four benchmarks in each suite (mean and std over 3 seeds):
We see that METIS attains the highest average in all six settings with gains of 1.2–2.0 points over No Curriculum, including Llama-3.1-8B on Math where it was behind PCL under the original evaluation set-up. METIS therefore improves the policy at a matched number of updates, rather than only reaching a comparable endpoint faster. We have updated the abstract and Table 1 in Sec. 5.1 with analysis accordingly in the revised manuscript.
W1 / Q1(b): Comparison with VCRL
Following your suggestion we added VCRL [2] as a baseline. VCRL is currently an arXiv preprint without released code, so we re-implemented it from the paper with the variance threshold and memory-bank settings given in the paper. We evaluated it under the same protocol as above:
We see that METIS outperforms VCRL on both model backbones. In our setup VCRL improves over No Curriculum but not following the exact number reported in the original VCRL paper. We believe the difference comes from the benchmark suite and differnet hyper-parameters for training and inference. We also note that the gains reported for VCRL are measured per step without normalizing for compute (which is a concern that was also raised in its public reviews on OpenReview). This indicates that VCRL step requires more rollouts than a METIS step because it filters prompts after rollout, while METIS instead predicts the variance before rollout and also covers continuous rewards. Also another advantage of METIS as suggested by the reviewer is that METIS works in both discrete and continuous reward settings while VCRL only solves discrete rewards. We have added the additional experiment result with VCRL in Appendix B.1 in the revised manuscript.