acceptodds
Under review as a conference paper at ICLR 2027

Learning from Error Structure: Selective Supervision for Reasoning Post-Training

Abstract

Reinforcement learning with verifiable rewards (RLVR) and supervised finetuning provide complementary signals for reasoning post-training. As recent approaches increasingly combine them, a key question is where additional supervised correction is most valuable. Existing approaches often rely on rollout accuracy, using failure frequency as a proxy for supervision need. Our empirical analysis shows that this reliance on rollout accuracy can mask important differences in failure structure. Even at the same rollout accuracy, errors may either repeatedly collapse onto the same semantic mode or span diverse reasoning strategies. Motivated by this observation, we introduce LESS (Learning from Error Structure: Selective Supervision), which allocates auxiliary supervision according to recurrent failure structure. LESS groups incorrect rollouts by their solution method and final answer, measures error concentration with semantic entropy, and selectively applies verified reference demonstrations to recurrent-error questions while retaining standard GRPO for all rollouts. Experiments with Qwen2.5-Math-7B show that LESS significantly improves performance relative to existing methods across multiple mathematical, scientific, and broader reasoning benchmarks. These results suggest that effective supervision depends not only on how often a model fails, but also on how it fails.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Models, Reasoning, Reinforcement Learning, Supervised Fine-Tuning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_bsADOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes LESS, a selective-supervision method built on GRPO. Every rollout group receives the regular RL update. When a question's incorrect rollouts concentrate in a few method–answer clusters, its verified solution enters a FIFO buffer for an auxiliary SFT update after GRPO. A fixed guide summarizes methods, extracts answers, and judges semantic equivalence.

The motivation is that equal rollout accuracy can reflect different learning states: a model may be exploring diverse unsuccessful approaches or repeatedly using one failed approach. LESS uses the structure of the errors to allocate demonstrations to the latter case.

Critical strengths
  • The paper identifies a meaningful limitation of accuracy-based supervision allocation: questions with the same rollout accuracy may have very different error structures and may not benefit equally from an additional demonstration.
  • LESS is easy to integrate into a GRPO pipeline, and all selected questions remain in the RL pathway. The matched-budget comparisons against random, reverse, low-accuracy, and rollout-entropy selection make the evaluation more informative than a comparison with pure RL or pure SFT alone.
Critical weaknesses
  • The current ablations do not isolate or validate the proposed error-structure signal. Incorrect rollouts are first partitioned by exact normalized answer and then clustered by method, so final-answer recurrence may account for much of the routing decision. Answer-only, method-only, and method–answer entropy should be compared under the same realized SFT budget. The fixed guide also controls summaries, answer normalization, and semantic-equivalence judgments, while the clustering is order-dependent and uses only eight rollouts per question. A human audit and stability analysis are needed to establish that the selected questions reflect meaningful error structure.
  • The evidence does not yet establish the reliability of the reported gain or the stability and cost of interleaved training. LESS improves substantially over pure SFT and pure RL, but its margins over matched selectors are smaller. The paper reports fixed-checkpoint sampling variation rather than independent training runs. It also provides no trajectories showing what happens immediately before and after auxiliary SFT updates, despite alternating GRPO and SFT without a reported KL penalty. Finally, the guide-inference cost is omitted from the resource comparison.
Questions for authors
  • Assessment-critical evidence: Under the same realized number of demonstrations and SFT optimizer steps, how do answer-only, method-only, and method–answer entropy compare? How stable are the resulting selections under human annotation, rollout-order permutations, guide reruns, prompt changes, or a second guide?
  • Assessment-critical evidence: Do LESS and the closest matched-budget selectors maintain their ranking across independent training seeds? Please also provide curves with SFT updates marked, including held-out reward, policy KL, response entropy or length, SFT loss, and immediate pre/post-SFT changes.
  • Clarification and cost accounting: How was alpha=0.2 chosen, what were the realized admission rate, number of unique demonstrations, and total SFT optimizer steps, and what additional guide calls, tokens, latency, and cost did LESS require?
3 comments

We thank the reviewer. We present additional routing, supervision-response, and training-dynamics evidence.

Representation and routing reliability

LESS groups incorrect rollouts by normalized answer and then clusters methods within each answer group. This distinguishes different approaches producing the same wrong answer. In an offline analysis of 2,738 historical batches using identical fixed per-batch quotas, Method–Answer and Answer-only routing differed in 4,312 of 36,751 selected occurrences (11.73%; Jaccard 0.790). This changes selections, but does not establish end-to-end superiority over simpler representations.

Two annotators blindly judged 500 incorrect-trajectory pairs (Cohen's κ = 0.74). Against adjudicated labels, gpt-5.4-mini achieved F1 = 0.90 (7.0% false merges, 10.7% false splits; 4% unresolved pairs excluded). Across 200 batches, changing joint-summary input order with fixed clustering order changed 1.8% of selection decisions; changing greedy clustering order alone changed 1.5%. Replacing the guide with gpt-6-sol gave Jaccard 0.88 and 2.6% flips, falling to 0.5% beyond a 0.05 entropy distance from the cutoff.

Table R3. Guide agreement and selection sensitivity.

Audit or perturbationResult
Human agreement / guide F1κ = 0.74 / F1 = 0.90
Summary-order / clustering-order flips1.8% / 1.5%
Cross-guide Jaccard / flips0.88 / 2.6%
Cross-guide flips away from cutoff0.5%

These results concern selection stability, not downstream performance. Independent summarization and fallback frequency were not measured.

Direct benefit from additional SFT

We stratified questions by initial eight-rollout accuracy (0/8–6/8), then independently applied one SFT update using one reference demonstration per question from the same checkpoint. Model/optimizer states were reset; no GRPO was performed. Gains were evaluated with 32 fresh generations before and after SFT. Across 185 questions in each entropy group, low entropy produced larger gains in six of seven strata. Equal-strata gains were +3.13 versus +0.83 pp (difference +2.30 pp; 95% CI [+0.21, +4.38]); the question-weighted difference was +2.40 pp (CI [+0.41, +4.38]). Each stratum CI includes zero; this tests immediate response, not unseen-question generalization.

Interleaved training dynamics

Figure R1 (Figure_R1_LESS_training_dynamics_6panel.png) shows the actual 600-step Qwen2.5-Math-7B LESS run, with all 160 SFT update steps marked. Held-out accuracy rises from approximately 49% at step 50 to 57% at step 600. SFT loss generally declines with fluctuations, and response-token entropy varies without sustained collapse. PPO approximate KL is near zero for much of training but has pronounced late spikes, the largest near step 600.

The bottom panels report the change at the next rollout following each SFT-associated step:

ΔMt=Mt+1−Mt.\Delta M_t=M_{t+1}-M_t.

Most KL changes are close to zero with occasional outliers; entropy changes are of both signs. Because GRPO and SFT updates are interleaved, these are descriptive diagnostics, not isolated causal effects of SFT. PPO approximate KL is not fixed-reference KL, and the GRPO configuration uses no explicit KL penalty.

Seeds and supervision budget

Two additional explicitly seeded LESS runs (42, 123) achieved five-benchmark step-300 averages of 48.88 and 48.80, versus 48.70 for a prior default-configuration run (sample SD ≈ 0.09 pp). The earlier run lacks full seed records. This does not establish significance of final margins over the matched selectors.

Table 5 reuses the original Table 1 LESS result. The main run naturally completed 160 FIFO-triggered SFT updates of 64 demonstrations each (10,240 exposures), without an ablation cap. The four alternative routers were capped at 160 updates. All five consumed 10,240 exposures (36.94–39.02M SFT loss tokens); LESS used 8,876 unique demonstrations, 1,364 repeat exposures, and 37,704,844 loss tokens. Its router admitted 10,240/37,950 eligible occurrences, or 26.98%, rather than nominal α = 0.20; inclusive ties increased counts in 189/600 steps. Among tested α values, 0.20 scored 51.0/61.4 (math-science/OOD), versus 41.9/52.8 at 0.50 and 42.4/53.8 at 0.80. These α runs were not budget-matched, so performance changes also reflect supervision volume. This cap is not part of Algorithm 1.

Cost

The guide required 54,077 logical calls, approximately 165.35M input and 3.46M output tokens, and an estimated $139.56 API cost. Routing blocking totaled 16.92 minutes in the 62-hour local run. GPU-hours were 496 for LESS, 416 for ReLIFT, and 584 for LUFFY (external guide computation excluded). LESS incurs greater local GPU and API costs than ReLIFT.

Dear Reviewer,

Hope you are doing well. As the discussion period is coming to an end soon, we would greatly appreciate it if you could let us know whether our responses have properly addressed your concerns. Please feel free to share any further comments or remaining questions, and we will promptly provide additional clarifications to improve our manuscript. Thank you very much for your time and valuable feedback.

Thanks for the additional experiments. The one-step SFT test is helpful, and the guide checks address much of my concern about clustering reliability. I am still unsure how much the method summaries contribute, though. The 11.73% difference shows that they change which questions are selected, but not whether those changes improve training. Do you have a budget-matched comparison with answer-only grouping?

The new seed results are also encouraging, but they only cover LESS at step 300. My concern is whether its advantage over the other selectors, especially Low Acc., holds at the final checkpoint across seeds. Could you provide that comparison?

One smaller point: since the α experiments use different amounts of supervision, please qualify the claim that the drop comes from including higher-entropy questions.

Reviewer_1pDwOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how to allocate auxiliary supervised demonstrations during reasoning post-training with RLVR. It argues that rollout accuracy alone can obscure differences in failure structure: questions with the same success rate may exhibit either recurrent or diverse incorrect reasoning patterns. LESS constructs method-answer representations of incorrect rollouts, clusters semantically equivalent representations using a fixed external guide, and computes normalized error-only semantic entropy. Questions in the batch-relative low-entropy tail contribute verified reference demonstrations to an auxiliary SFT buffer, while all rollout groups remain in the GRPO update. The demonstrations come from a filtered DeepSeek-R1-generated math dataset; the external guide supplies the routing signal. The main Qwen2.5-Math-7B experiments report improvements in aggregate performance across mathematical/scientific and broader reasoning benchmarks. Additional model comparisons and allocation ablations support the usefulness of the proposed signal, although its robustness and the proposed explanation of supervision value warrant further examination.

Critical strengths
  1. The paper identifies a useful distinction between failure frequency and failure structure. Figure 2(a) demonstrates that identical rollout accuracy can correspond to different semantic error-cluster configurations. This motivates looking beyond correctness statistics when allocating additional supervision.

  2. LESS provides a straightforward way to incorporate this signal into a hybrid training pipeline: every rollout group remains in GRPO, while selected questions receive auxiliary SFT. The empirical evidence includes independently reproduced GRPO, ReLIFT, and LUFFY comparisons under a shared main training protocol. Under the reported matched auxiliary-SFT budget, Table 5 also shows improvements over lowest-accuracy selection. These results support a practical contribution beyond the motivating observation.

Critical weaknesses
  1. The reliability of the routing signal needs further validation. LESS relies on an external guide to summarize incorrect traces and judge semantic equivalence. Appendix A.3 acknowledges rollout-order dependence and a string-matching fallback when guide calls or parsing fail, but their effects on question selection and final performance are not quantified. With eight rollouts per question, a small manual audit and sensitivity checks for rollout order and guide choice would strengthen confidence in the signal.

  2. The results support low-entropy allocation, but more direct evidence is needed that these questions benefit more from additional supervision. Figure 2(c) and Table 5 are informative, although the Random baseline samples only zero-accuracy questions, while LESS selects from questions with at least two incorrect rollouts. Low Acc. provides a better control, but comparing low- and high-entropy questions at the same rollout accuracy under matched SFT exposure would better isolate the value of error concentration. The paper states that SFT budgets are matched; reporting the actual demonstrations and tokens consumed would make this easier to assess.

Questions for authors
  1. How stable are entropy rankings and selected-question sets under rollout-order permutations and reasonable changes to the guide or summary prompt? Could a small manually checked sample also quantify clustering agreement and the frequency of fallback/indeterminate cases?

  2. For the Table 5 ablations, how is the matched SFT budget enforced in practice? Please report the actual supervised updates, consumed demonstration examples/tokens, and admission rates. Within the same rollout-accuracy strata, do low-entropy questions benefit more from matched SFT exposure than higher-entropy questions? This would strengthen the interpretation of error concentration as an allocation signal.

  3. Can the authors clarify the GPQA subset and scoring protocol for every method in Table 1? Appendix A.2 explicitly identifies GPQA-Diamond for RL w/ SFT Loss and SFT then RL, but does not make the corresponding subset equally explicit for the other entries. Also, is there evidence that the gains over Low Acc., ReLIFT, and LUFFY persist across training seeds? The fixed-checkpoint decoding analysis in Appendix C.2 addresses a different source of variability.

  4. Table 3 reports 13.3 for LESS on LLaMA-3.1-8B-Instruct, compared with 17.1 for the reported Instruct model, despite LESS outperforming the reproduced RL baseline at 9.6. Are these entries directly comparable under the same evaluation settings, and what explains the remaining degradation from initialization? This would clarify the limits of the cross-family result.

  5. Could the authors report guide-call counts, input/output tokens, API cost, and end-to-end latency, and clarify what is included in Table 2's GPU-hours? Resource comparisons with ReLIFT and LUFFY would also help assess the performance-cost trade-off. How does the guide overhead change as the rollout count increases?

Clarification of routing robustness, supervision-budget matching, and evaluation comparability would materially affect my assessment.

3 comments

We thank the reviewer. We address routing stability, matched supervision, evaluation, and resources.

Q1. Guide agreement and selection stability

Two annotators blindly evaluated 500 incorrect-trajectory pairs (Cohen's κ = 0.74). Against adjudicated labels, gpt-5.4-mini achieved F1 = 0.90 (7.0% false merges; 10.7% false splits); 4% unresolved pairs were excluded. Across 200 batches, changing joint-summary input order changed 1.8% of selections (fixed clustering order); changing greedy clustering order alone changed 1.5%. Replacing gpt-5.4-mini with gpt-6-sol gave selection Jaccard 0.88 and a 2.6% flip rate, decreasing to 0.5% for questions more than 0.05 normalized-entropy units from the original cutoff. These indicate selection stability, not invariance. Exact-string fallback is described in Appendix A.3; independent summarization and fallback frequency remain unmeasured.

Q2. Direct supervision benefit and matched budgets

We compared low- and high-entropy questions at matched initial accuracy (0/8–6/8). Each question received one independent SFT update using one demonstration from the same checkpoint, with model/optimizer resets and no GRPO. We evaluated accuracy using 32 fresh generations before and after SFT. Across 185 questions per group, low entropy had a larger gain in six of seven strata. Equal-strata gains were +3.13 pp (Low SE) and +0.83 pp (High SE), a +2.30 pp difference (95% bootstrap CI [+0.21, +4.38]); the question-weighted difference was +2.40 pp (CI [+0.41, +4.38]). Individual stratum intervals include zero. This measures immediate, not unseen-question, gains.

Table 5 reuses the main LESS run, which naturally completed 160 FIFO-triggered updates of 64 demonstrations. The four alternative routers were capped at 160 updates to match its 10,240 realized exposures. Table R2 reports actual SFT consumption.

Table R2. Realized auxiliary SFT budgets; loss tokens exclude prompts and padding.

MethodUnique demosExposuresRepeatsUpdatesLoss tokens
LESS8,87610,2401,36416037,704,844
Full SE8,96310,2401,27716039,023,971
Random9,12910,2401,11116038,184,000
Reverse9,05510,2401,18516036,936,000
Low Acc.8,26210,2401,97816038,612,000

The original LESS run admitted 10,240 of 37,950 eligible question occurrences:

radmission=10,24037,950×100%=26.98%.r_{\mathrm{admission}}=\frac{10{,}240}{37{,}950}\times100\%=26.98\%.

The nominal α = 0.20 can be exceeded because boundary ties are retained; selection counts increased in 189/600 steps. Update counts and exposures match across methods, whereas unique coverage, token counts, and update timing may differ. Random samples 20% of 0/8-accuracy questions, whereas Low Acc. shares LESS's eligibility and quantile mechanism. The α sensitivity runs in Table 4 were not matched for actual supervision volume.

Q3. GPQA protocol and independent seeds

Every GPQA result in Table 1 refers to GPQA-Diamond (198 questions). For independently evaluated methods, we permute the four options and remap labels, sample one answer per question at temperature 0.6 with an 8,192-token limit, and score the extracted option by exact match; invalid options count as incorrect. Earlier RLVR scores come from LUFFY and were not independently rerun. Additional LESS runs with seeds 42 and 123 gave five-benchmark step-300 averages of 48.88 and 48.80 versus 48.70 for a previous default-configuration run (sample SD ≈ 0.09 pp). The earlier run lacks a complete seed record; this is not a matched-seed test of final baseline margins.

Q4. LLaMA initialization

We confirm that LESS and the independently trained RL baseline were initialized from LLaMA-3.1-8B-Base, not Instruct. We will correct Section 4.2 and Appendix A.1. With the same Base initialization, LESS scores 13.3 versus RL's 9.6 (+3.7 pp). The 17.1 Instruct result is a separately reported reference from LUFFY, not the initialization of LESS; LESS does not surpass that reference. We will clarify the Table 3 caption and result provenance.

Q5. Guide overhead and scaling

The main run used 54,077 logical guide calls (34,223 summaries; 19,854 equivalence judgments), around 165.35M input and 3.46M output tokens, with $139.56 estimated API cost. Routing waiting plus synchronous preparation totaled 16.92 minutes (0.455% of 62 hours); 11.14 hours of background routing execution largely overlapped with training. Local GPU-hours were 496 for LESS (8×62), 416 for ReLIFT (8×52), and 584 for LUFFY (8×73); guide inference costs are additional. Summary inputs grow with rollout count/length; greedy clustering can require O(K²) comparisons for K errors. Costs were measured at K = 8.

Dear Reviewer,

Hope you are doing well. As the discussion period is coming to an end soon, we would greatly appreciate it if you could let us know whether our responses have properly addressed your concerns. Please feel free to share any further comments or remaining questions, and we will promptly provide additional clarifications to improve our manuscript. Thank you very much for your time and valuable feedback.

Thank you for the additional experiments. My remaining concern is whether the advantage reported in Table 1 persists across training seeds. On the six in-distribution benchmarks, LESS averages 51.0, compared with 49.5 for both ReLIFT and LUFFY. Table 1 does not specify how many training seeds contributed to each score, and the additional seed results at step 300 cover LESS alone. Given the uncertainty about the reproducibility of this 1.5-point gain, I maintain my rating of 2.

Reviewer_fpMsOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper looks at which problems should get extra supervision when RLVR and SFT are combined for reasoning post-training. The main claim is that, for problems with the same rollout accuracy, reference demonstrations help more when the wrong answers mostly fall into one error mode. Based on this, the authors propose LESS. LESS runs standard GRPO on all rollouts. It also uses an external LLM (gpt-5.4-mini) to summarize each wrong rollout as a (solution method, final answer) pair, clusters these summaries, and computes an error-only semantic entropy. Problems in the bottom 20% of entropy in each batch then get an auxiliary SFT update on DeepSeek-R1 solutions. On six math and science benchmarks and three OOD benchmarks, LESS beats hybrid baselines such as ReLIFT and LUFFY, and it does so with fewer demonstrations (10K).

Critical strengths

The motivation is clear and intuitive. The observation that problems with the same rollout accuracy can fail in very different ways is convincing, and the paper backs it up by showing that these differences actually exist.

The method fits naturally into existing RL training. It keeps the GRPO update on all rollouts and only decides which problems receive additional SFT, so the roles of supervision allocation and RL training are cleanly separated.

The results suggest that using demonstrations selectively is a practical option. Based on the reported usage, the method uses only 10K demonstrations, compared with the full 46K used by SFT-based baselines, and still achieves higher average performance.

Critical weaknesses

W1 The central hypothesis, that lower error entropy predicts larger gains from additional SFT, is not tested directly. The comparison with accuracy-only selection helps, but it mixes the effect of entropy with other factors. A cleaner test would compare gains on low-entropy and high-entropy problems while holding rollout accuracy and the amount of additional SFT fixed.

W2 It is unclear whether the more involved pipeline is needed. Simpler criteria, such as answer-only entropy, the share of the most common wrong answer, or embedding-based clustering of the raw solutions, are not compared against. Without these baselines, it is hard to tell what the Method-Answer summaries and the LLM-based equivalence judgments actually add.

W3 Because rollouts are summarized jointly, each summary may depend on the other rollouts in the same context, which could affect both the clusters and the problems chosen for supervision. The paper does not examine how stable the clustering, the entropy ranking, and the final SFT set are when rollouts are summarized independently, or when the input order to the summarizer is changed while the clustering order stays fixed.

W4 The paper states that the SFT budget is matched across selection methods, but does not explain how this was implemented. With possible ties at the quantile boundary and the FIFO queue, the actual amount of training could differ between methods. Reporting the number of demonstrations, SFT updates, and training tokens actually used by each method would resolve this. It is also unclear whether the budget was controlled in the α experiments.

W5 The improvements over ReLIFT, LUFFY, and accuracy-only selection appear to come from a single training run. The decoding analysis in Appendix C.2 covers only one checkpoint and does not capture variance across training seeds. Results over multiple seeds, along with some measure of uncertainty on the differences between methods, are needed to show that the gains are reliable.

W6 The cost of the external guide is not reported. Method summarization and semantic equivalence judgments require additional API calls, so the number of calls, tokens, cost, and latency should be included. Without this, it is difficult to judge whether the gains over accuracy-only selection and other hybrid methods are worth the added cost.

Questions for authors

Please see the Weaknesses section.

I also have a small question: why did you choose GRPO as the main RL algorithm?GRPO is known to have length and difficulty biases from its normalization terms, so I would be curious to see whether the results still hold with Dr. GRPO, which removes these biases.

3 comments

We thank the reviewer. We provide a controlled test of the allocation hypothesis and clarify routing, budgets, and costs.

W1. Does lower error entropy predict greater SFT benefit?

We stratified questions by initial accuracy (0/8–6/8) from eight rollouts of a fixed checkpoint. Each low- or high-entropy question then received one independent SFT update on its own reference demonstration. Model and optimizer states were reset; no GRPO was performed. Gains were estimated from 32 fresh generations before and after SFT:

ΔSFT(q)=Apost(32)(q)−Apre(32)(q).\Delta_{\mathrm{SFT}}(q)=A_{\mathrm{post}}^{(32)}(q)-A_{\mathrm{pre}}^{(32)}(q).

Table R1. Accuracy-controlled immediate SFT gains, in percentage points (pp).

Initial acc.N low/highLow SEHigh SEDifference95% bootstrap CI
0/830/301.560.52+1.04[−1.56, +3.65]
1/830/304.061.04+3.02[−0.99, +7.03]
2/830/305.100.94+4.17[−1.25, +9.59]
3/830/305.630.73+4.90[−1.61, +11.41]
4/830/301.041.56−0.52[−6.15, +5.10]
5/822/222.840.57+2.27[−2.63, +7.17]
6/813/131.680.48+1.20[−6.61, +9.01]
Equal-strata185/1853.130.83+2.30[+0.21, +4.38]
Question-weighted185/1853.280.88+2.40[+0.41, +4.38]

Low-SE gains are larger in six of seven strata, and both aggregate confidence intervals exclude zero. Individual stratum intervals include zero. This supports immediate responsiveness, not unseen-question generalization.

W2. Why Method–Answer?

LESS groups incorrect rollouts by normalized final answer, then clusters methods within each answer group. Unlike Answer-only entropy, this distinguishes different approaches producing the same wrong answer. In an offline analysis of 2,738 historical batches under identical fixed per-batch quotas, switching from Method–Answer to Answer-only replaced 4,312 of 36,751 selected occurrences (11.73%; Jaccard 0.790). This changes routing but does not establish downstream superiority over simpler representations. We will make that limitation explicit.

W3. Guide reliability

Two annotators blindly judged 500 incorrect-trajectory pairs (Cohen's κ = 0.74). Against adjudicated labels, gpt-5.4-mini achieved F1 = 0.90, with 7.0% false merges and 10.7% false splits; 4% unresolved pairs were excluded. Across 200 batches, permuting joint-summary input order while fixing clustering order changed 1.8% of selection decisions; changing greedy clustering order alone changed 1.5%. Switching to gpt-6-sol gave Jaccard 0.88 and 2.6% selection flips, falling to 0.5% for questions more than 0.05 entropy units from the threshold. These results concern routing stability; independent summarization remains untested.

W4. Realized SFT budgets

Table 5 reuses the original Table 1 LESS run. Its FIFO procedure naturally completed 160 SFT updates with 64 demonstrations per update (10,240 exposures), without an ablation-specific cap. The four alternative selectors were capped at 160 updates to match this realized budget. All five consumed 10,240 exposures, while loss-token totals ranged from 36.94M to 39.02M. LESS consumed 8,876 distinct demonstrations, including 1,364 repeat exposures and 37,704,844 loss tokens. Its admission rate was 10,240/37,950 = 26.98%; inclusive quantile ties increased selections in 189/600 steps. Admission and SFT consumption are distinct counts, although their totals coincide in this run. Table 4's α experiments did not match realized SFT budgets, so differences can reflect both selection and supervision volume.

W5. Training seeds

Two additional LESS runs (seeds 42 and 123) scored 48.88 and 48.80 at step 300 on five math benchmarks; an earlier default-configuration run scored 48.70 (sample SD ≈ 0.09 pp). Its complete random state was not retained. These intermediate results do not establish uncertainty in the final margins over other methods.

W6. Guide cost

The guide made 54,077 logical calls (34,223 summaries; 19,854 equivalence judgments), consuming approximately 165.35M input and 3.46M output tokens, with an estimated API cost of $139.56. Routing waiting and synchronous preparation totaled 16.92 minutes (0.455% of 62 hours of local training); background execution largely overlapped with GPU work. Local GPU-hours were 496 for LESS, 416 for ReLIFT, and 584 for LUFFY. LESS achieved 51.0/61.4 math-science/OOD averages versus 49.5/58.7 and 49.5/57.7, respectively, but used more GPU time and external API resources than ReLIFT.

GRPO versus Dr. GRPO

We used GRPO to retain a common RL backbone for the reproduced comparisons. Our implementation omits length and standard-error normalization, but is not equivalent to Dr. GRPO. Transfer to Dr. GRPO remains untested.

Dear Reviewer,

Hope you are doing well. As the discussion period is coming to an end soon, we would greatly appreciate it if you could let us know whether our responses have properly addressed your concerns. Please feel free to share any further comments or remaining questions, and we will promptly provide additional clarifications to improve our manuscript. Thank you very much for your time and valuable feedback.

Thank you for the additional experiments and clarifications. However, W2 still does not establish that Method–Answer improves final performance over simpler selection criteria. The additional seed results in W5 concern an intermediate checkpoint and do not establish the reliability of the final gains over baselines. W4 also confirms that the α experiments mix changes in problem selection with changes in the SFT budget. These gaps leave the proposed pipeline’s added value and the robustness of its performance gains insufficiently supported. I therefore maintain my Weak Reject rating

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.