- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes LESS, a selective-supervision method built on GRPO. Every rollout group receives the regular RL update. When a question's incorrect rollouts concentrate in a few method–answer clusters, its verified solution enters a FIFO buffer for an auxiliary SFT update after GRPO. A fixed guide summarizes methods, extracts answers, and judges semantic equivalence.
The motivation is that equal rollout accuracy can reflect different learning states: a model may be exploring diverse unsuccessful approaches or repeatedly using one failed approach. LESS uses the structure of the errors to allocate demonstrations to the latter case.
Critical strengths
- The paper identifies a meaningful limitation of accuracy-based supervision allocation: questions with the same rollout accuracy may have very different error structures and may not benefit equally from an additional demonstration.
- LESS is easy to integrate into a GRPO pipeline, and all selected questions remain in the RL pathway. The matched-budget comparisons against random, reverse, low-accuracy, and rollout-entropy selection make the evaluation more informative than a comparison with pure RL or pure SFT alone.
Critical weaknesses
- The current ablations do not isolate or validate the proposed error-structure signal. Incorrect rollouts are first partitioned by exact normalized answer and then clustered by method, so final-answer recurrence may account for much of the routing decision. Answer-only, method-only, and method–answer entropy should be compared under the same realized SFT budget. The fixed guide also controls summaries, answer normalization, and semantic-equivalence judgments, while the clustering is order-dependent and uses only eight rollouts per question. A human audit and stability analysis are needed to establish that the selected questions reflect meaningful error structure.
- The evidence does not yet establish the reliability of the reported gain or the stability and cost of interleaved training. LESS improves substantially over pure SFT and pure RL, but its margins over matched selectors are smaller. The paper reports fixed-checkpoint sampling variation rather than independent training runs. It also provides no trajectories showing what happens immediately before and after auxiliary SFT updates, despite alternating GRPO and SFT without a reported KL penalty. Finally, the guide-inference cost is omitted from the resource comparison.
Questions for authors
- Assessment-critical evidence: Under the same realized number of demonstrations and SFT optimizer steps, how do answer-only, method-only, and method–answer entropy compare? How stable are the resulting selections under human annotation, rollout-order permutations, guide reruns, prompt changes, or a second guide?
- Assessment-critical evidence: Do LESS and the closest matched-budget selectors maintain their ranking across independent training seeds? Please also provide curves with SFT updates marked, including held-out reward, policy KL, response entropy or length, SFT loss, and immediate pre/post-SFT changes.
- Clarification and cost accounting: How was
alpha=0.2chosen, what were the realized admission rate, number of unique demonstrations, and total SFT optimizer steps, and what additional guide calls, tokens, latency, and cost did LESS require?
3 comments
Dear Reviewer,
Hope you are doing well. As the discussion period is coming to an end soon, we would greatly appreciate it if you could let us know whether our responses have properly addressed your concerns. Please feel free to share any further comments or remaining questions, and we will promptly provide additional clarifications to improve our manuscript. Thank you very much for your time and valuable feedback.
Thanks for the additional experiments. The one-step SFT test is helpful, and the guide checks address much of my concern about clustering reliability. I am still unsure how much the method summaries contribute, though. The 11.73% difference shows that they change which questions are selected, but not whether those changes improve training. Do you have a budget-matched comparison with answer-only grouping?
The new seed results are also encouraging, but they only cover LESS at step 300. My concern is whether its advantage over the other selectors, especially Low Acc., holds at the final checkpoint across seeds. Could you provide that comparison?
One smaller point: since the α experiments use different amounts of supervision, please qualify the claim that the drop comes from including higher-entropy questions.
We thank the reviewer. We present additional routing, supervision-response, and training-dynamics evidence.
Representation and routing reliability
LESS groups incorrect rollouts by normalized answer and then clusters methods within each answer group. This distinguishes different approaches producing the same wrong answer. In an offline analysis of 2,738 historical batches using identical fixed per-batch quotas, Method–Answer and Answer-only routing differed in 4,312 of 36,751 selected occurrences (11.73%; Jaccard 0.790). This changes selections, but does not establish end-to-end superiority over simpler representations.
Two annotators blindly judged 500 incorrect-trajectory pairs (Cohen's κ = 0.74). Against adjudicated labels, gpt-5.4-mini achieved F1 = 0.90 (7.0% false merges, 10.7% false splits; 4% unresolved pairs excluded). Across 200 batches, changing joint-summary input order with fixed clustering order changed 1.8% of selection decisions; changing greedy clustering order alone changed 1.5%. Replacing the guide with gpt-6-sol gave Jaccard 0.88 and 2.6% flips, falling to 0.5% beyond a 0.05 entropy distance from the cutoff.
Table R3. Guide agreement and selection sensitivity.
These results concern selection stability, not downstream performance. Independent summarization and fallback frequency were not measured.
Direct benefit from additional SFT
We stratified questions by initial eight-rollout accuracy (0/8–6/8), then independently applied one SFT update using one reference demonstration per question from the same checkpoint. Model/optimizer states were reset; no GRPO was performed. Gains were evaluated with 32 fresh generations before and after SFT. Across 185 questions in each entropy group, low entropy produced larger gains in six of seven strata. Equal-strata gains were +3.13 versus +0.83 pp (difference +2.30 pp; 95% CI [+0.21, +4.38]); the question-weighted difference was +2.40 pp (CI [+0.41, +4.38]). Each stratum CI includes zero; this tests immediate response, not unseen-question generalization.
Interleaved training dynamics
Figure R1 (
Figure_R1_LESS_training_dynamics_6panel.png) shows the actual 600-step Qwen2.5-Math-7B LESS run, with all 160 SFT update steps marked. Held-out accuracy rises from approximately 49% at step 50 to 57% at step 600. SFT loss generally declines with fluctuations, and response-token entropy varies without sustained collapse. PPO approximate KL is near zero for much of training but has pronounced late spikes, the largest near step 600.The bottom panels report the change at the next rollout following each SFT-associated step:
ΔMt=Mt+1−Mt.
Most KL changes are close to zero with occasional outliers; entropy changes are of both signs. Because GRPO and SFT updates are interleaved, these are descriptive diagnostics, not isolated causal effects of SFT. PPO approximate KL is not fixed-reference KL, and the GRPO configuration uses no explicit KL penalty.
Seeds and supervision budget
Two additional explicitly seeded LESS runs (42, 123) achieved five-benchmark step-300 averages of 48.88 and 48.80, versus 48.70 for a prior default-configuration run (sample SD ≈ 0.09 pp). The earlier run lacks full seed records. This does not establish significance of final margins over the matched selectors.
Table 5 reuses the original Table 1 LESS result. The main run naturally completed 160 FIFO-triggered SFT updates of 64 demonstrations each (10,240 exposures), without an ablation cap. The four alternative routers were capped at 160 updates. All five consumed 10,240 exposures (36.94–39.02M SFT loss tokens); LESS used 8,876 unique demonstrations, 1,364 repeat exposures, and 37,704,844 loss tokens. Its router admitted 10,240/37,950 eligible occurrences, or 26.98%, rather than nominal α = 0.20; inclusive ties increased counts in 189/600 steps. Among tested α values, 0.20 scored 51.0/61.4 (math-science/OOD), versus 41.9/52.8 at 0.50 and 42.4/53.8 at 0.80. These α runs were not budget-matched, so performance changes also reflect supervision volume. This cap is not part of Algorithm 1.
Cost
The guide required 54,077 logical calls, approximately 165.35M input and 3.46M output tokens, and an estimated $139.56 API cost. Routing blocking totaled 16.92 minutes in the 62-hour local run. GPU-hours were 496 for LESS, 416 for ReLIFT, and 584 for LUFFY (external guide computation excluded). LESS incurs greater local GPU and API costs than ReLIFT.