- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
SURGE is an on-policy distillation method that applies teacher log-probability feedback only at tokens that pass two gates: a low process score on newline-delimited steps, and high student next-token entropy. A frozen copy of the initial student supplies the step scores and a response-wide reference KL term. The main experiments distill Gemma4-31B into Gemma4-E4B; a second pairing distills Qwen3.5-122B-A10B into Qwen3.5-2B. The paper reports higher point-estimate accuracy than dense OPD at lower estimated distillation FLOPs, with gate, masking, and reference-KL ablations.
Critical strengths
1.The Gemma accuracy gains versus dense OPD are internally consistent. On the five splits in Table 5, SURGE’s reported accuracy is higher than the paper’s dense OPD run on every task, and the abstract’s 8.61%, 40.93%, 49.89%, and 66.60% figures recompute from Table 5 under Appendix G. That at least shows the main table is not an arbitrary collage, and that restricting teacher supervision to selected positions can produce a reported accuracy increment in this setting.
2.Selective teacher supervision is a useful empirical question. The recipe is restatable: student on-policy tokens, teacher log-probabilities only where the gates fire, student-only inference. The claimed compute saving should be marked unverified rather than counted as a demonstrated efficiency result.
Critical weaknesses
1.Empirical validity and computational attribution are not established. The efficiency claim is that gated entropy reduces teacher compute. Appendix D already states that the supervision mask is not a multiplier on teacher-body cost: the teacher still forwards the full prefix at selected positions, while the output projection may be counted only there. Table 2’s densest task, Competition Math (63.16% active), is the cheapest versus OPD (−74.9%), while the sparsest, DynaHate (28.51% active), saves only 35.6%. Cross-task sparsity-versus-saving is not a causal identification, because sequence length, update count, and the body versus vocabulary-projection mix can differ. Evaluation lengths of 334 vs 342 are not training FLOPs. Final configurations are written as not retained. The paper therefore does not show, under matched training budgets, that the reported FLOP drop is an effect of sparse routing.
The evaluation populations also change how the numbers should be read. Table 10 lists GPQA Diamond as 178 train + 198 evaluation; the public Diamond set is typically 198 items. Competition Math is cited as Hendrycks MATH (7,500/5,000) but run as 10,000/2,500. MMLU-Pro is split 9,625/2,407 from the official 12,032-item pool. A documented, disjoint custom split used by every compared method can be a valid experiment; it is not the standard leaderboard. The sharpest numerical issue is the Qwen GPQA figures 42.50 / 52.50 / 77.50: a single equal-weight binary score on 198 items would imply 84.15 / 103.95 / 153.45 correct, none of which is an integer. That is a denominator inconsistency (another N, a mean over runs, or a reporting error) that the authors should resolve; it is not, as it stands, confirmed contamination.
2.The proposed mechanism is not isolated. The micro-gate H > τ drops low-entropy, high-divergence positions. TIP, which this paper cites, treats both high-entropy tokens and low-entropy tokens with large teacher–student disagreement as informative, so SURGE keeps the first region and misses the second; that is incomplete coverage, not the opposite of TIP. The tables also do not show that high-entropy selection alone retains the dense corrective signal. The macro-gate scorer is the frozen initial student, with no gold labels or external verifier (§3.1, Appendix E.1). That is self-scoring or a weak process-quality proxy: the live questions are whether it carries information beyond student confidence, whether it reinforces the student’s own errors, and how much it adds over an entropy-only gate — not whether the name PRM is allowed. Strict masking in Table 3 also replaces the reduction denominator Z with A (Appendix E.2), mixing coverage with normalization. In Table 4, β=0 drives Math and MMLU accuracy to 0 while active tokens drop to 0. Collapse is evidence that this recipe needs the reference term to keep teacher feedback alive; it does not yet separate a generic reference regularizer from a SURGE-specific gating–regularization interaction. Related work lists SE-KD, TIP, TA-OPD, and TGOPD, but the tables compare only dense OPD, SFT, GRPO, and a self-score PRM that also uses fewer mean FLOPs than SURGE (8.74e16 vs 1.89e17).
Questions for authors
1.Under matched training configs (same length caps, epochs / update counts, and the same FLOP decomposition), can retained run logs rather than approximate ranges show that the FLOP gap versus OPD comes from fewer teacher output positions rather than other training-cost differences? Does Appendix D separate teacher-body FLOPs from output-projection FLOPs counted only at selected positions?
2.How were the 178 GPQA Diamond training items chosen relative to the 198 evaluation items and to the public 198-item Diamond set? Are train and eval disjoint? The Qwen accuracies 42.50 / 52.50 / 77.50 cannot be a single equal-weight binary score on 198 items; is the denominator 200, 80, or a mean over runs?
3.The micro-gate omits TIP’s second region (low-entropy, high-divergence). Is there a matched-teacher-budget comparison to entropy-only selection, or to putting that second region back? How much independent gain does the macro-gate add over entropy-only?
4.Can the strict-mask versus SURGE gap be rerun with the same reduction (Z versus A) so that only inactive-position reference coverage changes? For Table 4, if β=0 is not allowed to drive the active set to empty, is the reference term still necessary — and is the collapse a generic regularizer failure or an interaction with this gate?
2 comments
Follow-up on compute accounting and the new GPQA runs
Thank you for the matched GPQA runs and the admissions on the original 178 split and Qwen denominators. I am keeping Rating 1. The new results recast the two central claims rather than supporting them.
Under the matched GPQA protocol, SURGE matches dense OPD at 20.37% (37/42/42 vs 40/39/42) but does not save work: 15,897 teacher rows / 362.68 vs 3,072 / 507.38 (eager) or 373.19 (shortcut). Fewer output rows is not a FLOP saving. You already write that the share attributable to routing remains unresolved.
The dual-gate controls also do not isolate the method. The joint gate changed 9 of 9,216 entropy selections (20.37% vs entropy 21.55%). Ranking vs shuffle is +0.17 [-2.69, +3.03]. With a shared , the reference term is not needed (33.85% vs 37.24%). The original 178 overlap and Qwen 42.50 / 52.50 / 77.50 remain unresolved.
Three questions:
- Do any logs remain for the original Gemma Table 5 runs? If not, will the 49.89% abstract saving be recast as an unmatched run-level difference?
- Please post the truncated J.14 / J.15 numbers (SE-KD, TA-OPD, TGOPD, TIP) under the same protocol.
- Is the quota shortcut and cache in the submitted method, or only in the rebuttal?
Thank you for checking the Gemma numbers and for pressing us on the compute accounting and GPQA split. We appreciate your point that sparse teacher feedback is worth testing under matched settings.
W1 / Q1. Do fewer active positions explain the reported FLOP gap?
We measured the reduction in the output projection on identical inputs. On 32 Qwen3 MATH rollouts, both implementations processed 9,216 teacher context positions. Gathering before projection reduced the output rows from 6,088 to 571 (J.1). Appendix D counts teacher context processing and output projection separately.
On public GPQA, we matched the 128 rollouts, 32 updates, optimizer and 256-token training cap. The table gives means across three seeds.
SURGE projected 80.68% fewer teacher rows at the same mean accuracy. The shortcut and cache cut eager SURGE's estimated work by 26.45% and preserved all 96 logged objectives and gradient norms, and all per-question grades. Total work was still 2.90% above dense OPD (J.7, J.11). We report 74.90% for MATH and 37.25% for GPQA as original run-level differences. The share attributable to routing remains unresolved. We agree that the original self-scoring PRM is cheaper (8.74 × 10¹⁶ versus 1.89 × 10¹⁷ reported mean FLOPs), with SURGE 2.81 accuracy points higher on average (I.2).
W1 / Q2. How were the GPQA splits and denominator determined?
The source and overlap of the original 178 training items remain unresolved. We agree that 42.50%, 52.50% and 77.50% are incompatible with one equal-weight binary evaluation on 198 questions. The original Qwen3.5 denominator remains unresolved, and we exclude that row from the cross-family summary (I.2).
For the new runs we removed Diamond from GPQA Main by record ID and normalized question text, trained on 128 Main questions and evaluated all 198 Diamond questions. SURGE got 37, 42 and 42 correct across the three seeds, and dense OPD got 40, 39 and 42 (J.7). The original Competition Math (10,000/2,500) and MMLU Pro (9,625/2,407) use the study splits in Appendix C.
W2 / Q3. What does the macro-gate add, and what about low-entropy disagreement?
On labeled steps, the Gemma student scored 0.828 AUROC versus 0.499 for entropy-based confidence. Its fixed cutoff caught 32 of 96 incorrect steps. Whether specific missed errors persist through training remains open. On Qwen3, the student scored 0.509 versus 0.515 for confidence (J.13). In the sampled GPQA runs, the joint gate changed nine of 9,216 top-entropy selections and scored 20.37% versus 21.55% for entropy. Ranking scored 24.07% and shuffling scored 23.91%, a difference of +0.17 points [−2.69, +3.03] (J.11).
With the sampled loss on Gemma, ranking beat shuffling by 8.85 points on boxed accuracy. We kept those grades and recovered answers from uncapped responses missing a box. The difference moved to −1.30 points [−5.47, +2.86], showing how much the measured advantage depended on the requested format. Under shared categorical reverse KL, this check narrowed Gemma's ranking deficit against entropy from 35.16 to 2.08 points and against TIP from 35.94 to 2.60 points. On GPQA, ranking still trailed entropy, TIP and shuffling under both grading rules (J.16).
We agree that the entropy threshold misses confident disagreements. We reserved 19 of 96 GPQA positions per update for low-entropy tokens and compared forward-KL ranking with uniform draws from the same region. They scored 22.05% and 20.88%, respectively (difference +1.18 points [−1.52, +3.87]). Ranking needed roughly five times the teacher projection rows of eager SURGE (J.14). J.14 and J.15 also report SE-KD, TA-OPD and TGOPD. TIP
W2 / Q4. Does the reference term still help when the gate stays active?
With the same valid-token denominator Z and 192 teacher positions per update on Qwen3 MATH, full coverage scored 33.85% and active-only coverage scored 37.24% (difference −3.39 points [−7.81, +1.04]). Mean pre-clipping norms were 0.486 and 0.480, and both arms stayed below the clipping limit. The β=0 sparse arm kept the full quota in every update and scored 35.16%. Dense OPD scored 34.64% with reference regularization and 35.42% without it. Enforcing the quota avoided the original collapse in this setup. The difference between the sparse and dense reference effects was −0.52 points [−5.47, +4.43] with signs varying across seeds (J.8). The intervals resample questions with training seeds fixed.
We hope you will reconsider your score in light of the revised compute claim and the new controls. Please let me know if there are any other problems you'd like me to address before the discussion period ends.