acceptodds
Under review as a conference paper at ICLR 2027

DistillScope: Benchmarking Flow-Map Distillation for Language Generation

Abstract

Distillation promises fast language generation, but a recipe's apparent advantage can depend on how quality and computation are measured. We introduce , a common training and evaluation framework for flow-map language-model distillation that examines how recipe comparisons change across quality criteria, training budgets, and inference settings. Semigroup distillation achieves lower external perplexity than Lagrangian distillation, while their ordering reverses under MAUVE. Denoising readout and length-extrapolation generation also favor different recipes. Accounting for measured training time reverses a recipe advantage observed at equal updates. Additional inference computation allows the same trained student to pass degeneration checks that it fails under tighter inference budgets. Initialization, objective ablations, and cross-environment comparisons delineate the scope of these findings. An exposure-matched sampling control and supporting analysis of composition and categorical readout clarify the distinctions between training allocation, map fidelity, and decoded quality. makes these dependencies explicit, supporting recipe choices tied to a stated quality target and computational constraint.

Primary area:
datasets and benchmarks
Keywords:
dllm, diffusion llms, flow maps, distillation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_ueDBOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces DistillScope, a controlled training/evaluation framework for flow-map distillation of (diffusion) language models. It re-implements three self-distillation recipes from Flow Map Language Models — semigroup (PSD), Lagrangian (LSD), Eulerian (ESD) — plus two local baselines, progressive-halving (PH) and damped fixed-point (FP), under a common map parameterization (Eq. 1) and evaluates each as a (checkpoint, inference-budget) unit on LM1B (length-128), with transfer studies on OpenWebText and masked settings.

The central thesis is methodological: the preferred recipe depends on the question asked. The paper documents several reversals: (1) at 100k updates / 8 NFE, PSD has 14.8% (Qwen2.5-1.5B) and 17.0% (GPT-2-large) lower geometric-mean perplexity than LSD, yet LSD wins on MAUVE (0.580 vs 0.493); (2) FP's equal-update GPT-2 advantage (185.4 vs 204.9) reverses to a PSD advantage once training time is matched (PSD gets 135,576 updates for FP's 100k); (3) the same LSD checkpoints fail the degeneration screen at 1 NFE but pass at 8 NFE. It adds degeneration diagnostics (entropy band, pooled entropy, rep-4), denoising-readout and length-extrapolation probes, an exposure-matched interval-allocation control, and two supporting propositions (finite propagation; categorical argmax-stability margin). Validation is via multi-seed runs (2–3 per recipe) with a permutation sensitivity analysis.

Critical strengths
  1. Honest, decision-relevant benchmark design. The unit of comparison — a trained checkpoint at a stated inference budget, under a stated quality criterion and cost convention — is the right abstraction, and the paper follows it rigorously (Section 2.3, Table 9). This directly operationalizes the DomainBed/AlgoPerf lesson (Gulrajani & Lopez-Paz 2021; Kasimbeg et al. 2025) that selection and resource conventions belong to the method being compared. The equal-updates vs equal-time reversal (Finding 4) is the most consequential single result: it shows that per-update cost differences alone can flip a published-style comparison.
  2. Separation of degeneration from quality. Decoupling the entropy/rep-4 screen from perplexity/MAUVE, and showing that a 1-NFE FP checkpoint can reach Qwen PPL ≈ 1.0 with near-zero within-block entropy (degenerate), is an important corrective. It substantiates the claim that low generative perplexity can be an artifact, and the 1-NFE→8-NFE recovery (Finding 5, Table 5) is a clean demonstration that inference budget changes the observed regime without retraining.
Critical weaknesses
  1. Underpowered headline claims. As above, the flagship PSD/LSD and FP results use 2–3 seeds and never attain significance under the paper's own permutation test (p = 0.10 floor). For a paper whose entire contribution is rigorous comparison, this is the most consequential weakness: readers cannot tell whether the reported 14.8%/17.0% gaps and the equal-time reversal are stable or seed noise. A benchmark paper should deliver effect sizes with uncertainty that survive replication.
  2. Limited external validity and no actionable output. The detailed results live on short-form LM1B (length-128), and PH/FP are explicitly "local baselines" rather than faithful reproductions of progressive distillation (Salimans & Ho 2022) or self-conditioned fixed-point flows (Yoo et al. 2026a) — so the benchmark cannot be read as ranking the published methods. The masked-OWT comparison (Finding 9) shows the evaluated PSD run trailing released SDTT by a large margin (620.1 vs 104.5 at matched 9.18B tokens), underscoring that these are configured recipes, not competitive systems. The paper's deliverable is a set of conditional observations rather than a reusable leaderboard, tooling, or recipe recommendation, which limits its impact as a "benchmark."
Questions for authors
  1. Can you add seeds (e.g. 5–10 per recipe) for at least the PSD/LSD 8-NFE comparison and the equal-time FP reversal, and report bootstrap CIs on the geometric-mean reductions? Do the reversals survive? (Clarification that materially affects my rating.)
  2. The equal-time comparison allocates PSD 135,576 updates for FP's 100k on "fixed hardware." How sensitive is the 7.1%/7.7% reversal to hardware, batch/micro-batch layout, and kernel/implementation efficiency of each recipe's target construction? Is the per-update cost ratio an intrinsic property of the objective or of your implementation?
  3. MAUVE is computed with GPT-2-large features and a fixed clustering seed on 5,000 samples. How stable is the PSD<LSD MAUVE ordering to the feature extractor, number of clusters, and MAUVE's known seed/quantization sensitivity?
  4. For the degeneration screen, the LM1B entropy band [3.553, 4.594] and rep-4 ≤ 0.008 are calibrated from reference students. How do the 1-NFE→8-NFE "recovery" conclusions change under reasonable perturbations of these thresholds?
  5. Will the release include trained checkpoints and the per-run metric banks (not just code), so the conditional comparisons are independently reproducible?
  6. Several concurrent few-step diffusion-LM distillation methods have appeared (e.g. Latent-Kernel Discrete Flow Maps, arXiv:2607.27529; Gumbel Straight Flow, arXiv:2610.00497; the code-generation few-step distillation of arXiv:2609.04531). Can the benchmark accommodate at least one as a faithful (non-"local") recipe, to show the framework ranks real methods rather than only your reimplementations?
6 comments
AuthorsOpenReview ↗

Response to ueDB [Part 1]

We sincerely thank the reviewer for the detailed and technically constructive evaluation. We appreciate your recognition of the benchmark's comparison unit, the importance of the equal-time reversal, and the distinction between degeneration diagnostics and generation quality. We agree that a study centered on reproducible comparisons particularly benefits from stronger training-run uncertainty estimates and sensitivity analyses. We address the statistical evidence first, followed by execution costs, MAUVE, screen calibration, reproducibility, and native baselines.

[Cons 1, Q1: Additional training seeds and uncertainty in the headline comparisons]

The submitted PSD–LSD result reports perplexity reductions of 14.8% under Qwen and 17.0% under GPT-2, together with an exact three-versus-three permutation calculation. With only (63)=20\binom{6}{3}=20 possible run-label assignments, the smallest attainable two-sided pp-value for the stated absolute-difference test is 0.10. This does not negate the observed effect, but it limits the statistical resolution available from three runs per recipe.

The expanded replication protocol uses eight independent training runs per recipe for PSD–LSD at 100k updates, with EMA weights and 8 NFE. The PSD–FP comparison also uses eight runs per recipe, comparing equal 100k-update checkpoints with the PSD checkpoints at their matched-time endpoints. Training runs are the resampling units. For PSD–LSD, each bootstrap replicate computes the difference in mean log perplexity, Δ\Delta, and reports the corresponding PSD reduction as

100(1−exp⁡(Δ))%.100\bigl(1-\exp(\Delta)\bigr)\%.

For PSD–FP, a positive excess means FP has higher geometric-mean perplexity than PSD. Estimating the change between equal-update and equal-time comparisons preserves dependence between checkpoints from the same training trajectory. The original cohorts and the separately collected Table 12 metric-comparison cohort remain distinguishable rather than being pooled without a common-protocol check.

Expanded eight-run comparisons at 8 NFE.

ComparisonScorerEffect estimateRun-bootstrap 95% interval
PSD perplexity reduction relative to LSD, 100k updatesQwen+14.6%[+11.0%, +18.0%]
PSD perplexity reduction relative to LSD, 100k updatesGPT-2+16.8%[+13.1%, +20.4%]
FP perplexity excess relative to PSD, 100k updatesQwen+3.9%[+1.4%, +6.5%]
FP perplexity excess relative to PSD, 100k updatesGPT-2−9.1%[−12.8%, −5.3%]
FP perplexity excess relative to PSD, matched timeQwen+7.8%[+4.0%, +11.7%]
FP perplexity excess relative to PSD, matched timeGPT-2+7.3%[+3.5%, +11.4%]
Change in FP excess from equal updates to equal timeGPT-2+16.4 percentage points[+10.8, +22.0] percentage points

The pattern is consistent with both headline observations. PSD retains its perplexity advantage over LSD, while FP's 9.1% lower GPT-2 perplexity at equal updates changes to 7.3% higher perplexity at matched time. The interval for the change in the FP–PSD contrast is particularly informative because it examines the reversal itself, not only its two endpoints. The expanded MAUVE replication gives an LSD-minus-PSD gap of +0.083, with a training-run bootstrap interval of [+0.061, +0.105]. These quantities should be interpreted at the training-run level; generating more output samples or changing scorers does not create additional independent fitted models.

AuthorsOpenReview ↗

Response to ueDB [Part 2]

[Q2: Hardware and implementation dependence of matched training time]

We agree that the update-cost ratio is a property of an implemented procedure on specified hardware, not an intrinsic constant of the mathematical objective. PSD and FP differ in target construction, while realized cost also depends on kernels, memory utilization, microbatching, and precision. The submitted time-matched allocation of 135,576 PSD updates for 100k FP updates corresponds to an FP-to-PSD update-cost ratio of approximately 1.356 in that timing experiment. The rounded A6000 costs in Table 23 are a separate reference measurement and should not be used to reconstruct this exact ratio.

For hardware configuration hh, the relevant cost ratio is

rh=cFP,hcPSD,h,r_h=\frac{c_{\mathrm{FP},h}}{c_{\mathrm{PSD},h}},

where each cc measures the full training update, including target computation and optimizer work. The additional profiles cover A6000, RTX 6000 Ada, and A100 hardware, using effective batch size 128 with the indicated microbatch and accumulation layouts. The quality comparisons use the PSD allocation associated with the measured ratio for each setting, rather than applying one hardware's ratio universally.

Matched-time hardware sensitivity. Positive perplexity excess indicates that PSD has lower perplexity. Brackets report run-bootstrap 95% intervals.

Hardware, microbatch ×\times accumulationFP/PSD update costFP excess Qwen PPLFP excess GPT-2 PPL
A6000, 32 ×\times 41.32+6.7% [+3.0%, +10.5%]+5.8% [+2.3%, +9.4%]
RTX 6000 Ada, 64 ×\times 21.28+5.5% [+2.0%, +9.1%]+4.5% [+0.7%, +8.5%]
A100, 64 ×\times 21.24+4.0% [+0.8%, +7.4%]+3.1% [+0.3%, +6.2%]

The measured cost advantage of PSD varies across execution settings, and the matched-time perplexity difference becomes smaller as the ratio approaches one. Nevertheless, FP has higher perplexity under both scorers in all three reported confirmations. The evidence therefore supports the direction of the reversal in these environments, not a hardware-independent cost ratio or a universal latency claim. The distinction between a timing-based budget allocation and a separately retrained hardware-specific model remains important when interpreting such comparisons.

[Q3: MAUVE sensitivity to feature extractors, cluster counts, and sampling]

The submitted MAUVE comparison uses GPT-2-large features, 5,000 samples, a fixed clustering seed, and a specified PCA configuration. We agree that the robustness of the ordering should be checked beyond that particular feature and quantization choice.

The expanded sensitivity analysis evaluates the same six Table 12 checkpoints using five regenerated output banks, ten clustering seeds, and cluster counts of 100, 250, and 500, with the original cluster configuration retained for reference. Feature representations include GPT-2-large, GPT-2-XL, and a fixed sentence encoder. Regenerating output banks probes sampling variability, whereas varying clustering on fixed outputs isolates variation due to the feature-space partition. Absolute MAUVE values remain associated with their respective feature and clustering protocols.

MAUVE sensitivity across feature representations. Gaps are LSD minus PSD; each reported range spans evaluation configurations, not independent training runs.

Feature representationMean MAUVE gapObserved gap rangeSettings favoring LSD
GPT-2-large+0.084[+0.041, +0.124]150/150
GPT-2-XL+0.071[+0.025, +0.115]150/150
Fixed sentence encoder+0.048[−0.006, +0.096]146/150

LSD maintains the higher mean MAUVE in every tested GPT-2-based configuration and in 146 of 150 sentence-encoder configurations. The four exceptions and the slightly negative minimum gap in the sentence-encoder results are important: they show that the result is robust over a broad tested set, but not invariant to feature representation. The 150 configurations per representation are evaluations of the same fitted models, so they describe measurement sensitivity, not 150 additional training replications. The narrower regenerated-bank and clustering-seed analysis is also reported in our response to Reviewer KXc8.

AuthorsOpenReview ↗

Response to ueDB [Part 3]

[Q4: Sensitivity of degeneration recovery to screen thresholds]

We agree that a binary degeneration screen can obscure how close a checkpoint is to a threshold. The LM1B screen is calibrated using reference text and student outputs (Appendix C.1), and its interpretation is deliberately diagnostic rather than a claim of semantic quality.

The submitted results already contain a useful borderline example: at 25k updates and 4 NFE, LSD has repeated-four-gram fraction 0.00844, just above the cutoff of 0.008. Changing the cutoff to 0.010 changes this clause's outcome. The exact inference step where a checkpoint first passes is therefore more sensitive than the main comparison of 1 versus 8 NFE.

The additional threshold audit rescored the original generation banks over three values each for the lower entropy bound (3.453, 3.553, 3.653), upper entropy bound (4.494, 4.594, 4.694), repetition ceiling (0.006, 0.008, 0.010), and pooled-entropy ceiling (8.8, 9.0, 9.2). This gives 81 joint screen settings, including the submitted thresholds.

LSD checkpointSettings failing at 1 NFESettings passing at 8 NFE
12k updates, raw81/8181/81
25k updates, raw81/8181/81
50k updates, EMA81/8181/81
100k updates, EMA81/8181/81

The 1-NFE failure and 8-NFE pass persist under all tested threshold combinations for these checkpoints. This supports the endpoint contrast without implying that every intermediate transition is equally stable. Reporting the continuous entropy, pooled-entropy, and repetition measurements alongside pass/fail status remains essential: passing this screen rules out specified aggregate pathologies, not incoherence or other semantic failures.

[Cons 2, Q5: Practical guidance, external validity, and reproducibility]

We appreciate the concern that a benchmark should provide more than a collection of conditional observations. DistillScope's actionable unit is a recipe, checkpoint, and inference setting assessed under a stated objective and resource constraint. The submitted evidence already supports different choices for different questions:

Intended comparisonReported resultScope
Lower external perplexity at 100k updates and 8 NFEPSD over LSDCore LM1B configurations
Higher MAUVE at the same operating pointLSD over PSDTable 12 evaluation protocol
Lower perplexity in the PSD–FP matched-time comparisonPSD under both scorersMeasured implementation and two-run cohort
Avoiding the specified 1-NFE degeneration screenPSD and PH pass in both 100k EMA runsDiagnostic status only, not a semantic-quality guarantee

This table is intentionally conditional. It is not a recommendation to deploy PSD, LSD, or any other recipe without evaluating the target environment. For example, the original masked OpenWebText PSD configuration substantially underperforms released SDTT, and the additional environment-tuned comparison still favors native SDTT. Similarly, PH and FP are local baselines, not faithful reproductions of every method in their respective lineages. The native-method comparisons consequently remain separate from comparisons intended to isolate the configured flow-map recipes.

For independent inspection, the benchmark's comparison records identify the training corpus, tokenizer, teacher and student configurations, recipe provenance, initialization, optimizer and checkpoint treatment, training updates and measured time, inference budget, sampler, metric configuration, and screen outcome. The associated reproducibility scope covers student checkpoints, per-run generation banks, scorer NLLs, diagnostic statistics, MAUVE inputs or extraction instructions, training and evaluation configurations, timing records, and run-level manifests with seeds and software versions. These artifacts enable readers to recompute the reported comparisons without treating a regenerated sample bank as an independent model replication.

Reviewer_2i5UOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2(+1)
Read review
Paper summary

DistillScope provides a common framework for comparing (five) flow-map distillation recipes for few-step language generation. It examines how their relative performance changes with evaluation metrics, training budgets, and inference steps. The main findings show that a recipe’s advantage can reverse when switching from perplexity to MAUVE or from equal training updates to equal training time.

Critical strengths

The paper examines how evaluation choices affect conclusions about distillation methods. The comparisons across fixed checkpoints, inference budgets and distinguishing training time from update counts are useful. The reported metrics on perplexity, MAUVE, and degeneration diagnostics provide some informative signals when comparing different training settings.

Critical weaknesses

The benchmark does not yet establish that its flow-map configurations represent well-performing implementations of the methods being compared. The gap against a released baseline is concerning: in the masked OpenWebText task, the PSD run achieves PPL 620.1, compared with 104.5 for released SDTT at matched training-token exposure and eight network evaluations. These procedures still differ in optimization and curriculum, and the flow-map recipes receive no environment-specific tuning. Consequently, it remains unclear whether the comparisons reflect meaningful differences between distillation methods or limitations of the tested configurations.

Questions for authors
  1. Do the quality rankings persist under matched training time? Could the authors report both perplexity and MAUVE for all five recipes at common training-time budgets and inference settings?
  2. What do the metric disagreements mean for readers? A blinded human evaluation of PSD and LSD outputs, assessing coherence and diversity separately, would help establish whether their contrasting metric advantages correspond to meaningful differences in text quality.
5 comments
AuthorsOpenReview ↗

Response to 2i5U [Part 1]

We sincerely thank the reviewer for the thoughtful evaluation and for recognizing the value of fixed-checkpoint comparisons and separating training time from optimizer updates. Your concerns focus on whether the tested procedures are competitive, whether the metric differences persist at common compute budgets, and whether they correspond to differences that readers can perceive in generated text. We address these points with the existing controls and additional comparisons below.

[Cons 1: Configuration quality and the comparison with released SDTT]

We agree that the masked OpenWebText gap between the submitted PSD configuration and released SDTT is substantial: 620.1 versus 104.5 perplexity at 8 NFE and matched final training-token exposure. This is a comparison of complete procedures, not an isolated comparison of their objectives. The methods differ in curriculum, target construction, optimization, and checkpoint treatment, and the original masked flow-map configurations did not receive environment-specific tuning (Appendix E.2).

Two checks in the submission help identify what this comparison does and does not establish. First, locally trained native SDTT achieves perplexities of 42.2 and 42.5 at its first-round, 512-NFE design point, close to the released first-round checkpoint at 43.6 and below the teacher at 52.1 (Appendix E.2). This supports the first-round implementation, but it does not establish competitiveness at 8 NFE. Second, the submitted PSD trajectory improves from 665.7 at 10k updates to 537.5 at 35k, then reaches 620.1 at 70k; the 70k EMA checkpoint scores 958.1 (Table 17). More updates and EMA treatment alone do not close the observed gap in that trajectory.

The additional environment-specific comparison holds the 70k-update, batch-128, length-1,024, 8-NFE evaluation point fixed. PSD is selected through a bounded search over learning rate, warmup, and diagonal/off-diagonal loss balance, while preserving its defining objective. Native SDTT receives an equal search GPU-hour budget. Held-out validation banks determine configurations, and the final evaluations use three independent training runs. Search costs are accounted for separately from the matched training-token exposure.

Masked OpenWebText, 8 NFE. Perplexity follows the local SDTT scoring convention. The ±\pm values denote standard deviations across the newly trained runs.

ProcedureTraining updatesGenerative perplexity ↓\downarrow
Submitted PSD, raw weights, one trajectory70k620.1
Environment-tuned PSD70k146.8 ±\pm 6.9
Locally trained native SDTT, seven rounds70k108.9 ±\pm 4.6
Released SDTT, round 770k104.5

The selected PSD configuration substantially narrows the gap, from 620.1 to 146.8, but native SDTT remains stronger at the tested operating point. The locally trained SDTT result also lies near the released checkpoint. These results show why the original gap should not be attributed entirely to the distillation objective, while still acknowledging an important limitation of the evaluated PSD procedure. The controlled LM1B recipe study and native-system comparisons answer different questions and should not be treated as a single leaderboard.

AuthorsOpenReview ↗

Response to 2i5U [Part 2]

[Q1: All five recipes under matched training time]

The submitted PSD–FP comparison shows that accounting for per-update cost can reverse a result: FP leads on GPT-2 perplexity at equal updates, while PSD leads after receiving 135,576 updates for the measured training time of FP's 100k updates (Section 3.4). We agree that evaluating all five recipes at common time budgets makes the implications clearer.

The expanded fixed-configuration comparison uses 24 and 36 A6000 GPU-hours, three training runs per recipe and budget, and common inference settings of 1, 2, 4, and 8 NFE. Timing includes target construction, derivative calculations, optimization, and EMA updates. Data, teacher initialization, hardware, and checkpoint treatment are held fixed. PH retains its four-round curriculum within each budget, and checkpoints are selected by elapsed training time rather than test performance. The table below shows the 8-NFE results; separately selected configurations can be evaluated without conflating tuning with cost accounting.

Five-recipe comparison at 8 NFE. Values are arithmetic means over three runs; screen status counts independent training runs.

Training budgetRecipeQwen PPL ↓\downarrowGPT-2 PPL ↓\downarrowMAUVE ↑\uparrowScreen passes
24 GPU-hoursPSD162.8198.60.4913/3
24 GPU-hoursLSD248.7313.50.5383/3
24 GPU-hoursPH198.2234.60.1643/3
24 GPU-hoursFP186.9211.80.2183/3
24 GPU-hoursESD10.716.10.0090/3
36 GPU-hoursPSD151.7182.40.5143/3
36 GPU-hoursLSD197.4250.10.5743/3
36 GPU-hoursPH189.5223.70.1833/3
36 GPU-hoursFP170.8196.30.2473/3
36 GPU-hoursESD8.612.40.0080/3

At both training budgets, PSD has the lowest external perplexity among screen-passing recipes, whereas LSD has the highest MAUVE. The extremely low perplexities of ESD should not be interpreted as superior generation: none of its three runs passes the degeneration screen, and its MAUVE remains near zero. Increasing the time budget improves several measured scores but does not eliminate the disagreement between the two quality criteria. This extends the paper's central observation beyond a single PSD–FP comparison, while retaining the distinction between diagnostic validity and sample quality.

[Q2: Human assessment of the PSD–LSD metric disagreement]

We agree that perplexity and MAUVE alone cannot determine whether people find one recipe more coherent or meaningfully diverse. The submitted results establish lower external-model perplexity for PSD and higher feature-space distributional similarity for LSD, but neither metric is a human judgment of text quality.

The additional blinded evaluation uses the three PSD and three LSD checkpoints from Table 12 at 100k updates and 8 NFE. It samples 100 unfiltered outputs per checkpoint, giving 600 outputs in total. Recipe identity, training run, and metric scores are hidden from raters. Three independent five-point judgments per output assess coherence, including intelligibility, continuity, and internal consistency. Meaningful diversity is assessed through three five-point judgments per ten-output batch, balancing batches across runs and distinguishing topical or phrasing variation from incoherent variation. Intervals reflect evaluation variability conditional on these six fitted checkpoints; outputs and raters are not treated as additional training replications.

Eye-pass human evaluation. Higher values indicate better ratings. Four human raters participated.

DimensionPSD meanLSD meanLSD minus PSD, conditional 95% interval
Output coherence (1 to 5)2.983.04+0.06 [−0.12, +0.24]
Meaningful diversity (1 to 5)2.873.35+0.48 [+0.23, +0.73]

LSD receives higher ratings for meaningful diversity, consistent with the direction of its MAUVE advantage. The coherence difference is small, and its interval includes both positive and negative values, so the comparison does not establish a coherence advantage or equivalence. The moderate absolute coherence ratings also matter: these results characterize a relative trade-off between two recipes, not uniformly high-quality generation. The AI-only descriptive audit in Appendix C.4 is a separate analysis and is not used as a substitute for these human ratings.

Together, the matched-time and human evaluations clarify what the metric disagreement means. External perplexity measures predictability under a specified scorer, MAUVE measures feature-space distributional similarity under a specified protocol, and human ratings assess the properties represented in the annotation criteria.

AuthorsOpenReview ↗

Sincerely looking forward to further discussions

Thank you again for your thoughtful feedback. We would be incredibly grateful to know if these revisions successfully resolve your initial questions, or if there are any remaining aspects you would like us to clarify before the discussion period ends.

Reviewer_KXc8OpenReview ↗
Rating
4(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces DistillScope, a common training and evaluation framework for comparing five procedures for distilling flow-map language models. It examines how their relative performance varies with the evaluation metric, training budget, and number of generation steps. The primary comparisons are conducted on LM1B, with supplementary experiments on OpenWebText and larger models. Key findings include a reversal between PSD and LSD when using perplexity versus MAUVE, a reversal between PSD and the local fixed-point baseline when matching training time instead of updates, and shifts in degeneration-screen results when the same checkpoint is evaluated with more inference steps.

Critical strengths

1 - The paper addresses a valuable comparison problem. Identical training updates do not necessarily imply identical training costs, and a method's performance at one inference budget may not reflect its behavior at another. By making these conditions explicit, the work provides interpretation of results and can support practical method selection.

2 - The main metric reversal is concrete and has been replicated. In Table 12, all three PSD runs show lower perplexity, whereas all three LSD runs show higher MAUVE. This usefully qualifies a preference for PSD based on perplexity alone. The equal-training-time comparison offers a concrete example in which the comparison convention influences the conclusion.

Critical weaknesses

1 - The comparisons do not establish the best achievable performance of the method families. The procedures differ in losses, target construction, projections, and optimization choices. It is unclear whether they received comparable hyper-parameter selection or tuning effort. The results therefore describe the procedures as they were set up, I'm not sure if practitioners can confidently rely on them when choosing among the underlying methods.

2 - The empirical contribution is incremental, and its strongest evidence is limited in scope. Prior work already compares distillation objectives, sampling budgets, and multiple quality metrics. The contribution here lies in the specific additional comparisons rather than in a fundamentally new evaluation approach. Most detailed results concern short LM1B sequences with two or three runs per condition, while the extensions offer limited evidence that the main conclusions hold up in other settings.

Questions for authors

1 - Clarification on comparison fairness: How were the learning rates, loss configurations, and other recipe-specific hyper-parameters selected? Was there comparable validation or tuning effort across methods? This would help distinguish differences between the configured procedures from differences between the method families.

2 - Clarification of the main result: How sensitive is the PSD–LSD MAUVE ordering to evaluation randomness, particularly regenerated sample banks or clustering seeds? Table 12 shows consistent ordering across training runs, but additional evaluation sensitivity results would strengthen confidence in this central finding.

3 - Implementation clarification: Equation 9 specifies the ESD target for interior times with (t<1), while the shared training setup includes full-interval boundary examples with (t=1). How are these boundary examples handled for ESD?

5 comments
AuthorsOpenReview ↗

Response to KXc8 [Part 1]

We sincerely thank the reviewer for the careful and constructive evaluation. We appreciate your recognition of the practical value of explicit resource accounting and the consistent PSD–LSD metric reversal across training runs. We address the questions about configuration fairness, the scope of the contribution, evaluation sensitivity, and the ESD boundary condition below.

[Cons 1, Q1: Comparison fairness, hyperparameter selection, and method selection]

We agree that a common training protocol does not by itself establish that each method family has received comparable tuning. DistillScope compares specified training procedures, not the best attainable result from every method family. In the core LM1B study, the recipes share the teacher initialization, architecture, data, batch size, optimizer family, and evaluation protocol. Their target construction, projections, and losses remain recipe-specific, as detailed in Table 1 and Appendix B.3. The learning rate is 3×10−43\times10^{-4} for PSD, LSD, ESD, and FP, and 10−310^{-3} for progressive halving (PH), with AdamW coefficients (0.9,0.95)(0.9,0.95) and 1,000 warmup updates. LSD uses entmax 2.0, FP uses four damped iterations with α=0.3\alpha=0.3, and PH uses four rounds.

To distinguish configuration effects from the observed recipe differences, the additional tuning comparison allocates the same search GPU-hour budget to each recipe. The search varies learning rate and off-diagonal loss weight without changing the defining target construction. Configurations are selected on validation banks separate from final evaluation, with three independent training runs per selected configuration. This is a bounded and explicitly costed search rather than a claim of exhaustive optimization.

Additional comparison at 100k updates and 8 NFE. Each row uses configurations selected separately for the indicated validation target.

Selection and evaluation targetPSDLSD
Qwen perplexity ↓\downarrow160.2184.9
GPT-2 perplexity ↓\downarrow197.6232.8
MAUVE ↑\uparrow0.5210.594

The preference remains metric-dependent after the bounded search. PSD has lower external perplexity under both scorers, whereas LSD has higher MAUVE. Importantly, the rows represent separately selected configurations and should not be read as three metrics from one common checkpoint. The result supports the more limited, practitioner-facing conclusion that a recipe comparison should specify both its selection criterion and its resource budget. It does not establish a universal ranking of the underlying method families.

[Cons 2: Contribution relative to prior work and transfer beyond short LM1B sequences]

We agree that prior work has compared distillation objectives, inference budgets, and quality metrics. The contribution of DistillScope is the controlled change in the conclusion when these comparison conditions vary, rather than the introduction of a new distillation objective or quality metric. Three findings make this distinction concrete:

  1. Quality criterion. In Table 12, every PSD run has lower generative perplexity than every LSD run under both external scorers, while every LSD run has higher MAUVE. The observed MAUVE ranges are 0.479 to 0.518 for PSD and 0.559 to 0.602 for LSD.
  2. Training cost. At equal updates, FP has lower GPT-2 perplexity than PSD (185.4 versus 204.9). At matched measured training time, FP instead has 7.1% higher GPT-2 perplexity and 7.7% higher Qwen perplexity (Section 3.4 and Appendix D.2).
  3. Inference budget. The same LSD checkpoints at 12k, 25k, 50k, and 100k updates fail the specified degeneration screen at 1 NFE and pass at 8 NFE (Section 3.5). This comparison changes inference computation without retraining the student.

The paper also reports cases where these observations do not transfer directly. On continuous-flow OpenWebText, the 1-NFE entropy outcomes differ from LM1B, and all six evaluated runs pass the full screen at 2 NFE (Appendix D.7). In the masked OpenWebText setting, one LSD run becomes more degenerate as NFE increases (Appendix E.4). These are useful limits on any recommendation based only on the core LM1B results.

The additional continuous-flow OpenWebText comparison provides a more direct test of the PSD–LSD metric ordering on a second corpus with longer sequences. With three training runs per recipe at 8 NFE, the ratios of mean PSD to LSD perplexity are 0.92 under Qwen and 0.94 under GPT-2, while LSD's mean MAUVE is higher by 0.043. Thus, the tested OpenWebText setting reproduces the direction of the cross-metric reversal. The magnitude and the broader recipe rankings remain specific to the training and evaluation conditions, rather than establishing environment-independent superiority.

AuthorsOpenReview ↗

Response to KXc8 [Part 2]

[Q2: Sensitivity of MAUVE to regenerated output banks and clustering seeds]

The submitted three-run comparison establishes disjoint observed MAUVE ranges, with the lowest LSD score (0.559) exceeding the highest PSD score (0.518). However, differences across fitted models and variation from MAUVE evaluation are distinct sources of uncertainty. We therefore examine the six checkpoints from Table 12 with five regenerated banks of 5,000 outputs and ten clustering seeds per bank, retaining GPT-2-large features, the reference bank, PCA setting, and original clustering configuration.

MAUVE evaluation perturbationSettings with LSD mean greater than PSD mean
Five regenerated banks, original clustering seed5/5
Original output banks, ten clustering seeds10/10
Regenerated banks crossed with clustering seeds50/50

Across the 50 crossed settings, the mean LSD-minus-PSD MAUVE gap is 0.084, with an observed range of 0.059 to 0.108. The ordering therefore persists under the tested sampling and clustering perturbations, even though the size of the gap varies. These are repeated evaluations of six trained checkpoints, not 50 independent training runs. Feature-extractor and cluster-count sensitivity are considered separately in our response to Reviewer ueDB.

[Q3: ESD treatment of the full-interval boundary]

Thank you for identifying this important implementation detail. Equation 9 defines the ESD target for interior intervals with t<1t<1 and contains a factor proportional to (1−t)−1(1-t)^{-1}. It is therefore not valid to evaluate that expression by directly substituting t=1t=1. The full-interval boundary requires an explicit ESD-specific convention, separate from the interior formula and the shared batch allocation.

The boundary audit checks the implemented targets, losses, and gradients on full-interval examples against an independent calculation. All tested boundary cases produce finite values and agree with the independent calculation. This numerical check is distinct from the symbolic definition of the boundary rule, which should be stated explicitly alongside Equation 9 and Appendix B.3. The PSD–LSD and PSD–FP conclusions use separate models and are not inferred from the ESD boundary check.

Taken together, these clarifications reinforce the intended interpretation of DistillScope: a recipe comparison is meaningful when the configuration, quality criterion, training allocation, and inference budget are stated, and its transfer limits are measured rather than assumed.

AuthorsOpenReview ↗

Sincerely looking forward to further discussions

Thank you again for your thoughtful feedback. We would be incredibly grateful to know if these revisions successfully resolve your initial questions, or if there are any remaining aspects you would like us to clarify before the discussion period ends.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Rebuttal

Summary of Rebuttal

We thank all three reviewers for their careful reading and constructive feedback. The reviewers recognized the value of comparing distillation recipes under explicit quality criteria, training costs, and inference budgets. During discussion, KXc8 confirmed that the sampling and clustering checks answered the MAUVE question and that OpenWebText strengthened the evidence beyond LM1B. Reviewer 2i5U increased the soundness assessment. Our rebuttal adds the following evidence in direct response to the reviewers' requests:

  1. Eight-run replication of the main findings (@Reviewer ueDB). We expanded the PSD–LSD and PSD–FP comparisons to eight independent training runs per recipe at eight network evaluations (NFE), with training-run bootstrap intervals. PSD's geometric-mean perplexity reductions relative to LSD are 14.6% [11.0%, 18.0%] under Qwen and 16.8% [13.1%, 20.4%] under GPT-2. LSD retains a MAUVE advantage of +0.083 [0.061, 0.105]. FP's GPT-2 perplexity changes from 9.1% lower than PSD at equal updates to 7.3% higher at matched time; the change is 16.4 percentage points [10.8, 22.0]. All intervals are 95%.

  2. Concrete choices across all five recipes (@Reviewer 2i5U, @Reviewer KXc8). At 24 and 36 A6000 GPU-hours and eight NFE, with three runs per recipe and budget, PSD has the lowest mean external perplexity among screen-passing recipes, while LSD has the highest mean MAUVE. PSD also has better mean scores than progressive halving (PH) and FP; ESD's low perplexity accompanies failed screens. The separate 100k-update comparison identifies PSD and PH as screen-passing one-NFE candidates. These results give practitioners an explicit choice based on their quality target and resource budget.

  3. Fair configuration selection and native baselines (@Reviewer KXc8, @Reviewer 2i5U, @Reviewer ueDB). Equal search budgets and held-out selection, performed separately for each metric, preserve the PSD–LSD ordering. On masked OpenWebText, environment-specific tuning improves PSD perplexity from 620.1 to 146.8. Locally trained native SDTT reaches 108.9, close to the released checkpoint's 104.5, identifying the stronger procedure in that setting. Under its native protocol, Gumbel Straight Flow reproduction closely matches published design points: 117.6 ± 4.8 at one NFE and 101.8 ± 3.6 at four NFE, versus published 114.9 and 99.7 (± denotes SD).

  4. Transfer beyond the core LM1B setting (@Reviewer KXc8). Three runs per recipe on continuous-flow OpenWebText reproduce the metric reversal at eight NFE: PSD/LSD mean-perplexity ratios are 0.92 under Qwen and 0.94 under GPT-2, while LSD's mean MAUVE is higher by 0.043. The ordering therefore appears on a second corpus with longer sequences as well as after bounded configuration selection.

  5. Robustness to evaluation and screen choices (@Reviewer KXc8, @Reviewer ueDB). Across regenerated sample banks, clustering seeds, and cluster counts, LSD has higher MAUVE in 150/150 GPT-2-large settings, 150/150 GPT-2-XL settings, and 146/150 sentence-encoder settings. These are repeated evaluations of six checkpoints. Separately, all four tested LSD checkpoints retain their one-NFE screen failure and eight-NFE pass under all 81 joint threshold settings. This tests both the metric reversal and the inference-budget result against the requested evaluation perturbations.

  6. Hardware-aware cost comparisons (@Reviewer ueDB). We profiled full update costs on A6000, RTX 6000 Ada, and A100, including target construction and optimizer work. Recomputing matched-time allocations using each setting's measured costs preserves PSD's perplexity advantage over FP under both scorers. This directly tests whether the conclusion depends on the original timing setup.

  7. Human assessment of the metric disagreement (@Reviewer 2i5U). Four blinded raters assessed coherence and meaningful diversity using 600 outputs from six 100k-update checkpoints at eight NFE. LSD receives higher meaningful-diversity ratings: 3.35 versus 2.87, a difference of +0.48 [conditional 95% interval: +0.23, +0.73]. This connects its MAUVE advantage to useful topical and phrasing variation recognized by human readers, with coherence assessed separately.

  8. Implementation checks and reproducibility (@Reviewer KXc8, @Reviewer ueDB). In all tested ESD boundary cases, targets, losses, and gradients are finite and agree with an independent calculation. We also specify the reproducibility artifacts: student checkpoints, per-run generation banks, scorer NLLs, MAUVE inputs, diagnostics, configurations, timing records, and seed/version manifests.

These additions strengthen DistillScope's central contribution: reproducible recipe comparisons that translate quality criteria, training computation, and inference budgets into concrete method-selection guidance.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.