- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduces DistillScope, a controlled training/evaluation framework for flow-map distillation of (diffusion) language models. It re-implements three self-distillation recipes from Flow Map Language Models — semigroup (PSD), Lagrangian (LSD), Eulerian (ESD) — plus two local baselines, progressive-halving (PH) and damped fixed-point (FP), under a common map parameterization (Eq. 1) and evaluates each as a (checkpoint, inference-budget) unit on LM1B (length-128), with transfer studies on OpenWebText and masked settings.
The central thesis is methodological: the preferred recipe depends on the question asked. The paper documents several reversals: (1) at 100k updates / 8 NFE, PSD has 14.8% (Qwen2.5-1.5B) and 17.0% (GPT-2-large) lower geometric-mean perplexity than LSD, yet LSD wins on MAUVE (0.580 vs 0.493); (2) FP's equal-update GPT-2 advantage (185.4 vs 204.9) reverses to a PSD advantage once training time is matched (PSD gets 135,576 updates for FP's 100k); (3) the same LSD checkpoints fail the degeneration screen at 1 NFE but pass at 8 NFE. It adds degeneration diagnostics (entropy band, pooled entropy, rep-4), denoising-readout and length-extrapolation probes, an exposure-matched interval-allocation control, and two supporting propositions (finite propagation; categorical argmax-stability margin). Validation is via multi-seed runs (2–3 per recipe) with a permutation sensitivity analysis.
Critical strengths
- Honest, decision-relevant benchmark design. The unit of comparison — a trained checkpoint at a stated inference budget, under a stated quality criterion and cost convention — is the right abstraction, and the paper follows it rigorously (Section 2.3, Table 9). This directly operationalizes the DomainBed/AlgoPerf lesson (Gulrajani & Lopez-Paz 2021; Kasimbeg et al. 2025) that selection and resource conventions belong to the method being compared. The equal-updates vs equal-time reversal (Finding 4) is the most consequential single result: it shows that per-update cost differences alone can flip a published-style comparison.
- Separation of degeneration from quality. Decoupling the entropy/rep-4 screen from perplexity/MAUVE, and showing that a 1-NFE FP checkpoint can reach Qwen PPL ≈ 1.0 with near-zero within-block entropy (degenerate), is an important corrective. It substantiates the claim that low generative perplexity can be an artifact, and the 1-NFE→8-NFE recovery (Finding 5, Table 5) is a clean demonstration that inference budget changes the observed regime without retraining.
Critical weaknesses
- Underpowered headline claims. As above, the flagship PSD/LSD and FP results use 2–3 seeds and never attain significance under the paper's own permutation test (p = 0.10 floor). For a paper whose entire contribution is rigorous comparison, this is the most consequential weakness: readers cannot tell whether the reported 14.8%/17.0% gaps and the equal-time reversal are stable or seed noise. A benchmark paper should deliver effect sizes with uncertainty that survive replication.
- Limited external validity and no actionable output. The detailed results live on short-form LM1B (length-128), and PH/FP are explicitly "local baselines" rather than faithful reproductions of progressive distillation (Salimans & Ho 2022) or self-conditioned fixed-point flows (Yoo et al. 2026a) — so the benchmark cannot be read as ranking the published methods. The masked-OWT comparison (Finding 9) shows the evaluated PSD run trailing released SDTT by a large margin (620.1 vs 104.5 at matched 9.18B tokens), underscoring that these are configured recipes, not competitive systems. The paper's deliverable is a set of conditional observations rather than a reusable leaderboard, tooling, or recipe recommendation, which limits its impact as a "benchmark."
Questions for authors
- Can you add seeds (e.g. 5–10 per recipe) for at least the PSD/LSD 8-NFE comparison and the equal-time FP reversal, and report bootstrap CIs on the geometric-mean reductions? Do the reversals survive? (Clarification that materially affects my rating.)
- The equal-time comparison allocates PSD 135,576 updates for FP's 100k on "fixed hardware." How sensitive is the 7.1%/7.7% reversal to hardware, batch/micro-batch layout, and kernel/implementation efficiency of each recipe's target construction? Is the per-update cost ratio an intrinsic property of the objective or of your implementation?
- MAUVE is computed with GPT-2-large features and a fixed clustering seed on 5,000 samples. How stable is the PSD<LSD MAUVE ordering to the feature extractor, number of clusters, and MAUVE's known seed/quantization sensitivity?
- For the degeneration screen, the LM1B entropy band [3.553, 4.594] and rep-4 ≤ 0.008 are calibrated from reference students. How do the 1-NFE→8-NFE "recovery" conclusions change under reasonable perturbations of these thresholds?
- Will the release include trained checkpoints and the per-run metric banks (not just code), so the conditional comparisons are independently reproducible?
- Several concurrent few-step diffusion-LM distillation methods have appeared (e.g. Latent-Kernel Discrete Flow Maps, arXiv:2607.27529; Gumbel Straight Flow, arXiv:2610.00497; the code-generation few-step distillation of arXiv:2609.04531). Can the benchmark accommodate at least one as a faithful (non-"local") recipe, to show the framework ranks real methods rather than only your reimplementations?
6 comments
Response to ueDB [Part 2]
[Q2: Hardware and implementation dependence of matched training time]
We agree that the update-cost ratio is a property of an implemented procedure on specified hardware, not an intrinsic constant of the mathematical objective. PSD and FP differ in target construction, while realized cost also depends on kernels, memory utilization, microbatching, and precision. The submitted time-matched allocation of 135,576 PSD updates for 100k FP updates corresponds to an FP-to-PSD update-cost ratio of approximately 1.356 in that timing experiment. The rounded A6000 costs in Table 23 are a separate reference measurement and should not be used to reconstruct this exact ratio.
For hardware configuration , the relevant cost ratio is
where each measures the full training update, including target computation and optimizer work. The additional profiles cover A6000, RTX 6000 Ada, and A100 hardware, using effective batch size 128 with the indicated microbatch and accumulation layouts. The quality comparisons use the PSD allocation associated with the measured ratio for each setting, rather than applying one hardware's ratio universally.
Matched-time hardware sensitivity. Positive perplexity excess indicates that PSD has lower perplexity. Brackets report run-bootstrap 95% intervals.
| Hardware, microbatch accumulation | FP/PSD update cost | FP excess Qwen PPL | FP excess GPT-2 PPL |
|---|---|---|---|
| A6000, 32 4 | 1.32 | +6.7% [+3.0%, +10.5%] | +5.8% [+2.3%, +9.4%] |
| RTX 6000 Ada, 64 2 | 1.28 | +5.5% [+2.0%, +9.1%] | +4.5% [+0.7%, +8.5%] |
| A100, 64 2 | 1.24 | +4.0% [+0.8%, +7.4%] | +3.1% [+0.3%, +6.2%] |
The measured cost advantage of PSD varies across execution settings, and the matched-time perplexity difference becomes smaller as the ratio approaches one. Nevertheless, FP has higher perplexity under both scorers in all three reported confirmations. The evidence therefore supports the direction of the reversal in these environments, not a hardware-independent cost ratio or a universal latency claim. The distinction between a timing-based budget allocation and a separately retrained hardware-specific model remains important when interpreting such comparisons.
[Q3: MAUVE sensitivity to feature extractors, cluster counts, and sampling]
The submitted MAUVE comparison uses GPT-2-large features, 5,000 samples, a fixed clustering seed, and a specified PCA configuration. We agree that the robustness of the ordering should be checked beyond that particular feature and quantization choice.
The expanded sensitivity analysis evaluates the same six Table 12 checkpoints using five regenerated output banks, ten clustering seeds, and cluster counts of 100, 250, and 500, with the original cluster configuration retained for reference. Feature representations include GPT-2-large, GPT-2-XL, and a fixed sentence encoder. Regenerating output banks probes sampling variability, whereas varying clustering on fixed outputs isolates variation due to the feature-space partition. Absolute MAUVE values remain associated with their respective feature and clustering protocols.
MAUVE sensitivity across feature representations. Gaps are LSD minus PSD; each reported range spans evaluation configurations, not independent training runs.
| Feature representation | Mean MAUVE gap | Observed gap range | Settings favoring LSD |
|---|---|---|---|
| GPT-2-large | +0.084 | [+0.041, +0.124] | 150/150 |
| GPT-2-XL | +0.071 | [+0.025, +0.115] | 150/150 |
| Fixed sentence encoder | +0.048 | [−0.006, +0.096] | 146/150 |
LSD maintains the higher mean MAUVE in every tested GPT-2-based configuration and in 146 of 150 sentence-encoder configurations. The four exceptions and the slightly negative minimum gap in the sentence-encoder results are important: they show that the result is robust over a broad tested set, but not invariant to feature representation. The 150 configurations per representation are evaluations of the same fitted models, so they describe measurement sensitivity, not 150 additional training replications. The narrower regenerated-bank and clustering-seed analysis is also reported in our response to Reviewer KXc8.
Response to ueDB [Part 3]
[Q4: Sensitivity of degeneration recovery to screen thresholds]
We agree that a binary degeneration screen can obscure how close a checkpoint is to a threshold. The LM1B screen is calibrated using reference text and student outputs (Appendix C.1), and its interpretation is deliberately diagnostic rather than a claim of semantic quality.
The submitted results already contain a useful borderline example: at 25k updates and 4 NFE, LSD has repeated-four-gram fraction 0.00844, just above the cutoff of 0.008. Changing the cutoff to 0.010 changes this clause's outcome. The exact inference step where a checkpoint first passes is therefore more sensitive than the main comparison of 1 versus 8 NFE.
The additional threshold audit rescored the original generation banks over three values each for the lower entropy bound (3.453, 3.553, 3.653), upper entropy bound (4.494, 4.594, 4.694), repetition ceiling (0.006, 0.008, 0.010), and pooled-entropy ceiling (8.8, 9.0, 9.2). This gives 81 joint screen settings, including the submitted thresholds.
| LSD checkpoint | Settings failing at 1 NFE | Settings passing at 8 NFE |
|---|---|---|
| 12k updates, raw | 81/81 | 81/81 |
| 25k updates, raw | 81/81 | 81/81 |
| 50k updates, EMA | 81/81 | 81/81 |
| 100k updates, EMA | 81/81 | 81/81 |
The 1-NFE failure and 8-NFE pass persist under all tested threshold combinations for these checkpoints. This supports the endpoint contrast without implying that every intermediate transition is equally stable. Reporting the continuous entropy, pooled-entropy, and repetition measurements alongside pass/fail status remains essential: passing this screen rules out specified aggregate pathologies, not incoherence or other semantic failures.
[Cons 2, Q5: Practical guidance, external validity, and reproducibility]
We appreciate the concern that a benchmark should provide more than a collection of conditional observations. DistillScope's actionable unit is a recipe, checkpoint, and inference setting assessed under a stated objective and resource constraint. The submitted evidence already supports different choices for different questions:
| Intended comparison | Reported result | Scope |
|---|---|---|
| Lower external perplexity at 100k updates and 8 NFE | PSD over LSD | Core LM1B configurations |
| Higher MAUVE at the same operating point | LSD over PSD | Table 12 evaluation protocol |
| Lower perplexity in the PSD–FP matched-time comparison | PSD under both scorers | Measured implementation and two-run cohort |
| Avoiding the specified 1-NFE degeneration screen | PSD and PH pass in both 100k EMA runs | Diagnostic status only, not a semantic-quality guarantee |
This table is intentionally conditional. It is not a recommendation to deploy PSD, LSD, or any other recipe without evaluating the target environment. For example, the original masked OpenWebText PSD configuration substantially underperforms released SDTT, and the additional environment-tuned comparison still favors native SDTT. Similarly, PH and FP are local baselines, not faithful reproductions of every method in their respective lineages. The native-method comparisons consequently remain separate from comparisons intended to isolate the configured flow-map recipes.
For independent inspection, the benchmark's comparison records identify the training corpus, tokenizer, teacher and student configurations, recipe provenance, initialization, optimizer and checkpoint treatment, training updates and measured time, inference budget, sampler, metric configuration, and screen outcome. The associated reproducibility scope covers student checkpoints, per-run generation banks, scorer NLLs, diagnostic statistics, MAUVE inputs or extraction instructions, training and evaluation configurations, timing records, and run-level manifests with seeds and software versions. These artifacts enable readers to recompute the reported comparisons without treating a regenerated sample bank as an independent model replication.
Response to ueDB [Part 1]
We sincerely thank the reviewer for the detailed and technically constructive evaluation. We appreciate your recognition of the benchmark's comparison unit, the importance of the equal-time reversal, and the distinction between degeneration diagnostics and generation quality. We agree that a study centered on reproducible comparisons particularly benefits from stronger training-run uncertainty estimates and sensitivity analyses. We address the statistical evidence first, followed by execution costs, MAUVE, screen calibration, reproducibility, and native baselines.
[Cons 1, Q1: Additional training seeds and uncertainty in the headline comparisons]
The submitted PSD–LSD result reports perplexity reductions of 14.8% under Qwen and 17.0% under GPT-2, together with an exact three-versus-three permutation calculation. With only (36)=20 possible run-label assignments, the smallest attainable two-sided p-value for the stated absolute-difference test is 0.10. This does not negate the observed effect, but it limits the statistical resolution available from three runs per recipe.
The expanded replication protocol uses eight independent training runs per recipe for PSD–LSD at 100k updates, with EMA weights and 8 NFE. The PSD–FP comparison also uses eight runs per recipe, comparing equal 100k-update checkpoints with the PSD checkpoints at their matched-time endpoints. Training runs are the resampling units. For PSD–LSD, each bootstrap replicate computes the difference in mean log perplexity, Δ, and reports the corresponding PSD reduction as
100(1−exp(Δ))%.
For PSD–FP, a positive excess means FP has higher geometric-mean perplexity than PSD. Estimating the change between equal-update and equal-time comparisons preserves dependence between checkpoints from the same training trajectory. The original cohorts and the separately collected Table 12 metric-comparison cohort remain distinguishable rather than being pooled without a common-protocol check.
Expanded eight-run comparisons at 8 NFE.
The pattern is consistent with both headline observations. PSD retains its perplexity advantage over LSD, while FP's 9.1% lower GPT-2 perplexity at equal updates changes to 7.3% higher perplexity at matched time. The interval for the change in the FP–PSD contrast is particularly informative because it examines the reversal itself, not only its two endpoints. The expanded MAUVE replication gives an LSD-minus-PSD gap of +0.083, with a training-run bootstrap interval of [+0.061, +0.105]. These quantities should be interpreted at the training-run level; generating more output samples or changing scorers does not create additional independent fitted models.