- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes KIS, which uses a corpus unigram prior and fixed token-embedding geometry to compute the conditional entropy of a single-token Gaussian corruption channel, then normalizes entropy reduction into an information coordinate. During training, KIS shifts the median of the noise distribution to the information midpoint; during sampling, it fits interior log-SNR nodes to information quantiles using a positive affine transformation. The method does not query the denoiser. Experiments cover pretraining calibration for ELF-B and sampling calibration for frozen ELF, FLM, and LangFlow models. Several settings improve substantially—for example, ELF-B's LM1B GenPPL drops from 121.2 to 90.1—but some ELF-L and LangFlow results regress.
Critical strengths
- KIS can be applied before training or to a frozen model at sampling time without repeated denoiser evaluations. Sampling calibration preserves monotonicity and uses a constrained transformation of the original grid.
- The experiments cover both training and inference, several model representations, and different sampling budgets. The paper reports substantial GenPPL improvements in low-step settings such as FLM-B.
Critical weaknesses
- is computed from a fixed-embedding, single-token Gaussian channel and does not include context. For ELF, calibration additionally relies on a T5-small lookup-embedding proxy with a global RMS scale adjustment; this does not establish that the proxy matches the information curve of the actual contextual latents.
- Improvements are not universal: some ELF-L and LangFlow results regress. The main metrics are GPT-2 Large GenPPL and unigram token entropy, and the main results do not provide enough seed or confidence-interval information to assess stability.
Questions for authors
- For ELF, can you quantify how the T5 lookup-embedding proxy relates to predictive uncertainty or denoising loss on actual contextual latents? This would affect my assessment of the central proxy assumption.
- How many independent random seeds were used for each training and sampling comparison? Is the corpus token prior estimated only from the training split?
5 comments
Response to Reviewer Sx98 (2/4)
Q2 — ELF-L Algorithm 2 Audit & LangFlow Scope Boundary
Reviewer Sx98 (Q2): Improvements are not universal: how do the ELF-L and LangFlow results affect the main conclusions?
Answer: Applying Algorithm 2 uniformly lowers ELF-L GenPPL at both small and full step budgets, while LangFlow reflects train-test schedule shift when overriding a co-trained schedule post hoc.
- ELF-L implementation audit: We audited all five pretrained models across LM1B and OWT (Tables 2, 4–6, 8, 9, 12): FLM-B [4], ELF-B/M [2], and LangFlow [1] already used affine-fit (using direct-quantiles only in w/o linear fit ablations), whereas ELF-L's evaluation config had used direct-quantiles (unified configs are in the supplementary code). Applying Algorithm 2 to ELF-L lowers GenPPL at both small ( and at ) and full step budgets (shown alongside raw quantiles below), and matched ELF-L training also lowers GenPPL across all four settings:
| ELF-L Frozen-Sampling Schedule | LM1B () GenPPL | OWT () GenPPL |
|---|---|---|
| Default schedule | 108.9 | 23.3 |
| Direct quantiles (initial submission) | 110.3 | 24.1 |
| Positive affine fit (Algorithm 2, corrected) | 107.1 | 22.8 |
- LangFlow post-hoc boundary: Online schedules cannot calibrate frozen fixed-schedule checkpoints post hoc and incur early-training warmup lag ( GenPPL for online EMA vs. for KIS in Table 10); on LangFlow [1], whose velocity network already co-adapted with its Gumbel schedule during training, post-hoc replacement shifts the midpoint log-SNR noisier relative to the training distribution:
| Model | Corpus | Mode and Step Budget | Default GenPPL | KIS GenPPL | |
|---|---|---|---|---|---|
| ELF-L without encoder | LM1B | Matched training decoded at | 109.1 | 104.3 | |
| ELF-L with encoder | LM1B | Matched training decoded at | 108.9 | 104.8 | |
| ELF-L without encoder | OWT | Matched training decoded at | 30.9 | 29.2 | |
| ELF-L with encoder | OWT | Matched training decoded at | 31.1 | 28.8 | |
| ELF-L with encoder | LM1B | Paired frozen sampling at small budget | 180.8 | 165.2 | |
| ELF-L with encoder | LM1B | Paired frozen sampling at full budget | 108.9 | 107.1 | |
| ELF-L with encoder | OWT | Paired frozen sampling at small budget | 194.8 | 176.4 | |
| ELF-L with encoder | OWT | Paired frozen sampling at full budget | 23.3 | 22.8 | |
| LangFlow [1] | LM1B | Paired frozen sampling at small budget | 238.4 | 244.1 | |
| LangFlow [1] | LM1B | Paired frozen sampling at full budget | 92.2 | 94.1 | |
| LangFlow [1] | OWT | Paired frozen sampling at small budget | 192.5 | 198.2 | |
| LangFlow [1] | OWT | Paired frozen sampling at full budget | 36.5 | 37.3 |
↳ Manuscript updates: Sec. 1, Secs. 4.1–4.2, Sec. 5, and App. D.3 (Table 8).
Response to Reviewer Sx98 (3/4)
Q3 — Diversity Metrics, Temperature Matching, & Llama-3-8B Evaluation
Reviewer Sx98 (Q3): Scope of GPT-2 Large GenPPL and unigram token entropy as quality and diversity metrics.
Answer: GenPPL reductions persist after matching corpus-level diversity via logit temperature and under independent Llama-3-8B evaluation.
- Training & ELF sampling diversity: In Table 9 below, KIS training on ELF-B LM1B lowers GenPPL while moving token and -gram diversity closer to held-out LM1B text, and across ELF-B/M/L sampling and remain within of default.
- Diversity-matched frontier & Llama-3-8B: On FLM-B [4] OWT sampling, matching baseline diversity at categorical logit temperature lowers both GPT-2 Large () and Llama-3-8B () GenPPL, and Llama-3-8B preserves all OWT rankings.
↳ Manuscript updates: Secs. 4.1–4.2 and App. D.3 (Table 9).
| Model, Corpus, and Evaluation Mode | GenPPL GPT-2 Large | GenPPL Llama-3-8B | nats | Distinct-2 | Distinct-3 | nats |
|---|---|---|---|---|---|---|
| Held-out LM1B reference text, | n/a | n/a | 4.3 | 0.62 | 0.88 | 10.96 |
| Held-out OWT reference text, | n/a | n/a | 5.3 | 0.48 | 0.81 | 12.93 |
| ELF-B on LM1B Training, , Table 1 Default KIS | 121.2 79.3 | n/a | 3.4 3.6 | 0.52 0.55 | 0.78 0.81 | 10.74 10.80 |
| FLM-B on LM1B Training, , Table 1 Default KIS | 96.9 92.5 | n/a | 4.3 4.3 | 0.61 0.61 | 0.87 0.87 | 10.92 10.92 |
| FLM-B on LM1B Sampling, , Default KIS | 98.0 94.0 | n/a | 4.3 4.3 | 0.61 0.61 | 0.87 0.87 | 10.92 10.91 |
| FLM-B on OWT Sampling, , Default KIS at | 62.2 46.3 | 48.4 37.9 | 5.3 5.1 | 0.47 0.45 | 0.80 0.78 | 12.89 12.85 |
| FLM-B on OWT Sampling, , KIS at | 62.2 47.9 | 48.4 39.0 | 5.3 5.2 | 0.47 0.46 | 0.80 0.79 | 12.89 12.88 |
| FLM-B on OWT Sampling, , KIS diversity-matched () | 62.2 48.9 | 48.4 39.6 | 5.3 5.3 | 0.47 0.47 | 0.80 0.80 | 12.89 12.89 |
| FLM-B on OWT Sampling, , KIS at | 62.2 51.4 | 48.4 41.2 | 5.3 5.4 | 0.47 0.48 | 0.80 0.81 | 12.89 12.91 |
| ELF-B on LM1B Sampling, , Default KIS | 121.2 115.1 | n/a | 3.4 3.4 | 0.52 0.52 | 0.78 0.78 | 10.74 10.74 |
| ELF-B on OWT Sampling, , Default KIS | 23.6 21.8 | 21.4 19.8 | 5.2 5.2 | 0.46 0.46 | 0.79 0.79 | 12.88 12.87 |
| ELF-M on OWT Sampling, , Default KIS | 21.7 20.6 | 19.9 19.0 | 5.2 5.2 | 0.46 0.46 | 0.79 0.79 | 12.88 12.88 |
| ELF-L on OWT Sampling, , Default KIS | 23.3 22.8 | 21.1 20.7 | 5.3 5.3 | 0.47 0.47 | 0.80 0.80 | 12.89 12.89 |
| LangFlow on OWT Sampling, , Default KIS | 36.5 37.3 | 31.2 31.8 | 5.3 5.2 | 0.47 0.46 | 0.80 0.79 | 12.89 12.87 |
Response to Reviewer Sx98 (1/4)
Q1 — Quantifying the T5 Lookup Proxy vs. Contextual Latents in ELF
Answer: Static T5 lookup captures the slope of the denoiser's SNR-weighted loss profile, and contextual centroids close the remaining scale offset.
↳ Manuscript updates: Sec. 4.4 (Table 6), App. E.2 (Table 10, Fig. 5), and App. E.3 (Fig. 7).