- Rating
- 2
- Confidence
- 1
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes LEDFlow, a training free sampler, which absorbs lowest entropy positions in discrete flow through selective absorption.
Critical strengths
Emperical analysis. The emperical analysis is good. Multiple samplers are compared on a range of tasks (multimodal udnerstanding, text-to-image generation, constrained reasoning) and multiple datasets. The proposed approach consistently performs well, achieving top ranks across the board.
Critical weaknesses
Theory
Definition 1: This is a very key point so the argument should be strengthened. Is this a realistic assumption to make? Additional validation should be performed or clearer justification on why this is the case.
Presentation The presentation could be improved and simplified. Some more minor suggestions to improve clarity:
-
Global lookahead/info gain: This takes up a significant amount of room/discussion, e.g. at the end of Section 4. As this is just a comparison, it would be clearer to move such discussions to the main body. It can detract from the main focus, and the additional discussion e.g. in line 85-87 feels misplaced.
-
In the abstract, mentioning 0.845 solve accuracy is not very useful without closest comparison mentioned. Either remove it or add other results for comaprison.
-
L155-161: Everything relies crucially on the denoiser itself. Please promote anything important from the appendix to the main body to strengthen these claims, or add some experimental validation around the denoiser itself, e.g. how entropy correlates with correctness empirically.
-
Figure 2 caption says LEDflow rises monotonically. This just happens to be the case in one instance? Can error bars be added to this plot, or multiple lines per run.
-
In figure 1 it looks like entropy is normalised 0..1, but this isnt the case in the paper (L194). A note should be added to the fig to avoid confusion.
-
Algorithm 1 could benefit from adding in line natural language comments, or replacing some of the heavy notation.
-
The appendix is very difficult to parse in its current structure.
Questions for authors
The main point to clarity is around Theorem 1 (and Definition 1), this entropy-error regularity. How often is this idealized case violated? Its mentioned 95.5% of sampled Sudoku states? What is the impact when it's violated, e.g. in the 4.5% of cases? The paper crucially relies on this, so more analysis is warranted.
-
In the results tables -- Can you please confirm how the rank is generated, are the scores averaged over 5 runs, and then the rank based on these averages? Or is the rank computed for each run and averaged?
-
In figure 2, its clear that earlier stopping could help the uniform flow. Can early stopping criteria be compared to as well, e.g. adding conditions exploring just earlier stopping rather than selective absorption.
5 comments
Replies to Reviewer sqK1: Questions
Questions:
Q1 Violation of idealised case:
We have detailed theoretical justifications in Theory Weakness Def. 1 above. Regarding the empirical evidence, we show that Def. 1 satisfies 95.5% of sampled reachable states for the model we use in main text, where a relaxed envelope in Eq. 50: , where , further holds on 99.4% of the states. Table 19 (attached below) also provides details that shows error on each cell rises monotonically with entropy (now App. F.3, Table 19 in the revised manuscript), and cells that violate the bound are wrong.
| (bin mean) | 0.00 | 0.24 | 0.46 | 0.64 | 0.79 | 1.08 |
|---|---|---|---|---|---|---|
| Per-cell error | 0.015 | 0.120 | 0.222 | 0.343 | 0.399 | 0.526 |
Q2 Rank generation:
We compute the scores of each run from five seeds and aggregate the scores on each dataset. We then rank those aggregate scores and then compute the average ranking.
Q3: Early stopping:
The absorbing state can somehow be considered as early stopping at each position. Nevertheless, we interpret your idea of early stopping as early stopping over the entire puzzle. On that regard, we pretrain another two checkpoints which 1) replaces the position-wise absorbing indicator with a single early stopping indicator, and 2) adds an additional early stopping indicator by training an additional sigmoid layer to take in the position-wise absorbing indicator. We undergo the same evaluation on Nikoli dataset (App. G.1, Table 22):
| Procedure | Accuracy |
|---|---|
| Uniform flow | 0.410 |
| Single early stopping indicator | 0.364 |
| Early stopping on position indicator | 0.452 |
| Entropy selective absorption | 0.845 |
It shows that 1) the position wise indicators contain signals to validate the model's internal thoughts on each position, which is consistent with our comparison at Table 14 at App. F.1 between pretraining without indicator versus pretraining with indicator but not using it at inference, and 2) the current early stopping decisions seem to be overconfident in both designs, so that in training, it loses the learning on trajectories where the low-entropy positions can be refined at later steps. We are happy to discuss further if you have further early stopping criteria which you would like to test.
Please let us know if you have further concerns!
References
[1] Yang et al. Improving sampling for masked diffusion models via information gain. ICML, 2026.
If you have any further concerns regarding our replies, please let us know. We are happy to discuss further in the remaining three days of rebuttal period. Thanks!
Replies to Reviewer sqK1: Weaknesses
Thank you for your comments. We have updated the paper with a new PDF file uploaded with your comments accordingly (updates in blue). Please find our replies as follows:
Weaknesses:
Theory
Def. 1:
We provide the following example using Sudoku for its unique trajectory solution. The ϕ-regularity follows the relationship between predictive entropy and probability assigned to the correct digit. Given state s where previous absorbed positions are correct, at each remaining position j, the true conditional distribution should assign probability one to the solution digit yj∗, so that the KL divergence from the denoiser's predicted distribution Qθ(⋅∣s,j) is −logQj(yj∗). Now let pjmax=maxzQj(z) and define log probability ratio Δj(s)=log(pjmax/Qj(yj∗)), we have H(Qj)=EZ∼Qj(−logQj(Z)), which is greater than or equal to −logpjmax, so that Dj(s)=−logpjmax+Δj(s)≤H(Qj)+Δj(s). When the correct digit has the highest predicted probability, the ratio is zero so the predictive entropy bounds KL divergence. If the ratio is less than cH(Qj) for some non-negative constant c, ϕ(h)=(1+c)h then satisfies Def. 1. Next, regarding empirical evidence, we show that Def. 1 satisfies 95.5% of sampled reachable states for the model we use in main text, where a relaxed envelope in Eq. 50: KL(p1∣skq(a1j)Q1∣tkθ(a1j∣sk))≤ϕ0+ϕ(Hθj(sk)), where ϕ0≥0, further holds on 99.4% of the states. Table 19 (attached below) also provides details that shows error on each cell rises monotonically with entropy (now App. F.3, Table 19 in the revised manuscript).
Nevertheless, we conducted a further out-of-distribution scenario for Def. 1 by fitting a monotone envelope on the puzzles, which we found the envelope covers 95% trajectories and 99.9% of positions. For selected cells with entropy less than 0.05, at most 0.41% are wrong (App. F.3, Table 17).
Presentation
W1: Global lookahead/info gain:
Our primary contribution introduces ordering in uniform discrete flow via absorbing state, with a stepwise greedy decoding on entropy. There are two questions: 1) Why entropy (Thm. 1) and 2) Why greedy decoding because a recent work [1] proposes that "selecting locally certain tokens ... leads to suboptimal generation", motivating our choice of Top K-margin and Info-Gain as SOTA baselines of each angle. In particular, LEDFlow ranks positions using the current prediction. Lookahead temporarily fixes each candidate to its most likely token, reruns the denoiser, and scores how uncertain the remaining positions would be. We added one line (L316 - 318) to explain why is global lookahead a key idea that we have to analyse and compare with.
0.845 mention:
We add the 0.410 from ablation baseline into both abstract (L24 - 25) and introduction (L87 - 88) to ease comparison.
Justification of entropy and correctness:
Line 287 already cites [1] regarding negative relationship between prediction entropy and reasoning accuracy, which we further verify with Table 19 at App. F.3 (attached below):
As per your suggestion, we add the line with citation [1] at L181 - 182 to improve clarity.
Figure 2 results:
It is run on five seeds (0 - 4) to prevent bias from single instance. We add a line (L172) to clarify it clearly in main text with reference pointer towards Appendix F.1, as well as error bands on Figure 2(a).
Figure 1 note:
We add a note at Figure 1 caption (L67) to state explicitly that the entropy values visualised there are normalised to ease reading.
Algo 1 notation:
Well noted with many thanks. We incorporate some natural language descriptions in Algo 1.
Appendix structure and wordings:
We have added a new page of content for the Appendix and updated wordings in Appendix accordingly to ease reading.