- Rating
- 2(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper studies inference-time control for fine-tuned masked diffusion language models (LLaDA-8B) on globally constrained problems, including ZebraLogic, Nurse Rostering, and JSSP. The authors use mean confidence over masked tokens to trigger corrective inference, guide lookahead search and backtracking, and selectively invoke Best-of-N sampling. The approach improves over standard greedy decoding on the same LLaDA checkpoint and outperforms matched-scale autoregressive baselines.
Critical strengths
-
The paper is clear and easy to follow.
-
Selectively triggering additional inference using late-phase confidence is a simple and practical idea. It substantially reduces NFE compared to unconditional intervention while leaving most already-correct greedy trajectories untouched.
-
The paper evaluates its approach on well-chosen tasks spanning constraint satisfaction and combinatorial optimization.
Critical weaknesses
1. Separating the known benefits of adaptive-order decoding from Blackboard's contribution
A central theme of the paper is that dLLMs can outperform AR models on globally constrained problems. However, prior work has already demonstrated the benefits of adaptive-order unmasking on puzzles such as Sudoku and ZebraLogic ([1, 2]). In this paper, most of the gap between LLaMA-8B and LLaDA-8B is already present under standard confidence-greedy decoding (+43.8 pp on ZebraLogic-Hard and +59.2 pp on Nurse Rostering), compared to Blackboard's additional gains of +12.0 pp and +3.4 pp. These additional gains are meaningful, but the framing should more clearly distinguish Blackboard's contribution from the established benefits of adaptive-order decoding.
[1] Ye et al., Beyond Autoregression: Discrete Diffusion for Complex Reasoning and Planning, in ICLR 2025.
[2] Kim et al., Train for the Worst, Plan for the Best: Understanding Token Ordering in Masked Diffusions, in ICML 2025.
2. The Gibbs bound does not justify mean confidence as a search criterion
In Section 3.2, the paper motivates mean confidence as a proxy for global coherence through the Gibbs bound
\leq \frac{|M|}{\beta}\psi(\mathcal{C}(z)).$$ However, this bound only controls the expected excess potential above $V^\star(z)$, the minimum potential attainable from the current state, rather than $V^\star(z)$ itself. When comparing search branches, what matters is whether a partial state can still lead to a valid or optimal solution. A wrong branch ($V^\star(z)>0$) can still have $\mathcal{C}(z)\approx 1$ if the remaining completions concentrate on a single least-bad solution. Conversely, a valid branch ($V^\star(z)=0$) can have lower confidence if probability is split across multiple optimal completions. Mean confidence can therefore favor a dead-end branch over a valid one. These limitations may also help explain some of the empirical results. On Nurse Rostering, unconditional confidence-guided correction overwrites already-correct rosters and reduces accuracy from 73.0% to 57.4% (Table 16), raising concerns about the reliability of local search decisions. On JSSP, where multiple optimal solutions exist, the pre-test trigger precision is only 21.4% (Table 21). These results raise questions about mean confidence as both a branch-ranking score and a failure-detection signal, neither of which is justified by the Gibbs bound. >**3. Lack of direct validation of confidence-guided reasoning trajectories** The paper presents BACKTRACK and SEARCH as ways to maintain global coherence, but does not directly evaluate the quality of their local decisions. For CSPs like ZebraLogic and Nurse Rostering, a useful reference would be the deduction trace of an efficient solver that derives later assignments from earlier ones through explicit, deterministic rules. Comparing the model's commitment order with this reference would help determine whether its search follows the problem's deductive dependencies rather than simply selecting high-confidence cells. Since both tasks also have a unique ground-truth solution $x^\star$, the authors could directly check whether BACKTRACK rejects incorrect fills and whether SEARCH selects branches consistent with $x^\star$. Without these analyses, it remains unclear whether confidence-guided search actually captures the logical structure of the problem. >**4. Overstated "revisable canvas" framing** The paper emphasizes a "persistent, revisable canvas" where earlier decisions can be revised, but Algorithm 1 implements a much more limited form of revision. When the trigger fires, the model discards the greedy output and restarts with a blank answer table. During the second pass, previously accepted cells are never revisited; BACKTRACK only cancels the immediately proposed fill. On JSSP and ATSP, mean confidence is not used to rank partial search states at all. It only triggers Best-of-10 sampling, where the best candidate is selected using the exact task objective (makespan or tour length). These experiments demonstrate confidence-guided inference and selective computation, but not the general canvas revision implied by the framing.Questions for authors
-
On ZebraLogic-Hard and Nurse Rostering, how closely does the model's unmasking order align with the deduction trace of an efficient reference solver? When the current partial state is consistent with the unique solution , how often does BACKTRACK reject an incorrect versus a correct fill? Similarly, how often does SEARCH select a branch consistent with when such a branch exists among the candidates?
-
In SEARCH, how does mean confidence compare with alternative scoring rules, such as cumulative rollout log-probability or mean predictive entropy, under the same candidate pool and search budget?
-
Could you clarify how NFE is counted in SEARCH? Section 4.1 describes branching over the top- candidates at each masked cell and running a depth- rollout per branch. With repeated SEARCH calls, a naive cost of candidate evaluations per call could amount to thousands per puzzle, making the reported averages of 129 NFE for Blackboard and 280 for always-on (Table 1) surprisingly small. Could you provide the exact SEARCH pseudocode and clarify the branching scope and NFE accounting?
5 comments
Response to Reviewer wEKX - Part2
W2 / Q2. The Gibbs Bound and Empirical Utility of Mean Confidence (Continued)
(1) Alternative SEARCH scoring (Q2). We compared scoring rules using identical candidate pools, rollout depth (d = 3), and per-call evaluation budgets, changing only the SEARCH score.
| Triggered puzzles solved (NFE) | ZL (n = 92) | NR (n = 98) |
|---|---|---|
| Mean confidence (ours) | 76.1% (558) | 21.4% (522) |
| Negative mean entropy | 72.8% (560) | 19.4% (532) |
| Cumulative rollout log-probability | 18.5% (1466) | 15.3% (1343) |
| Log-probability + confidence-gated commitment | 23.9% (1677) | 12.2% (1696) |
Mean confidence performs comparably to negative mean entropy but substantially outperforms cumulative rollout log-probability. Both mean confidence and negative mean entropy evaluate the predictive state induced by a candidate continuation, aggregating confidence or uncertainty across the remaining masked positions rather than scoring the likelihood of the sampled path. In contrast, log-probability-selected paths fail to increase remaining-state confidence in 58% / 78% of SEARCH calls (ZL / NR), leading to repeated search and substantially higher NFE without comparable accuracy gains. Applying the same confidence-based commitment rule does not close this gap, highlighting the value of state-level evaluation under globally coupled constraints.
(2) Selective intervention (NR and JSSP). Beyond individual commitments, Blackboard uses late-phase trajectory-level mean confidence to decide whether additional inference is warranted, avoiding unnecessary intervention on reliable greedy trajectories.
On NR, unconditional correction reduces accuracy from 73.0% to 57.4%, whereas selective triggering achieves 95.9% precision for greedy failures and improves accuracy to 76.4%, with 19 recoveries and only 2 regressions. On JSSP, confidence instead triggers additional sampling, with the exact task objective selecting the final solution. Despite imperfect trigger precision (21.4% on training traces; 41.2% on test), retaining the greedy schedule prevents regressions, while 41 triggered instances improve to optimality.
Together, these results show that mean confidence need not perfectly evaluate every state to be useful. By exploiting this signal across evolving partial states and inference stages, Blackboard coordinates search and selective correction to improve end-to-end performance despite imperfect individual judgments.
W3 / Q1. Validating State-Dependent Inference Decisions
We directly evaluated commitment and search decisions against the unique ground-truth solution on ZebraLogic-Hard (ZL) and Nurse Rostering (NR).
(1) Deductive structure and error propagation.
We ran exact constraint propagation from the currently committed cells to examine whether the model's commitment order reflects logical dependencies.
| ZL | NR | |
|---|---|---|
| Commits a propagation-forced cell when available | 98.8% | 89.2% |
| Commit accuracy on cells not yet resolved by propagation | 80.3% | 94.7% |
The model frequently commits deductively determined cells when available, though not in the solver's exact sequence. Incorrect commitments have cascading consequences: after the first wrong fill, greedy's next-commit error rate rises from 1.6% to 63.5% (ZL) and from 1.5% to 50.9% (NR), consistent with the constraint propagation observed in Appendix D.3. This motivates evaluating alternative partial states rather than relying on a single irreversible greedy trajectory.
(2) Coordinated commitment and correction.
On triggered puzzles, the α-gate coordinates BACKTRACK and SEARCH using mean confidence over evolving partial states. BACKTRACK commits or defers proposed fills according to their effect on predictions across remaining masked positions, while SEARCH evaluates alternative 3-step continuations. Together, they adapt commitment order and guide corrective inference.
BACKTRACK regulates commitment rather than classifying correctness. It defers 29.6% / 73.0% of correct proposals (ZL / NR), alongside 0/1 and 24/45 incorrect proposals, respectively. Importantly, 100% / 98.6% of deferred correct fills are eventually completed correctly, illustrating how postponed commitments can be resolved as the partial state evolves.
SEARCH selects ground-truth-consistent branches in 85.7% / 34.1% of cases (random: 19.0% / 19.3%). On ZL, all 13 observed potential incorrect greedy commitments routed to SEARCH were corrected.
Together, these results show how Blackboard exploits evolving partial states to coordinate commitment, deferral, and corrective search, rather than merely selecting locally high-confidence cells.
Response to Reviewer wEKX - Part3
W4. Scope of the Revisable Canvas
As discussed in W1–W3, Blackboard refers to a state-level inference interface enabled by the joint use of any-order prediction, partial-state conditioning, and model-internal mean confidence, rather than unrestricted revision of previously committed tokens. Its central capability is to evaluate and adapt decisions over evolving partial solutions under globally coupled constraints. For ZebraLogic and Nurse Rostering, Algorithm 1 implements this perspective through selective restarting, local backtracking, and search over alternative partial-state continuations. BACKTRACK tentatively fills a cell and reverts (remasks) it if mean confidence over the remaining masked positions decreases. While previously accepted commitments are not arbitrarily revisited, this local revision, together with SEARCH, allows Blackboard to adapt the commitment order and redirect inference before potentially unfavorable decisions become irreversible, as supported by our decision-level analysis in W3/Q1. For JSSP and ATSP, confidence serves a different role: it determines when additional complete-solution sampling is warranted, while the exact task objective selects among candidates. These experiments instantiate confidence-based selective computation rather than partial-state revision.
Thus, Blackboard is an inference-time perspective that supports different forms of state-dependent computation, with task-specific implementations rather than a single universal revision operator.
Q3. SEARCH Implementation and NFE Accounting
The corrective procedure can be summarized as follows:
Correction (restart from blank answer region):
while masked cells remain:
evaluate current partial state # 1 NFE
if mean confidence < α: # SEARCH
for each masked cell i and top-k value v:
evaluate depth-d continuation # ~d NFE
accept best improving continuation,
or fall back to greedy commitment
else: # BACKTRACK
evaluate candidate commitments
accept a non-decreasing-confidence fill,
or invoke SEARCH if none is accepted
One NFE corresponds to one model forward pass. As the reviewer correctly notes, a SEARCH call evaluates the top-k values at each remaining masked position through depth-d rollouts, requiring approximately |M|kd model forward passes, all of which are included in the reported NFE. The total inference cost is:
NFE = NFE_greedy + I(trigger) × NFE_correction
Indeed, corrective inference can be computationally expensive: on ZebraLogic-Hard, triggered puzzles incur an average of 558 NFE. However, Blackboard's confidence-guided inference strategy selectively allocates this additional computation. Adaptive greedy decoding already constructs solutions through state-dependent commitments, while mean confidence determines when further correction is warranted.
On ZebraLogic-Hard, 81.6% of puzzles retain their greedy solutions, with only 18.4% triggering correction. Among these, SEARCH is invoked approximately 2.1 times on average, with its cost decreasing as fewer masked positions remain. Consequently, Blackboard achieves an overall average of 129 NFE, substantially below the cost of applying corrective search to every puzzle.
Taken together, these additional analyses clarify Blackboard's contribution beyond adaptive-order decoding, provide direct evidence for the practical utility of state-level mean confidence, and validate its role in coordinating commitment and corrective search.*
We hope this evidence, together with the clarified theoretical scope and implementation details, addresses the reviewer's soundness concerns and supports reconsideration of the overall assessment.
Response to Reviewer wEKX - Part1
We appreciate the reviewer's careful technical scrutiny of our work and the detailed questions concerning Blackboard's foundations and implementation. These comments motivated us to further examine the proposed inference mechanisms through additional controlled experiments, direct decision-level validation, and implementation clarifications.
Our responses address five key concerns:
W1. Separating Adaptive-Order Decoding from Blackboard
We agree that prior work [Ye et al., 2025; Kim et al., 2025] has established the benefits of adaptive-order decoding. Importantly, adaptive-order commitment is a foundational component of Blackboard, rather than a separate or competing mechanism. As stated in our original submission (Lines 67–73), our central claim is that the standard masked diffusion objective provides a state-level inference interface for problem solving under complex global constraints. Building on confidence-based adaptive decoding, which we already recognize as a standard dLLM inference strategy (Lines 131–139), Blackboard exploits any-order prediction and partial-state conditioning to enable state-dependent search, commitment deferral, and correction. This is particularly valuable when individual assignments are coupled through global constraints, where locally plausible decisions can compromise the feasibility of the entire solution. Our paper already quantifies these additional gains over adaptive greedy decoding using the same fine-tuned checkpoint (Lines 380–391). To further clarify these contributions, we conducted a forced left-to-right control using the same LLaDA weights:
The first improvement captures the established benefits of adaptive-order commitment (+14.0 / +28.2 / +6.0 pp), while the second demonstrates the additional gains from Blackboard's state-dependent inference procedure (+12.0 / +3.4 / +10.2 pp). These gains are empirically distinguishable but mechanistically connected. Together, they demonstrate how the dLLM's any-order inference interface and state-level confidence signal enable richer inference-time problem solving beyond greedy decoding on globally constrained problems.
W2 / Q2. The Gibbs Bound and Empirical Utility of Mean Confidence
We acknowledge that our Gibbs analysis does not guarantee global feasibility or optimal branch selection. Rather, it provides theoretical motivation for mean confidence as a state-level inference signal. As defined in Section 3.2 (Lines 218–226), the bound concerns excess potential relative to the best completion compatible with the current partial state; Appendix B (Lines 785–788) explicitly notes its idealized nature and the distinction from model-derived confidence.
Building on this theoretical motivation, the practical question is whether model-derived mean confidence provides a useful signal for guiding inference. Although individual confidence judgments may favor incorrect branches, Blackboard uses this signal across evolving partial states and multiple inference stages to guide commitment, deferral, search, and selective intervention. We examine its empirical utility through the following additional experiments.