- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper examines how MDMs can improve from better decoding strategies, based on the current state. The authors introduce a framework to map decoding decisions into five axes. The contributions are the framework for measuring opportunities to improve decoding decisions, using an oracle at the final state. The authors also introduce an adaptation procedure that specifies when to apply an alternative action and which one.
The evaluation is done on three models and ten tasks, with interventions and then the normal continuation. Their stated finding is that opportunities and the predictability of opportunities varies significantly. The experiments show mild utility improvements of 3-6 percentage points over a static policy.
Critical strengths
Their framework separates the predictability of an alternative decoding step from the increase in utility. This helps, as it narrows the headroom provided by the oracle and allows for benchmarking alternative decoding methods.
Critical weaknesses
- The reported numbers do not cleanly reconcile: for LLaDA-8B on constrained JSON filling, Table 10 reports an oracle gap of 0.0287, and Table 13 reports the fixed and oracle utility of 0.9078 and 0.9875, respectively, implying a gap of 0.0797, which is impossible by Equation 4. Note that I am having trouble reconciling the numbers in general, as the result tables in the appendix are 20 pages with formatting problems. Further, Table 10 reports the 100% coverage lift at 0.0594, which exceeds the Table 10 oracle gap. It is unclear to me how the policy improves beyond the oracle gap, which should not be possible. This number substantiates the headline result in the abstract and is the basis for the selective adaptation argument, thus reducing confidence in the contribution.
- The aggregated gains provide limited evidence for the added value of the proposed selective gating. Dream's gain is largely driven by one dataset; GSM8k. Selective decoding improves only 3/10, 4/10, 3/10 tasks across the three models. Further, the AUC of the state-level detector is 0.54-0.57, which is slightly better than guessing. This reduces my confidence in the proposed method's contribution.
- The headline decoding settings gives zero end-to-end utility: LaDA-8B Constrained JSON / region is the abstract’s headline (0.854 AUROC, 56.9% capture). End-to-end, Fixed, Always and Selective all score 0.000, while the reference decoder scores 0.320.
Questions for authors
- Could the authors clarify the numerical consistencies stated above, and reconcile the headline numbers used in the table 5 and the abstract?
- Could the authors clarify whether the realized lift of 0.0594 at 100% coverage exceeds the oracle gap stated in Table 10?
- How much of the improvement of is attributable to gating rather than to the alternative action? Dream's GSM8k improvement contributes about 80% of the aggregate gain. Please also provide uncertainty estimates for these numbers to inspect significance.
- What are the validation set sizes and sources for each task? Please further clarify what role the validation data plays and also evaluate the held-out performance on completely unseen datasets.
5 comments
Official Rebuttal [2/5]
W2: The aggregated gains provide limited evidence for the added value of the proposed selective gating. Dream's gain is largely driven by one dataset; GSM8k. Selective decoding improves only 3/10, 4/10, 3/10 tasks across the three models. Further, the AUC of the state-level detector is 0.54-0.57, which is slightly better than guessing. This reduces my confidence in the proposed method's contribution.
A2: The criticism of a universal improvement claim is justified. Mean AUROC 0.54–0.57 is modest, and the large Dream/GSM8K improvement at 100% coverage does not measure gating. We therefore focus on where the detector and controller yield an empirically supported benefit, including matched-coverage Random controls and disjoint final-test prompts. Low positive-opportunity prevalence can make performance difficult to estimate, but does not by itself force AUROC toward 0.5; we do not use it to dismiss the reported mean AUROC.
| Model | Fixed | Selective | Sel − Fixed | Tasks improved | Tasks worsened | Ex-GSM8K lift | Sel − Random macro |
|---|---|---|---|---|---|---|---|
| LLaDA-8B | 0.478 | 0.523 | +0.045 | 5/10 | 0/10 | +0.050 | +0.033 |
| LLaDA-1.5 | 0.317 | 0.391 | +0.074 | 6/10 | 0/10 | +0.082 | +0.062 |
| Dream-7B | 0.428 | 0.497 | +0.070 | 3/10 | 1/10 | +0.027 | +0.020 |
On the new final-test split, Selective improves 5/10, 6/10, and 3/10 tasks (the original submission reported 3/10, 4/10, and 3/10 on the preliminary evaluation). To separate timing from other effects, we group the 30 settings in Table 18 by coverage: nine have 0% coverage (Selective never deviates from Fixed); three have 100% coverage (LLaDA-8B/TD07 +0.055, Dream/GSM8K +0.450, Dream/Multi-Span Cloze 0.000), which reflect an alternative-rule effect rather than timing; and five have partial coverage but identical Fixed, Always, Selective, and Random utilities. Timing can therefore be evaluated in the remaining 13 settings. Excluding all three 100%-coverage settings, the macro Sel − Fixed gain is +0.044 (LLaDA-8B, 9 tasks), +0.074 (LLaDA-1.5, 10 tasks), and +0.031 (Dream, 8 tasks); the Ex-GSM8K column above still includes LLaDA-8B/TD07. We will add this categorization to Appendix I.6.
For a compact view of state-aware timing, here are the strongest and most uncertain cases, with paired 95% bootstrap CIs. These are all compared on the same prompt sets per setting.
| Model | Task | Always − Fixed (95% CI) | Sel − Random (95% CI) | Sel − Fixed (95% CI) |
|---|---|---|---|---|
| LLaDA-8B | Constrained JSON Fill | -0.043 [-0.075, -0.011] | +0.046 [+0.033, +0.059] | +0.053 [+0.025, +0.081] |
| LLaDA-8B | Unique List Commit | -0.205 [-0.233, -0.177] | +0.185 [+0.154, +0.216] | +0.215 [+0.184, +0.246] |
| LLaDA-1.5 | Carry RTL | -0.035 [-0.064, -0.006] | +0.106 [+0.063, +0.149] | +0.130 [+0.100, +0.160] |
| LLaDA-1.5 | CSV Missing Cells | -0.002 [-0.018, +0.014] | +0.031 [-0.002, +0.064] | +0.040 [+0.028, +0.051] |
| LLaDA-1.5 | JSON Mode Eval | -0.039 [-0.244, +0.167] | +0.184 [+0.043, +0.326] | +0.222 [-0.006, +0.454] |
| Dream | Carry RTL | -0.100 [-0.116, -0.083] | +0.136 [+0.120, +0.152] | +0.169 [+0.152, +0.185] |
In 12/30 settings, unconditional switching has a negative point estimate and Selective beats Fixed, Always, and Random. These 12 are all of the 13 timing-evaluable settings except Dream/Unique List, where Selective is −0.006 [−0.017, +0.005] relative to both Fixed and Random. The three selective contrasts have positive 95% intervals simultaneously in 10/12 settings; eight also have a negative Always − Fixed interval. The two exceptions have intervals crossing zero, and we do not claim universal significance or multiple-comparison adjustment. In LLaDA-8B/CJSON, the one-step detector AUROC is 0.854. In a distinct Dream Carry/cardinality test, AUROC is 0.802, with end-to-end Sel − Random +0.066 [0.050, 0.083]. These results support selective gains in specific regimes rather than generally strong detection.
Official Rebuttal [3/5]
W3: The headline decoding settings gives zero end-to-end utility: LaDA-8B Constrained JSON / region is the abstract’s headline (0.854 AUROC, 56.9% capture). End-to-end, Fixed, Always and Selective all score 0.000, while the reference decoder scores 0.320.
A3:The original 0.000 end-to-end CJSON result is a real failure, not a reporting typo. An audit showed that its preliminary prompts differed in template/constraint composition from calibration, causing the chosen region action to fail output extraction. Also, one-step utility need not translate to repeated decoding because the visited-state distribution changes. Below we report the original pilot and the corrected new, disjoint, matched-generator 500-prompt test separately; they are not a controlled before/after comparison.
| LLaDA-8B / CJSON / region | n | Ref | Fixed | Stage | Always | Random | Selective | Coverage |
|---|---|---|---|---|---|---|---|---|
| Original mismatched pilot | preliminary | 0.320 | 0.000 | — | 0.000 | — | 0.000 | — |
| New generator-matched final test | 500 | 0.920 | 0.908 | 0.912 | 0.865 | 0.915 | 0.961 | 0.250 |
On the new split, Sel − Fixed = +0.053 [0.025, 0.081], Sel − Always = +0.096 [0.065, 0.127], and Sel − Random = +0.046 [0.033, 0.059] (paired 95% bootstrap). The original failure reveals sensitivity to distribution shift; the revised claim will be restricted to the matched-generator evaluation and will not infer end-to-end success from AUROC alone.
Q1: Could the authors clarify the numerical consistencies stated above, and reconcile the headline numbers used in the table 5 and the abstract?
A1:The abstract and Tables 5–6 and 12–15 use 640 held-out states with a 20-prompt validation-selected fixed action. Original Table 10 instead uses 800 states with a hindsight-selected fixed action. The exact headline quantities are:
| Source | Evaluation definition | Value |
|---|---|---|
| Original Table 10 oracle gap | 800 states; hindsight fixed | 0.0287 |
| Tables 12–13 oracle gap | 640 held-out; validation fixed | 0.0797 [0.0625, 0.0984] |
| Table 4 detector AUROC | 640 held-out; validation fixed | 0.854 |
| Tables 4–5 capture at 10% coverage | 640 held-out; validation fixed | 56.9% |
| Tables 4–5 lift at 10% coverage | 640 held-out; validation fixed | +0.0453 |
| Tables 5 / 13 / 15 lift at 100% coverage | 640 held-out; validation fixed | +0.0594 |
The related utilities are Fixed 0.9078, Oracle 0.9875, and full-coverage Diagnostic 0.9672. We will relabel the estimands in the abstract, main tables, and appendix so these figures are not treated as interchangeable.
Q2: Could the authors clarify whether the realized lift of 0.0594 at 100% coverage exceeds the oracle gap stated in Table 10?
A2: No. The applicable candidate-set oracle gap is 0.0797, not the original Table 10 value 0.0287, because the 100%-coverage diagnostic uses the same 640 held-out states and validation-selected fixed action as the former.
| Quantity (640 held-out states) | Value |
|---|---|
| Validation-selected Fixed | 0.9078 |
| Candidate-set Oracle | 0.9875 |
| Diagnostic at 100% coverage | 0.9672 |
| Oracle − Fixed | +0.0797 [0.0625, 0.0984] |
| Diagnostic − Fixed | +0.0594 |
| Oracle − Diagnostic | +0.0203 |
Therefore +0.0594 ≤ +0.0797; no bound is violated under the consistent evaluation definition. The separate hindsight-fixed 800-state gap is not the relevant bound.
Official Rebuttal [1/5]
We thank the reviewer for checking the numerical results so closely. The inconsistent captions and difficult appendix layout made key comparisons appear invalid; we apologize for that presentation failure. We have traced the reported quantities to their evaluation definitions, audited the tables, and expanded the end-to-end tests to address the substantive concern about selective gating. We respond to each point below.
A1: The apparent violation is caused by comparing different state sets and different fixed baselines. The table below reconciles the exact values for LLaDA-8B/Constrained JSON Fill/region; the full-coverage lift and the oracle gap are compatible when measured on the same 640 held-out states.
Thus 0.9672 − 0.9078 = +0.0594 ≤ 0.9875 − 0.9078 = +0.0797 (74.5% recovery; oracle − diagnostic = 0.0203). The original 0.0287 Table 10 gap used 800 states and a hindsight-selected baseline, so it cannot be used as the upper bound for this held-out diagnostic lift. The same 640-state hindsight-fixed gap is 0.0125, making clear that validation-fixed opportunity includes calibration loss. We audited all 104 model–task–axis settings with held-out transition-level utilities (Tables 12–13; the remaining 16 settings, HumanEval for all three models and LLaDA-8B GSM8K, have no compatible held-out branch table) and found zero violations: the oracle utility is never below the Fixed, Stage, or Diagnostic utility. We have corrected two legacy Table 10 cells (GSM8K/region 0.0137 → 0.0050; HumanEval 0.0048 → 0.0482)and reorganized the appendix into separate, explicitly labeled tables (Appendix I).