- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes DCLED, a training-free decoding method for scoring candidate answers on factuality benchmarks. It builds on SLED style logit evolution and adds three parts. A confidence gate runs evolution only at token positions where the top final-layer probability is below a threshold. A contrastive term pushes scores toward the average early-layer distribution and away from mass that appears only at the final layer, and a boost term scales the step size by top-token concentration. The method is tested on TruthfulQA (MC1, MC2, MC3) and HotpotQA ranking accuracy with five Llama, Qwen and Mistral checkpoints against vanilla scoring, DoLa and SLED. It gets the best TruthfulQA MC2 on all five and the best HotpotQA accuracy on four, with large HotpotQA gains such as 53.0% to 78.3% on Llama-3.2-1B, but the TruthfulQA gains over SLED are not significant and latency goes up by 12 to 64% over SLED.
Critical strengths
-
The confidence gate is a simple, low-cost idea, and the ablation shows that removing it drops HotpotQA accuracy by 14 to 19 points on every checkpoint.
-
Remark 3.1 writes the evolution target as (1 + 2αc) Platent - 2αc Pmature, which makes it plain that only αc sets where the logits go.
-
The evaluation covers three model families and five sizes, plus a Llama-3.2-3B result in the appendix.
-
The paper reports latency next to accuracy and states plainly that DCLED is slower than both SLED and vanilla scoring.
-
The failure case analysis in Appendix D, especially the point that the gate skips positions where the model is confidently wrong, is concrete and leads to clear next steps.
Critical weaknesses
-
The HotpotQA ranking setup, including how candidates are made, how many there are and how distractors are chosen, is not described anywhere in the paper.
-
DCLED has six tuned hyperparameters picked from 30 random trials on just 100 examples, and the paper itself notes that validation and test rankings disagree for both Qwen checkpoints.
-
The significance tests ignore that all methods are scored on the same examples, even though paired tests such as McNemar or a paired bootstrap are easy to run on outputs the authors already have.
-
The TruthfulQA numbers do not add up, since 817 questions minus 100 disjoint tuning examples should leave 717, yet the paper evaluates on 790.
-
Candidate scores are summed log probabilities with no length normalization, and it is unclear whether vanilla and SLED also use relative-top filtering, so some of the gain may come from the filter or from length effects and not from evolution.
Questions for authors
-
How are HotpotQA candidate sets built, how many candidates does each question have, and where do the distractors come from?
-
Do vanilla, DoLa and SLED all use the same relative-top filter with ϵ equal to 0.1, and what happens to DCLED if the filter is removed?
-
Since K is up to 250 steps and Figure 2 shows logits reaching the target within about ten steps, how does DCLED compare with simply scoring gated positions using q from Remark 3.1 directly with no iterations?
-
Can you explain the 790 versus 717 TruthfulQA count and report paired tests on example-level outputs for the DCLED versus SLED comparisons?
-
Can you report results with the official DoLa and SLED code, or explain why your DoLa falls below vanilla when the original paper shows gains on TruthfulQA?