acceptodds
Under review as a conference paper at ICLR 2027

DCLED: Dynamic Confidence-Aware Layer-wise Evolution Decoding for Factuality-Oriented Candidate Scoring

Abstract

Large language models often assign high likelihood to fluent but factually incorrect candidate answers in knowledge-intensive benchmarks. We introduce DCLED (Dynamic Confidence-Aware Layer-wise Evolution Decoding), a training-free inference-time method for factuality-oriented multiple-choice and ranking tasks. DCLED combines three mechanisms inside iterative logit evolution: an early-layer consensus anchor with a mature–latent contrastive term, a confidence gate that applies evolution only at uncertain token positions, and a confidence boost that scales the evolution step with top-token concentration. Across TruthfulQA and HotpotQA with five Llama, Qwen, and Mistral checkpoints, DCLED obtains the highest TruthfulQA MC2 on all five checkpoints and the highest HotpotQA ranking accuracy on four of five. The largest gain occurs with Llama-3.2-1B on HotpotQA, where accuracy increases from 52.99% to 78.27% over vanilla decoding (47.7% relative) and from 57.41% to 78.27% over SLED. HotpotQA gains over SLED are statistically significant for four of five checkpoints, whereas TruthfulQA MC2 gains over SLED (+0.013 to +0.055) are not significant after multiple-comparison correction. DCLED adds 12–64% latency relative to SLED and 15–185% relative to vanilla decoding. Ablations identify confidence gating as the dominant mechanism on HotpotQA, while the contrastive term and confidence boosting provide checkpoint-dependent refinements. DCLED is therefore best viewed as a parameter-preserving method for confidence-conditioned candidate scoring when hidden states are accessible, not as a universal replacement for factuality-oriented decoding baselines. Code is available at https://anonymous.4open.science/r/DCLED-Dynamic-Confidence-Aware-Layer-wise-Evolution-Decoding-A564/README.mdanonymous. Benchmarks: https://huggingface.co/datasets/truthfulqa/TruthfulQA and https://huggingface.co/datasets/hotpotqa/HotpotQA.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Factuality-oriented decoding, Hallucination mitigation, Contrastive decoding, Confidence gating, Training-free decoding

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_R41UOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes DCLED, a training-free decoding method for scoring candidate answers on factuality benchmarks. It builds on SLED style logit evolution and adds three parts. A confidence gate runs evolution only at token positions where the top final-layer probability is below a threshold. A contrastive term pushes scores toward the average early-layer distribution and away from mass that appears only at the final layer, and a boost term scales the step size by top-token concentration. The method is tested on TruthfulQA (MC1, MC2, MC3) and HotpotQA ranking accuracy with five Llama, Qwen and Mistral checkpoints against vanilla scoring, DoLa and SLED. It gets the best TruthfulQA MC2 on all five and the best HotpotQA accuracy on four, with large HotpotQA gains such as 53.0% to 78.3% on Llama-3.2-1B, but the TruthfulQA gains over SLED are not significant and latency goes up by 12 to 64% over SLED.

Critical strengths
  1. The confidence gate is a simple, low-cost idea, and the ablation shows that removing it drops HotpotQA accuracy by 14 to 19 points on every checkpoint.

  2. Remark 3.1 writes the evolution target as (1 + 2αc) Platent - 2αc Pmature, which makes it plain that only αc sets where the logits go.

  3. The evaluation covers three model families and five sizes, plus a Llama-3.2-3B result in the appendix.

  4. The paper reports latency next to accuracy and states plainly that DCLED is slower than both SLED and vanilla scoring.

  5. The failure case analysis in Appendix D, especially the point that the gate skips positions where the model is confidently wrong, is concrete and leads to clear next steps.

Critical weaknesses
  1. The HotpotQA ranking setup, including how candidates are made, how many there are and how distractors are chosen, is not described anywhere in the paper.

  2. DCLED has six tuned hyperparameters picked from 30 random trials on just 100 examples, and the paper itself notes that validation and test rankings disagree for both Qwen checkpoints.

  3. The significance tests ignore that all methods are scored on the same examples, even though paired tests such as McNemar or a paired bootstrap are easy to run on outputs the authors already have.

  4. The TruthfulQA numbers do not add up, since 817 questions minus 100 disjoint tuning examples should leave 717, yet the paper evaluates on 790.

  5. Candidate scores are summed log probabilities with no length normalization, and it is unclear whether vanilla and SLED also use relative-top filtering, so some of the gain may come from the filter or from length effects and not from evolution.

Questions for authors
  1. How are HotpotQA candidate sets built, how many candidates does each question have, and where do the distractors come from?

  2. Do vanilla, DoLa and SLED all use the same relative-top filter with ϵ equal to 0.1, and what happens to DCLED if the filter is removed?

  3. Since K is up to 250 steps and Figure 2 shows logits reaching the target within about ten steps, how does DCLED compare with simply scoring gated positions using q from Remark 3.1 directly with no iterations?

  4. Can you explain the 790 versus 717 TruthfulQA count and report paired tests on example-level outputs for the DCLED versus SLED comparisons?

  5. Can you report results with the official DoLa and SLED code, or explain why your DoLa falls below vanilla when the original paper shows gains on TruthfulQA?

Reviewer_Ek1DOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes DCLED, a training-free method for factuality-oriented candidate scoring using intermediate-layer distributions. It applies iterative logit evolution selectively at low-confidence token positions, combining an early-layer consensus target, a mature–latent contrastive term, and confidence-dependent step scaling. Experiments on TruthfulQA and HotpotQA across five main checkpoints report improvements over vanilla scoring and a shared-pipeline SLED implementation. The strongest gains occur on HotpotQA; TruthfulQA improvements over SLED are not statistically significant after multiple-comparison correction.

Critical strengths

The results suggest that selectively applying logit evolution can be substantially better than applying it indiscriminately. Table 2 reports consistent HotpotQA degradation when confidence gating is removed. The largest main-result improvement, from 52.99% to 78.27% on Llama-3.2-1B, makes selective intervention a potentially useful direction for candidate scoring.

The paper gives a balanced account of its limitations. It reports additional latency, checkpoint-dependent component benefits, non-significant TruthfulQA improvements over SLED, and cases where DoLa or gated SLED performs better. Restricting the claimed scope to finite-candidate scoring also improves the interpretability of the contribution.

Critical weaknesses

The evaluation protocol contains unresolved inconsistencies and missing details. Section 4.1 states that TruthfulQA contains 817 questions, that 790 are evaluated, and that 100 disjoint tuning examples are selected from the corpus. These counts cannot all describe disjoint subsets of the stated corpus. For HotpotQA, interpretation of ranking accuracy depends on candidate construction, but distractor sources, candidate counts, and tie handling are insufficiently specified. These are material reproducibility concerns because the reported gains depend on the exact evaluation population and candidate sets.

The main comparison does not establish the benefit over the closest alternative. Section 3.6 states that the SLED implementation differs from the original latent-distribution estimator. Moreover, SLED_CG, which adds only confidence gating, is evaluated in a small diagnostic that Section 4.4 explicitly says is not comparable with Table 1. Since gating accounts for much of the observed improvement, a full matched comparison with gated SLED is needed to establish the contribution of the additional contrastive and boosting terms relative to their computational cost.

Questions for authors

Please reconcile the TruthfulQA counts and provide the exact tuning and evaluation split definitions. Could the discrepancy be a reporting error, or were tuning examples obtained from a different source?

How are the HotpotQA candidate sets constructed? Please specify distractor sources, candidate counts, score ties, and the behavior when relative-top filtering suppresses every candidate.

Can SLED_CG be evaluated on the same full test sets with a matched tuning budget? Please also clarify the material differences between the implemented SLED baseline and the original method.

Reviewer_KbjwOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes DCLED, a training-free inference-time method for factuality-oriented candidate scoring using intermediate transformer-layer representations. The method combines three components: (1) confidence-based gating that restricts refinement to uncertain token positions, (2) a contrastive correction based on the difference between final-layer and early-layer token distributions, and (3) confidence-dependent scaling of iterative logit updates.

The authors evaluate DCLED on TruthfulQA and HotpotQA using five checkpoints from the Llama, Qwen, and Mistral families. The reported results demonstrate substantial improvements over vanilla candidate scoring on several settings, particularly HotpotQA. For example, Llama-3.2-1B improves from 52.99% to 78.27% ranking accuracy. The ablation results suggest that confidence gating is particularly important for these improvements.

Critical strengths
  1. Practical training-free approach. DCLED requires no parameter updates, external retrieval, or additional training. This makes it potentially applicable to frozen language models when internal hidden states are accessible.

  2. Promising empirical improvements on HotpotQA. The improvements over vanilla decoding and SLED are substantial for several checkpoints, particularly Llama-3.2-1B and Mistral-7B

Critical weaknesses

Main weakness: the evolution target is not necessarily a valid probability distribution, in Section 3.4, Remark 3.1, Equations (9)–(10), the paper interprets its evolution rule as gradient descent toward a target distribution:

q=(1+2αc)Platent−2αcPmature.q = (1 + 2\alpha_c)P_{\mathrm{latent}} - 2\alpha_c P_{\mathrm{mature}}.

Although the authors correctly observe that the coordinates of q sum to one, they are not guaranteed to be nonnegative. For example, consider:

  • P_latent = (0.1, 0.9)
  • P_mature = (0.9, 0.1)
  • alpha_c = 0.4

Then:

q=1.8(0.1,0.9)−0.8(0.9,0.1)=(−0.54,1.54).q = 1.8(0.1, 0.9) - 0.8(0.9, 0.1) = (-0.54, 1.54).

Thus, q is not a valid probability distribution.

This raises several concerns:

  1. Invalid optimization interpretation. Although the update can be written as the gradient of a formal cross-entropy-like objective with a signed target, it cannot generally be interpreted as minimizing cross-entropy between two probability distributions.

  2. When q contains negative coordinates, the corresponding formal objective can be unbounded below. Consequently, the iterative updates need not converge to a finite optimum without additional constraints.

  3. The resulting behavior may depend critically on the chosen logit clipping bounds, step-size schedule, and number of iterations rather than convergence toward a well-defined probability target.

  4. The paper does not establish convergence or stability guarantees for the proposed iterative updates.

While the authors acknowledge that q can contain negative coordinates and introduce clipping safeguards, these safeguards do not resolve the underlying issue with interpreting q as a probability distribution.

The authors should either reformulate the optimization objective explicitly using a signed target and analyze its stability, or modify the target construction to guarantee that q lies on the probability simplex.

Questions for authors

Please see the comments above for the theoretical motivation part.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.