- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
Summary
- This paper proposes LUCiD, a contrastive decoding method with a learned premature distribution and a token-wise gate.
- The method improves SQuAD performance on some models. However, its novelty and general effectiveness remain unclear.
Overall Assessment
- The method is computationally efficient and shows promising results on LLaMA-8B.
- However, the methodological contribution is limited, and several design choices lack justification.
- More competitive baselines and stronger evidence of generalization are needed.
Critical strengths
- S1. The method adds few trainable parameters and has relatively low inference overhead.
- S1. The paper includes experiments across five models and several ablation studies.
Critical weaknesses
W1. Limited novelty and motivation
- The method mainly combines a learned residual probe with a gating mechanism.
- The paper does not clearly explain why learning the premature distribution should reduce hallucinations.
- It is unclear whether the gains come from contrastive decoding or simply from supervised logit correction. A comparison with simpler learned baselines would help.
W2. Fixed layer selection
- The probe is always placed at the middle layer, regardless of the model or task.
- Table 4 shows that the optimal layer varies across models. For LLaMA-3B, a later layer performs substantially better.
- The paper should justify this fixed choice and compare it with adaptive layer selection, as in DoLa, or validation-based selection.
W3. Insufficient comparison with recent methods
- The baselines are limited to Vanilla, DoLa, and SLED.
- Recent methods such as ALW and ActLCD are cited but not evaluated.
- Comparisons with more recent contrastive decoding methods are needed to establish the contribution.
W4. Unclear training and evaluation setup
- The model is trained on MetaMathQA, ARC-Challenge, and SciQ. Yet SQuAD is used as the main evaluation benchmark.
- Evaluating on SQuAD is reasonable for testing out-of-distribution generalization. However, the motivation for this particular setup is unclear.
- SQuAD F1 alone is also insufficient to support broad claims about hallucination reduction.
- More direct factuality benchmarks and a clearer comparison between in-domain and out-of-domain performance are needed.
W5. Inconsistent performance across tasks
- The gains are largely driven by LLaMA-8B. Improvements on other models are small, and LLaMA-3B performs worse than Vanilla.
- Several general capability benchmarks also show substantial drops. For example, Mistral-12B drops from 0.389 to 0.140 on GPQA-Diamond under greedy decoding.
- HellaSwag performance also decreases across all five models under greedy decoding.
- These results raise concerns about the method's general applicability. The paper provides limited analysis of these failures.
Questions for authors
See the weakness section.