- Rating
- 3(+1)
- Confidence
- 2(−2)
- Informativeness
- 1(−1)
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
- This paper proposes E-CALR, a training-free decoding method that uses predictive commitment and cross-layer evidential conflict to refine final-layer logits.
- The method shows improvements over existing layer-wise decoding baselines across nine LLMs.
Critical strengths
- The method requires no additional training and can be applied to existing LLMs.
- The experiments cover multiple model families, and the ablation studies examine the contributions of commitment and conflict weighting.
Critical weaknesses
Weaknesses and Questions
-
Model scale and architecture: All evaluated models are dense models with at most 13B parameters. Experiments on MoE architectures and larger models (e.g., 20B+ parameters) would help establish whether the method generalizes to different architectures and scales.
-
Decoding efficiency: The paper provides a theoretical complexity analysis but lacks wall-clock decoding speed comparisons. Please report generation throughput (tokens/sec) and latency against Base, DoLa, and SLED, including the overhead of extracting intermediate-layer logits. If E-CALR is slower, this trade-off should be quantified and discussed as a limitation.
-
Generalization across tasks: The evaluation focuses primarily on factual QA and reasoning benchmarks. It remains unclear how the proposed logit refinement affects non-knowledge-intensive tasks. Experiments on tasks such as summarization, translation, and general instruction following would help identify potential performance degradation.
-
Intermediate-layer selection: The method selects every other intermediate layer (odd-indexed layers), but the motivation for this choice is unclear. Why this selection strategy? An ablation comparing odd/even layers, all layers, and different layer subsets would be useful, ideally under comparable computational budgets.
-
Relationship between conflict and factual errors: It is unclear when the proposed conflict score reflects genuinely harmful intermediate-layer information. Predictive commitment does not imply factual correctness, and a confident intermediate prediction may still be correct. In such cases, suppressing its contribution could hurt performance. A more systematic analysis of the relationship between conflict scores and actual prediction errors, including cases where suppression hurts, would strengthen the justification for the method.
Minor Comment
- The description of the FACTOR evaluation in Section 4.1 could be made slightly more concise to avoid repetitive wording.
Questions for authors
I mentioned in the weakness section.
2 comments
Response to Reviewer weaZ’s Follow-up
Thank you for taking the time to reread our paper, consider the additional experiments, and update your assessment. We appreciate your thoughtful engagement and candid feedback.
Your comments helped us clarify how conflict weighting contributes to refinement, examine its effects on both correct and incorrect predictions, and extend the evaluation across model architectures, scales, and tasks. We will incorporate these analyses and experimental results into the revised manuscript, with a clearer presentation of the method’s contribution, scope, and computational cost.
We appreciate the care you put into this review and your kind wishes.
Thank you for the additional experiments and detailed responses. The more I read the paper, the more I feel that something is still missing.
However, considering the authors' efforts, I will raise my score from 2 to 3. I will also lower my confidence, as I feel I may not have enough expertise in this area.
I appreciate the authors' hard work and wish them good luck.