- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes B-L-V, a cognitively inspired decomposition for diagnosing LLM reasoning failures. The framework decomposes a task’s standard answer into Belief, Logic, and Value components. Belief represents the knowledge and premises needed for a solution; Logic represents the inference steps; and Value represents output requirements and verification constraints. Model outputs are then compared with this reference decomposition and assigned a primary failure locus. The paper evaluates B-L-V on TruthfulQA, TAL-SCQ5K, and SemEval-2023 Task 4, using Qwen2.5 models at 1.5B, 3B, and 7B scales. It compares direct answering, chain-of-thought prompting, and B-L-V prompting. The reported results show that the B-L-V prompt often improves task accuracy, particularly on TruthfulQA and the mathematical reasoning benchmark.
Critical strengths
- The paper addresses an important question beyond final-answer accuracy: where and why does reasoning fail?
- The Belief-Logic-Value decomposition is intuitive and easy to communicate, and the use of a fixed model-independent reference is a useful idea for making analyses comparable across models.
Critical weaknesses
- The paper does not directly validate its main claim that errors can be reliably attributed to B, L, or V.
- The performance gain from B-L-V prompting is confounded with a stronger and longer structured reasoning prompt. There is no length-matched generic checklist baseline or component ablation.
- The datasets are assumed to isolate B, L, and V respectively, but this assumption is debatable. Truthfulness errors are not necessarily purely knowledge errors, mathematical errors are not necessarily purely logic errors, and the “Value” construct differs between answer verification and human-value classification.
- The annotation reliability methodology is insufficiently clear. Since free-form annotations are embedded and clustered before computing Fleiss’ Kappa, more details are needed about clustering, label alignment, and what agreement the metric actually measures.
Questions for authors
- Can the authors conduct a direct human study in which independent annotators assign B/L/V failure labels to model outputs and report attribution agreement?
- Can the authors include a length-matched structured-prompt baseline with generic headings or a shuffled taxonomy?
- Can the authors ablate B-only, L-only, V-only, B+L, and B+L+V prompting?
I will keep my score unchanged
I checked other reviewers' comments and feel my judgement is in general consistent with others. The author does not give any rebuttal so far. I will keep my "weak rejection" score as is.