- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This work does uncertainty estimation for LLMs. PULSE uses a small, trained transformer model to learn a mapping from token probabilities to response correctness. Basically, I think the authors try to model uncertainty estimation as a regression problem that can be learned directly from token probabilities.
However, in this paper, the motivation and problem definition are unclear. The authors also acknowledged that their method design won’t help. For example, “adds nothing over these free scalars on its own” and “models sequence order in a smaller increment” in abstract. The experimental setting is also not fair. For example, most of the baselines in the paper are training-free, whereas PULSE requires training.
Critical strengths
The authors conduct extensive ablation studies
Critical weaknesses
- Lack of correct position against the previous works. line 13-14: “yet the strongest methods resample the model, while single-pass methods trade accuracy for cost.” Not true. Many sampling-based methods won’t affect the performance. For example, TokUR doesn’t affect generation quality, which is the main baseline used in this paper. Lines 128-130: “We answer this empirically by learning the readout instead, and find that the answer is: not much." The authors said the gap this paper wants to fill is “not much”.
- Overclaim: Line 25: “PULSE is the strongest option we test, never significantly worse than the best baseline”: not true. See table 1 Llama-3.1-8B DeepScaler and gemma-4-E2B MATH500, PULESE is not the best one.
- The motivation is confusing. For example, the authors said that “compressing the whole generation into one handcrafted statistic” would “discard the temporal structure”. But it seems the authors' method still uses a scalar as the uncertainty. The problem this paper wants to study is also not so clear to me. In Line 047, it says, “asks a narrower, more testable question,” but followed by many questions: “how much response-correctness signal …. ”, “where ….”, “how much is ….”, “How much…..”, “how much …”. I’m trying to understand, but I really cannot grasp the gap this paper aims to fill.
- An important component of this paper's conclusions concerns “response length”. The authors aim to show the importance of response length for uncertainty using experiments. But this conclusion has been widely studied by previous work [1~3].
- The experimental results in this paper make me feel that the author doesn’t think the proposed method can really work. For example, in section 5.3, the authors acknowledged that the length itself is sufficient, and that supervised learning and sequence modeling in PULSE contribute little. They also said “PULSE’s remaining edge is the top-k evidence itself, not architectural sophistication on top of it," in Line 349-353.
[1] SAUP: Uncertainty Propagation on LLM Agent (ACL 2025)
[2] Sequential Probabilistic Uncertainty Estimation for Parallel Multi-Agent Reasoning Systems
[3] From Sequence to Structure: Relational Uncertainty Propagation for LLM Agents
Questions for authors
- Please see the weaknesses.
- The presentation needs substantial improvement.
- I’m confused about the design that combines the top-k probability of each token to do simple regression. Could the author provide any supportive experimental evidence or theoretical proof?