- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper proposes a scoring framework for quantifying the uncertainty of CoT responses in black-box omni-modal large models. The method maps multimodal inputs and generated CoT reasoning steps into an external shared semantic space, computes the coherence and grounding of the reasoning chain, and aggregates the confidence of multiple sampled answers. Its core feature is that it does not rely on the logits, hidden states, or additional calls of the evaluated model, making it applicable to both closed-source and open-source models. Experiments show that the method achieves systematic gains over baselines such as self-consistency on UNO-Bench, especially in multimodal open-ended answer scenarios, and is complementary to answer-level dispersion signals.
Critical strengths
-
The problem is well chosen and the setting is practical. This work focuses on uncertainty quantification for black-box omni-modal large language models, with the goal of assessing the confidence of chain-of-thought (CoT) answers without access to logits, hidden states, or gradients, which is of practical relevance to closed-source models and real-world deployment.
-
Reproducibility is relatively good. The release of code and the evaluation pipeline facilitates reproduction and future comparisons.
Critical weaknesses
-
Inconsistent logic. The paper says existing methods need model internals while its method does not, but the method still uses model embeddings.
-
Small and unstable performance gains, with weak motivation and introduction. The average improvement is only about 0.01–0.02 AUROC and is inconsistent across splits and generators; some settings are even below random or show losses. The introduction is also too short, has only one citation, and the motivation is too weak to lead naturally to the proposed method.
-
No systematic analysis of auxiliary-step errors, plus poor figures and readability. The paper does not analyze how CoT segmentation errors or answer-clustering errors affect the final confidence. Figure 1 leaves too much blank space, and Figure 2 is likely AI-generated and too information-dense, reducing readability and weakening the paper’s rigor.
Questions for authors
see weakness
We thank the reviewer, especially for W3, which prompted an analysis that uncovered a pipeline defect. We address the weaknesses in turn.
W1: The method uses model embeddings
Our claim concerns access to the evaluated model. Our scores require no logits, hidden states, gradients, or additional calls from that model. We instead use a separate, frozen encoder, E5-Omni 3B, which has no access to the evaluated model's parameters or activations.
This allows our method to work with closed API models, unlike UMPIRE without a proxy model. However, we agree that the submitted abstract and introduction do not explain this distinction clearly. We will replace "require access to model internals" with "require access to the evaluated model's internals" and state our dependencies upfront: an external omni-modal encoder, an auxiliary segmentation model, and a small labelled calibration set for the aggregated score. This is the method's actual cost model, comparable to semantic entropy's use of a clustering LLM.
W2: Small gains, weak motivation, and limited citations
Introduction. We agree that a single citation is insufficient. We will explain how existing multimodal uncertainty methods either access the evaluated model's internals, re-invoke it, or discard its reasoning trace. We will cite relevant work for each, including Uncertainty-o (arXiv:2506.07575), which Reviewer qdwg correctly notes is missing.
Empirical gains. The reviewer's characterization is accurate: gains are small, inconsistent across splits and generators, and sometimes below chance. Our new experiments make the results less favorable. With identical answer selection across methods (R1, 100 runs), the split-averaged gain over self-consistency falls from +0.0121 to +0.0038 for Gemini and remains negative for MiniCPM (-0.0034). A baseline that embeds the full generation without reasoning segmentation matches or beats chain geometry in three of four settings. We therefore cannot defend the submitted empirical claim as it stands.
Below-chance results. These occur in Gemini-Audio and are explained by the pipeline defect discussed under W3.
W3: Errors in auxiliary steps
This was an important request. R4 analyzes both auxiliary steps, and R5 reports the clustering defect.
Segmentation errors. Degenerate segmenter output (m1), for which
$S_{\text{smooth}}$,$S_{\text{dens}}$, and$\Delta_{\text{bwd}}$are undefined, affects 17 of 24,016 Gemini chains (0.07%) and 6,151 of 37,300 MiniCPM chains (16.5%). MiniCPM rates range from 2.7% on Audio to 22.1% on Visual. Our implementation assigns these cases the maximum score of 1.Removing these chains raises MiniCPM candidate-level AUROC by 3 to 4 points:
$S_{\text{smooth}}$0.5175 to 0.5503,$S_{\text{goal}}$0.5255 to 0.5618, and$S_{\text{dens}}$0.5329 to 0.5724. Gemini changes by less than 0.001. Thus, segmentation introduces a generator-specific error, and assigning the maximum score is inappropriate. The system should abstain when segmentation fails.Step count alone has AUROC 0.4904 for Gemini and 0.5105 for MiniCPM, so it does not explain the scores, although individual scores correlate with chain length (
$\rho(S_{\text{goal}},m)=-0.336$for Gemini and$\rho(G,m)=+0.227$for MiniCPM).Answer-clustering errors. We also found a pipeline defect. On single-choice splits, LLM clustering agrees with exact option matching on 76.9% of Gemini candidate pairs and 89.8% of MiniCPM pairs. Agreement falls to 39.7% on Gemini-Audio, compared with 96.4% on Gemini-Visual.
The cause is a regex bug in our answer extraction. Therefore, 63.8% of Gemini-Audio candidates reached the clusterer with the entire reasoning chain instead of the answer. Among 419 questions where all six candidates state the same option letter, 331 (79.0%) were split into multiple clusters, including 220 into six singletons.
We re-clustered single-choice questions using the stated option letter, a label-free procedure. Gemini-Audio Self-Consistency AUROC rises from 0.4524 to 0.7600, its tie rate falls from 48.2% to 3.3%, and answer accuracy rises from 0.7954 to 0.8079. Gemini-MC AUROC rises from 0.5946 to 0.6616, and MiniCPM-Visual from 0.6848 to 0.7264.
This defect explains the below-chance Gemini-Audio results. It also invalidates the submitted Audio failure-mode analysis and part of our complementarity argument. A complete corrected rerun is beyond the rebuttal period.
Figures. We agree both figures need revision. Figure 1's whitespace comes from resizing the TikZ figure to
0.3\columnwidthinside a full-column float. We will redraw it at natural width and move the caption text into the body.Figure 2 is too dense. We will replace it with a two-panel vector TikZ figure showing only step embeddings, media embeddings, and the answer embedding, moving aggregation details to the appendix. We will also extend the appendix's AI-use statement to cover figure preparation.