acceptodds
Under review as a conference paper at ICLR 2027

Estimating Uncertainty of Omnimodal Large Language Models via External Omnimodal Embeddings

Abstract

Recent work has investigated Omni-modal Large Language Models (OLLMs), models which can take multiple modalities and their combinations as input, as a way forward to have systems really capable of interacting with their environment. Uncertainty quantification (UQ) for OLLMs, however, remains an open problem: existing methods largely require access to model internals, or expensive self-verbalisation approaches. We propose a model-agnostic UQ framework that scores a Chain-of-Thought response entirely through external omni-modal embeddings, measuring the grounding of the reasoning in the multimodal input and its internal coherence. The resulting confidence estimates require no access to logits, hidden states or gradients of the evaluated model, rather relying on the segmented reasoning trace and, for the aggregated score, a labelled calibration set that we show can be as small as 25 examples. On UNO-Bench, with both a closed and an open model, our scores yield small gains over self-consistency (systematic with the closed generator, mixed with the open one) and are competitive with self-verbalisation and internal-state baselines, especially in open-ended, omni-modal questions where previous state-of-the-art baselines struggle.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
uncertainty quantification, omnimodal LLMs, multimodal semantic representations

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_8dHZOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes a scoring framework for quantifying the uncertainty of CoT responses in black-box omni-modal large models. The method maps multimodal inputs and generated CoT reasoning steps into an external shared semantic space, computes the coherence and grounding of the reasoning chain, and aggregates the confidence of multiple sampled answers. Its core feature is that it does not rely on the logits, hidden states, or additional calls of the evaluated model, making it applicable to both closed-source and open-source models. Experiments show that the method achieves systematic gains over baselines such as self-consistency on UNO-Bench, especially in multimodal open-ended answer scenarios, and is complementary to answer-level dispersion signals.

Critical strengths
  1. The problem is well chosen and the setting is practical. This work focuses on uncertainty quantification for black-box omni-modal large language models, with the goal of assessing the confidence of chain-of-thought (CoT) answers without access to logits, hidden states, or gradients, which is of practical relevance to closed-source models and real-world deployment.

  2. Reproducibility is relatively good. The release of code and the evaluation pipeline facilitates reproduction and future comparisons.

Critical weaknesses
  1. Inconsistent logic. The paper says existing methods need model internals while its method does not, but the method still uses model embeddings.

  2. Small and unstable performance gains, with weak motivation and introduction. The average improvement is only about 0.01–0.02 AUROC and is inconsistent across splits and generators; some settings are even below random or show losses. The introduction is also too short, has only one citation, and the motivation is too weak to lead naturally to the proposed method.

  3. No systematic analysis of auxiliary-step errors, plus poor figures and readability. The paper does not analyze how CoT segmentation errors or answer-clustering errors affect the final confidence. Figure 1 leaves too much blank space, and Figure 2 is likely AI-generated and too information-dense, reducing readability and weakening the paper’s rigor.

Questions for authors

see weakness

1 comment

We thank the reviewer, especially for W3, which prompted an analysis that uncovered a pipeline defect. We address the weaknesses in turn.

W1: The method uses model embeddings

Our claim concerns access to the evaluated model. Our scores require no logits, hidden states, gradients, or additional calls from that model. We instead use a separate, frozen encoder, E5-Omni 3B, which has no access to the evaluated model's parameters or activations.

This allows our method to work with closed API models, unlike UMPIRE without a proxy model. However, we agree that the submitted abstract and introduction do not explain this distinction clearly. We will replace "require access to model internals" with "require access to the evaluated model's internals" and state our dependencies upfront: an external omni-modal encoder, an auxiliary segmentation model, and a small labelled calibration set for the aggregated score. This is the method's actual cost model, comparable to semantic entropy's use of a clustering LLM.

W2: Small gains, weak motivation, and limited citations

Introduction. We agree that a single citation is insufficient. We will explain how existing multimodal uncertainty methods either access the evaluated model's internals, re-invoke it, or discard its reasoning trace. We will cite relevant work for each, including Uncertainty-o (arXiv:2506.07575), which Reviewer qdwg correctly notes is missing.

Empirical gains. The reviewer's characterization is accurate: gains are small, inconsistent across splits and generators, and sometimes below chance. Our new experiments make the results less favorable. With identical answer selection across methods (R1, 100 runs), the split-averaged gain over self-consistency falls from +0.0121 to +0.0038 for Gemini and remains negative for MiniCPM (-0.0034). A baseline that embeds the full generation without reasoning segmentation matches or beats chain geometry in three of four settings. We therefore cannot defend the submitted empirical claim as it stands.

Below-chance results. These occur in Gemini-Audio and are explained by the pipeline defect discussed under W3.

W3: Errors in auxiliary steps

This was an important request. R4 analyzes both auxiliary steps, and R5 reports the clustering defect.

Segmentation errors. Degenerate segmenter output (m1), for which $S_{\text{smooth}}$, $S_{\text{dens}}$, and $\Delta_{\text{bwd}}$ are undefined, affects 17 of 24,016 Gemini chains (0.07%) and 6,151 of 37,300 MiniCPM chains (16.5%). MiniCPM rates range from 2.7% on Audio to 22.1% on Visual. Our implementation assigns these cases the maximum score of 1.

Removing these chains raises MiniCPM candidate-level AUROC by 3 to 4 points: $S_{\text{smooth}}$ 0.5175 to 0.5503, $S_{\text{goal}}$ 0.5255 to 0.5618, and $S_{\text{dens}}$ 0.5329 to 0.5724. Gemini changes by less than 0.001. Thus, segmentation introduces a generator-specific error, and assigning the maximum score is inappropriate. The system should abstain when segmentation fails.

Step count alone has AUROC 0.4904 for Gemini and 0.5105 for MiniCPM, so it does not explain the scores, although individual scores correlate with chain length ($\rho(S_{\text{goal}},m)=-0.336$ for Gemini and $\rho(G,m)=+0.227$ for MiniCPM).

Answer-clustering errors. We also found a pipeline defect. On single-choice splits, LLM clustering agrees with exact option matching on 76.9% of Gemini candidate pairs and 89.8% of MiniCPM pairs. Agreement falls to 39.7% on Gemini-Audio, compared with 96.4% on Gemini-Visual.

The cause is a regex bug in our answer extraction. Therefore, 63.8% of Gemini-Audio candidates reached the clusterer with the entire reasoning chain instead of the answer. Among 419 questions where all six candidates state the same option letter, 331 (79.0%) were split into multiple clusters, including 220 into six singletons.

We re-clustered single-choice questions using the stated option letter, a label-free procedure. Gemini-Audio Self-Consistency AUROC rises from 0.4524 to 0.7600, its tie rate falls from 48.2% to 3.3%, and answer accuracy rises from 0.7954 to 0.8079. Gemini-MC AUROC rises from 0.5946 to 0.6616, and MiniCPM-Visual from 0.6848 to 0.7264.

This defect explains the below-chance Gemini-Audio results. It also invalidates the submitted Audio failure-mode analysis and part of our complementarity argument. A complete corrected rerun is beyond the rebuttal period.

Figures. We agree both figures need revision. Figure 1's whitespace comes from resizing the TikZ figure to 0.3\columnwidth inside a full-column float. We will redraw it at natural width and move the caption text into the body.

Figure 2 is too dense. We will replace it with a two-panel vector TikZ figure showing only step embeddings, media embeddings, and the answer embedding, moving aggregation details to the appendix. We will also extend the appendix's AI-use statement to cover figure preparation.

Reviewer_aP9MOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper estimates confidence using reasoning coherence and input grounding measured by external multimodal embeddings. It tests these scores on UNO-Bench with Gemini and MiniCPM, against self-consistency and other confidence baselines.

Critical strengths

Using external embeddings is practical when model internals are unavailable. The second encoder, calibration analysis, and feature-selection experiments help test the idea. The paper also reports where results with MiniCPM are mixed. This is a useful setting to study because answer agreement alone can miss shared errors. Measuring grounding against the input offers a different signal, and testing two generators helps expose where that signal does and does not transfer.

Critical weaknesses

The methods do not always score the same answer. Section 4 breaks majority-vote ties using each method’s confidence score, and ties occur on 28.9% of Gemini questions and 14.2% of MiniCPM questions. This changes answer selection as well as confidence estimation. The average AUROC improvements over self-consistency are 0.0119 for Gemini and 0.0044 for MiniCPM. I am not assuming tie-breaking explains those gains, but the current comparison cannot separate the effects.

It is also unclear how much the reasoning-step geometry adds. In Appendix G, the fitted aggregate usually selects just one feature, and adding whole-generation grounding improves it further. A baseline using that simpler feature family alone seems necessary. This would test whether segmenting the reasoning contributes beyond embedding the response as a whole.

Questions for authors

Can you repeat the AUROC comparison with fixed tie-breaking so every method returns identical answers? On that evaluation, how does a validation-selected whole-generation grounding baseline compare with the chain-based method? A persistent gain under both controls would substantially strengthen the case for the proposed method.

1 comment

We thank the reviewer for identifying two essential controls. We ran both across the same 100 randomized runs and test items as the main experiments (R1). Both support the reviewer's concerns. A pipeline defect we uncovered also helps explain the results (R5).

Q1 / W1: Fixed answer selection

We added a fixed protocol in which all methods return the same cluster self-consistency answer, using the lowest-index cluster in tied majorities. Thus, AUROC differences reflect confidence estimation alone. Results show split-averaged AUROC and paired mean differences from Self-Consistency:

GeneratorSelf-ConsistencyPublishedFixed
Gemini0.59270.6048 (+0.0121)0.5965 (+0.0038)
MiniCPM0.65590.6515 (-0.0044)0.6525 (-0.0034)

Under fixed selection, Gemini's per-split differences are Audio +0.0051***, MC +0.0029***, MO -0.0047***, and Visual +0.0119***. MiniCPM's are Audio -0.0007 (n.s.), MC -0.0071***, MO +0.0005 (n.s.), and Visual -0.0062***.

Roughly two thirds of Gemini's published gain therefore comes from method-dependent answer selection, not confidence estimation. The remaining gain is only +0.0038 and comes mainly from Visual, while MO becomes negative. MiniCPM remains negative. The reviewer was right that the original comparison conflated these effects.

Q2 / W2: Segmentation-free grounding baseline

We implemented ansG3, a third pool containing the three answer-level grounding scores ($G^{\max}_{\text{ans}}$, $G^{\text{avg}}_{\text{ans}}$, and $G^{\min}_{\text{ans}}$). It embeds each generation as a whole and aligns it with the media pool, without segmenting the reasoning chain. We used the same validation selection and experimental protocol.

Generator / protocolchain5 (segmented)ansG3 (no segmentation)all8
Gemini, published0.60480.60150.6081
Gemini, fixed0.59650.60030.5992
MiniCPM, published0.65150.66160.6537
MiniCPM, fixed0.65250.65780.6539

The segmentation-free baseline matches or beats chain geometry in three of four settings. It is also the only pool with a positive split-averaged difference on MiniCPM under either protocol. Under fixed Gemini selection, ansG3 is chosen in all 100 Audio runs and 78 of 100 MO runs. On the canonical partition, it scores 0.5717 versus 0.5679 for the best chain score on Gemini-MC, and 0.7344 versus 0.7345 on MiniCPM-Audio. Chain scores lead clearly in only two of eight cells: Gemini-Visual and MiniCPM-MO.

We therefore cannot claim that reasoning segmentation adds value beyond embedding the full response.

Additional disclosure: Clustering defect

We found a regex defect in final-answer extraction affecting Gemini-Audio. It caused the clusterer to split semantically identical answers. Among 419 Gemini-Audio questions where all six candidates gave the same option letter, 331 (79.0%) were split into multiple clusters, including 220 split into six singletons.

We re-clustered single-choice questions using the stated option letter, a label-free procedure that reads only the generation:

Generator / splitnTied majorities, before → afterSelf-Consistency AUROC, before → after
Gemini Audio47948.2% → 3.3%0.4524 → 0.7600
Gemini MC66411.1% → 7.5%0.5946 → 0.6616
Gemini Visual9784.4% → 4.1%0.7594 → 0.7684
MiniCPM Audio8508.2% → 4.7%0.7194 → 0.7424
MiniCPM Visual9908.5% → 6.1%0.6848 → 0.7264

This defect explains the 28.9% tie rate flagged by the reviewer and the below-chance Audio result in Table 1. It also undermines the submitted Audio failure analysis and part of our complementarity argument: answer-dispersion baselines appear strong on Gemini-Audio partly because they use answer embeddings and bypass the faulty clustering.

Conclusion

The reviewer asked for persistent gains under both controls. We do not find them. With fixed answers, the gain is just +0.0038 for Gemini and negative for MiniCPM. The segmentation-free baseline matches or beats chain geometry in three of four settings. Moreover, the clustering procedure underlying the submitted results and these new comparisons must be corrected, and the affected results recomputed.

Reviewer_da4AOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes an uncertainty estimation framework for omnimodal LLMs using external omnimodal embeddings. It measures coherence among reasoning steps, convergence toward the final answer, and grounding in multimodal inputs.

Critical strengths

The paper explores an interesting combination of within-chain reasoning geometry and alignment between reasoning steps and multimodal evidence. The approach is intuitive and well motivated and works with proprietary models without access to hidden states. The method is evaluated on closed and open source models, including in the multimodal setting, and an analysis of the scoring components and embedding models is given.

Critical weaknesses
  1. Reasoning models may explore multiple ideas that are later rejected as either incorrect or irrelevant to the problem. Although the encoder processes the complete reasoning step, it is unclear how the similarity score handles these, or if it distinguishes between incorrect or irrelevant ideas. This is one of the key differences between reasoning models and non-reasoning models, so is important to investigate.

  2. The accuracy improvements in Self-Consistency are small on several subsets in Tables 1 and 2. For example, AUROC increases from 0.7234 to 0.7272 on Gemini-Visual (+0.0038) and from 0.5461 to 0.5507 on MiniCPM-MC (+0.0046). Performance decreases on Gemini-MC from 0.5630 to 0.5622 and from 0.6782 to 0.6700 on MiniCPM-MO. The improvements are small enough where I question the reliability and practical benefit relative to additional embedding and reasoning-segmentation.

Questions for authors

How does the score handle reasoning traces that consider incorrect options but still reach the final correct answer? A comparison between reasoning traces that directly produce the correct answer and traces that have incorrect or irrelevant intermediate reasoning steps would clarify this.

For the geometry-weighted answer-selection variant, could the authors report the number of evaluated questions that change from incorrect to correct and vice versa relative to ordinary majority voting? It is also good to provide representative examples from both categories, including the candidate answers, reasoning traces, confidence scores, and voting weights.

2 comments

We thank the reviewer for raising two precise questions that we could address experimentally. We conducted both analyses: one supports our approach, while the other does not.

W1 / Q1: How does the score handle traces that consider incorrect options but reach the correct answer?

This is an important question for a geometric score, and we had not previously investigated it. Evidence R3 classifies 24,016 Gemini and 37,300 MiniCPM chains as option-eliminating (ELIM), self-correcting (BACKTRACK), or direct (DIRECT), and evaluates each group separately. The results are:

GeneratorTypeShareAccuracyMean SsmoothS_{\text{smooth}}Mean SdensS_{\text{dens}}AUROC SsmoothS_{\text{smooth}}AUROC SdensS_{\text{dens}}
GeminiDIRECT69.4%0.72800.81100.57900.50690.4897
GeminiELIM15.5%0.79230.80960.58640.49710.4917
GeminiBACKTRACK15.1%0.58640.81690.59540.51370.5149
MiniCPMDIRECT67.9%0.59650.93310.86100.49050.5052
MiniCPMELIM1.8%0.72600.90630.80050.57620.6341
MiniCPMBACKTRACK30.4%0.56390.91290.81300.59440.5966

Three main findings:

  • Option elimination is not penalized by our coherence scores. For Gemini, ELIM and DIRECT chains differ by less than 0.008 in mean scores. However, their accuracy differs substantially (0.7923 vs. 0.7280). Our scores therefore fail to capture whether a candidate explicitly rejects an incorrect option.
  • Backtracking produces different results across generators. For MiniCPM, 30.4% of candidates backtrack (64 steps on average). Within this group, SsmoothS_{\text{smooth}} and SdensS_{\text{dens}} achieve AUROCs of 0.5944 and 0.5966, versus 0.4905 and 0.5052 for DIRECT chains. For Gemini, the corresponding scores provide little discrimination. Thus, trace-based signals are informative for MiniCPM's longer self-correcting chains but largely uninformative for Gemini's traces.
  • This reveals a limitation and a research direction. A step-level distinction between asserted and retracted reasoning is missing from our current representation. We will clarify this limitation.

W2 / Q2: How often does geometry-weighted selection change the answer?

Evidence R2 reports answer changes across all 3,730 questions per generator. Since the individual scores are label-free, no validation split is required.

Generator / SignalChangedIncorrect → CorrectCorrect → IncorrectNetAccuracy: SC → Weighted
Gemini, SdensS_{\text{dens}} (best)847 (22.7%)10087+130.7359 → 0.7394
Gemini, SgoalS_{\text{goal}}834 (22.4%)9085+50.7359 → 0.7373
Gemini, GavgG_{\text{avg}}837 (22.4%)8087-70.7359 → 0.7340
Gemini, aGmeanaG_{\text{mean}} (worst)837 (22.4%)7186-150.7359 → 0.7319
MiniCPM, aGmaxaG_{\text{max}} (best)349 (9.4%)5147+40.5987 → 0.5997
MiniCPM, GavgG_{\text{avg}} (worst)361 (9.7%)4649-30.5987 → 0.5979

The gains and losses are nearly symmetric: the best signal yields only 13 net corrections, while the worst causes 15 net errors for Gemini. Overall, geometry-weighted selection does not meaningfully improve accuracy. We will state this explicitly rather than present it as a promising direction.

W3: Are the gains worth the additional embedding and segmentation costs?

We agree that the gains are too small to justify the additional complexity. The four values cited by the reviewer are correct: +0.0038 on Gemini-Visual, +0.0046 on MiniCPM-MC, -0.0008 on Gemini-MC, and -0.0082 on MiniCPM-MO. Further experiments (R1, 100 runs), conducted following Reviewer aP9M's request, reinforce this concern:

  • Fixed answer selection: When all methods return the same answer, the split-averaged gain over self-consistency falls from +0.0121 to +0.0038 for Gemini and remains negative for MiniCPM (-0.0034). Thus, much of the reported Gemini gain comes from answer selection rather than confidence estimation.
  • Segmentation-free baseline: Embedding the entire generation and scoring its grounding against the media matches or outperforms the five chain-geometry scores in three of four settings (Gemini fixed: 0.6003 vs. 0.5965; MiniCPM published: 0.6616 vs. 0.6515; MiniCPM fixed: 0.6578 vs. 0.6525).

Finally, R5 reveals a pipeline defect: for 63.8% of Gemini-Audio candidates, clustering used the entire reasoning chain rather than the final answer. This occurred because of a regex issue in the pipeline. Correcting this issue on the single-choice questions raises self-consistency AUROC from 0.4524 to 0.7600 and reduces the tie rate from 48.2% to 3.3%.

This correction invalidates our original Audio failure-mode analysis and part of the complementarity argument. We apologise for the oversight.

Dear authors, thank you for your rebuttal, I'll rise my score.

Reviewer_qdwgOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper propose a framework to estimate uncertainty for Omnimodal Large Language Models (OLLMs), and utilize the uncertainty for following up hallucination detection. During the uncertainty estimation, this work mainly focuses on two aspects, Grounding and Coherence. The grounding score refers to whether the model thinking process is related to initial question. During the calculation process, fine-grained step-level grounding scores are first processed, and then aggregated to full grounding score through statistic operations such as mean and max. The coherence score refers to whether model thinking steps and final answer is consistent. The calculation process considers aspects like coherence between current step and next step, coherence between any pair of two steps. Following-up statistic operation leads to several coherence scores. Final model confidence is derived based on the obtained grounding score and coherence score. The overall principle is that more grounding, more coherent the model response, the higher the model confidence. This work adopts several metrics, including threshold-based, AUROC, ECE, etc, for hallucination detection. The main evaluation is conducted on UNO-Bench, with two models, Gemini 3.1 Flash Lite and MiniCPM-o 4.5. The evaluation results show mariginal improvement over baseline methods.

Critical strengths
  1. The studied topic of this work is critial. This work focuses on estimating uncertainty of OLLM and faciliate following-up hallucination detection. It is important to propose effective methods to quantify the uncertainty of OLLM and identify unwanted or high-risk model repsonse, which can ensure safer deployment of those models.
Critical weaknesses
  1. Unreasonable method design. It is unreasonable that high variance of model thinking steps means the model is inconfident. Given a complex question, it is common that the model analyzes from various aspects, and it is natural that the variance between those steps is high.
  2. Novelty problem. The author is recommmended to add proper discussion and comparison with Uncertainty-o (EMNLP'26) [1]. This work already proposes a model-agnostic framework to quantify the uncertainty of OLLM. It proposes a unfied method to estimate uncertainty of multimodel model capable not only understanding, but also generation. Moreover, it considers 5 modalites, text, image, video, audio, and point cloud. The domain of this paper is fully included within Uncertainty-o. [1] Zhang, R., Zhang, J., Fei, H., & Zheng, Z. . Uncertainty-o: One model-agnostic framework for unveiling uncertainty in large multimodal models.
  3. Low performance. The proposed method shows low results during experimental evaluation. From Tab. 1 and Tab. 2, the proposed method yields lower performance than baselines across most sub split. For the average results, the method only shows quite marginal improvement.
  4. Writing problem. The paper has many hard writing, such as "a closed and an open model" in abstract (L023-L024). The author is recommended to thoroughly check the paper to avoid those writing issues.
Questions for authors

See Weakness.

2 comments

We thank the reviewer for the feedback. We address the weaknesses below. We agree with two, agree with the substance of one while clarifying its interpretation, and believe the remaining concern partly reflects a misreading of the results. We also disclose an issue identified in our pipeline while preparing this response (evidence R5), as it affects W3.

W1: Interpreting variation across reasoning steps

We agree that high variance across reasoning steps does not necessarily indicate low model confidence. Our method makes no such a priori assumption: Cchain\mathcal{C}_{\text{chain}} is a validation-fitted logistic regression (with a single-feature fallback), so each score's relationship with correctness is estimated per split, including cases where it reverses.

The Platt slopes in our calibration appendix illustrate this: all Gemini-Visual chain scores have strongly positive slopes (+4.06 to +4.56), whereas four of five Gemini-Audio scores are negative (SsmoothS_{\text{smooth}}: -0.08, SgoalS_{\text{goal}}: -0.02, SdensS_{\text{dens}}: -0.25, GG: -0.29). Thus, greater geometric regularity can correspond to lower accuracy, as the reviewer suggests.

We investigated whether legitimate multi-perspective reasoning is penalized (R3). Across 24,016 Gemini and 37,300 MiniCPM chains, we distinguished option elimination (ELIM), backtracking (BACKTRACK), and direct reasoning (DIRECT). For Gemini, ELIM chains (15.5% of candidates) are more accurate than DIRECT chains (0.7923 vs. 0.7280), with comparable coherence and slightly higher semantic density (0.5864 vs. 0.5790). For MiniCPM, however, ELIM chains have lower smoothness (0.9063 vs. 0.9331) despite higher accuracy (0.7260 vs. 0.5965). Thus, the relationship between trace regularity and correctness depends on reasoning style, a limitation we will clarify.

For MiniCPM, 30.4% of candidates contain backtracking (64 steps on average). Within these chains, SsmoothS_{\text{smooth}} and SdensS_{\text{dens}} achieve AUROCs of 0.5944 and 0.5966, respectively, versus 0.4905 and 0.5052 for DIRECT chains.

W2: Relationship to Uncertainty-o

We thank the reviewer for pointing us to Uncertainty-o. Its omission was an oversight, and we will add it to related work. However, three distinctions remain:

Computational cost. Uncertainty-o's Multimodal Prompt Perturbation uses five additional samples by default (Table 9 and Cookbook 4), requiring approximately six evaluated-model invocations per question, plus Qwen2.5-7B and OneLLM-7B for semantic checking and captioning. Our method requires no additional evaluated-model calls beyond the original generations; its only additional cost is one external encoder pass.

What is measured. Uncertainty-o estimates Shannon entropy over semantic clusters of final answers, without using reasoning traces. Our method instead measures trace geometry and alignment with non-textual inputs. Semantic Entropy is already among our baselines: it achieves 0.5882 AUROC versus 0.5927 for self-consistency (Additional Baselines appendix), offering little improvement at our budget.

Omni-modal setting. Although Uncertainty-o covers five modalities and 18 benchmarks, its evaluations pair one non-text modality with text. It does not evaluate questions jointly involving audio, vision, and text, as in 100% of our MC and MO splits.

We will sharpen our novelty claims around two aspects: no additional evaluated-model invocations and trace-level rather than answer-level evidence.

W3: Performance relative to baselines

We agree that our empirical performance requires careful assessment. In split-averaged AUROC, our method leads both main tables (GavgG_{\text{avg}}: 0.6102 with Gemini; Cchain\mathcal{C}_{\text{chain}}: 0.6603 with MiniCPM). On Gemini's omni-modal open-answer split, SsmoothS_{\text{smooth}} achieves 0.6880, while all baselines remain below 0.64. Nevertheless, no method consistently dominates individual splits, and our average improvement is modest.

Under the fixed-answer-selection control requested by Reviewer aP9M (R1, 100 runs), the gain over self-consistency decreases from +0.0121 to +0.0038 for Gemini and remains negative for MiniCPM (-0.0034). A segmentation-free baseline also matches or outperforms our chain-geometry in three of four generator-by-protocol settings.

Crucially, R5 reveals a pipeline error: 63.8% of Gemini-Audio candidates were clustered using entire reasoning chains rather than final answers. This inflated the tie rate to 48% and reduced self-consistency AUROC from its corrected value of 0.7600 to 0.4524. We therefore cannot ask the reviewer to accept the original empirical claims as they stand.

The trace-based signals, redundancy characterization (0.91–0.94), complementarity with answer dispersion, and Audio failure-mode analysis must be re-evaluated using corrected clusterings. This exceeds the available rebuttal period. We apologize for the oversight and will correct the pipeline and revise the associated experiments and claims.

Thanks for your response, I will maintain my score.

Comments on the paper

1 comment

Dear reviewers,

Thank you for your reviews. Please read the authors’ response and follow up in the discussion, briefly indicating whether your main concerns have been addressed and highlighting any remaining questions. Please also update your review and rating where appropriate.

Many thanks for your time and effort.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.