- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes OmniConfess, a training-free method for reducing hallucinations in omni-modal models. It first generates and freezes a candidate response, then removes evidence channels one at a time and rescores the same tokens under fixed prefixes. These changes are used to estimate which channels support each prediction and to guide either a revised judgment or local rewriting of suspect spans. The authors evaluate the method on OmniHalluBench, a 3,540-example collection drawn from six benchmarks, using three backbones. They report improvements over several inference-time baselines and analyse the method's components, generation quality, and computational cost.
Critical strengths
S1. Fixing the candidate and its prefixes provides a clear way to separate the effect of changing evidence from the effect of generating a different continuation. The token-by-channel measurements also give the method a concrete basis for deciding which parts of an answer to revise. The same framework handles both judgment and free-form tasks without additional training, which makes this approach practically useful.
S2. The evaluation covers text, image-text, and audio-visual tasks across three backbones, with comparisons against several families of inference-time methods. The ablations examine anchoring, channel-wise scoring, and correction separately, and the main results show substantial improvements in several settings. The paper also reports inference costs. Overall, the breadth of this approach is impressive.
Critical weaknesses
W1. Important evaluation and implementation details are missing.
The paper names GPT-4o and human experts for the multidimensional quality assessment in Section 5.4, but it does not identify the LLM judge used for the GAV scores in Table 1.
The decoding settings, revision threshold , thresholds for assigning diagnostic states, and baseline hyperparameters are also not specified in enough detail. Appendix E largely describes the methods rather than giving their actual configurations, such as the number of Self-Consistency samples or the VCD contrast coefficient. These choices can affect both quality and cost. Without them, it is difficult to reproduce the results or determine whether the baselines were appropriately adapted to the tasks and backbones.
W2. The reported Nemotron improvement does not agree with Table 1.
Figure 6(b) and Section 5.5 report an average F1 improvement of +7.50 for OmniConfess on Nemotron. However, Table 1 gives 61.57 for OmniConfess and 53.06 for Base, a difference of 8.51. Other entries agree with the table. Thus, the authors should clarify the value.
W3. The headline F1 improvement is not tied to a clear comparison.
The conclusion states that F1 improves by up to 13.3 points, but the source of this figure is unclear. I could not identify the dataset, backbone, and comparator that produce this value.
Questions for authors
Q1. Are the prompting conditions identical across methods? Section 4.1 introduces task-specific evidence constraints prior to the OmniConfess pipeline. Do Base and all baselines share identical generation prompts? Additionally, Appendix A.1 omits generation prompts for PhD and RAGTruth, and Section 4.1 references correction prompts for that are missing. Please provide these missing prompts and clarify how prompt consistency was enforced across models.
Q2. How are evidence interventions and correction rules implemented? Please specify key hyperparameters and definitions: the revision threshold , the relevance/contribution thresholds for assigning , and task-specific definitions of . How is constructed (token deletion, empty input, feature masking, or noise replacement)? For text-only benchmarks (PubMedQA, RAGTruth), how are evidence channels defined and isolated from questions and instructions? Providing baseline decoding settings and full hyperparameter specifications is critical to interpreting the measured probability shifts.
Q3. Why do certain baselines perform significantly worse than Base? On Qwen2.5-Omni, PhD F1 drops from 71.68 (Base) to 21.16 (GD) and 32.12 (PD). On RAGTruth, ToT achieves only 12.08 Token-F1 versus 38.25 (Base). Could you analyze representative failures to determine if these drops stem from intrinsic method flaws, formatting mismatches, or prompt adaptations? Furthermore, since MAD is evaluated only on CMM and AVHBench, its average is computed over a non-identical benchmark set; please include a matched-subset comparison or explicitly flag this discrepancy.
Q4. How should the efficiency metrics in Table 2 be interpreted? OmniConfess reports 0.82s latency on HaloQuest versus 0.93s for Base, despite adding channel-wise rescoring after candidate generation. On AVHBench, it reports 164.1 output tokens versus 25 for Base. Please clarify these timing measurements (e.g., per-example averages, caching, batching, hardware handling) and specify what the reported token counts include (candidate generation, revisions, or final response only). Computational costs for teacher-forced rescoring should be distinguished and reported separately to properly contextualize the efficiency claims.
Q5. Correct the appendix examples and explanations. In Figure 12 (PhD cat example), the Ground Truth field incorrectly contains the motorbike text from Figure 11. In Appendix G, the text claims Figure 11 contrasts "person" vs. "man," but neither word appears in the displayed reference, and the OmniConfess answer already starts with "A man adjusts..." Please update these figures with the intended predictions and ground-truth references to ensure the qualitative analysis can be verified.
Q6. Positioning this work with preference-trained hallucination mitigation. A discussion comparing OmniConfess to preference learning approaches such as HA-DPO, SENTINEL (ICCV 2025), and TPR (NeurIPS 2025) would help contextualize the contribution. For instance, SENTINEL highlights early object hallucination propagation, complementing OmniConfess's finding that evidence dependence is front-loaded; similarly, TPR's topic-level rewriting relates closely to local span repair. Please elaborate on what OmniConfess adds to diagnosing and correcting these phenomena.
I would be happy to revise my assessment if the authors can clarify these points and provide further details.
10 comments
Response to Reviewer bZg4 Q2
Q2. How are evidence interventions and correction rules implemented?
Please specify key hyperparameters and definitions: the revision threshold , the relevance/contribution thresholds for assigning , and task-specific definitions of . How is constructed (token deletion, empty input, feature masking, or noise replacement)? For text-only benchmarks (PubMedQA, RAGTruth), how are evidence channels defined and isolated from questions and instructions? Providing baseline decoding settings and full hyperparameter specifications is critical to interpreting the measured probability shifts.
Response. We thank the reviewer for raising these points about the reproducibility of our evidence interventions, correction rules, and baseline decoding settings.
Channel relevance and evidence interventions. The rule indicates whether evidence from channel should be used to answer question in task . To clarify how this rule is applied, we added the table below to the Evidence Interrogation paragraph of the “OmniConfess Implementation Details” appendix. For each dataset, it shows how channel relevance is determined and which inputs are modified or retained during the intervention.
| Dataset | How channel relevance is determined | How the input evidence is modified |
|---|---|---|
| PhD | The task designates the image as relevant and the misleading textual context as irrelevant. | We remove the misleading context before tokenization, retaining the image, question, and task instructions. |
| CMM | A keyword rule assigns audio or video as the relevant channel from the question text. | We remove audio and video in separate passes. Each pass omits the corresponding media tensor and placeholder tokens, retaining the other modality, question, and task instructions. |
| PubMedQA | The PubMed abstract is the relevant evidence channel. | We delete the entire Passage: line containing the abstract before tokenization, retaining the question, task instructions, and output-format constraints. |
| RAGTruth | The retrieved passage is the relevant evidence channel. | We pass an empty passage field to the prompt constructor, retaining the question, task instructions, and output-format constraints. Both views score the same candidate tokens under the same answer prefixes. |
| HaloQuest | The image is the relevant evidence channel. | We omit the image tensor and its placeholder tokens, then rescore the fixed candidate with the same question and task instructions. |
| AVHBench | Both audio and video are task-relevant. Audio and visual lexicons associate content words in the candidate captions with their corresponding channels. | We vary video frame sampling and resolution while keeping audio and text fixed, then compare agreement on content words across the resulting captions. |
Diagnostic states and correction thresholds. We revised the text following Eq. (9) to define three diagnostic states and added their assignment rules to the “OmniConfess Implementation Details” appendix. For channel-removal interventions, we subtract the ablated score from the full-input score. For judgment tasks, the score is the candidate’s log-probability margin over its fixed alternative. For token-level diagnosis on RAGTruth and HaloQuest, it is the selected token’s log probability. A relevant channel is marked under-reliant when the signed change in judgment margin is below 1.0. For token-level diagnosis on RAGTruth and HaloQuest, the corresponding threshold on the signed change in log probability is 0.5. An irrelevant channel is marked over-reliant when the absolute difference exceeds 2.0. The remaining cases are marked aligned.
These discrete states support post-hoc interpretation. Correction uses continuous evidence scores with separate thresholds. The main configuration uses to determine when to revise a judgment in Eq. (14). For free-form generation, identifies weak evidence support, and governs whether to initiate local revision. The same appendix reports the threshold selection protocol and sensitivity analysis.
Baseline decoding settings. Appendix E, “Backbone and Baseline Details,” reports the decoding configurations and key hyperparameters used for the baselines in our experiments.
Response to Reviewer bZg4 Q3 & Q4 & Q5 & Q6
Q3. Why do certain baselines perform significantly worse than Base?
On Qwen2.5-Omni, PhD F1 drops from 71.68 (Base) to 21.16 (GD) and 32.12 (PD). On RAGTruth, ToT achieves only 12.08 Token-F1 versus 38.25 (Base). Could you analyze representative failures to determine if these drops stem from intrinsic method flaws, formatting mismatches, or prompt adaptations? Furthermore, since MAD is evaluated only on CMM and AVHBench, its average is computed over a non-identical benchmark set; please include a matched-subset comparison or explicitly flag this discrepancy.
Response. We thank the reviewer for suggesting a closer analysis of why these baselines underperform Base. To investigate what may account for the performance gaps on PhD and RAGTruth, we examined representative outputs from GD, PD, and ToT and added a “Baseline Failure Analysis” subsection to the “Results by Task Family” appendix. We also revised the table caption to clarify that MAD was evaluated only on CMM and AVHBench, so its average is computed over those two subsets.
Q4. How should the efficiency metrics in Table 2 be interpreted?
OmniConfess reports 0.82s latency on HaloQuest versus 0.93s for Base, despite adding channel-wise rescoring after candidate generation. On AVHBench, it reports 164.1 output tokens versus 25 for Base. Please clarify these timing measurements (e.g., per-example averages, caching, batching, hardware handling) and specify what the reported token counts include (candidate generation, revisions, or final response only). Computational costs for teacher-forced rescoring should be distinguished and reported separately to properly contextualize the efficiency claims.
Response. Thank you for the suggestion. We reviewed and revised Table 2 and added an analysis of the efficiency results in the main text. We clarified the measurement protocol in the Inference Efficiency (RQ5) subsection. We measure end-to-end wall-clock latency per example at a batch size of one, synchronize CUDA at both timing boundaries, and average the measurements after excluding warm-up runs. Token counts include all tokens generated during inference, including those from discarded candidates and revisions. Teacher-forced rescoring generates no new tokens. Its runtime is included in end-to-end latency and was not measured separately.
Q5. Correct the appendix examples and explanations.
In Figure 12 (PhD cat example), the Ground Truth field incorrectly contains the motorbike text from Figure 11. In Appendix G, the text claims Figure 11 contrasts "person" vs. "man," but neither word appears in the displayed reference, and the OmniConfess answer already starts with "A man adjusts..." Please update these figures with the intended predictions and ground-truth references to ensure the qualitative analysis can be verified.
Response. We thank the reviewer for pointing out the inconsistencies in the appendix examples. We corrected the fields in Figures 11 and 12, including the Ground Truth field in the PhD cat example, and revised the “AVHBench Case Study” discussion in Appendix G to match the content shown in Figure 11.
Q6. Positioning this work with preference-trained hallucination mitigation.
A discussion comparing OmniConfess to preference learning approaches such as HA-DPO, SENTINEL (ICCV 2025), and TPR (NeurIPS 2025) would help contextualize the contribution. For instance, SENTINEL highlights early object hallucination propagation, complementing OmniConfess's finding that evidence dependence is front-loaded; similarly, TPR's topic-level rewriting relates closely to local span repair. Please elaborate on what OmniConfess adds to diagnosing and correcting these phenomena.
Response. We thank the reviewer for highlighting the connections between early hallucination propagation, targeted rewriting, and our work. We expanded the “Omni-Modal Hallucination and Preference Training” paragraph in Related Work to discuss HA-DPO, SENTINEL, and TPR.
SENTINEL targets sentences where object hallucinations first appear. Our analysis tracks dependence on input evidence during generation and shows that it is stronger at earlier positions. TPR rewrites selected semantic topics to construct preference pairs for training. OmniConfess uses channel-specific evidence dependence to guide local correction at inference time. It rescores the same fixed candidate after removing each evidence channel in turn and checks the resulting token-level score changes against the task’s evidence requirements to locate suspect spans. It then regenerates those spans while preserving the rest of the response, without additional model training.
Response to Reviewer bZg4 W1 & W2 & W3 & Q1
Thank you very much for your time and effort in reviewing our paper. We sincerely appreciate your feedback. Below, we respectfully provide our detailed responses to address your concerns.
W1. Important evaluation and implementation details are missing.
Response. We thank the reviewer for raising these points about the reproducibility of our evaluation and implementation. In Evaluation Metrics under Experimental Setup, we now specify that GPT-4o assigns the GAV scores in Table 1 on a 0–10 scale using the task-specific evaluation prompt.
We revised the text following Eq. (9) to specify the three diagnostic states: aligned (0), under-reliant (−), and over-reliant (+). The new “OmniConfess Implementation Details” appendix documents the rules and thresholds for assigning these states, the decoding setup (greedy decoding with at most 512 new tokens), and the correction thresholds and how we selected them. It also reports a threshold sensitivity analysis on Qwen2.5-Omni-7B.
In Appendix E, “Backbone and Baseline Details,” we report the baseline configurations used in our experiments. For example, Self-Consistency samples four reasoning paths at temperature 0.6, and VCD uses a contrast coefficient of 0.1 and a noise standard deviation of 0.1.
W2. The reported Nemotron improvement does not agree with Table 1.
Response. Thank you for the suggestion. The correct average F1 improvement on Nemotron is 8.51 points (61.57 − 53.06 in Table 1). We corrected the value from 7.50 to 8.51 in Figure 6(b) and Section 5.5.
W3. The headline F1 improvement is not tied to a clear comparison.
Response. Thank you for the suggestion. We checked the result against Table 1 and revised Discussion and Conclusion to specify that OmniConfess improves Token-F1 by 17.3 points over the base model on RAGTruth with Qwen2.5-Omni-7B.
Q1. Are the prompting conditions identical across methods?
Response. Thank you for pointing out the ambiguity in our description of the prompts. Base and all baselines share the same task-level initial generation templates. OmniConfess uses its own templates, each containing instructions for both generation and correction. We revised the “Prompting Setup” subsection in Section 4.1 to clarify this and added the generation prompts for PhD and RAGTruth to Appendix A.1.