- Rating
- 2
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This work proposes a cross-modal hallucination mitigation algorithm that is training-free and it leverages the fact that uni-modal inputs for the prompt-relevant modality contains the information necessary to answer the prompt correctly. The authors justify the motivation to their work empirically and perform experiments on two cross-modal hallucination benchmarks as well as a general omni-understanding benchmark to show the efficacy of their approach. The proposed method steers the representation from omni-modal inputs to uni-modal representations hoping that such steering would correct the model's response.
Critical strengths
- The paper is very well written and easy to follow and the authors have added empirical justifications to most of their design choices (see W3 below).
- The proposed technique is a simple intervention and can be applied at test time without manipulating the inputs as some other test-time approaches do.
- The authors also show the performance of their approach on a general benchmark which helps judge how much the proposed approach generalize beyond fixing cross-modal hallucinations.
Critical weaknesses
W1. The approach in the case of using multiple modalities falls back to the default omni-modal LLM.
- If I understand correctly, the approach in the case of a prompt/question that relies on both the modalities falls back to the default omni-LLM reference.
- Considering the above, my questions is that can the authors report how they have performed the inference on Daily-Omni in Table-4. Also, can the authors share some stats on how frequently does this happen in case of DailyOmni is that the inference is falling back to the omni-LLM (i.e. the query is classified as requiring both the modalities).
W2. Single modality performance and motivation
- Section 3 and Figure 3 show how good single modality inputs are on tasks of AVHBench and CMM however, it only shows these results on the samples that were wrong with omni-modal reference. It would be good to see the performance of relevant unimodal input compared to omni-modal inputs across the entire sets of CMMBench and AVH bench.
- [Minor, and not the main point of the paper. No need to respond.] The recovery rates are significant but not close to a 100% meaning that the single modality is still not perfect to solve the task.
W3. Some design choices are not justified
- The authors apply depth-decayed residual intervention (or activation-steering) but its justification is somewhat losely stated in L249-250 without any citations. Although the ablations do justify this choice to some extent however, the performance reported in Figure 6 across different variations are within 1% of each other.
- The choice to use 3 consecutive layers for intervention is also not stated. The ablation in Figure 4 shows that choosing late layers is the best but why chose consecutive 3 layers in the first place and why choosing 3 layers?
W4. Comparison to baselines on general benchmarks and compute
- [W4.1] The authors should add a comparison to the other test-time baselines mentioned in Table 1 in the DailyOmni Table 3. Additionally, the authors should also report the performance on other tasks of AVHBench and CMM except just the cross-modal hallucination tasks.
- [W4.2] The authors should also add comparison to the test-time baselines on the basis of compute (FLOPs or inference time or number of forward loops or similar) to justify that CaRE uses similar compute as those.
Questions for authors
In addition to responding to the weaknesses above, can the authors also respond to the following:
Q1. The DailyOmni number seem very close to the baseline which can be attributed to noise. Can the authors report average and standard deviation across a few runs?
Overall I believe that this paper's idea is simple yet effective however there are some areas where I would like more justification from the authors. I would consider to increase my rating for this paper based on the author's rebuttal and the feedback from other reviewers. I wish the authors good luck for the rebuttals!