- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper propsoes Learning to Verify (L2V), a dual-loop agent framework for multimodal hallucination mitigation. The inner loop repeatedly verifies and revises a main agent's response using image evidence, while the outer loop uses reference responses, counterfactual interventions, and held-out replay to refine the verifier's textual criteria. The approach is evaluated with four MLLM backbones on POPE-COCO, POPE-GQA, and CHAIR.
Critical strengths
- overall motivation, and inner/outer loop structure are presented clear
- the paper identifies a meaningful limitation iterative self-correction, i.e., a verifier can itself make errors so repeatedly applying the same verifier does not necassarily improve the response.
- the experiments are reasonably extensive
Critical weaknesses
- my main concern is that 'learning' is insufficiently specified. the paper described s_t as the verifier's criteria and it says that verifier revises these criteria based on recurring errors. what conceretely are these criteria and the optimization objective?, apprently outerloop is the main contribution
- final learned object is a set of verifier instructions, the paper itself provides substatial prior work on instruction optimization, prompt evolution, evaluator prompt refinement etc, so Learning to Verify is the central contirbution, and that requires evidence if the entire update is couterfactual guided refinement or simpler feedback based berifier prompt optimizer.
- the empirical scope should be widened. the paper motivates hallucinations broadly in terms of objects, attributes, relations, quantities, and insufficient evidence. The final evaluation consists of POPE and CHAIR, these are object hallucination benchmarks, the literature points to HallusionBench, MME, and NOPE for broader object hallucination mitigation.
- the outerloop uses LLaVA-Instruct-150K reference responses to identify discrepencies, only 800 images are used for deedback and 200 for held out replay. Also, the final utility U and candidate acceptance are reference controlled, so the possibility is part of the gain comes from learning task/reference-response.
- minor: the paper may benefit from further proofreading, several writing errors, 'metraics' , 'vairous',
Questions for authors
- clarification on what exactly s_0 and s_t, initial criteia, representative acceptance criteria,
- definition of U, and details around it
- can the work be compared agains a generic verifier prompt optimization baseline,
- also does the learned verifier becomes accurate?
- and how the learned criteria immprove perofrmance on hallucination involving attributes, quantities or composition reasoning.
Thank the authors for the detailed rebuttal, which addresses several of my earlier clarification concerns. However, the central issue remains that the independent contribution of counterfactual attribution is not fully isolated. The direct-feedback baseline uses a different optimization budget, with 2,229 calls versus 1,444 for L2V, and the candidate/adoption procedures also differ. The outer-loop and inner-loop ablations therefore still contain confounding factors, while no matched-budget comparison with a generic prompt optimizer is provided. I therefore maintain my original rating.