- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces ColonExplainVQA, a colonoscopy video question-answering dataset containing 7,962 examples across nine tasks, with original and human-corrected explanations. The authors highlight that correct answers may still rely on hallucinated visual evidence and propose Qwen-Colon-8B to address this issue. The training pipeline combines evidence-curated supervised fine-tuning (SFT) with the Colon Function Layer (CFL) followed by direct preference optimization (DPO). Experiments on a manually reviewed test set demonstrate improvements in answer accuracy and reference-consistency metrics over multiple baseline models.
Critical strengths
- Novel and meaningful benchmark
The paper addresses an important limitation of conventional medical VQA evaluation by distinguishing answer correctness from supporting visual evidence. ColonExplainVQA retains original and minimally corrected explanations, enabling targeted analysis of hallucinated observations. The independent annotation process, clinical adjudication, and manually verified test set strengthen the reliability of the dataset.
- Encouraging results with efficient ablations
Qwen-Colon-8B achieves competitive results across answer accuracy and reference-consistency metrics. The comparisons between original and curated SFT supervision, together with vanilla, random-layer, last-layer, and CFL-guided DPO, provide useful evidence for the effectiveness of the overall approach. In particular, the results suggest that both supervision quality and selective adaptation influence model performance.
Critical weaknesses
-
The proposed Reference Contradiction Rate (RCR) primarily measures explicit contradictions between generated and curated conclusions, rather than directly evaluating all unsupported visual claims. As acknowledged in Section 4.1 and Appendix A.2, details absent from the reference are treated as neutral, and the absence of contradiction does not guarantee complete visual grounding. Although the judgments are human-verified, it remains unclear how reliably the evaluation captures unsupported but non-contradictory visual claims. This limits the interpretation of the reported hallucination reductions, particularly the 0.95% rate among correct answers, as a comprehensive measure of visual-evidence reliability.
-
The benefits of human-corrected visual explanations are not sufficiently isolated from those of answer correction. Section 3.1.2 reports that 28.2% of examples involve answer changes, while 27.2% retain the answer but revise the explanation. However, the SFT ablation in Table 2 changes both answer labels and explanatory content simultaneously. Similarly, DPO preferences include both incorrect-answer responses and hallucinated explanations with correct answers. Consequently, the extent to which the observed improvements specifically arise from correcting visual evidence, rather than improved answer supervision, remains unclear.
Questions for authors
-
Appendix A.2 treats claims absent from the reference conclusion as neutral rather than contradictory. Could the authors clarify whether such claims are directly verified against the original videos during human review? In particular, can the reported 0.95% hallucination rate capture unsupported visual claims that do not explicitly contradict the reference?
-
Given the two correction types reported in Section 3.1.2, could the authors isolate the effects of answer correction and conclusion-only correction? For example, a comparison with answer-corrected-only supervision would help establish the additional benefit of human-corrected visual explanations.
2 comments
Dear Reviewer 7r1E,
As the rebuttal period comes to an end, we hope to make the best use of the remaining time for further discussion. We have addressed all comments and concerns raised in your review with utmost care, and hope that our rebuttal and revisions have met your expectations.
We look forward to further discussion. Thank you again for your time and effort!
Response to Reviewer 7r1E
Thank you for your detailed review and constructive feedback. We address your points below.
W1 and Q1: Direct-video validation of conclusions
Overview. We add a direct-video audit and compare its metrics with RCR (
Appendix A.3, manuscript Table 4), clarifying the scope of the original 0.95% reference-contradiction rate (Abstract;Appendix A.2).The original 0.95% is the reference-contradiction rate among correctly answered questions on the full 796-question test set. Human review verified reference consistency; reference-neutral claims were not systematically checked against videos.
We therefore conducted a direct-video audit on a shared, task-stratified random sample of 239/796 questions (~30%), balancing review workload with coverage of all eight tasks. Sampling was independent of correctness and RCR labels. We evaluated 2,629 original conclusions from 11 models. Two of three reviewers independently checked each video and its conclusions, with model identities hidden and references initially withheld. A conclusion was flagged if any visual claim contradicted or lacked support in the video, including reference-neutral claims. All three reviewers jointly resolved disagreements.
Table 1. Direct-video audit on the shared 239-question subset (%). Best in bold; second-best underlined.
On the audited subset, reference-based RCR and direct-video VUR have a mean absolute difference of 1.52 percentage points. For Qwen-Colon-8B, the rates are 18.83% and 19.25%, respectively, differing by 0.42 percentage points.
Under direct-video review, our model retains the lowest VUR (19.25%) and VURc (3.02%, 6/199), versus 29.29% and 10.20% for the best baselines, respectively, and the highest VJA (80.75%, versus 66.53%). These results support RCR's usefulness for model-level comparison and demonstrate that our model's advantage persists when all visual claims are assessed directly against the videos, including reference-neutral claims.
W2 and Q2: Separating answer and conclusion correction
Overview. We add matched full-split SFT controls to isolate answer and conclusion correction (
Appendix B.1, manuscript Table 5).We added SFT controls with identical training examples, initialization, budgets, checkpoint-selection rules, and the same 796 test questions. Q0/Q1 supervise only original/corrected answers. M0 uses original answers and conclusions; MA, MC, and MAC correct answers, conclusions, or both, respectively. Terminal option markers in MA/MC were adjusted for consistency with supervised answers.
Table 2. Separating answer correction and explanation correction (%). SFT-only comparisons. Best in bold; second-best underlined.
The gains cannot be attributed to answer correction alone. MA provides the requested control with correct reference answers and original conclusions. MA and MAC share identical answer supervision and matched training settings, differing only in conclusion correction. MAC improves all five metrics over MA, including more than halving CR_correct (9.39%→3.98%). This shows that correcting answer labels alone does not account for the improvement in conclusion consistency.
Together with Q0→Q1, which establishes the benefit of answer correction, these controls support complementary contributions from answer correction and conclusion correction: curated conclusions provide useful supervision beyond correct answer labels, improving both answer correctness and conclusion consistency.
We hope these results address your concerns.