- Rating
- 2
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper investigates sycophantic behavior in Video-LLMs when faced with user pushback. The authors argue that existing evaluations typically focus only on whether a model abandons an initially correct answer, which may mistakenly regard a stubborn strategy of never changing its answer as robust. To address this limitation, the paper introduces a two-sided evaluation protocol that assesses both whether the model maintains its answer when the initial response is correct and whether it accepts valid corrections when the initial response is wrong. Based on this setting, the authors propose Confidence-Gated Acceptance (CGA), which uses the model's internal confidence in its initial response to determine whether user feedback should be accepted. Experiments across five video question-answering benchmarks and multiple models show that CGA outperforms the naive acceptance and never-change baselines in most settings.
Critical strengths
- The two-sided formulation identifies an important evaluation flaw in existing sycophancy studies.
- The empirical study is relatively broad.
Critical weaknesses
- The reported +50–57 point gains on Qwen3-VL appear to be substantially affected by unparseable baseline outputs, as shown in the authors’ supplementary material. I inspected the three Qwen3-VL JSONL files provided by the authors and found that 60% of the naive responses on MVBench and 66.5% on PerceptionTest cannot be parsed into valid answer options. These cases are automatically counted as incorrect by the authors’ analysis scripts. This is a serious issue because the reported gains may therefore conflate improved sycophancy handling with differences in output validity.
- The evaluation protocol explicitly couples the correctness of the user claim with the correctness of the first answer: if the first answer is correct, the user always provides an incorrect claim; if the first answer is wrong, the user always provides the gold answer. Consequently, a method that can reliably detect whether the first answer is incorrect can naturally perform well on the proposed two-sided task. This makes it unclear whether CGA performs genuine claim arbitration or primarily first-answer error detection.
- In the supplementary code, the “first answer” is not the answer actually generated by the model. Instead, it is constructed directly as the argmax over the option logits. This does not fully match the interactive protocol described in the paper.
Questions for authors
Please see the weaknesses above. I have serious concerns about the validity and interpretation of the experimental results. If the authors can clearly address these issues and provide convincing clarification or additional evidence, I would be willing to reconsider my current assessment and potentially raise my score.