acceptodds
Under review as a conference paper at ICLR 2027

They Know When They're Unsure: Confidence-Gated Acceptance as a Two-Sided Defense against Sycophancy in Video Language Models

Abstract

Video large language models (Video-LLMs) can answer visually grounded questions correctly, yet systematically reverse those correct answers when a user pushes back with a confident but wrong claim—a multimodal form of sycophancy. We first show that this failure must be measured two-sidedly: a defense should hold correct answers (should-hold) while still correcting genuine mistakes (should-flip). Under a two-sided balanced-accuracy metric, six language-level defenses—including self-consistency, chain-of-thought, an explicit "be confident" instruction, and our own structured visual-anchoring prompt—all collapse to the trivial never-change baseline (), because the model cannot express its own uncertainty in language. Yet the internal signal exists: first-answer probability predicts first-answer correctness with AUROC – across all models. We exploit this with Confidence-Gated Acceptance (CGA), a training-free defense that accepts a user's claim only when the model's own answer probability is low, using a single extra forward pass and one cross-validated threshold. Across five video-QA benchmarks and multiple open models, CGA significantly beats naive acceptance and the never-change baseline in all eleven evaluated cells, with the largest gain (pt) on the poorly-calibrated Qwen3-VL-8B. A threshold learned on Video-MME also transfers to TempCompass without retuning.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
multimodal sycophancy, video language models, confidence calibration, selective prediction, training-free defense

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_H9EFOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
1
Read review
Paper summary

This paper investigates sycophantic behavior in Video-LLMs when faced with user pushback. The authors argue that existing evaluations typically focus only on whether a model abandons an initially correct answer, which may mistakenly regard a stubborn strategy of never changing its answer as robust. To address this limitation, the paper introduces a two-sided evaluation protocol that assesses both whether the model maintains its answer when the initial response is correct and whether it accepts valid corrections when the initial response is wrong. Based on this setting, the authors propose Confidence-Gated Acceptance (CGA), which uses the model's internal confidence in its initial response to determine whether user feedback should be accepted. Experiments across five video question-answering benchmarks and multiple models show that CGA outperforms the naive acceptance and never-change baselines in most settings.

Critical strengths
  1. The two-sided formulation identifies an important evaluation flaw in existing sycophancy studies.
  2. The empirical study is relatively broad.
Critical weaknesses
  1. The reported +50–57 point gains on Qwen3-VL appear to be substantially affected by unparseable baseline outputs, as shown in the authors’ supplementary material. I inspected the three Qwen3-VL JSONL files provided by the authors and found that 60% of the naive responses on MVBench and 66.5% on PerceptionTest cannot be parsed into valid answer options. These cases are automatically counted as incorrect by the authors’ analysis scripts. This is a serious issue because the reported gains may therefore conflate improved sycophancy handling with differences in output validity.
  2. The evaluation protocol explicitly couples the correctness of the user claim with the correctness of the first answer: if the first answer is correct, the user always provides an incorrect claim; if the first answer is wrong, the user always provides the gold answer. Consequently, a method that can reliably detect whether the first answer is incorrect can naturally perform well on the proposed two-sided task. This makes it unclear whether CGA performs genuine claim arbitration or primarily first-answer error detection.
  3. In the supplementary code, the “first answer” is not the answer actually generated by the model. Instead, it is constructed directly as the argmax over the option logits. This does not fully match the interactive protocol described in the paper.
Questions for authors

Please see the weaknesses above. I have serious concerns about the validity and interpretation of the experimental results. If the authors can clearly address these issues and provide convincing clarification or additional evidence, I would be willing to reconsider my current assessment and potentially raise my score.

Reviewer_H4jhOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies the sycophancy problem of video large language models when answering multiple-choice questions: after the model answers, the user incorrectly asserts another option, and the model should not follow the wrong instruction but should stick to its own correct answer. The authors claim that this sycophancy should be evaluated from 2 dimensions, holding when the first answer is correct (should-hold) and accepting the correction when the first answer is wrong (should-flip). Then the authors propose a CGA confidence-gated acceptance method, which evaluates the first-answer option probability, accepting the user’s claim if it is lower than a threshold and holding otherwise.

Critical strengths
  1. The authors’ CGA method is simple, training-free, and low-cost.
  2. A 5-fold evaluation is used to select the threshold instead of fixing it.
  3. The experiments cover five video question-answering benchmarks and five models.
Critical weaknesses
  1. Under the evaluation setting of this paper, the user always gives the correct answer in should-flip cases, while CGA does not look at what the user says at all. Therefore, the two-sided score of CGA is mathematically equal to the balanced accuracy of “using the maximum softmax probability to judge whether the first answer is right or wrong” [1]. Moreover, the authors do not provide findings specific to the sycophancy problem, nor do they make any method design targeted at sycophancy. In addition, the evaluation lacks the case of “the first answer is wrong, and the user asserts another wrong option”: in this case, CGA will accept, and the result is still wrong.
  2. Stengel-Eskin et al. [2] have already proposed, earlier than the authors, that a model needs to both resist negative persuasion and accept positive persuasion, and showed that optimizing only one side harms the other side, and this paper does not cite it. MM-SY [3] has already found that mitigating sycophancy makes vision-language models more stubborn and less receptive to corrections, and proposed a training-free balancing method; although this paper cites this work, the listed authors do not match the original paper. In addition, the recent concurrent work MUSE [4] decomposes conformity into sycophantic conformity and uncertainty-driven conformity, which aligns with the core intuition of CGA, and the authors should discuss the relationship between the two.
  3. There are multiple inconsistencies in the experimental table data. For example, for the same cell Video-MME + InternVL3-8B, H/F is 245/155, and CGA is 68.6 in Table 1, while they are 137/63 and 74.0 in Table 2. The difference calculation in Table 2 is wrong: vs. naive should be 74.0 − 55.3 = 18.7, but the table shows +16.4. The CGA and naive of GPT-4o are both 78.8, but the difference is −4.5; relative to never-change, it should be +28.8, but the table shows +24.4. The threshold τ = 0.86 does not appear in the threshold grid ({0.3, …, 0.9, 0.95}) given in Appendix A. The two ablation variants in Table 3 (2D and claim-rank) have identical values on both models. These inconsistencies make the conclusions drawn from the experiments hard to be convincing.
  4. The baselines are insufficient. Self-consistency only uses k=3 [9], the number of samples is too small, and it is not sufficient as a fair uncertainty baseline. Commonly used uncertainty estimation methods are missing, such as P(True) [5], verbalized confidence plus threshold [6, 7], and semantic entropy [8]; existing sycophancy mitigation methods are also missing, such as PBT [2], the SFT/DPO of MM-SY [3], and the key-frame selection and representation intervention proposed by VISE [10].

Reference List:

[1]Hendrycks, Dan, and Kevin Gimpel. "A baseline for detecting misclassified and out-of-distribution examples in neural networks." arXiv preprint arXiv:1610.02136 (2016).

[2] Stengel-Eskin, Elias, Peter Hase, and Mohit Bansal. "Teaching models to balance resisting and accepting persuasion." Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers). 2025.

[3] Li, Shuo, et al. "Have the VLMs lost confidence? A study of sycophancy in VLMs." International Conference on Learning Representations. Vol. 2025. 2025.

[4] Guo, Kevin H., et al. "It's Not Always Sycophancy: Measuring LLM Conformity as a Function of Epistemic Uncertainty." arXiv preprint arXiv:2605.27288 (2026).

[5] Kadavath, Saurav, et al. "Language models (mostly) know what they know." arXiv preprint arXiv:2207.05221 (2022).

[6] Lin, Stephanie, Jacob Hilton, and Owain Evans. "Teaching models to express their uncertainty in words." arXiv preprint arXiv:2205.14334 (2022).

[7] Xiong, Miao, et al. "Can llms express their uncertainty? an empirical evaluation of confidence elicitation in llms." International Conference on Learning Representations. Vol. 2024. 2024.

[8] Kuhn, Lorenz, Yarin Gal, and Sebastian Farquhar. "Semantic uncertainty: Linguistic invariances for uncertainty estimation in natural language generation." arXiv preprint arXiv:2302.09664 (2023).

[9] Wang, Xuezhi, et al. "Self-consistency improves chain of thought reasoning in language models." arXiv preprint arXiv:2203.11171 (2022).

[10] Zhou, Wenrui, et al. "Flattery in motion: Benchmarking and analyzing sycophancy in video-llms." Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2026.

Questions for authors
  1. Please claim what new contribution or method the CGA method brings to the sycophancy problem.
  2. Please supplement the case of “the first answer is wrong and the user asserts another wrong option”, and report the performance of CGA and the baselines in this case.
  3. Please supplement how CGA performs compared with these baselines, such as the mitigation methods of PBT and MM-SY.
  4. Please explain why the sample sizes and scores of Video-MME + InternVL3-8B differ between Table 1 and Table 2; the difference calculation in Table 2 is wrong; why is the difference −4.5 for GPT-4o when CGA and naive are equal.
  5. Please explain how τ = 0.86 was obtained given the threshold grid? Why do the two variants in Table 3 have exactly the same values?
  6. In which turn and conditioned on what, is the p_r1 that CGA relies on computed? If it comes from the first-turn generation, why is one extra forward pass needed? If the first generated token is not an option letter, how is p_r1 obtained?
Reviewer_2ZnQOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies when video language models should retain their answers or accept user corrections. It proposes Confidence-Gated Acceptance (CGA), which accepts a suggested answer when the model’s initial-answer probability falls below a threshold and otherwise retains the initial answer. Experiments across five video-QA benchmarks evaluate both answer preservation and error correction, alongside prompting baselines, gating-signal ablations, and cross-dataset threshold transfer.

Critical strengths
  1. Evaluating both resistance to incorrect feedback and acceptance of correct feedback addresses a practically relevant trade-off. It prevents a policy that never changes its answer from appearing effective solely because it avoids harmful revisions, accounting for both resistance to sycophancy and correction ability.
  2. CGA requires no model-parameter updates and uses a single confidence threshold with low additional computational cost.
Critical weaknesses
  1. Li et al. [1] already evaluate resistance to misleading feedback alongside acceptance of correct feedback and discuss the risk of making models stubborn. This challenges the paper’s repeated characterization of prior evaluation as exclusively one-sided. Furthermore, Hendrycks and Gimpel [2] established softmax-based error detection, while Geifman and El-Yaniv [3] studied confidence-threshold selection.
  2. Section 4.1 and Appendix A guarantee that the user supplies the correct answer whenever the model’s initial answer is wrong. Under this construction, CGA’s two-sided balanced accuracy is mathematically equivalent to its balanced accuracy at classifying initial answers as correct or incorrect. It can therefore perform well without evaluating the suggested correction itself. However, low confidence in the initial answer does not establish that the user’s alternative is trustworthy, particularly when both answers are wrong. The evaluation excludes this case, so the results establish useful error detection but provide insufficient evidence that CGA reliably distinguishes helpful corrections from misleading feedback.

[1] S. Li, T. Ji, X. Fan, et al., “Have the VLMs Lost Confidence? A Study of Sycophancy in VLMs,” in International Conference on Learning Representations (ICLR), 2025. [2] D. Hendrycks and K. Gimpel, “A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks,” in International Conference on Learning Representations (ICLR), 2017. [3] Y. Geifman and R. El-Yaniv, “Selective Classification for Deep Neural Networks,” in Advances in Neural Information Processing Systems, 2017.

Questions for authors
  1. How does CGA handle an incorrect initial answer followed by another incorrect answer from the user? Can authors provide an evaluation that varies correction reliability rather than guaranteeing correct feedback whenever the model is wrong?
  2. Did you directly compare explicitly elicited verbal confidence against internal answer probability using the same threshold-selection procedure? Poor performance from CoT or “be confident” prompts alone does not establish that models cannot express useful uncertainty in language.
  3. Which results in Table 2 are authoritative? For InternVL3-8B, 74.0 minus 55.3 is 18.7, whereas the reported improvement is 16.4. For GPT-4o, both scores are 78.8, but the reported difference is −4.5.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.