- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper argues that existing inference-time anti-sycophancy interventions are usually evaluated only when the user's suggested answer is wrong, even though the same interventions may discard beneficial corrections when the user is right. For each question, the authors construct matched conditions with a wrong hint, a correct hint, and no hint. They propose accept-or-answer: the VLM is first run on the opinion-free question, and the user's candidate is scored by the logit margin of its first token relative to the model's top token. If this margin exceeds a threshold learned from labeled calibration data, the system returns the user's answer; otherwise, it continues decoding the model's answer from the opinion-free cache. Across 17 settings covering seven benchmarks, multiple opinion phrasings, and three local backbones, the method reaches 72.2% mean accuracy, compared with 69.3% for the best fixed policy in each setting selected post hoc.
Critical strengths
-
The paper shows that LQCD can gain 52 points when the hint is wrong while losing 47 points when it is correct. The decomposition into capitulation and beneficial-correction cells explains why an intervention that always pushes back must sacrifice useful corrections. This analysis can materially change how methods are ranked when only wrong-hint accuracy has previously been reported.
-
The method is simple, reuses computation, and exposes an interpretable decision signal. Accept-or-answer needs only one prefill of the opinion-free question; when it rejects the hint, it can continue decoding from the same KV cache. The candidate margin directly asks whether the model considered the answer plausible before seeing the user's opinion, and it outperforms policies based only on the model's own confidence.
-
The empirical analysis is detailed and candid about statistical uncertainty. The authors also disclose that the final policy was chosen after inspecting several margin-based alternatives, rather than presenting the entire study as preregistered confirmation.
Critical weaknesses
-
Most headline results use constructed wrong hints that are often implausible. Natural and plausible-error settings should be summarized separately as a primary result and expanded with real interaction data.
-
The method assumes that the neutral question x_G and candidate h have already been parsed correctly, but end-to-end parsing causes a large performance drop. Candidate extraction, question rewriting, and final selection should be evaluated jointly on natural human messages.
Questions for authors
-
In a fully end-to-end setting without gold x_G or h, using independent human messages that may contain explanations, hedging, or multiple candidates, what is the net improvement of accept-or-answer over no-op, counter-instruction, and always-remove when parsing failures are counted as final errors?
-
What happens if the first-token margin is replaced by length-normalized full-candidate log probability with common semantic equivalents aggregated, particularly on natural hints, multi-token answers, and the subset in which the candidate and model answer share their first token?
3 comments
Follow-up to Reviewer rsUn
Dear Reviewer rsUn,
With the discussion period now past the halfway mark, we wanted to thank you again for your insightful comments—particularly on evaluating end-to-end message parsing jointly (Table R1) and testing whole-candidate log-probability scoring alongside separate primary breakdowns for plausible and natural hints (Table R2, §5.1). Please let us know whether our response and the highlighted revision address your questions; if anything remains open, we are very happy to discuss further.
Thank you for the author's detailed reply. This paper is simple yet effective, and I have decided to maintain a positive grade.
Response to Reviewer rsUn
Thank you for highlighting how the matched correct-hint control reshapes method rankings, and for proposing the two exact stress tests—end-to-end message parsing and full-candidate scoring—that anchor our revision.
W1: Elevating plausible and natural hints as primary results (General Response A, Abstract, §5.1, Tables 2 & 16).
The Abstract and §5.1 now report performance broken down explicitly by hint regime against the unified post-hoc best fixed baseline (cross-fitted tuned LQCD included across Table 2, and fixed LQCD a = −0.5 plus balanced instruction added to Table 16):
W2 & Q1: Joint end-to-end evaluation net of parsing errors (General Response B, Appendix A, Table R1).
Qwen3-VL-8B wrote 1,095 natural messages across eight styles (including hedged, two-candidate, reported speech, statement-first and chatty context) for all 800 GQA colour queries and all 295 real human-annotator VQAv2 queries. Qwen3-VL-4B extracted (x_G, h) text-only, missing extractions counted wrong (0 missing), τ was cross-fitted out of fold on gold benchmark queries (zero overlap with test messages), and every baseline answered the same messages:
The drop from gold inputs (70.4%, 70.5%) comes mainly from extraction: h is recovered exact on 91.1% of GQA messages vs 59.0% on VQAv2 (where 4B often extracts full clauses), rewritten x_G alone costs always-remove 6.8 and 7.1 pts, and end-to-end decisions match gold decisions on 93.8% and 68.1%.
Q2: Whole-candidate log-probability and synonym aggregation (General Response B, §6, Table R2).
Teacher-forcing four whole-candidate scores on the cached opinion-free prefill (zero extra prefills) yields 17-setting means of 72.2% / 72.3% / 72.6% / 72.1% (length-norm / total log-prob / full-seq margin / length-norm + static-list synonyms, never annotator answers) vs 72.2% first-token margin. On your three target subsets:
Because length-normalised variants lose 1.6–2.2 pts on TextVQA and OK-VQA (LLaVA), §6 recommends selecting the whole-candidate score alongside τ on calibration queries when multi-token candidates are frequent.
Presentation. To improve clarity, Figure 1c now shows paired differences with 95% CIs, §3 derives LQCD's a ≥ 0 bound, §4.1 contrasts AoA directly against S2VA, and §5.1 separates random, plausible and natural hints.