- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper separates two phenomena that are often conflated in discussions of human-AI interaction: style convergence (interlocutors adopting each other's wording) and sycophancy (models changing substantive positions under user pressure). Evidence comes from three sources: (1) an observational analysis of the HUMAINE corpus (125k conversations with pairwise votes), where users adopt model-introduced vocabulary above chance baselines (d = 0.33 to 0.60) and adoption predicts votes at AUC 0.609; (2) scripted counterfactual audits of 16 models, branching identical conversation prefixes into frozen follow-ups that vary emotional register or expressed stance, scored by two independent channels (rank agreement rho = 0.75); and (3) 699 model-model conversations showing semantic asymmetry collapses when the human is removed. The headline result is a null: the corpus style ranking does not correlate with audited capitulation across the 16 models (rho = -0.14, p = .609), supporting the title claim that mirroring and sycophancy are distinct.
Critical strengths
The audit methodology is careful and unusually transparent: frozen branches from identical prefixes, two independent scoring channels that agree (rho = 0.75), a disclosed conflict (the auditor model is also an audited target, with a sensitivity analysis excluding it), and external validation of the audit style ranking against the corpus (rho = 0.68). This matters for my assessment because it makes the measurements themselves credible, whatever one thinks of the conclusions drawn from them.
Critical weaknesses
-
The headline claim outruns the evidence. The dissociation rests on a rank correlation across 16 models. The paper's own Fisher interval at n = 16 is [-0.62, 0.41], so a moderate positive association "in models at large is not excluded" (the tighter interval of [-0.37, 0.04] captures only measurement noise within these 16 models, not sampling of models). A title-level general claim ("Mirroring Is Not Sycophancy") supported by a null over 16 points, where excluding the auditor model moves the estimate from -0.14 to +0.03, is weak foundation. This is consequential because the dissociation is the paper's main contribution; the component findings (models capitulate under stance pressure, users adopt model vocabulary) are individually closer to confirmatory than novel.
-
The practical stakes claimed for preference evaluation do not survive the paper's own controls. Convergence predicts votes only marginally before controlling for quality (beta = 0.10, p = .067) and not significantly after (beta = 0.07, p = .182), and the authors concede they cannot distinguish "convergence influences the judgment" from "users engage more with replies they already prefer." No intervention experiment is attempted. In addition, the register result ("tone changes style, not substantive advice") rests on the paper's least reliable measurement (advice-change ranking reliability 0.40, versus 0.85 to 0.92 for the other rankings), which the authors acknowledge.
Questions for authors
-
Given the Fisher interval and the sensitivity of rho to excluding the auditor model, would you consider restating the central claim as "no positive association observed among 16 current models" rather than the categorical title? If you believe the stronger claim is warranted, on what basis?
-
All rubric scoring and stance scoring is done by claude-sonnet-4-6. Have you replicated the capitulation ranking with a judge from a different vendor, as you did for direction labels elsewhere?
-
How should readers weigh the register-advice conclusion given the 0.40 reliability of that ranking?
-
An intervention that interrupts vocabulary convergence (e.g. paraphrasing model replies before the user sees them) would directly test the claimed threat to preference evaluation.
Thank you for a thorough and precise review. We agree with the main criticisms and plan to withdraw this submission and rework the paper.
The headline claim. You are right. The correlation across 16 models is -0.14. Treating the models as a sample, the interval is -0.62 to 0.41. Leaving out Claude Opus 4.8, which is the auditor and the least capitulating model, gives +0.03. The text says a sample of 16 models cannot exclude a moderate association, but the title does not carry that qualification, and it should. Leaving out any single model gives estimates between -0.33 and +0.03, so no one model produces a positive association. We have no basis for the categorical claim.
Q1: restating the claim. Yes. We will restate it as no positive association among these models. We will add more models and report how large an association the data rule out.
Q2: a judge from a different vendor. We had not done this for capitulation. The capitulation score, stance score, probe scores and quality ratings all came from claude-sonnet-4-6. Since your review we rescored the stance and probe scores with Gemini 3.5 Flash and GPT-4.1, using the original prompts. Agreement with the Claude judge on individual replies is 0.95 for the stance score and 0.83 to 0.87 for the probe score. The correlation of stance range with corpus accommodation is -0.23 under both new judges (-0.38 under the original). Claude Opus 4.8 stays among the three least capitulating models. One limit: we did not rescore the whole-branch rating behind the -0.14. We will in the revision.
Q3: the register-advice result. The 0.40 is the reliability of ranking models by judged advice change. The claim that tone changes commitment and not the recommendation rests mainly on scoring 1,208 replies on each seed's binary choice. The recommended option differed from the neutral reply in 9.1% of assertive cases, against 7.6% between two replies to the same follow-up (interval -4 to +7 points). That rules out a large effect but not a modest one. We will say so.
Q4: an intervention. Agreed. We will run it in the revision or remove the claim.
The vote result. The figures you quote are correct, but they come from the 1,953 pairs with quality ratings, about a fifth of the data. We have now rated all 10,798 usable pairs with GPT-4.1. On the full sample, adding quality lowers the convergence coefficient from 0.24 to 0.21, and it stays significant (p < .001). A single rating of the first reply is a noisy control, so the intervention is still the direct test.
Thank you again.