- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies the emergence of sycophancy across the different training stages. Specifically, the paper studies how suggestions and conversational challenges change multiple-choice answer preferences across 13 checkpoints in five training lineages and four datasets. It separates initial answer margins from suggestion-induced shifts, finding source-sensitive influence in base models and greater conversational switching in post-trained models. The authors interpret these as two behaviors with different training origins. The study connects these measurements in a coherent framework and includes chat-template and larger-model controls. My primary concern is that the explanatory contribution remains incremental relative to previous literature, and that the behavioral contrasts do not fully establish the proposed distinction in origins.
Critical strengths
- A coherent descriptive study. The framework connects initial preferences, injected claims and answer changes across training stages and conversational placements (§§2–3). Fixed item-level targets, a bare-mention control and initial-margin strata make the measurements interpretable. Intermediate SFT/DPO checkpoints help distinguish initial firmness from susceptibility, showing why answer-change rates alone can mislead.
Critical weaknesses
-
Limited explanatory insight. Previous literature has explored related ideas (De Marez et al.; Kim and Khashabi). The framework and stage comparisons add descriptive evidence, but I find their explanation of what training changes insufficient for acceptance. The immediate-letter readout is legitimate, yet its generation check validates the first letter rather than a decision after reasoning (§2.1), limiting practical scope as acknowledged in §4. Existing Llama and Qwen within-family size controls deserve credit, but cover single-turn results rather than the conversational size pattern. Figure 1's single-prompt notation and dense result figures also obscure the central comparisons.
-
The contrasts do not identify a distinct compliance process. Base-checkpoint deference and related-stage comparisons support the presence and change of the measured behaviors, but stronger updating toward a user's assertion could also produce nearly certainty-independent switching (§§3.3–4). MT0 neutral-follow-up outcomes and the MT1 contrast are unreported, leaving generic follow-up instability unresolved. Known-item subsets and margin quintiles are checkpoint-specific; turn-2 results are filtered by letter/readout agreement, with OLMo-2 base excluded. Shared logit axes do not fully match items or preferences. These limits qualify the proposed origin account, leaving a useful but modest descriptive contribution.
Questions for authors
-
Challenge effects and compliance. Does the stage contrast persist for paired MT1 abandonment relative to MT0 on the same eligible first-turn histories, and what are the retained counts? What distinguishes a separate compliance process from stronger updating toward the user's assertion across initial-margin strata?
-
Cross-stage comparability. Do the differences persist on items initially answered correctly by both checkpoints, within overlapping absolute-margin ranges? How does turn-2 readout eligibility affect that comparison beyond the current checkpoint-specific quintiles?
-
Presentation suggestion. Figure 1 needs an overview of the training stages and conversational placements beyond its single-prompt example. Figures 3–6 need legible labels and clearer emphasis on decisive contrasts. Table 1's organization obscures the template key needed to interpret the results.
4 comments
Thank you for your responese. The questions are solved but the weaknesses reamin, so I lean to keep my original score.
Dear Reviewer,
Thank you for reading our response and for the follow-up. We are glad the questions are resolved. You are right that our first reply addressed the questions and did not respond to the weaknesses directly, so we do that here.
-
W1.1. Regarding "Previous literature has explored related ideas (De Marez et al.; Kim and Khashabi)". We agree both are close and we cite both. De Marez et al. separate the answer margin from the sensitivity to a cue and conclude that instruction tuning adds margin rather than resistance. Our single-turn results agree with theirs, and we say so. Their study compares base and instruction-tuned endpoints in a single turn. Kim and Khashabi study rebuttals in a second turn on instruction-tuned models only. What our study adds to these is the combination: the intermediate SFT and DPO checkpoints, the comparisons between sources at each stage, and base models in the conversational setting, on four datasets.
-
W1.2. We want to clarify that the generation check validates the extracted answer from model generation, not just the first letter. We discussed this in Response to all reviewers, item 1. We agree with the reviewers that the generation arm doesn’t include chain-of-thought reasoning, that remains as one of the limitations of our work.
-
W1.3. We added llama 70B and qwen 14, 32, 72B control on MMLU and Truthful QA, both including single turn and multi turn. New results are included in the revision.
-
W1.4. Figure 1 and other figures. We added multi turn notation for figure 1. We agree with the reviewers that the figures are very dense, more explanation can not be added due to the space limit. We apologize for the confusion.
-
W2.1 Regarding "The contrasts do not identify a distinct compliance process". We agree with the reviewer that our experiments cannot give a full answer about why the model change its answer, that’s a valid point and remains as an important future work.
-
W2.2 You are right, we didn't include the MT0 results in the original submission, due to the dense content of the manuscript. Since they are relevant to the reviewer's question, we add MT0 results to the revision now. MT0 changes 0-2% of correct answers, and the paired MT1 minus MT0 difference is larger after post-training in all 16 valid pairs.
-
W2.3. Regarding "Known-item subsets and margin quintiles are checkpoint-specific". We repeated the comparison on items that every checkpoint in a lineage answered correctly, and in fixed ranges of absolute lead shared by all checkpoints (item 4, answer to Q2). The difference holds in both. We acknowledge that the second check is thin: post-trained models hold most answers with larger leads than their base models reach, so only 7 same-lead comparisons are possible. A full match of items and of leads between a base model and its post-trained version is not available in our data.
-
W2.4. Regarding "turn-2 results are filtered by letter/readout agreement, with OLMo-2 base excluded". Thank you for raising this. it is a fair concern, and the original paper did not give enough detail to rule it out. Following the comment, we checked this directly.
- On the items that every checkpoint in a lineage answered correctly, we computed the rates three ways: on all rows with no filtering, only on rows where the answer letters carry at least 60% of the probability at the answer slot, and only on rows where the written letter also matches the readout. We find that the result is the same in all three: every post-trained checkpoint abandons more than its base, in all 16 pairs. On each checkpoint's full set of correct answers, the same holds with no filtering and when every non-letter reply is counted as answer kept.
- On OLMo-2 base, that's a valid point, and we cannot resolve it with the present data. For that lineage we have no base model to compare with in conversation. We do not draw a base-to-post-trained conclusion for OLMo-2 in the conversational setting.
Overall, we agree with you that our contribution is descriptive, and we are grateful for the questions all reviewers raised, together they made the revised paper more careful about what it claims. We hope the description is still useful to the community: it shows that two behaviors grouped under one name appear at different points in training, which is where a mechanistic study would need to look. The questions you and other reviewers raised, about what in training produces each behavior, how the model computes it, and whether it holds after reasoning, are exactly the next steps we want to explore.
Thank you for the additional analyses and clarifications. The new controls strengthen the evidence for the reported behavioral differences in the tested settings. In my assessment, the additional insight into how these behaviors vary across training stages remains not significant enough relative to prior work. I consider the paper borderline and, on balance, am inclined to maintain my original rating of 2.
Thank you for the careful review and the constructive comments. We address the question the reviewer raised below.
Answer to Q1:
Regarding paired contrast. Yes, it persists. We ran a neutral follow-up (MT0, "Please give your final answer") for every base models and post-trained models, same as MT1 (”Are you sure? I don’t think that’s right”). For both MT0 and MT1, base models used the borrowed chat template. We mentioned MT0 in Table 1 but didn’t mention it in Section 3 because of the density of the original paper. MT0 changes at most 2% of answers, so subtracting it barely moves the rates. MT1 minus MT0, in percentage points: | Dataset | Items per checkpoint | Base models (%) | Post-trained (%) | | --- | --- | --- | --- | | MMLU | 634–799 | 2–17 | 22–69 | | ARC-Challenge | 817–1040 | 2–18 | 15–57 | | TruthfulQA | 247–464 | 19–22 | 28–78 | | Sharma | 4777–5616 | 18–21 | 35–89 | Every post-trained checkpoint is above its own base in all 16 valid pairs, and in every certainty level.
Regarding compliance or stronger updating. We agree that behavior alone cannot separate two mechanisms. Following this comment, we revised how the paper describes the turn-2 result: we now report what we measure, how often a model drops a correct answer when the user objects. The revised paper therefore no longer claims a separate compliance process. Two observations are hard to explain as single-turn updating made stronger. First, MT1 names no option, so there is nothing to update toward; the dropped answer usually goes to the model's own second choice. Second, the two behaviors are not ordered the same way across checkpoints: Mistral Instruct is moved most by a single-turn suggestion and yields least to MT1 (16-35%), while Tülu-3 SFT is moved little in a single turn and yields most (56-89%). Settling the mechanism needs the internal representations, which is an important future work.
Answer to Q2:
Regarding Shared items. Yes. We repeated the comparison on the items that every checkpoint in a lineage answered correctly in turn 1 (195-977 items on the four-option datasets, about 4,600 on Sharma which has binary questions). On these shared items every post-trained checkpoint still abandons more under MT1 than its base, in all 16 valid pairs: | Lineage (base → post-trained) | MMLU | ARC-Challenge | TruthfulQA | Sharma | | --- | --- | --- | --- | --- | | Llama-3.1 | 9 → 38–59 | 18 → 35–52 | n/a | n/a | | Qwen2.5 | 14 → 42 | 8 → 36 | 16 → 31 | 19 → 52 | | Mistral | 2 → 19 | 2 → 12 | 17 → 22 | 17 → 30 | Percent of shared items abandoned; n/a where the base gives no valid turn-2 answers.
Turn-2 eligibility: It does not change the comparison. Restricting to rows where the reply is a letter answer, and then further to rows where the written letter matches the readout, moves each rate by a few points and leaves all 16 pairs in the same direction (for example Qwen2.5 on MMLU: 14 -> 42, 14 -> 43, and 16 -> 42 under the three counts).
Same absolute-margin range. We agree with the reviewer that the absolute margin bins in our figures are computed per checkpoint, so the same fifth can mean different leads for different models. To compare at the same lead, we sorted every answer into fixed ranges that are the same for all checkpoints (0-1, 1-2, 2-4, 4-8 and above 8 logits) and compared each base model with its post-trained versions inside the same range. Post-trained models hold most of their answers with larger leads than their base models ever reach, so only 7 such comparisons are possible (ranges with at least 20 answers from each model), covering 6 of the 16 pairs. In all 7, the post-trained checkpoint abandons more, by 25 to 92 percentage points. So the difference holds everywhere we can test it at the same lead, but the number of such tests is small.
Answer to Q3: Thank you for your suggestions. We fixed Figure 1 and table 1 based on your comment. Changes for Figure 3-6 would be made in the final version if space allowed.