- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper introduces a benchmark of 450 synthetic text artefacts and 5,400 challenge arguments for studying changes in evaluative judgments. Models initially score each artefact over 20 runs and then revise their scores after valid or invalid arguments. Updates are classified relative to twice the baseline standard deviation and aggregated using Bradley–Terry argument weights. The paper reports separate responsiveness and compliance scores for ten models.
The research question is important, but the submission does not sufficiently validate the interpretation of its measurements. In particular, a valid observation about an artefact does not necessarily warrant revising an existing evaluation. Model-dependent challenge selection further complicates cross-model comparisons, and the absence of neutral rescoring controls prevents attribution of changes to the supplied arguments. The paper also has no Related Work section, so the contribution is never distinguished from closely related published evaluations. Together, these concerns undermine the benchmark's central claims.
Critical strengths
- Separately reporting responses to valid and invalid arguments is preferable to rewarding resistance alone.
- The construction pipeline includes substantial human review and efforts to remove generation artefacts and other superficial cues.
- Repeated baseline scoring, artefact-level bootstrap intervals, threshold sensitivity analyses, and the weighted/unweighted comparison provide useful descriptive checks. The shared 100-artefact comparison also addresses part of the concern arising from the smaller Opus evaluation.
Critical weaknesses
1. Argument validity does not establish that a score revision is warranted
The main construct-validity problem is the gap between a true, relevant observation and a justified change in an existing score. The model has already read the artefact before receiving the challenge. A valid argument can repeat a strength or weakness already incorporated into that score. Maintaining the score may therefore be appropriate, while changing it could double-count the same evidence.
The paper acknowledges this ambiguity in Sections 5 and 5.1, but still draws stronger conclusions about responsiveness and resistance to correction. The reasoning in Section 4.3 (lines 403–418) does not resolve the ambiguity: the fact that a model produces different scores across independent runs does not imply that it should revise a particular judgment after receiving an already-considered observation. Unconditional sampling variability is not evidence that a conditional revision is warranted.
Consequently, the benchmark cannot reliably distinguish justified stability from stubbornness. The observed lack of upward revision by Opus is a descriptive finding, but the current evidence does not establish that those revisions should have occurred. A targeted human assessment of whether particular updates are warranted, or a controlled distinction between previously considered evidence and genuinely corrective information, is needed to support the intended interpretation.
2. Model-dependent challenge selection confounds the comparisons
Section 3.2 selects the challenge direction using each model's own mean baseline score. Models scoring an artefact at least 50 receive lowering arguments; the others receive raising arguments. Different models can therefore receive different arguments for the same artefact.
This matters substantially because Section 4.2 reports large directional asymmetries. GPT-4o receives only lowering challenges, whereas GPT-6 Astra receives 326 lowering and 124 raising challenges. GLM-5.3 receives 408 and 42, respectively, with valid-following rates of 84.7% and 6.3% in those groups. Aggregate responsiveness therefore mixes model behavior with baseline scoring habits, artefact composition, and argument selection.
Recomputing results on the same 100 artefacts does not fully resolve this issue because the models retain their own challenge directions and thresholds. The present results do not establish how the ranking would behave under a common challenge distribution. A shared subset evaluated in both directions, with direction-specific results and a fixed aggregation rule, would provide a more interpretable comparison. Endpoint effects can be handled separately rather than by changing the evaluated challenge set across models.
3. Threshold sensitivity does not replace a neutral rescoring control
The baseline standard deviation describes repeated initial evaluations. It does not characterize the distribution of score changes when a model is asked to reassess within an existing conversation. Such changes may reflect reconsideration, contextual anchoring, or conversational effects even without a substantive argument.
Figures B3 and B4 show sensitivity to the scoring threshold, but they do not identify how much of the observed change is attributable to the feedback. The paper appropriately describes these thresholds as practical scoring rules rather than calibrated significance tests. Nevertheless, the missing neutral rescoring condition remains essential to interpreting the reported following probabilities. Valid/invalid contrasts are informative, but do not by themselves calibrate either absolute rate.
Moreover, an identical absolute update can count as following for one model and stability for another because the threshold depends on each model's variability. The analysis should include a neutral reassessment condition and a direction-only request, then report the incremental effect of substantive and invalid arguments relative to these controls. These are tests of the current benchmark's interpretation, not peripheral extensions.
4. Human validation does not clearly cover the final rewritten dataset
The reported human review precedes extensive normalization: Section 2.3 states that 2,560 of 5,400 arguments were rewritten more substantially, in addition to 2,819 arguments receiving tail removal. Rewriting almost half of the argument pool can change factual support and validity, even when preservation is an explicit generation requirement.
As described in Section 2.4, final verification focuses on residual construction cues, with human adjudication of uncertain or failed automated decisions. It is not clear that the final rewritten arguments underwent an independent validity reassessment. The earlier agreement statistics therefore cannot automatically be interpreted as validation of the released versions.
The 94.1% raw agreement also needs context because 97.6% of overlapping ratings accepted the assigned label. Annotators assessed supplied labels, and all were members of the author team. This does not invalidate their work, but strengthens the need for a blind audit of final argument texts, with class-specific results and disagreement analysis. For a benchmark whose central distinction is valid versus invalid feedback, this evidence is fundamental.
5. The paper has no Related Work section, so the contribution is never positioned against the closest published work
The paper has no Related Work section. For a benchmark paper, this is a serious omission: nowhere does the paper explain how Big-Big-SycoBench differs from existing sycophancy evaluations in task, feedback construction, ground truth, controls, or metrics. The introduction names a few prior benchmarks in one sentence each, but it lists them without comparison. As a result, the novelty claim is asserted rather than demonstrated. The omission matters because the closest published works already cover much of the proposed design:
- SycoBench-600 [1], published in Findings of ACL 2026, explicitly evaluates correct-suggestion acceptance, wrong-suggestion following, stubbornness, and correction selectivity. The submission cites it in lines 85–86, but does not sufficiently explain what is gained, or what validity is lost, when moving from answer correctness to argument validity in subjective scoring.
- SycEval [2], published at AIES 2025, distinguishes updates toward correct and incorrect answers and studies rebuttals with different rhetorical strengths. It is absent from the references despite its relevance to the proposed separation of beneficial and harmful updates.
- Sharma et al. [3], published at ICLR 2024, already evaluates feedback sycophancy on text passages, including arguments and poems, by comparing feedback under different user-preference cues. The submission cites this paper primarily for preference-training effects rather than explaining the relationship to its evaluative task.
The combination of subjective scoring, curated validity labels, repeated baselines, and argument weighting may offer a distinct contribution. However, that contribution needs to be stated narrowly and validated, especially because removing answer-level ground truth creates the ambiguity in weakness 1. A dedicated Related Work section that systematically compares task, feedback construction, ground truth, controls, and metrics is needed to clarify the advance. My concern is insufficiently established novelty and validity, not an allegation of plagiarism.
References
- Debu Sinha. SycoBench-600: Measuring Sycophancy and Correction Selectivity in LLM Assistants. Findings of ACL, 2026.
- Aaron Fanous et al. SycEval: Evaluating LLM Sycophancy. AIES, 2025.
- Mrinank Sharma et al. Towards Understanding Sycophancy in Language Models. ICLR, 2024.
Questions for authors
- What evidence distinguishes justified retention from failure to incorporate a valid argument? Can a blinded audit determine whether score changes are actually warranted in a representative subset?
- Do the main comparisons hold when all models receive the same raising and lowering arguments on shared artefacts, with a fixed direction mixture?
- How often do models cross the update threshold after neutral reassessment or a direction-only request, without substantive feedback?
- What fraction of the final, substantially rewritten arguments received fresh human validity judgments? Can the final-label audit be reported separately from the pre-rewrite annotations?
- What specific measurement capability does the benchmark establish beyond the published evaluations above, and which experiment validates that capability?
5 comments
W3 and Q3
We considered reassessment-only prompts when designing the benchmark, but came to the conclusion that they are not neutral. Asking a model to evaluate the same artefact again can itself signal dissatisfaction with its initial judgment and encourage revision. A direction-only request likewise explicitly pressures the model to change its score. These conditions would measure responses to additional conversational interventions, rather than provide an intervention-free estimate of scoring variability.
W4 and Q4
We will clarify steps 2-4 of the Benchmark creation process. (2) A human audit of all artefact+arguments by 12 annotators, (3) Rewrites based on the notes of the annotators by GPT-5.6 Sol high, (4) Targeted human checks and one final sweep with a different model, Opus 5 high.
In Step 4, an author reviewed 30 targeted arguments, all six edited artefacts, and a separate 60-argument sample balanced between valid and invalid arguments, requested corrections, and rechecked the revised arguments. This audit mainly flagged leftover score references. Then, Claude Opus 5 screened all 5,400 arguments for residual scoring cues. Its 78 flagged arguments were repaired following author approval. An author then individually reviewed all 55 uncertain cases and all 20 provider-refused cases, identifying another 53 and two arguments requiring repair, respectively. This yielded 133 repairs: 130 received a fresh Opus clearance, while three received targeted assistant verification following provider refusals. These checks concerned scoring cues, rather than reassessment of argument validity.
We will also report the original 94.1% agreement alongside the 97.6% acceptance of assigned labels. Annotators were author-team members who could see those labels, creating a risk of confirmation bias. We will explicitly acknowledge this and the absence of an independent, blinded validity audit of the final dataset as limitations.
W5 and Q5
We agree that the paper needs a dedicated Related Work section, and will add one explicitly comparing tasks, feedback construction, reference labels, controls, and metrics.
Our specific contribution is to assess revision of subjective evaluations using reference labels for the support of the challenging argument, without imposing a uniquely correct overall score. This addresses settings such as evaluating essays, proposals, and reviews, where reasonable evaluators can disagree about overall quality while still distinguishing supported criticism from invalid pressure.
SycoBench-600 and SycEval evaluate updating relative to answer correctness: correction and misleading pressure are assessed according to whether the initial answer is wrong or right. Our benchmark instead presents valid and invalid arguments against the same existing evaluation. Moreover, genuine strengths and weaknesses can coexist in one artefact, allowing argument validity and raising/lowering direction to be varied separately. Sharma et al. already study subjective feedback, but their feedback experiment varies user-preference cues and measures feedback positivity. We study subsequent revision of an established evaluation under arguments whose support is explicitly assessed.
Although the author's rebuttal partially addresses my concerns, I do not find it sufficiently comprehensive; therefore, I will not be increasing my score at this stage. Given that there is still some time remaining, I suggest that the authors undertake substantial revisions to the paper (such as adding a "Related Work" section) to facilitate further discussion among the other reviewers and the Area Chair in the next stage.
W1 and Q1
We agree that a valid argument need not warrant changing an existing score: the argument may already have been considered, or may not warrant changing the overall evaluation. This is also why we do not claim that Astra’s judgments are generally better than Opus’s. We agree that variation across independent initial evaluations does not establish that a particular conditional revision is warranted, and will revise Section 4.3 accordingly. To further examine this distinction, we performed two additional analyses.
(1) We annotated the reasoning traces of score updates for all 2,700 evaluated arguments per model for GLM-5.3 and Kimi K3, using one randomly selected repetition per argument from the original runs. When GLM and Kimi follow valid arguments, they explicitly endorse them in 97.7% and 94.7% of cases, respectively. Crucially, when their updates are classified as stable, they still endorse or partly endorse valid arguments in 99.4% of cases for both models. Conversely, when following invalid arguments, GLM and Kimi explicitly reject them in 44.7% and 63.8% of cases, respectively; when classified as stable under invalid feedback, both reject the arguments in over 97% of cases. These findings show that score revision and expressed acceptance of feedback are distinct. The full table appears in our response to Reviewer 1oZM.
(2) We collected 120 new update justifications each from Astra and Opus, using their own saved initial assessments on 20 matched artefacts. We annotated endorsement using the same categories as in (1), with a separate annotation pass identifying whether the model stated that it had already accounted for the argument. For valid-feedback cases classified as stable, we obtain:
These results identify prior consideration as one explanation offered by Opus for retaining its evaluation. They do not independently verify that the observation was already incorporated or establish whether retention was warranted. Accordingly, we interpret low responsiveness as limited score revision under valid feedback, rather than sufficient evidence of stubbornness. The full table appears in our response to Reviewer 1oZM.
We thank the reviewer for prompting this distinction and will update Section 4.3, the Discussion, and the Conclusion accordingly.
W2 and Q2
Yes, Figure B1 shows that the mean initial scores are mostly >50/100 for all models, which indeed means that they get mostly lowering challenges. However, Figure 5 shows that models are also more inclined to follow valid arguments advocating for a lower score than for a higher score. As the new experiment below shows, this also holds for invalid arguments for Luna and Terra.
In response to the reviewer’s suggestion, we evaluated Astra, Luna, and Terra on the same 20 artefacts using five repetitions of the six arguments in the previously untested direction. We combined these 600 new responses per model with 600 matching original-direction responses. The resulting dataset covers all 12 arguments per artefact, with five repetitions each, so all three models receive the same arguments in both directions. The table reports unweighted following rates, with 300 responses per cell.
Using this combined dataset, we find that all three models follow valid lowering arguments more frequently than raising arguments. Whereas Figure 5 compared different artefact groups selected for each direction, this experiment shows that the pattern persists when both directions are evaluated on the same artefacts. Luna and Terra also exhibit this asymmetry under invalid feedback.
To show how this behavior affects aggregation, we compare the original direction-selection rule with a fixed 50:50 raising/lowering mixture on this same subset:
Balanced aggregation gives greater weight to raising challenges, under which following rates are lower, and therefore reduces measured responsiveness, most strongly for Astra. Astra still resists all invalid arguments in both directions, while Terra follows fewer invalid arguments than Luna.
We thank the reviewer for the suggestion which provided us with more nuanced insights, and will clarify both this finding and its implications for interpreting aggregate scores in the revised manuscript.