acceptodds
Under review as a conference paper at ICLR 2027

Jagged Judges: Epistemic Stability Under Perturbation, Pressure, and Persistence

Abstract

LLM judges have become central infrastructure for model evaluations, online grading, and reward modeling. Judges are typically validated by accuracy on golden data, but accuracy says little about whether they are stable under reprompting, challenge, or sustained pushback. We introduce the Wiggle Framework, a unified stress test for epistemic stability in LLM judges. The framework decomposes judge robustness along three dimensions: Mechanical Consistency (stability under re-prompting and reframing), Single-turn Conviction (stability under a single challenge), and Multi-turn Persistence (stability under sustained or adaptive pressure). We use the framework to study 9 frontier models across 14 judging tasks spanning safety, toxicity, AI writing detection, and politicalresponse evaluation, emphasizing difficult or class-balanced subsets. At turn 10, task-average wiggle rates range from 19–71% under repeated static consensus pressure and 40–73% for a single adaptive LLM persuader; coverage under at least one of three persuaders is 62–91%. Successful flips are more often corrupting than corrective in aggregate, but a base-rate-controlled analysis shows that initially incorrect binary verdicts are conditionally more likely to flip than initially correct ones. Beyond the framework itself, we find that baseline jury majority strength is associated with item instability, although its predictive strength falls under held-out-judge evaluation. Taken together, this is the first apples-to-apples cross-dataset comparison of mechanical, conformity, and persuadability tests in a judging context.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
LLM-as-a-Judge, epistemic stability, robustness, sycophancy, judge calibration, adversarial persuasion, multi-turn debate, pressure, temperature, position bias

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_jaVTOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces the Wiggle Framework for measuring the stability of LLM judges under repeated prompting, single-turn challenges, and sustained or adaptive conversational pressure. The framework anchors each trajectory to an initial no-pressure verdict and reports mechanical consistency, single-turn conviction, and multi-turn persistence. The experiments evaluate nine frontier models on 14 judging tasks from six datasets covering safety, toxicity, AI-text detection, and political-response evaluation, using both binary and Likert scales. The reported results are that all tested judges change their verdicts under at least some pressure, adaptive multi-turn persuasion produces higher wiggle rates than most static challenges, flips are net-corrupting in aggregate but initially incorrect verdicts are more likely to flip conditionally, and jury majority strength is associated with item-level wiggle but is weaker under leave-one-judge-out analysis.

Critical strengths
  1. The paper brings several previously separate stability probes into one evaluation protocol and applies them to the same judges, items, and grading scales. This centralization enables direct comparisons between mechanical variation, single-turn challenges, and multi-turn persistence across multiple judging domains.

  2. The analysis goes beyond a single aggregate instability score. It reports per-task and per-model results, Binary versus Likert differences, correctness-conditioned flip rates, a representative-sample ablation, cross-task profile correlations, and a leave-one-judge-out analysis of jury majority strength.

Critical weaknesses

The Wiggle metric does not distinguish unjustified capitulation from justified belief revision. A judge that changes its verdict after a valid counterargument may be updating appropriately, while a change caused by a misleading argument may indicate susceptibility; both are counted as wiggles. L2 and L3 use model-generated arguments whose evidential quality is not independently controlled, and L6 permits the persuader to invent authorities, precedents, and rhetoric. The current results therefore support sensitivity to the tested persuasion prompts, but do not by themselves establish general epistemic instability.

Questions for authors
  1. Please compare the Wiggle Framework with the closest existing consistency, sycophancy, persuasion, and position-sensitivity protocols at the level of estimands and experimental controls. Which reported findings are new because the probes are centralized, and which reproduce known effects?

  2. Please independently assess the evidential quality of the L2-L6 arguments, or compare matched valid and misleading counterarguments while controlling the number of turns, approximate token budget, argument quantity, and persuader identity. Reporting whether verdict changes differ between valid and invalid challenges would clarify when wiggle indicates harmful capitulation versus appropriate updating.

Reviewer_7Zj5OpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes the Wiggle Framework to evaluate the epistemic stability of LLM judges, not just their accuracy relative to ground-truth labels. The framework comprises three dimensions: Mechanical Consistency, Single-turn Conviction, and Multi-turn Persistence, testing the changes in judgments relative to L0 verdicts under no-information perturbation, single-turn questioning, and continuous or adaptive pressure, respectively.

The authors conducted experiments on 9 frontier models and 14 judging tasks, covering safety, toxicity, AI-generated text detection, and political-response evaluation, while simultaneously examining Binary and Likert scoring. Results show that mechanical perturbation results in a lower wiggle rate, but fabricated consensus and adaptive persuaders can significantly alter judgments; pressure generally reduces accuracy, although initial errors are more easily corrected under the Binary verdict condition. The paper also analyzes the differences between different pressure types, scoring scales, models, and datasets, as well as the predictive role of jury majority strength on item-level instability.

Critical strengths

1.A unified and practically valuable evaluation framework. The Wiggle Framework unifies the comparison of Mechanical Consistency, Single-turn Conviction, and Multi-turn Persistence across the same items, judges, and rubrics, enabling apples-to-apples comparison of previously fragmented failure modes. This has direct value for deploying LLM judges, automated review, and agentic oversight.

2.The experiments are comprehensive and the analysis is cautious. The paper covers 9 models, 6 datasets, and two scoring scales, distinguishing between wiggle, corrective flip, and corrupting flip. The authors also report L6 mean-per-persuader and union coverage, and by controlling for L0 correctness, avoid misinterpreting overall corrupting flips as correct decisions, which are themselves more easily swayed.

Critical weaknesses

1.The construct validity of “epistemic stability” remains insufficient. All challenges are designed to push the decision away from L0, thus the wiggle simultaneously includes unprincipled capitulation and reasonable updates based on valid arguments. While corrective/corrupting analysis on ground-truth tasks provides some relief, L6 does not control for argument quality, verbosity, or rhetorical strategy, and Paired Prompts lack ground truth. Therefore, the current results more directly measure conversational susceptibility than the more general epistemic stability.

2.Core metrics still suffer from measurement and reproducibility issues. L5/L6 rely on the GPT-5 observer model to extract verdicts, but the paper only reports manual inspection of 100 transcripts, without providing a confusion matrix or agreement statistics. Repeat-based Mechanical Consistency in the appendix is ​​also described as per-trial disagreement and the item-level rate of “any change in 10 trials,” respectively; furthermore, some midpoint tie-breaking and runtime prompt-selection rules are not preserved. These issues can affect the interpretation and reproduction of core wiggle rates.

Questions for authors

1.Please clarify the unit of measurement for repeat-based Mechanical Consistency: is it the average disagreement rate of all repeated trials relative to L0, or the proportion of arbitrary changes in each item across 10 runs? Please explain the relationship between the different statements in the main text, Table 16, Figure 7, and Appendix C, and provide results under a consistent caliber where necessary.

2.Could you provide the confusion matrix validated by the observer model, the overall and sub-model/sub-task agreements, and the sensitivity analysis of recalculated L5/L6 wiggle rates on manually labeled subsets? This will directly impact my assessment of the reliability of the core multi-turn results.

3.How can I distinguish between reasonable updates in L6 due to valid evidence and unprincipled capitulation due to authority, verbosity, or other rhetorical tactics? If available documentation allows, could you analyze it by argument correctness or strategy type, or correspondingly narrow the interpretation of "epistemic stability"?

Reviewer_jybuOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces the Wiggle Framework to evaluate LLM-judge stability under mechanical perturbations, single-turn challenges, and sustained or adaptive multi-turn pressure. Experiments cover nine models and fourteen judging configurations across six datasets. The results reveal substantial verdict changes under pressure, varying susceptibility across models and tasks, and differences between Binary and Likert evaluations. The paper also analyzes corrective versus corrupting changes, initial correctness, and baseline jury agreement.

Critical strengths
  1. The experiments are comprehensive. The paper evaluates multiple models, judging configurations, and datasets, including sensitive domains, providing a broad empirical assessment of LLM-judge stability.
  2. The paper provides useful analyses of judge behavior under different types of perturbations and pressure, revealing several interesting patterns and differences across settings.
Critical weaknesses
  1. [a] already examines authority and bandwagon biases closely related to L3–L4, The paper should clarify how its findings go beyond these known phenomena.

[a] Justice or Prejudice? Quantifying Biases in LLM-as-a-Judge; https://arxiv.org/abs/2410.02736

  1. Section 5.2 states that first-turn Binary flip rates are 3–4 times the Likert rates across L2–L6, but Table 4 shows ratios of 1.5 at L4 and 2.6 at L5. Additionally, Appendix C.1 measures mechanical wiggle as the fraction of trials differing from L0, whereas Table 16 uses the fraction of items with any change across ten trials. These inconsistent definitions introduce ambiguity.
Questions for authors
  1. Would model stability rankings persist under matched reasoning budgets or when comparing the same model with reasoning enabled and disabled?
  2. Does L6 still outperform static strategies when total token budget, argument count, and dialogue turns are matched? Can the gains from adaptive feedback be separated from those due to argument diversity?
  3. In practice, can jury agreement predict subsequent judgment reversals more effectively than single-model repeated sampling or self-reported confidence at the same inference cost?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.