- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces the Wiggle Framework for measuring the stability of LLM judges under repeated prompting, single-turn challenges, and sustained or adaptive conversational pressure. The framework anchors each trajectory to an initial no-pressure verdict and reports mechanical consistency, single-turn conviction, and multi-turn persistence. The experiments evaluate nine frontier models on 14 judging tasks from six datasets covering safety, toxicity, AI-text detection, and political-response evaluation, using both binary and Likert scales. The reported results are that all tested judges change their verdicts under at least some pressure, adaptive multi-turn persuasion produces higher wiggle rates than most static challenges, flips are net-corrupting in aggregate but initially incorrect verdicts are more likely to flip conditionally, and jury majority strength is associated with item-level wiggle but is weaker under leave-one-judge-out analysis.
Critical strengths
-
The paper brings several previously separate stability probes into one evaluation protocol and applies them to the same judges, items, and grading scales. This centralization enables direct comparisons between mechanical variation, single-turn challenges, and multi-turn persistence across multiple judging domains.
-
The analysis goes beyond a single aggregate instability score. It reports per-task and per-model results, Binary versus Likert differences, correctness-conditioned flip rates, a representative-sample ablation, cross-task profile correlations, and a leave-one-judge-out analysis of jury majority strength.
Critical weaknesses
The Wiggle metric does not distinguish unjustified capitulation from justified belief revision. A judge that changes its verdict after a valid counterargument may be updating appropriately, while a change caused by a misleading argument may indicate susceptibility; both are counted as wiggles. L2 and L3 use model-generated arguments whose evidential quality is not independently controlled, and L6 permits the persuader to invent authorities, precedents, and rhetoric. The current results therefore support sensitivity to the tested persuasion prompts, but do not by themselves establish general epistemic instability.
Questions for authors
-
Please compare the Wiggle Framework with the closest existing consistency, sycophancy, persuasion, and position-sensitivity protocols at the level of estimands and experimental controls. Which reported findings are new because the probes are centralized, and which reproduce known effects?
-
Please independently assess the evidential quality of the L2-L6 arguments, or compare matched valid and misleading counterarguments while controlling the number of turns, approximate token budget, argument quantity, and persuader identity. Reporting whether verdict changes differ between valid and invalid challenges would clarify when wiggle indicates harmful capitulation versus appropriate updating.