acceptodds
Under review as a conference paper at ICLR 2027

Conformity Breaks Conformal Prediction

Abstract

Multi-agent LLM systems often rely on models that answer after seeing peer messages. Conformal prediction can help decide when an answer is reliable enough to act on, but calibration is usually done with the model answering alone. We identify and formalize a score-mechanism shift: the question stays the same, but peer messages change how the model scores its answers. We show that this one shift causes failures at three levels. First, peer pressure breaks overall coverage. Across four open-weight LLMs in our core experiments, coverage falls from a calibrated 90% to 74% under unanimous-wrong peer messages. More importantly, the average hides a much larger failure. On questions the model is already less sure about, coverage falls from 87% to 47%. An attacker can also identify such weak cases without knowing the correct answers, using only outputs from the target model or predictions from another model. Finally, the problem changes what the system actually does. A policy designed to defer uncertain cases instead acts on the wrong answer in 12% of cases where peer pressure flips the model's top choice. The coverage failure also holds across additional models and seven task types. Recalibrating on data with peer messages restores coverage, but makes the system defer more often. More broadly, conformal guarantees for multi-agent LLMs must account for the peer messages a model sees when it answers.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
conformal prediction, uncertainty quantification, distribution shift, multi-agent LLM systems, LLM safety, sycophancy, selective prediction

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_QauJOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper shows that the coverage guarantee of conformal prediction breaks when it is calibrated on solo answers and tested (evaluated) in multi-agent settings.

I do not find the problem and results interesting or valuable. It is well-known that the conformal prediction guarantees hold under exchangeability, unless some careful adjustments are made to tackle non-exchangeability. Moreover, the paper appears to be heavily AI-written.

Critical strengths

See summary.

Critical weaknesses

See summary.

Questions for authors

Why is the problem setting interesting and valuable? Why should the community care about this result?

2 comments

Dear Reviewer QauJ,

Thank you very much for your time and thoughtful feedback on our submission. We completely understand that you may have a busy schedule, so we just wanted to gently follow up as the discussion period draws to a close. Please let us know if our rebuttal has addressed your concerns or if there are any remaining questions we can help clarify.

Thank you for the response. I don't have any more questions.

Reviewer_WDrtOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies conformal prediction calibrated on solo LLM responses and evaluated after exposure to fabricated peer messages. Unanimous incorrect peers reduce marginal coverage and cause larger losses on selected subsets. Experiments span multiple models, tasks, and scoring rules. Calibration on pressured responses restores marginal coverage while increasing escalation.

Critical strengths

The experiments quantify substantial variation in susceptibility across models and show how aggregate coverage can conceal concentrated losses. Reporting both coverage and downstream wrong actions makes the consequences of reusing a solo-calibrated threshold concrete. The recalibration experiment also measures the associated cost in autonomous action.

Critical weaknesses

The central comparison omits the cited defense’s calibration protocol. Section 3.3 uses solo-calibrated thresholds, whereas Wang et al. (2026), Sections 3.4 and 4, calibrate debate outputs separately for each round and domain. Calibration on benign debate outputs followed by evaluation under corrupted peer messages is missing. Section 4.4 instead calibrates directly on attack-conditioned outputs. The reported losses therefore leave unresolved how much vulnerability persists when the defender follows the referenced method. This weakens the claim that the paper defeats a proposed multi-agent safety mechanism.

The label-free attack result stops at query selection. Section 2 uses the answer key to sample the advocated option from incorrect answers. Section 3.2 evaluates its observable selectors with these payloads. The reported 36–40 percentage-point losses thus establish effective targeting conditional on guaranteed-wrong messages. They leave the effectiveness of a complete attack without answer-key access unmeasured, including cases where the attacker inadvertently advocates the correct answer. This limits the practical reach of the attacker-without-answers result.

The formal analysis adds little beyond general score-distribution shift and misstates the finite-sample correction. Proposition 1 decomposes coverage differences and averages subgroup losses; it supplies no condition on peer messages that predicts the loss. The distinction from existing nonexchangeability analysis remains unclear (Barber et al., 2023). Moreover, Section 4.1 and Appendix E describe the estimated quantile as differing from the population quantile by 1/(n+1)1/(n+1). That term concerns marginal coverage discretization, not a bound on quantile estimation error. The connection between the population identity and the calibrated procedure is therefore incorrectly stated.

The verbalized-confidence extension is insufficiently specified to assess its strongest results. Appendix A describes an answer label and a 1–10 confidence rating, whereas LAC requires a score for every candidate label. Section 5.3 and Appendix I do not explain that conversion or specify a separate elicitation protocol. Wang et al.’s Section 3.2 elicits a full probability vector, so the citation does not resolve the gap. Table 14 also omits the proprietary-model event counts underlying the 96.9% headline. These omissions prevent checking whether the extension’s results depend on score construction or a small number of flipped cases.

Questions for authors

After calibration on benign debate outputs at the evaluated round, what coverage, automation rate, and overall wrong-singleton rate does the same attack produce? What coverage losses remain when both query selection and payload construction use no ground-truth labels, counting all attempted attacks? What finite-sample statement replaces the 1/(n+1)1/(n+1) quantile-error claim, and which conclusions about the random calibrated threshold follow from it? What exact per-label scoring rule produces Tables 13–14, and what numerator and denominator produce the 96.9% result?

Reviewer_oi6uOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper shows that iid assumption is experimentally not valid when there are peer messages in the context of LLMs. They show the average coverage falls from 90% to 74% when there are unanimous-wrong peer messages.

Critical strengths

None

Critical weaknesses
  • The paper relies on a single experimental observation, which is basically that the distribution shift can break the coverage guarantees, which is well known. The authors' contribution here is to show that LLM conformity can also cause this. Although the experimental observation is nice, it doesn't include the material for a standalone paper. I don't see any new understanding, theoretical contribution, or new experimental methodology. Besides, the paper itself is not that readable and is ambiguous sometimes (see def 1 and prop. 1)
Questions for authors

What's the purpose of Proposition 1? What's the implication of it in plain words?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.