acceptodds
Under review as a conference paper at ICLR 2027

Measure Before They Debate: Auditing LLM Response Composition

Abstract

Response probes measure how an LLM reacts to individual inputs, but using those measurements to predict responses to joint inputs requires a separate composition assumption. We study this assumption through a finite-input audit that holds the interface fixed, excludes combinations from calibration, and separates prediction error from response noise. In a joint study of responses, summed single-sender secants fall below common-input secants by and ; both approximate simultaneous intervals exclude zero. One configuration also answers all nonzero-target arithmetic queries correctly, although the prespecified global screen fails. A prospective test on eight new, separately calibrated propositions replaces numerical cards with prose comments. Across responses, of composition contrasts have approximate simultaneous intervals below zero; five remain unresolved. Addition retains the lowest observed overall held-out error, while averaging measured increments improves mixed-input prediction. Further tests show how calibration support redistributes error and how remeasuring archived replies changes a gain ordering. The audit distinguishes failed composition identities from useful predictions and makes sensitivity to input families and readouts explicit. The evidence covers numerical cards and controlled prose substitutions; it does not establish internal mechanisms or performance in unrestricted discussion.

Primary area:
foundation or frontier models, including LLMs
Keywords:
large language models, multi-agent systems, measurement validity, response composition, evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_RwYPOpenReview ↗
Rating
1
Confidence
1
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

Judging by Appendix C.4 it is something about measuring agreement in LLMs and how its initial opinion is overwritten by other participant's arguments, on a dataset of 12 debate statements. What the paper seems to be concerned with is that each other participant's argument does not change the opinion in isolation but that there are interaction effects that need to be taken into account when modeling.

Critical strengths

None.

Critical weaknesses

After 1.5 hours of trying, I did not understand the paper. I signaled this to the AC/SAC and was asked to continue reviewing. The paper in its entirety is unreadable. I invite anyone to read any paragraph to convince themselves.

  • Terms are used that are never defined
  • Sentences have no logical structure
  • The method section and its math lead nowhere
  • "Algorithm 1" is vague, full of undefined terms, defying the purpose of formalizing an algorithm
  • To cite the first sentence of the conclusion: "Evaluate response coefficients against their intended predictions: fix the display and readout, reserve combinations, propagate uncertainty, and report input-family and aggregate losses.".

Frankly, it reads like AI generated, but as long as ICLR has no clause for rejecting due to AI slop, I propose rejecting due to subpar presentation quality.

The appended codebase is also not understandable. Some 37 iterations of code in subfolders.

Even all this left aside, a dataset of 12 scenarios is just not enough.

Questions for authors

None.

Reviewer_bxvbOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper investigates whether the response of an LLM to joint inputs can be accurately predicted by summing its responses to individual, isolated inputs. The study is motivated by the fact that multi-agent systems often rely on this composition property for reasoning and coordination, yet it has not been empirically verified. The authors conducted a finite-input audit using numerical arithmetic tasks and controlled prose substitutions to test if isolated-response secants could predict joint-input behavior. The results show that composition identities frequently fail, as summed single-sender effects often fall below common-input secants.

Critical strengths
  • [S1] The paper studies a practical problem: whether an LLM’s response to a joint input can be predicted by additively composing its responses to each input to further uncover the effect of each message in a multi-agent systems.
Critical weaknesses
  • [W1] The clarity of the paper is significantly hindered by inconsistent or underutilized notation and poor presentation, making it really hard to understand the statement of this work. Specifically, the mathematical notations introduced in Section 3 (eg Δ\Delta) are rarely utilized in the subsequent analysis, making it difficult to follow the formal formulation of the experiments and evaluation metrics. Furthermore, several experimental details remain poorly defined; for instance, the specific configurations mentioned in lines 209–210 (e.g., C01/C03/S1/forward/uniform) and the labels L4, L10, and L11 in line 381 are not clearly explained. The experimented LLMs are never properly introduce in the experimental setups (eg the version of Gemini adopted in the experiments are not mentioned). The authors need to substaintially elaborate these details to ensure reproducibility and comprehensiveness.
    • The design intuition and motivation for certain experimental choices lack sufficient justification. For example, in the Related Work section (line 102), the authors do not clearly explain the rationale for extending existing debate trace and response probe research into the specific domains mentioned. A more robust argument for why these domains were chosen would strengthen the paper's overall contribution.
  • [W2] The core scenario (prompting an LLM with different arguments and weights and expecting it to organize and respond according to the correct weighted sum) may not reflect a practical multi-agent debate setting. A more realistic dataset or scenario for multi-agent debate would significantly strengthen and justify the motivation for measuring this behavior, since the current setup risks testing an artificial composition rule rather than a property that matters in actual interaction.
Questions for authors

Refer to the weaknesses section

Reviewer_DnwzOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies whether per-input response measurements of an LLM may be use to predict its response to joint inputs in multi-agent dynamics. The authors formalize this to a finite-input audit by fixing the the task, model, sampling policy, display, and numerical readout. Experiments consists of four studies, mostly on DeepSeek-V4.1-Flash. Generally the paper argues that changing peers one at a time cannot predict what happens when several peers are changed together.

Critical strengths

The paper turns a commonly assumed but rarely checked property, that single-input probe responses can be added to predict joint-input responses, into a precise, falsifiable statement (Eq. 1, Prop. 1). The test is carried out with four different experiments: (i) a joint audit on numerical stance cards with concurrent instructed-arithmetic controls in the same cohort (Sec. 4.1); (ii) a prospective prose test on eight pre-fixed propositions with scripted peer comments and held-out combinations (Sec. 4.2); (iii) calibration-support audits on Qwen and DeepSeek cohorts that vary the calibration set from 1 to 15 cells under a fixed layout (Sec. 4.3); and (iv) a readout audit that re-scores 216 archived replies with an LLM judge instead of self-reported tags (Sec. 4.4). Taken together, they conclude that the additive identity fails for both numerical and prose inputs even when instructed arithmetic is perfect, and yet that plain addition remains the lowest-error fixed predictor on held-out inputs.

Critical weaknesses
  1. The interface is far from any real multi-agent system: peer inputs are experimenter-written numbers or five near-identical comments differing only in a closing sentence, with fixed order, a neutral prior, repeated arguments, and a single round. At the same time, both headline studies rest on a single DeepSeek-V4.1-Flash. Qwen3-8B appears only in the calibration audit, where its arithmetic control failed. The authors acknowledge these limits (Sec. 5), but with one model and one artificial interface it is hard to know whether the composition failure, its magnitude, or even its direction would carry over to other models or to discussions where peers differ in argument content and confidence.

  2. In the numerical audit the group response exceeds the sum of single-sender responses (Table 4), whereas in the prose test support substitutions are sub-additive while opposition substitutions are super-additive, with 5 of 8 support contrasts unresolved (Table 8). The text states that the two designs "need not produce the same directional mechanism" (Sec. 3.1, App. C.2). Could the authors clarify whether they consider these the same phenomenon, and if so, what accounts for the sign asymmetry in the prose test? As written, the reader cannot tell whether the two studies reinforce each other or are measuring different effects.

  3. The main text is difficult to follow: it is dense and key design details are deferred to a 34-page appendix, and several claims point to an external "archived study report" that reviewers cannot evaluate.

Questions for authors

see weakness.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.