- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies attribution under interference in multi-agent LLM markets, distinguishing the effect of changing an agent’s own deployment from spillovers caused by changes to its peers. Using eight open-weight models in repeated Bertrand pricing markets, the authors apply a two-stage design to estimate direct, indirect, and overall effects on a profit-based collusion index. They report that estimated own effects vary with peer exposure and that self-play collusion rankings do not necessarily predict mixed-market performance. The paper also evaluates a query-based composition surrogate built from elicited reaction functions, finding better agreement with model-identity spillover directions than with prompt-level spillovers or own effects.
Critical strengths
-
Interference-aware causal evaluation. The paper provides a well-motivated application of causal inference under interference to multi-agent LLM evaluation. More importantly, its experiments demonstrate that aggregate deployment comparisons can yield conclusions that differ substantially, sometimes even in sign, from the agent-level effects of interest. This raises a meaningful concern about the interpretation of existing multi-agent evaluation results.
-
Informative validation of query-based auditing. The paper evaluates reaction-function composition against randomized experimental estimates rather than relying solely on theoretical motivation or qualitative examples. The contrasting results for model-identity spillovers, prompt-level spillovers, and own effects provide useful evidence about both the potential and the limitations of this lower-cost auditing approach.
Critical weaknesses
-
Missing direct inference for effect heterogeneity: The abstract claims significantly different own effects across peer-exposure settings, but Table 5 reports tests of individual effects rather than the contrast DE(1) − DE(0). Separate significance tests and sign differences between point estimates do not establish the significance of this contrast, leaving the stronger claim insufficiently supported.
-
Limited predictive reliability of the composition surrogate. The surrogate substantially underestimates own effects and correctly predicts prompt-level spillover directions in only 6 of 11 cases. Although the authors acknowledge these limitations, they do not systematically investigate the sources of these errors or clarify under what conditions the surrogate can reliably support causal attribution. This weakens the practical value of the proposed query-based auditing approach.
Questions for authors
-
Can the authors directly estimate DE(1) − DE(0) and report confidence intervals and tests that account for dependence between the estimates? Beyond the existing correction for the 66 main-effect tests, please clarify how multiplicity would be handled for these interaction contrasts.
-
Understanding the surrogate's prediction failures. Could the authors investigate why the surrogate substantially underestimates own effects and performs poorly on prompt-level spillovers? In particular, an ablation examining history dependence, stochastic approximation, and mean-price scoring would help clarify the sources of these errors and the surrogate's reliability.
4 comments
Thank you again for your suggestions. We added direct interaction tests with dependence-aware inference and multiplicity correction, supported by 32 fresh seeds per condition, alongside matched-history ablations examining history, noise, and scoring.
With the November 18 (AoE) discussion deadline approaching, we would very much appreciate your thoughts on these results. If they address your concerns, we would be grateful if you could reconsider your assessment.
Thank you for the detailed response and additional experiments. The paired interaction tests and multiplicity corrections adequately address my main statistical concern, so I maintain my positive assessment of the paper.
Thank you for your engagement and positive assessment of our paper.
Thank you sincerely for the constructive questions about inference and the surrogate. They motivated direct interaction tests and matched-history ablations. In the revised PDF, purple marks changes addressing your review and red marks shared changes. Footnotes identify the relevant reviewer IDs.
1. Directly testing DE(1) - DE(0).
Your point about the missing comparison is well noted and taken. We now compute H = DE(1) - DE(0) within each seed block before estimating its mean and uncertainty. This retains dependence between the effects and their shared mixed-market outcomes.
With 32 new seed blocks per condition, mean H is -0.345 (95% CI [-0.531, -0.159]) for Qwen3-8B reasoning and -1.265 [-1.571, -0.960] for replacement by Mistral-24B. Holm-adjusted p-values are 0.000675 and 3.12e-9. Student-ized bootstrap and leave-one-seed-out checks support both contrasts (Appendix L, Tables 13 and 14). The claim rests on these direct differences, rather than separate main-effect significance or point-estimate sign changes.
2. Accounting for multiplicity.
The original 22 interaction contrasts receive Holm correction separately from the original 66 main-effect tests. Independent replication has a two-contrast Holm family; Cournot and five-agent H tests form another two-test family; four paired history contrasts receive Holm-4 correction. Table 11 lists these families. This provides error control within each family, not jointly across all analyses. Please note that surrogate-error comparisons below (point 3) are exploratory.
3. What explains the surrogate error?
We ran matched Qwen3-8B markets for reasoning and ICL with visible histories of 1, 5, and 10 rounds, using 16 fresh seed blocks. Maps cross transient/persistent elicitation, residual noise OFF/ON, and profit at mean prices versus mean per-round profit. Each prediction is compared with its matched-history market target.
For reasoning, switching to mean per-round profit changes average own-effect MAE by -0.0054 (exploratory 95% CI [-0.0109, -0.0014]). Adding noise changes it by +0.0029 [-0.0146, 0.0245]; persistent versus transient elicitation changes it by -0.0258 [-0.0458, 0.0051]. Scoring explains a modest part of the error. Noise and persistent elicitation, however, do not establish a general remedy.
Shortening history from 10 previous rounds to 1 reduces reasoning own-effect MAE by 0.2969 [0.1561, 0.3546]. The ICL reduction is 0.0142 [-0.1576, 0.1597]. There is a catch: history also changes the target. For ICL, H at one round minus H at ten rounds is -0.505 [-0.752, -0.259], with Holm-adjusted p = 0.0022. Cumulative profit remains visible, so one-round history does not remove all memory.
Appendix N, Tables 18 through 22, reports the market effects, map configurations, and average error changes. Error intervals jointly resample 20 snapshot clusters and 16 seed blocks while preserving shared controls.
These checks identify conditions where the approximation improves without isolating a single failure mechanism. A cheaper audit still needs validation for the intervention and history regime being assessed. Your suggestions helped us turn that very crucial caution into a concrete observation of (i) when the surrogate succeeds and (ii) where it falls short if it does.