acceptodds
Under review as a conference paper at ICLR 2027

Auditing Behavioral Coverage in Model Ecosystems

Abstract

A target model can differ from each of its peers yet be closely approximated by their fixed combination. A similarly small reconstruction error can also depend almost entirely on one peer. We study these distinct sources of behavioral coverage through matched model responses. The Peer-Inexpressible Residual (\PIER) measures what remains after a fixed convex peer fit, and \DISCO estimates the fit and its residual on separate samples. Comparing the group with a selected single peer, and refitting after peer removal, identifies who supplies the coverage. In an eight-model MMLU-Pro audit, our primary response is the reference-answer probability averaged over cyclic option rotations. Six targets benefit from collective approximation under two stress families and matched squared- and absolute-error fitting. Qwen2.5 instead obtains essentially all available approximation quality from its reasoning-tuned sibling; the reverse direction gains a further – from other peers. Mistral's group fit reduces error by –. A separate complete-option-distribution analysis retains the Qwen directionality and Mistral's group advantage. Matched stress increases residuals for some targets and decreases them for others. In vision, context changes both the supporting peer combination and the concentration of residuals on individual images. These results characterize how accurately a collection reproduces a target, which peers supply that approximation, and which inputs account for the remaining gaps.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
quasi-experimental design, model behavioral coverage, model ecosystems

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_XWVGOpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
1
Presentation
1
Soundness
3
Read review
Paper summary

The paper aims to explore how much of a model behaviour can be approximated by behaviour of its peers, specifically via convex-peer fit. Peer removal allows quantitative measure of which peer contributes the most.

Critical strengths

None

Critical weaknesses
  • To claim approximation of model behaviour based on a single benchmark, MMLU Pro, is too weak a claim.
  • I don't see why this is an interesting problem to begin with. What do we expect to do with the "coverage"? Downstream, the only thing I can imagine is perhaps how to best construct an ensemble of models, but there is no clear evidence that the given analysis will help with that. I don't see how this is related to feature attribution, which is listed as related work.
  • I admit this is outside my area, but even then some parts were very difficult to read and understand.
Questions for authors
  • What can we do better with this information? And do we have any evidence for that?
Reviewer_9gRkOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes a framework for auditing behavioral coverage in model ecosystems, asking whether the response behavior of a target model can be reconstructed by a fixed convex combination of peer models. It introduces the Peer-Inexpressible Residual (PIER) to measure the remaining reconstruction gap, together with group-gain and peer-removal analyses to distinguish collective coverage from dependence on individual peers. The framework is evaluated primarily on instruction- and reasoning-tuned language models on MMLU-Pro, with additional analyses under input stress and in vision settings.

Critical strengths
  • The paper highlights an important distinction between pairwise model similarity and collective behavioral coverage: a target may be poorly approximated by every individual peer while still being well reconstructed by their combination. The directional Qwen/General Reasoner results provide a clear example of this distinction.
  • The paper evaluates coverage using held-out fitting/evaluation splits, multiple fitting losses, peer-removal analyses, stress interventions, and alternative response representations. This provides a relatively thorough characterization of both the amount and source of behavioral coverage.
Critical weaknesses
  • The main framework is built from a fixed convex reconstruction of peer responses, held-out residuals, comparison against a selected single peer, and leave-one-peer-out refitting. While the resulting auditing perspective is useful, the underlying methodology appears relatively straightforward and closely related to existing convex fitting / synthetic-control ideas.
  • The interpretation of PIER as “peer-inexpressible” behavior is relative to a fixed convex combination with input-independent, non-negative weights. Behaviors that cannot be represented by this restricted class may still be predictable from peers through signed, nonlinear, or input-dependent combinations. Therefore, it is not fully clear to what extent a large residual reflects genuinely distinctive target behavior versus limitations of the chosen approximation class. The paper itself notes that signed or input-dependent combinations would measure different forms of expressibility.
Questions for authors
  1. How should a large PIER be interpreted when the target may be predictable from peers using a richer reconstruction class, such as an input-dependent or nonlinear combination? To what extent is “behavioral distinctiveness” intended to be specific to the fixed-convex class rather than an intrinsic property of the target model?
  2. Could the authors clarify what they view as the main methodological novelty beyond applying fixed convex reconstruction, held-out evaluation, and peer-removal analysis to model-response traces? In particular, which aspects of the framework are intended to provide a broader contribution beyond the specific auditing formulation?
  3. How should the peer-removal effect be interpreted when several peers are highly redundant or substitutable? In such cases, removing any one peer may have little effect because the remaining peers can compensate, even though that peer still participates in an important source of coverage. Does the current framework capture such higher-order or coalition-level dependence, or would group-removal analyses be needed?
Reviewer_XfqAOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper attempts to answer the question of whether some aggregate of a family of models' responses can match the responses from a specific target model. It aims to answer how accurately a set of models can do this, as well as which peer models are contributing.

Critical strengths

The core idea is really interesting, understanding whether some model responses can be viewed as combinations of peers is really fascinating in terms of whether model behaviour covers a particiular behaviour space. I believe this work to be novel and interesting enough to warrant further distribution.

The experimental design is also solid. Featuring a robust set of experiments that inform one another. The results themselves are interesting and provide insight into model behaviour (I'm particularly interested in the asymmetry results and the superiority of recreation by a mix of peers rather than a single peer).

Critical weaknesses

Some equations are missing definitions for what certain aspects are. E.g., in Equation 1, what is the sub-index j, what is the function f, why introduce z and not use it? This pattern continues throughout the paper and impairs readability.

The paper seemingly invents terms where it's not needed. For example "DISCO" is undefined as an acronym but is just a constrained least squares error. Similarly PIER is just a residual of the fit. Reinventing terms for common operations and equations again hurts readability.

The largest model in the paper is 14B parameters, which is quite small. A few larger models in the 30Bish range would be interesting to see how these results scale.

Questions for authors

Can you add more explanations for some of the equations, and generally assume definitions in the paper are self contained?

Why introduce DISCO and PIER as new names when they could arguably be described with more conventional terms.

Would you be open to adding a roster of larger models?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.