acceptodds
Under review as a conference paper at ICLR 2027

Model Ecosystem Pruning via Joint Behavioral Coverage

Abstract

An individual redundancy audit asks whether a model can be replaced by its peers. It leaves open whether several such models can be removed together, because their replacements may depend on one another. We study model ecosystem pruning: finding a common retained subset and convex replacement weights that reproduce every original endpoint's declared response within a tolerance. Joint behavioral coverage preserves a family of responses rather than the performance of one aggregate predictor. We characterize the fixed convex aggregations it protects and derive a dependency-aware composition condition that bounds error accumulation along replacement chains. Experiments use 20 open-weight LLM judges on 3,724 comparison pairs under seven prompt configurations. At a predeclared tolerance of 0.20, three judges pass individual removal tests, but their joint removal fails in all five data splits. Separate exact analyses require all 20 judges at tolerances up to 0.12 for the fitting objective. Coverage-based selection improves reconstruction over accuracy-based selection, and backward-initialized 2-swap reaches the empirical optimum in 59 of 60 small search instances. Under benchmark shift, refitting weights recovers most of the gap to selecting a new subset. Joint substitutability, rather than individual redundancy alone, determines which ecosystem removals preserve the declared behavior.

Primary area:
datasets and benchmarks
Keywords:
model ecosystem pruning, LLM judges, joint redundancy, convex reconstruction, subset selection, total variation, panel compression, LLM evaluation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_msmmOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how to remove redundant models from a collection while preserving the behavior of the original models. They also introduce joint behavioral coverage, which measures how well a shared subset of retained models can reproduce the outputs of all original models. In the paper, author also present a theoretical analysis of dependencies between model replacements.

Experiments are performed on 20 open weighted LLM judges using 3724 comparison pairs equipped with seven prompt configurations. In conclusion based on their result, at a tolerance of 0.2, three judges can be removed individually; backward selection outperforms several baselines, while 2-swap achieves the best result in 59 of 60 cases.

Critical strengths

1). This paper proposes a clear formulation of joint model redundancy. Authors discuss about how multiple models can be removed together while preserving their original outputs. The comparison between individual and joint removal in Figure. 2 provides useful evidence for the proposed formulation. The experiments also compare multiple selection methods and use exact search on smaller cases to evaluate the quality of their method.

Critical weaknesses

1). Limited evidence for dependency analysis. The paper shows that removing three redundant judges can increase the reconstruction error to go beyond the tolerance of 0.2. But the joint error is only within the range from 0.208 to 0.216 across five splits (figure 2a), so the margin is fairly small. Also, authors do not test the dependency analysis on the 20 judge panel.

2). The conclusions depend closely on the evaluation setting, lack proof for generalization. The experiments use only one set of 20 LLM judges and fixed convex combinations for reconstruction. At a tolerance of 0.20, the method still needs to retain 18 judges. With the additional experiments, it still remains unclear how well the approach generalizes to other settings.

Questions for authors

1). For figure 2a, can authors provide the individual removal errors for the three judges and compare them with the joint error? and are the differences statistically significant?

2). Can authors further explain how theorem 2 relates to the joint-removal failures observed in Section 5.1, especially the dependencies among the three judges? This would help determine if the theory explains the experimental results.

3). How sensitive is the result of retaining 18 out of 20 judges at a tolerance of 0.20 to the reconstruction method? Can authors discuss the effect of using a calibrated / probability-preserving alternative?

Reviewer_KdfuOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper looks at subset selection. Specifically, the paper studies how to prune a collection of NN models (where each outputs a probability distribution) down to a minimal subset SS of size kk, such that the output distributions of any removed model (i.e., i∉Si \notin S) can be (approximately) reconstructed via a convex combination of the kept models. So, πi^(x,c)=∑j∈SWijpj(x,c)\hat{\pi_i}(x, c) = \sum_{j \in S} W_{ij} p_j (x, c), where WW is row stochastic supported on columsn SS, xx is a prompt, and cc is a context. One of the core parts of the paper is to analyze the "coverage" error of the subset SS, denoted by EP(S)E_P(S), which refers to the worst-case expected TVD between a true model distribtion and its best reconstruction using SS, across all models i∈{1,...,N}i \in \{1, ..., N\} and contexts cc, for x∼Pcx \sim P_c. The paper empirically explores a number of selection strategies, applied to LLM-as-a-Judge models.

Critical strengths

The problem setting / motivation is reasonable. In general, running large panels of LLM judges can quickly get expensive (and there is a lot of redundancy), so being able to reconstruct / approximate a full panel from a smaller subset is quite practical, and a nice formulation. The empirical part of the paper also seems sensible.

Critical weaknesses
  1. The paper is very difficult to read. Simple ideas are described with dense, repetitive, jargon-heavy and sometimes incomprehensible prose. Heavy revision is necessary.

  2. While the problem setting and formulation are reasonable, they (or the proposed techniques) are not particularly new. The subset search in Section 4 is closely related to standard convex-combination subset selection (see, e.g., the cited works of Vinué et. al and Sartipizadeh & Vincent). In the related work, the authors state the difference between this paper and the prior literature as connecting this "to executable joint removal, dependency-aware composition, and cross-context endpoint risk", but what exactly that means is not that clear to me.

  3. The theoretical results do not lend much to the paper, in my opinion. They are fairly elementary. Theorem 2 also never appears to actually be made use of in the actual algorithm that the paper uses, since in Section 4 WW is iteratively refit directly on SS via linear programming.

  4. The empirical results are not that strong. Specifically, it seems to me that Section 5 shows limited practical value for pruning this judge collection (keeping TV error ≤ 0.12 requires retaining all 20 judges, 18/20 are required for TV error ≤ 0.2, pruning down to k = 10 flips 25% of verdicts for the removed models per L346-351, etc), and simple baselines do nearly as well as any of the methods proposed.

Questions for authors

What should the main takeaway of Theorem 2 be? Are there any experiments that use M=(I−A)−1BM=(I−A)^{-1}B?

Given that pivoted QR matches backward coverage selection at k=10k = 10, and forward coverage selection is about the same as pivoted QR on average, what does the coverage-specific search add?

Reviewer_W23hOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

Formalizes joint behavioral coverage: retaining a shared subset of models that convexly reconstructs every original endpoint within a tolerance. Derives a dependency-aware composition bound showing how errors accumulate when substitutes themselves depend on removed models. Evaluates 20 LLM judges on 3,724 comparison pairs, showing that individual removability need not imply joint removability.

Critical strengths
  1. Distinguishing the preservation of individual model interfaces from aggregate prediction accuracy clarifies an underexplored pruning objective.
  2. The explicit common-support formulation and composition bound provide a rigorous explanation of replacement dependencies (Theorem 2).
  3. Exact finite-instance searches, diverse selection baselines, and cross-benchmark weight-refitting analyses strengthen the evaluation (Sections 5.2–5.4).
Critical weaknesses
  1. Empirical evidence comes from one 20-judge ecosystem, limiting claims about broader model populations or other output interfaces.
  2. Strict fidelity yields little compression: all 20 judges are needed at fitting tolerance ≤0.12 and 18 at tolerance 0.20 (Figure 2b).
  3. Fixed convex combinations are restrictive, while the supplied artifact bundle does not fully reproduce original inference or per-item outputs (Reproducibility Statement).
Questions for authors
  1. Does the same joint-versus-individual removal gap persist across other independently assembled ecosystems, beyond this judge panel?
  2. How does the pruning frontier change under a more expressive yet tractable context-dependent/nonlinear replacement map?
  3. Can the authors provide complete response caches and real inference-cost savings, accounting for different endpoint latencies and prices?

Comments on the paper

1 comment

Thanks for the response and the additional experiments. The comparison between individual and joint removal is clearer now, and the new experiments address my main concerns. I will raise my score from 2 to 3. Please include the new results and explanations in the revised paper. It would also be helpful to provide the input-dependent model’s parameterization and training details, along with convergence diagnostics for the fixed-convex baseline.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.