- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies how to remove redundant models from a collection while preserving the behavior of the original models. They also introduce joint behavioral coverage, which measures how well a shared subset of retained models can reproduce the outputs of all original models. In the paper, author also present a theoretical analysis of dependencies between model replacements.
Experiments are performed on 20 open weighted LLM judges using 3724 comparison pairs equipped with seven prompt configurations. In conclusion based on their result, at a tolerance of 0.2, three judges can be removed individually; backward selection outperforms several baselines, while 2-swap achieves the best result in 59 of 60 cases.
Critical strengths
1). This paper proposes a clear formulation of joint model redundancy. Authors discuss about how multiple models can be removed together while preserving their original outputs. The comparison between individual and joint removal in Figure. 2 provides useful evidence for the proposed formulation. The experiments also compare multiple selection methods and use exact search on smaller cases to evaluate the quality of their method.
Critical weaknesses
1). Limited evidence for dependency analysis. The paper shows that removing three redundant judges can increase the reconstruction error to go beyond the tolerance of 0.2. But the joint error is only within the range from 0.208 to 0.216 across five splits (figure 2a), so the margin is fairly small. Also, authors do not test the dependency analysis on the 20 judge panel.
2). The conclusions depend closely on the evaluation setting, lack proof for generalization. The experiments use only one set of 20 LLM judges and fixed convex combinations for reconstruction. At a tolerance of 0.20, the method still needs to retain 18 judges. With the additional experiments, it still remains unclear how well the approach generalizes to other settings.
Questions for authors
1). For figure 2a, can authors provide the individual removal errors for the three judges and compare them with the joint error? and are the differences statistically significant?
2). Can authors further explain how theorem 2 relates to the joint-removal failures observed in Section 5.1, especially the dependencies among the three judges? This would help determine if the theory explains the experimental results.
3). How sensitive is the result of retaining 18 out of 20 judges at a tolerance of 0.20 to the reconstruction method? Can authors discuss the effect of using a calibrated / probability-preserving alternative?
Thanks for the response and the additional experiments. The comparison between individual and joint removal is clearer now, and the new experiments address my main concerns. I will raise my score from 2 to 3. Please include the new results and explanations in the revised paper. It would also be helpful to provide the input-dependent model’s parameterization and training details, along with convergence diagnostics for the fixed-convex baseline.