- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes a framework that explains the differences between model behaviors and predicts their agreements on new samples given the explanation. Specifically, it designs specific prompts to generate explanations given a subset of samples where a pair of models agree or disagree with each other, and fine-tunes a LLM/VLM with LORA to predict the agreement labels based on explanations. Results show that the proposed method achieves higher accuracy than generic LLMs, and the generated explanations generalize for both new models and humans.
Critical strengths
(1) Understanding the key differences between models is an interesting and important problem.
(2) The paper takes into account a variety of tasks for evaluations, and also carries out human evaluations to validate the ideas.
(3) Extensive ablation study is provided to facilitate evaluation of contribution for individual components.
Critical weaknesses
(1) The technical contribution of the paper is rather limited. It essentially prompts off-the-shelf LLMs to generate explanations, and fine-tunes another LLM/VLM to perform the binary classification task. Except for the definition of labels, most method design and evaluation metrics are identical to OPeX, e.g., OPeX also optimizes with GRPO and LORA.
(2) While Table 1 shows that the proposed method is better than generic LLMs, it is unclear if the advantage comes from the proposed method or simply additional training. In addition, the method is only marginally better than GPT-4o, and the evaluation sample size is very small (60 samples). For a fair comparison, it would be reasonable to compare other methods that involve optimization on the training data or at least incorporating training data in the context.
(3) Related to the previous comment. The models being compared are inconsistent across different evaluation settings. For instance, GPT-4o is the strongest compared method in text-based tasks, but missing for evaluation on multi-modal tasks.
(4) The paper highlights explanations of model differences as its key contribution. Nevertheless, looking at Table 5, it appears that the proposed method is “memorizing” the detailed differences of individual training samples, and does not generate high-quality explanations as GPT-4o. This again raises questions about overfitting, i.e., whether the improvement in binary classification comes from additional training.
(5) The human evaluation only considers the task where the proposed method shows the largest improvement over GPT-4o (Table 1). It would be reasonable to conduct more comprehensive evaluations over different task settings.
(6) Alongside explaining the samples where a pair of models agree or disagree, it would be more interesting to systematically explain the characteristics of each model. For instance, instead of only feeding 40 training samples, how would the method generate explanations about the defining characteristics of a classifier giving a large dataset (e.g., 10K samples)?
Questions for authors
(1) Please justify the key technical novelty over OPeX, in addition to the difference of problem setup.
(2) How do you disentangle the performance gain from the proposed method and access to additional training data?
(3) Please report the performance of GPT-4o on multi-modal tasks.
(4) Looking at Table 1 and Table 2, it appears that the QWEN family has close-to-chance performance. Any idea why? This is a potential problem because the proposed method also chooses QWEN models as the default backbones.
(5) Please elaborate on the quality of explanation generated by the proposed method (refer to the weaknesses section).
2 comments
I thank the authors for the additional results and clarification. While it addresses a few concerned I raised in the initial review, I am still not fully convinced about the technical contribution of the paper. Specifically,
(1) I understand that the paper is not finetuning a LLM. Nevertheless, the proposed policy model is essentially a finetuned version of Qwen2.5-VL-32B-Instruct. The training algorithm and model design largely follows previous studies (e.g., OPex), with the major difference being the definition of labels. While the authors highlight a few subtle design choices and experiment setups, I don't think it bring sufficient technical novelty.
(2) I am aware that the paper uses performance averaged across multiple repeated runs. However, my point is that each run only considers a limited set of samples (40-60), which can lead to large variance in performance. For a more comprehensive evaluation, running on larger set is more preferrable.
(3) The rebuttal argues that the generated explanations focus on broader coverage. However, given the examples in the paper, I think this choice considerably deteriorate the interpretability of the explanations, as it reads like reciting every detailed element in the scene.
(4) The rebuttal does not address my concerns on human evaluation and the choice of backbone. For instance, if QWEN models have the worst performance in generating reasonable explanations, why should we use it as the backbone for policy model and observer?
As a result, I am leaning toward keeping my original score.
Response to Reviewer B8dL
We thank the reviewer for their review. We first clarify a fundamental misunderstanding regarding our training pipeline.
Clarification of SEAM's RL Fine-Tuning: SEAM is an explanation generation model trained with RL (GRPO) to capture behavioral differences between two classifiers. Its reward is computed by a separate Observer LLM, which predicts from the explanation whether the classifiers will converge or diverge on an input. RL fine-tuning updates SEAM only; the Observer is a frozen reward evaluator, not the model being trained.
W1/Q1: Technical Contribution and Novelty over OPeX: We do not fine-tune an LLM/VLM for binary classification; we fine-tune SEAM, the explanation generation policy, to improve the contrastive diagnosticity of its explanations. Beyond the problem setup, SEAM differs from OPeX through (1) a pairwise reward computed from agreement between two classifiers rather than a single model's predictions, (2) multimodal model comparison (SEAM-VL), and (3) explanations conditioned on synthetic samples (Table 6). Optimizing for individual model behavior does not transfer to the pairwise problem: OPeX evaluated directly on agreement labels obtains 0.62 recoverability and 0.58 simulatability, compared with 0.85 and 0.75 for SEAM.
W2/Q2: Evaluation and Baselines: Our evaluation is designed to rule out overfitting to training data: evaluation tasks are entirely disjoint from training tasks, the classifiers being compared are new instances never seen during training, and simulatability is measured on held-out samples not used to generate the explanation. The comparison is also controlled: all methods receive the same 40 conditioning samples in context and are evaluated by the same frozen Observer on the same held-out samples. Our baselines span:
The evaluation is not limited to 60 samples: 60 is the held-out set size for a single run, and each reported score for each task is averaged over 80 runs with freshly sampled conditioning and held-out sets. SEAM's improvement over GPT-4o is statistically significant over these runs and grows to 18.6 percentage points with human participants.
W3/Q3: GPT-4o Baseline for Multimodal Classifiers: Agreed. Please see below. We will include these results in the paper.
W4/Q5: Explanation Quality and Generalization Beyond Training Examples: The goal is not verbose explanations, but precise, actionable patterns that generalize beyond the conditioning examples. Memorization yields high recoverability but low simulatability, as with random forests (0.96 vs. 0.55; Table 11); SEAM drops only from 0.85 to 0.75. SEAM produces more rules, but its maximum number of conjuncts per rule (3.47) matches Llama-3.3-70B (3.43), reflecting broader coverage rather than instance-specific rules (Table 13). Table 5 shows that GPT-4o's concise explanations often miss diagnostic distinctions, though more detail does not always help, as on Car Evaluation.
W5: Human Evaluation: Human evaluations across multiple tasks and modalities are resource-intensive, limiting the feasibility of large-scale studies. We therefore focus on the Rice task, which has a moderate number of features (7), avoiding excessive cognitive load on participants.
W6: Scope: We agree this is an interesting but distinct problem: characterizing a single classifier over a large dataset, which prior work addresses (OPeX, MaNtLE, LIME, Anchors). Our work focuses on explaining behavioral differences between classifier pairs. For large datasets, informative samples can be selected with sample-discovery methods such as SLED, on whose samples SEAM can be conditioned (Table 6).
Q4: Choice of Backbone: The untuned Qwen models are near chance because they are not optimized for contrastive diagnosticity. RL fine-tuning raises the same backbone from 0.51 to 0.75 simulatability, showing that the gains come from our method rather than the backbone. This holds across backbones: SEAM-Llama improves Llama-3.1-8B from 0.49 to 0.67 (Figure 3a). We chose Qwen2.5-VL-32B-Instruct for its model capacity and multimodal capabilities.
We value your review, which has improved the clarity of our paper. Given these clarifications, we invite you to reconsider your evaluation scores.