- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
- The paper presents MEA, a framework in which an LLM agent is trained with RL (GRPO) to explain the predictions of classifiers across tabular, text, and vision data.
- The framework uses two agents: a Proposer that selects XAI tools such as LIME, SHAP, and Grad-CAM, and an Actor that runs these tools and produces an explanation identifying the relevant input elements. These elements are then perturbed, and the resulting change in the model’s behavior is scored with faithfulness metrics that also serve as the training reward.
Critical strengths
- The authors study generalization along several axes. Beyond held-out instances of the training datasets, they evaluate on unseen dataset-architecture pairs. They also test held-out question types and a concept-level question that lies entirely outside the training taxonomy. This goes well beyond a standard in-distribution evaluation.
- The base agent (the same Proposer–Actor harness, prompts, and tools, without GRPO training) serves as a baseline. This isolates the effect of training from the effect of the agentic harness itself. The comparisons against frontier LLMs, prompting strategies (CoT, ReAct, ToT), and an inference-time critic further help separate the contribution of RL from that of prompting.
- The evaluation is broad and seems fairly honest. Their analysis covers three modalities, six datasets, multiple classifier architectures per dataset, and ten question types spanning feature attribution, counterfactuals, and spurious features. It also compares against both general-purpose attribution methods and specialized counterfactual generators. The authors also report negative results, such as catastrophic forgetting under sequential training and cases where the base agent outperforms the trained model. This makes the positive results more credible.
Critical weaknesses
- The authors present their framework as producing natural-language explanations grounded in model behavior, but there is no mechanism to measure this. The Actor produces two separate outputs: a structured output identifying the relevant input elements, and a free-text explanation. All faithfulness metrics and the RL reward are computed from the structured output alone, and the text is never used. The authors should either measure agreement between the text and the structured output or revise their claims about faithful natural-language explanations.
- Faithfulness is evaluated with a single perturbation per instance. Standard evaluations typically aggregate over a range of perturbations (e.g., deletion/insertion curves over varying numbers of elements and multiple baselines), because single-point scores can be sensitive to these choices. It is therefore unclear whether the gains hold beyond this specific setup.
- The XAI toolkit available to the agents overlaps substantially with the evaluation metrics. For example, the sensitivity-analysis tool computes essentially the same leave-one-out probability drop that the Q1 metric measures. The reported gains may therefore partly reflect the agent learning to pick the tool that already approximates the metric, rather than a broader ability to explain.
- The question type determines both the Proposer’s instruction and the evaluation metric, creating an effective one-to-one mapping between them. The Proposer is also given the classifier, the prediction, and the input, and can infer the dataset from feature names, feature values, or the model name. It is therefore possible that the model memorizes which tool to use for each (dataset, classifier, question) tuple and applies this lookup table at test time.
- The paper claims that existing conversational explanation frameworks are unfaithful, but it never evaluates them or compares against them. The agentic baselines are only generic prompting strategies.
- The authors claim that “no existing work jointly optimizes a multi-agent pipeline end-to-end with RL across diverse modalities and question types.” This may be true with these qualifiers, but the literature review should be expanded to cover training-based approaches to faithful explanations, such as Malladi and Srivastava (2026); Cheah, Perez-Ortiz, Siegel et al. (2026); Hase and Potts (2026); Pres, Ruis, Ghebreselassie et al. (2026)
Questions for authors
- The paper doesn’t report how often each XAI tool is selected, or how much the choice varies across instances of the same dataset and question type. Did the authors analyze the frequency of each tool choice? I am especially interested in the distribution of tool choices conditioned on each (dataset, classifier, question) tuple. Do the authors observe any mode collapse in this distribution after GRPO training?
- The tool penalty in the reward function incentivizes the model to use fewer tools. I understand that its purpose is to discourage the Proposer from choosing every tool on every instance. However, could this penalty also lead to undesirable behavior? For example, could it push the Proposer to choose the same set of tools for every instance of a dataset?
- Did the authors analyze whether GRPO changes the Proposer’s behavior or the Actor’s behavior more? One way to check this could be to run one agent with the base weights and the other with the trained weights. This would roughly show how much of the gain comes from the Proposer versus the Actor.
- I understand that the authors constrain the size of the input region the Actor selects in its final output. How do the authors ensure the Actor follows these constraints? What happens if the Actor’s output does not satisfy them: is it cut down to the limit, rejected, or scored as is? This is a critical part of the framework and should be stated clearly. If the constraints are not enforced correctly, the agents could learn to exploit the evaluation pipeline and hack the reward.
- Did the authors try cheaper alternatives to RL, such as supervised fine-tuning? Hase and Potts (2026) found SFT-based training to be much more efficient than RL in their faithfulness-maximization setup. Could the same be true for this setup?
5 comments
Toolkit–metric overlap and tool collapse
We agree that sensitivity analysis (SA) is close to the Q1 metric, but both are standard methods, not designed around each other, and choosing a faithful explainer is part of what an explanation agent should learn. The question is whether MEA does more than pick that tool. We show that MEA beats every standalone tool on Q1, including SA (Fig. 4: tabular 0.195 vs. 0.150, text 0.265 vs. 0.237), and beats DiCE and Growing Spheres on Q6 (Table 1: 0.89 vs. 0.52).
Controls (500-question subset, Q4/Q10 excluded; paired gap MEA − base [95% CI]). No tools: neither agent gets XAI tools. Same tool: both agents must use only the tool MEA converges to. Own tool: MEA vs. that tool's raw output (Q1 deletion AOPC, training baseline).
| Modality | Free tool choice (paper) | No tools | Same tool | Share of gain kept with same tool | MEA vs. its own tool (Q1 AOPC) |
|---|---|---|---|---|---|
| Tabular | +0.156 [0.119, 0.197] | +0.112 [0.074, 0.154] | +0.089 [0.060, 0.124] | 57% | 0.130 vs. 0.097 (SA) |
| Text | +0.192 [0.140, 0.250] | +0.091 [0.048, 0.140] | +0.115 [0.068, 0.166] | 60% | 0.241 vs. 0.192 (SA) |
| Vision | +0.242 [0.181, 0.305] | +0.180 [0.113, 0.251] | +0.017 [−0.028, +0.067] | 7% | 0.428 vs. 0.422 (SmoothGrad) |
Without tools, *MEA is still significantly better in every modality, so training improves the agent's own explanatory ability. With the tool fixed, MEA keeps 57–60% of its gain on tabular and text and beats raw SA in 128/128 tabular and 64/64 text cells under unseen baselines and budgets: the Actor improves on the tool output rather than copying it. On vision, MEA is on par with SmoothGrad, consistent with Fig. 4.
Tool-choice distribution (per (dataset, classifier, question) tuple, sampled test subset):
| Modality | Base: modal plan (share), # distinct plans | MEA: modal plan (share), # distinct plans | MEA tuples with a single plan |
|---|---|---|---|
| Tabular | SA + SHAP (43%), 10 | SA (100%), 1 | 40/40 |
| Text | LIME + SA + SHAP (27%), 17 | SA (95%), 3 | 39/40 |
| Vision | GradCAM + IG + LIME (47%), 23 | SmoothGrad (98%), 4 | 34/40 |
The Proposer does collapse, as intended, onto one faithful tool per modality. This is not a lookup table: what varies per instance is how the Actor uses the tool output (above), and MEA also improves where no memorized mapping exists, namely on unseen dataset–architecture pairs (Table 2: 0.581 vs. 0.379 tabular, 0.689 vs. 0.625 text, 0.576 vs. 0.448 vision), held-out Q8/Q9 on tabular and text (Table 5), a concept-level question (Table 7), a classifier with a synthetic shortcut (Table 12), and a 4B backbone (Table 11).
Tool penalty. The penalty depends only on the number of tools/tasks, not on which, so it cannot favor a particular tool. We keep λ = 0.05 so faithfulness dominates; a short test with λ = 0.2 removed the learning signal on >98% of Q1 rollouts. In an exploratory run without the penalty, the agent still converged to about one tool per instance, so convergence comes from the faithfulness reward.
Comparison with conversational explanation frameworks
Our claim concerns their design: they route questions to fixed explainers and never verify answers against the target model. We added a TalkToModel-style baseline (TTM) [1] with an oracle parser, LIME/KernelSHAP, what-if and DiCE operations, and the per-question best of LIME and SHAP (tabular and text):
| CoT | ReAct | ToT | TTM | MEA | |
|---|---|---|---|---|---|
| Tabular | 0.487 | 0.400 | 0.407 | 0.640 | 0.783 |
| Text | 0.471 | 0.548 | 0.555 | 0.635 | 0.726 |
| All | 0.478 | 0.478 | 0.485 | 0.637 | 0.753 |
MEA − TTM = +0.115 [0.089, 0.143].
Proposer vs. Actor
We ran the suggested weight swap (B = base, T = GRPO; first letter = Proposer):
| BB | TB | BT | TT | Proposer effect | Actor effect | |
|---|---|---|---|---|---|---|
| Tabular | 0.520 | 0.682 | 0.709 | 0.783 | 0.118 [0.089, 0.148] | 0.145 [0.113, 0.178] |
| Text | 0.557 | 0.588 | 0.661 | 0.737 | 0.054 [0.025, 0.083] | 0.127 [0.094, 0.159] |
| Vision | 0.493 | 0.633 | 0.652 | 0.688 | 0.088 [0.051, 0.124] | 0.106 [0.078, 0.137] |
| All | 0.526 | 0.631 | 0.673 | 0.736 | 0.084 [0.065, 0.103] | 0.126 [0.107, 0.145] |
Both roles improve significantly in every modality. The Actor contributes about 60% of the gain (turning tool output into the answer), and the Proposer about 40% (committing to one faithful explainer instead of 2–3-tool hybrids).
Enforcement of the output-size constraint
We thank the reviewer for raising this. In the submission, the 25% output budget was specified only in the Actor prompt, and a few outputs of text modality exceeded it. We now enforce B = 25% of input characters: an over-budget output gets one revision turn, is then truncated deterministically if still over budget, and is scored with the unmodified evaluator.
Text results, base → MEA, budget enforced.
| Q1 | Q2 | Q3 | Q6 | Q8 | Q9 |
|---|---|---|---|---|---|
| 0.102 → 0.253 | 0.920 → 0.965 | 0.222 → 0.368 | 0.424 → 0.707 | 0.186 → 0.344 | 0.063 → 0.078 |
All gains hold. GRPO also reduced violations (Q1: 43% → 9%, Q2: 56% → 0%, Q9: 11% → 2%), so the agent did not exploit the budget. We will update the affected numbers and state this protocol in Sec. 3.3.
SFT as a cheaper alternative
SFT needs gold explanations, which do not exist here: faithfulness is defined by the target model's behavior, so targets would have to be mined from the agent's own rollouts. The base agent is weak on many question types (e.g., Q1: 0.10 tabular, 0.07 text), so few rollouts are worth imitating, and even high-scoring ones may contain poor plans or redundant tool calls. As a proxy, using the most faithful rollouts as 3-shot demonstrations recovers on average less than a third of the gap to MEA:
| Modality | Category | n | Gemini ZS → 3-shot | Claude ZS → 3-shot | Qwen ZS → 3-shot | MEA |
|---|---|---|---|---|---|---|
| Tabular | FA | 79 | 0.562 → 0.566 | 0.460 → 0.486 | 0.504 → 0.553 | 0.675 |
| CF | 57 | 0.671 → 0.712 | 0.668 → 0.700 | 0.667 → 0.686 | 0.884 | |
| SP | 13 | 1.000 → 1.000 | 0.563 → 0.883 | 0.536 → 0.936 | 1.000 | |
| Text | FA | 99 | 0.508 → 0.545 | 0.508 → 0.549 | 0.455 → 0.577 | 0.677 |
| CF | 71 | 0.559 → 0.558 | 0.446 → 0.482 | 0.505 → 0.509 | 0.748 | |
| SP | 24 | 0.874 → 0.829 | 0.575 → 0.862 | 0.832 → 0.957 | 1.000 | |
| Vision | FA | 79 | 0.504 → 0.548 | 0.517 → 0.544 | 0.527 → 0.491 | 0.693 |
| CF | 57 | 0.301 → 0.411 | 0.234 → 0.382 | 0.333 → 0.481 | 0.587 | |
| SP | 19 | 0.972 → 0.983 | 0.878 → 0.910 | 0.962 → 0.986 | 1.000 |
(FA = feature attribution, CF = counterfactual, SP = spurious feature.)
We will add a paragraph on training-based faithful explanations (Malladi and Srivastava, 2026; Cheah, Perez-Ortiz, Siegel et al., 2026; Hase and Potts, 2026; Pres, Ruis, Ghebreselassie et al., 2026) and qualify our novelty claim accordingly.
Thank you for supporting our work. We hope these clarifications and additional experiments address the reviewer's concerns, and we would be glad to discuss any remaining questions.
[1] D. Slack, S. Krishna, H. Lakkaraju, and S. Singh. Explaining Machine Learning Models with Interactive Natural Language Conversations Using TalkToModel. Nature Machine Intelligence, 2023.
We thank the reviewer for the thorough and positive review, and for recognizing that
our generalization study goes beyond in-distribution evaluation, thatour baselines isolate the contribution of RL, and thatthe evaluation is broad and honest. In response to your comments, we ran 14 new experiments and 1 additional analysis, detailed below. We hope the response below addresses your concerns.Agreement between the text and the structured output
The reward uses the structured output because it can be verified by intervening on the target model; the Actor writes the text from the same tool evidence. In response to the reviewer’s comment, we now measure their agreement directly on 462 paired explanations per method, using an LLM judge (Claude Sonnet 5.5; a second judge, Claude Opus 5.5, agrees on ~87% of decisions). All five agents use the same tools.
Agreement 95% of MEA's explanations are consistent or partially consistent with its structured output; only 1% contradict it. MEA's text is also the best grounded in the tool outputs:
Grounding in the tool outputs.
Human study 13 participants rated this consistency at 4.07 / 5 for vision.
We will state that the faithfulness metrics apply to the structured output, and report text grounding separately.
Robustness beyond a single perturbation
Each metric uses one perturbation matching the training reward, which keeps reward and evaluation consistent across question types and modalities. We agree that aggregate evaluations are more standard, so we re-scored over budgets (deletion AOPC, insertion AUC) and perturbation baselines, most unseen in training (tabular: mean (train), zero, median/mode, marginal, kNN; text: delete (train), <unk>, random word; vision: gray (train), black, blur, inpainting). We compare MEA against post-hoc explainers and agent baselines. All tests use matched budgets and paired bootstrap.
Our results show that gains are therefore not tied to the training perturbation.