acceptodds
Under review as a conference paper at ICLR 2027

MEA: A Reward-Driven Multi-Agent System for Faithful Model Explanations

Abstract

Recent years have seen the employment of a plethora of machine learning (ML) models in high-stakes domains, but they remain largely opaque to the practitioners who act on their predictions. While post-hoc explanation methods offer a lens into this model behavior, wielding them effectively demands expertise most domain experts lack: navigating high-dimensional outputs, selecting the best explanations, and synthesizing evidence across disparate tools. To this end, we present MEA, a multi-agent framework that removes the explanation knowledge barrier entirely: a agent selects and configures explanation tools based on the question and modality, while an agent is optimized end-to-end against faithfulness, transforming the outputs into natural language explanations grounded in model behavior across tabular, text, and vision modalities. Further, we introduce diverse question types spanning feature attribution, counterfactual reasoning, and spurious feature detection, each paired with a perturbation-based faithfulness metric. We find that frontier LLMs systematically produce unfaithful explanations. By optimizing against faithfulness rewards augmented with a modality-adaptive penalty, MEA consistently outperforms post hoc explainers, agentic, and closed-source baselines across six datasets, with reward-driven optimization yielding faithfulness gains of (tabular), (text), and (vision) over the untrained backbone. More broadly, our findings suggest that AI agents themselves can serve as a scalable, adaptable interface to ML explainability, opening a path toward natural-language explainability that generalizes beyond the fixed, single-purpose tools that have long defined the field.

Primary area:
interpretability and explainable AI
Keywords:
Explainable AI, Multimodal Agent, Post-hoc Explanation, Tool-Augmented Agents

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_PvoYOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary
  • The paper presents MEA, a framework in which an LLM agent is trained with RL (GRPO) to explain the predictions of classifiers across tabular, text, and vision data.
  • The framework uses two agents: a Proposer that selects XAI tools such as LIME, SHAP, and Grad-CAM, and an Actor that runs these tools and produces an explanation identifying the relevant input elements. These elements are then perturbed, and the resulting change in the model’s behavior is scored with faithfulness metrics that also serve as the training reward.
Critical strengths
  • The authors study generalization along several axes. Beyond held-out instances of the training datasets, they evaluate on unseen dataset-architecture pairs. They also test held-out question types and a concept-level question that lies entirely outside the training taxonomy. This goes well beyond a standard in-distribution evaluation.
  • The base agent (the same Proposer–Actor harness, prompts, and tools, without GRPO training) serves as a baseline. This isolates the effect of training from the effect of the agentic harness itself. The comparisons against frontier LLMs, prompting strategies (CoT, ReAct, ToT), and an inference-time critic further help separate the contribution of RL from that of prompting.
  • The evaluation is broad and seems fairly honest. Their analysis covers three modalities, six datasets, multiple classifier architectures per dataset, and ten question types spanning feature attribution, counterfactuals, and spurious features. It also compares against both general-purpose attribution methods and specialized counterfactual generators. The authors also report negative results, such as catastrophic forgetting under sequential training and cases where the base agent outperforms the trained model. This makes the positive results more credible.
Critical weaknesses
  • The authors present their framework as producing natural-language explanations grounded in model behavior, but there is no mechanism to measure this. The Actor produces two separate outputs: a structured output identifying the relevant input elements, and a free-text explanation. All faithfulness metrics and the RL reward are computed from the structured output alone, and the text is never used. The authors should either measure agreement between the text and the structured output or revise their claims about faithful natural-language explanations.
  • Faithfulness is evaluated with a single perturbation per instance. Standard evaluations typically aggregate over a range of perturbations (e.g., deletion/insertion curves over varying numbers of elements and multiple baselines), because single-point scores can be sensitive to these choices. It is therefore unclear whether the gains hold beyond this specific setup.
  • The XAI toolkit available to the agents overlaps substantially with the evaluation metrics. For example, the sensitivity-analysis tool computes essentially the same leave-one-out probability drop that the Q1 metric measures. The reported gains may therefore partly reflect the agent learning to pick the tool that already approximates the metric, rather than a broader ability to explain.
  • The question type determines both the Proposer’s instruction and the evaluation metric, creating an effective one-to-one mapping between them. The Proposer is also given the classifier, the prediction, and the input, and can infer the dataset from feature names, feature values, or the model name. It is therefore possible that the model memorizes which tool to use for each (dataset, classifier, question) tuple and applies this lookup table at test time.
  • The paper claims that existing conversational explanation frameworks are unfaithful, but it never evaluates them or compares against them. The agentic baselines are only generic prompting strategies.
  • The authors claim that “no existing work jointly optimizes a multi-agent pipeline end-to-end with RL across diverse modalities and question types.” This may be true with these qualifiers, but the literature review should be expanded to cover training-based approaches to faithful explanations, such as Malladi and Srivastava (2026); Cheah, Perez-Ortiz, Siegel et al. (2026); Hase and Potts (2026); Pres, Ruis, Ghebreselassie et al. (2026)
Questions for authors
  • The paper doesn’t report how often each XAI tool is selected, or how much the choice varies across instances of the same dataset and question type. Did the authors analyze the frequency of each tool choice? I am especially interested in the distribution of tool choices conditioned on each (dataset, classifier, question) tuple. Do the authors observe any mode collapse in this distribution after GRPO training?
  • The tool penalty in the reward function incentivizes the model to use fewer tools. I understand that its purpose is to discourage the Proposer from choosing every tool on every instance. However, could this penalty also lead to undesirable behavior? For example, could it push the Proposer to choose the same set of tools for every instance of a dataset?
  • Did the authors analyze whether GRPO changes the Proposer’s behavior or the Actor’s behavior more? One way to check this could be to run one agent with the base weights and the other with the trained weights. This would roughly show how much of the gain comes from the Proposer versus the Actor.
  • I understand that the authors constrain the size of the input region the Actor selects in its final output. How do the authors ensure the Actor follows these constraints? What happens if the Actor’s output does not satisfy them: is it cut down to the limit, rejected, or scored as is? This is a critical part of the framework and should be stated clearly. If the constraints are not enforced correctly, the agents could learn to exploit the evaluation pipeline and hack the reward.
  • Did the authors try cheaper alternatives to RL, such as supervised fine-tuning? Hase and Potts (2026) found SFT-based training to be much more efficient than RL in their faithfulness-maximization setup. Could the same be true for this setup?
5 comments

We thank the reviewer for the thorough and positive review, and for recognizing that our generalization study goes beyond in-distribution evaluation, that our baselines isolate the contribution of RL, and that the evaluation is broad and honest. In response to your comments, we ran 14 new experiments and 1 additional analysis, detailed below. We hope the response below addresses your concerns.

Agreement between the text and the structured output

The reward uses the structured output because it can be verified by intervening on the target model; the Actor writes the text from the same tool evidence. In response to the reviewer’s comment, we now measure their agreement directly on 462 paired explanations per method, using an LLM judge (Claude Sonnet 5.5; a second judge, Claude Opus 5.5, agrees on ~87% of decisions). All five agents use the same tools.

Agreement 95% of MEA's explanations are consistent or partially consistent with its structured output; only 1% contradict it. MEA's text is also the best grounded in the tool outputs:

Grounding in the tool outputs.

MEABaseClaude HaikuGemini FlashGPT-5.4-nano
Evidence claims supported ↑0.590.370.360.410.48
Fully grounded ↑0.490.250.190.300.30

Human study 13 participants rated this consistency at 4.07 / 5 for vision.

We will state that the faithfulness metrics apply to the structured output, and report text grounding separately.

Robustness beyond a single perturbation

Each metric uses one perturbation matching the training reward, which keeps reward and evaluation consistent across question types and modalities. We agree that aggregate evaluations are more standard, so we re-scored over budgets (deletion AOPC, insertion AUC) and perturbation baselines, most unseen in training (tabular: mean (train), zero, median/mode, marginal, kNN; text: delete (train), <unk>, random word; vision: gray (train), black, blur, inpainting). We compare MEA against post-hoc explainers and agent baselines. All tests use matched budgets and paired bootstrap.

QuestionModalityMEA rank across budgets × baselinesMEA vs. others
Q1Tabular1st under 5/5 baselines (deletion), 4/5 (insertion)Significantly better in all 64 unseen-baseline comparisons
Text1st under 3/3 (deletion and insertion)Ahead in 32/32 comparisons, 28 significant
Vision1st under 4/4 (deletion)—
Q8Tabular1st under 5/5 (recovery AOPC and flip rate)Mean fill: AOPC 0.154 vs. 0.093 / 0.074; flip 0.64 vs. 0.43 / 0.35
Text1st under 3/3AOPC 0.319–0.368 vs. 0.18–0.31; all significant
Q6Tabular1st at every edit magnitude and number of editsAll significant
Text1st at every edit fraction0.707 / 74% flip vs. 0.438 / 44% and 0.434 / 46%; all significant

Our results show that gains are therefore not tied to the training perturbation.

Toolkit–metric overlap and tool collapse

We agree that sensitivity analysis (SA) is close to the Q1 metric, but both are standard methods, not designed around each other, and choosing a faithful explainer is part of what an explanation agent should learn. The question is whether MEA does more than pick that tool. We show that MEA beats every standalone tool on Q1, including SA (Fig. 4: tabular 0.195 vs. 0.150, text 0.265 vs. 0.237), and beats DiCE and Growing Spheres on Q6 (Table 1: 0.89 vs. 0.52).

Controls (500-question subset, Q4/Q10 excluded; paired gap MEA − base [95% CI]). No tools: neither agent gets XAI tools. Same tool: both agents must use only the tool MEA converges to. Own tool: MEA vs. that tool's raw output (Q1 deletion AOPC, training baseline).

ModalityFree tool choice (paper)No toolsSame toolShare of gain kept with same toolMEA vs. its own tool (Q1 AOPC)
Tabular+0.156 [0.119, 0.197]+0.112 [0.074, 0.154]+0.089 [0.060, 0.124]57%0.130 vs. 0.097 (SA)
Text+0.192 [0.140, 0.250]+0.091 [0.048, 0.140]+0.115 [0.068, 0.166]60%0.241 vs. 0.192 (SA)
Vision+0.242 [0.181, 0.305]+0.180 [0.113, 0.251]+0.017 [−0.028, +0.067]7%0.428 vs. 0.422 (SmoothGrad)

Without tools, *MEA is still significantly better in every modality, so training improves the agent's own explanatory ability. With the tool fixed, MEA keeps 57–60% of its gain on tabular and text and beats raw SA in 128/128 tabular and 64/64 text cells under unseen baselines and budgets: the Actor improves on the tool output rather than copying it. On vision, MEA is on par with SmoothGrad, consistent with Fig. 4.

Tool-choice distribution (per (dataset, classifier, question) tuple, sampled test subset):

ModalityBase: modal plan (share), # distinct plansMEA: modal plan (share), # distinct plansMEA tuples with a single plan
TabularSA + SHAP (43%), 10SA (100%), 140/40
TextLIME + SA + SHAP (27%), 17SA (95%), 339/40
VisionGradCAM + IG + LIME (47%), 23SmoothGrad (98%), 434/40

The Proposer does collapse, as intended, onto one faithful tool per modality. This is not a lookup table: what varies per instance is how the Actor uses the tool output (above), and MEA also improves where no memorized mapping exists, namely on unseen dataset–architecture pairs (Table 2: 0.581 vs. 0.379 tabular, 0.689 vs. 0.625 text, 0.576 vs. 0.448 vision), held-out Q8/Q9 on tabular and text (Table 5), a concept-level question (Table 7), a classifier with a synthetic shortcut (Table 12), and a 4B backbone (Table 11).

Tool penalty. The penalty depends only on the number of tools/tasks, not on which, so it cannot favor a particular tool. We keep λ = 0.05 so faithfulness dominates; a short test with λ = 0.2 removed the learning signal on >98% of Q1 rollouts. In an exploratory run without the penalty, the agent still converged to about one tool per instance, so convergence comes from the faithfulness reward.

Comparison with conversational explanation frameworks

Our claim concerns their design: they route questions to fixed explainers and never verify answers against the target model. We added a TalkToModel-style baseline (TTM) [1] with an oracle parser, LIME/KernelSHAP, what-if and DiCE operations, and the per-question best of LIME and SHAP (tabular and text):

CoTReActToTTTMMEA
Tabular0.4870.4000.4070.6400.783
Text0.4710.5480.5550.6350.726
All0.4780.4780.4850.6370.753

MEA − TTM = +0.115 [0.089, 0.143].

Proposer vs. Actor

We ran the suggested weight swap (B = base, T = GRPO; first letter = Proposer):

BBTBBTTTProposer effectActor effect
Tabular0.5200.6820.7090.7830.118 [0.089, 0.148]0.145 [0.113, 0.178]
Text0.5570.5880.6610.7370.054 [0.025, 0.083]0.127 [0.094, 0.159]
Vision0.4930.6330.6520.6880.088 [0.051, 0.124]0.106 [0.078, 0.137]
All0.5260.6310.6730.7360.084 [0.065, 0.103]0.126 [0.107, 0.145]

Both roles improve significantly in every modality. The Actor contributes about 60% of the gain (turning tool output into the answer), and the Proposer about 40% (committing to one faithful explainer instead of 2–3-tool hybrids).

Enforcement of the output-size constraint

We thank the reviewer for raising this. In the submission, the 25% output budget was specified only in the Actor prompt, and a few outputs of text modality exceeded it. We now enforce B = 25% of input characters: an over-budget output gets one revision turn, is then truncated deterministically if still over budget, and is scored with the unmodified evaluator.

Text results, base → MEA, budget enforced.

Q1Q2Q3Q6Q8Q9
0.102 → 0.2530.920 → 0.9650.222 → 0.3680.424 → 0.7070.186 → 0.3440.063 → 0.078

All gains hold. GRPO also reduced violations (Q1: 43% → 9%, Q2: 56% → 0%, Q9: 11% → 2%), so the agent did not exploit the budget. We will update the affected numbers and state this protocol in Sec. 3.3.

SFT as a cheaper alternative

SFT needs gold explanations, which do not exist here: faithfulness is defined by the target model's behavior, so targets would have to be mined from the agent's own rollouts. The base agent is weak on many question types (e.g., Q1: 0.10 tabular, 0.07 text), so few rollouts are worth imitating, and even high-scoring ones may contain poor plans or redundant tool calls. As a proxy, using the most faithful rollouts as 3-shot demonstrations recovers on average less than a third of the gap to MEA:

ModalityCategorynGemini ZS → 3-shotClaude ZS → 3-shotQwen ZS → 3-shotMEA
TabularFA790.562 → 0.5660.460 → 0.4860.504 → 0.5530.675
CF570.671 → 0.7120.668 → 0.7000.667 → 0.6860.884
SP131.000 → 1.0000.563 → 0.8830.536 → 0.9361.000
TextFA990.508 → 0.5450.508 → 0.5490.455 → 0.5770.677
CF710.559 → 0.5580.446 → 0.4820.505 → 0.5090.748
SP240.874 → 0.8290.575 → 0.8620.832 → 0.9571.000
VisionFA790.504 → 0.5480.517 → 0.5440.527 → 0.4910.693
CF570.301 → 0.4110.234 → 0.3820.333 → 0.4810.587
SP190.972 → 0.9830.878 → 0.9100.962 → 0.9861.000

(FA = feature attribution, CF = counterfactual, SP = spurious feature.)

We will add a paragraph on training-based faithful explanations (Malladi and Srivastava, 2026; Cheah, Perez-Ortiz, Siegel et al., 2026; Hase and Potts, 2026; Pres, Ruis, Ghebreselassie et al., 2026) and qualify our novelty claim accordingly.

Thank you for supporting our work. We hope these clarifications and additional experiments address the reviewer's concerns, and we would be glad to discuss any remaining questions.

[1] D. Slack, S. Krishna, H. Lakkaraju, and S. Singh. Explaining Machine Learning Models with Interactive Natural Language Conversations Using TalkToModel. Nature Machine Intelligence, 2023.

Reviewer_Q2TnOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes MEA, which is a faithful explanation provider for a black box model behavior. It provides faithful natural language explanations. It consists of eo agents- proposer and actor. The proposer runs a set of explanation tools and obtains the outputs. The actor analyses the tools used in the outputs and synthesizes the natural language explanations. Both agents are trained with shared backbone using GRPO. It probes consistency, counterfactual and spurious correlation-based feature attribution across tabular, vision, and textual modalities.

Critical strengths
  1. The paper has broad scope across modalities like vision, tabular and text.
  2. The experimental setup is quite extensive. It covers 6 datasets and various classifier architectures.
  3. It shows that learning the explanations through the standard explanation policies, which are used to evaluate faithful explanations, indeed makes the explanations more faithful.
Critical weaknesses
  1. The Actor uses the tool calls and results obtained using the proposer. However, whether the natural language explanations are grounded in the proposer's output is not evaluated, creating an important gap in the evaluation. Can a human study help with this?
  2. What is the advantage of the dual agent configuration of the actor and the proposer over a single agent learning to call explainability tools and produce natural language explanations? Do the authors have any experiments to understand this?
  3. The evaluation of faithfulness happens using the same queries it was trained on. This is particularly concerning given the not-so-great OOD transfer to Q8/Q9 tasks (Table 5 in Appendix A.6 Q8/Q9 tasks), which shows inconclusive gains of MEA over the baselines.
  4. MEA is compared against strong LLMs like ChatGPT, Gemini, using different reasoning prompt strategies using base LLM, etc. However, they are not used in a few-shot manner, with only a few examples, when compared against MEA. How does a few-shot prompt perform on them? Given possible improvements using few-shot prompts, the GRPO training-based performance has to be compared.
  5. A model might correctly/incorrectly predict a question based on the same feature. For example, a simple model y=1[x[0]>0]y = 1[x[0]>0], x∈Rdx\in R^d can have two different predictions; however, out of d dimensions, x[0] remains responsible for both predictions. Hence, the objectives of Q4 and Q10 might not be apt.
  6. MEA should be compared against similar natural language explainers like [1], [2].
  7. Misprediction recovery is a good utility-centric measure that measures the percentage of times the mispredictions are flipped to correct predictions if the dominating features are removed. Given the user-centric approach of XAI, I encourage the authors to report misprediction recovery separately from probability-based recovery.
  8. From an end-user perspective, it is important to conduct a human study to see if they find MEA explanations more helpful and usable as compared to other natural language explainers.

[1] Towards Faithful Agentic XAI: A Verification Method and an Open-World Benchmark for Better Model Faithfulness. Arxiv 2605.27879.

[2] Directly Optimizing Natural Language Explanations for Behavioral Faithfulness: Simulatability and Recoverability. ICLR 2026.

Questions for authors

Can the authors conduct experiments and comment on the limitations asked in the Weaknesses section?

7 comments

We thank the reviewer for the detailed feedback, and for recognizing the broad scope across tabular, text and vision, the extensive experimental setup, and that training against faithfulness objectives makes explanations more faithful. In response to your comments, we ran seven new experiments, detailed below. We hope they address your concerns.We thank the reviewer for the detailed feedback, and for recognizing the broad scope across tabular, text and vision, the extensive experimental setup, and that training against faithfulness objectives makes explanations more faithful. In response to your comments, we ran seven new experiments, detailed below. We hope the response below addresses your concerns.

Is the text grounded in the Proposer's output?

The Actor writes its explanation only from the Proposer's plan and the tool outputs (Eq. 2, Fig. 11). Since the reward scores only the structured output, we now check the text separately.

Claim-level audit. We compare MEA with four agents that see the same tool outputs (462 paired explanations per method). An LLM judge (Claude Sonnet 5.5; a second judge, Claude Opus 5.5, agrees on ~87% of decisions) labels each claim supported or contradicted by the complete tool outputs.

MEABaseClaude HaikuGemini FlashGPT-5.4-nano
Evidence claims supported ↑0.590.370.360.410.48
Contradicted claims ↓0.120.180.210.160.18
Fully grounded ↑0.490.250.190.300.30

MEA's text is the most grounded and leads every baseline on the fully-grounded rate (paired McNemar, p < 10⁻⁹). In a human study (details below), participants rated the consistency between MEA's text and its answer at 4.04/5.

Dual vs. single agent

Separating planning from execution is a standard design for tool-using agents (Xu et al., 2023; Shen et al., 2024; Erdogan et al., 2025), since a single model that must both plan and act tends to make worse decisions (Erdogan et al., 2025; Shen et al., 2024). MEA follows it: the Proposer decides which evidence to gather, the Actor how to interpret it. This keeps the Actor’s context short (a single agent’s grows to 17k tokens, and LLMs use long contexts unreliably; Liu et al., 2024) and exposes the plan as an inspectable intermediate result, allowing us to inspect which tools were selected before any explanation is written and trace a failure to either planning or interpretation. It also makes training effects separable: in a weight-swap analysis, both roles contribute significantly to the GRPO gain of 0.210 (Proposer 0.084 [0.065, 0.103]; Actor 0.126 [0.107, 0.145]).

Held-out queries; misprediction recovery reported separately

We would like to highlight that the test instances are disjoint from training instances, MEA is further evaluated on paraphrased questions (within 1%, App. A.7), unseen dataset–architecture pairs (tabular 0.581 vs. 0.379, text 0.689 vs. 0.625, vision 0.576 vs. 0.448; Table 2), and held-out question types (Table 5). Following reviewer’s suggestion, we report misprediction recovery (MR) on held-out Q8: the share of misclassified instances that flip to the true class after masking the elements the agent blames.

ModalityProbability recovery, Base → MEAMR, Base → MEAΔMR [95% CI]
Tabular0.132 → 0.18456.9% → 77.8%+20.8 [+8.3, +33.3]
Text0.189 → 0.35328.7% → 44.0%+14.8 [+4.9, +24.6]
Vision0.069 → 0.05015.7% → 12.5%−3.3 [−11.6, +5.0]

On tabular and text, the decision-level gain is larger than probability recovery suggests. On vision the difference is not significant; the lower probability recovery comes from MEA proposing larger boxes (region ratio 0.63 vs. 0.43), to which the mask-based metric is sensitive. We will add MR to Table 5. Thank you for the suggestion!

Few-shot baselines

Demonstrations may teach frontier LLMs how to select and use the explanation tools, so we added 3-shot baselines: full Proposer and Actor (prompt, response) pairs from training questions, chosen as the most faithful per (modality, question type).

ModalityCategorynGemini ZS → 3-shotClaude ZS → 3-shotQwen ZS → 3-shotMEA
TabularFA790.562 → 0.5660.460 → 0.4860.504 → 0.5530.675
CF570.671 → 0.7120.668 → 0.7000.667 → 0.6860.884
SP131.000 → 1.0000.563 → 0.8830.536 → 0.9361.000
TextFA990.508 → 0.5450.508 → 0.5490.455 → 0.5770.677
CF710.559 → 0.5580.446 → 0.4820.505 → 0.5090.748
SP240.874 → 0.8290.575 → 0.8620.832 → 0.9571.000
VisionFA790.504 → 0.5480.517 → 0.5440.527 → 0.4910.693
CF570.301 → 0.4110.234 → 0.3820.333 → 0.4810.587
SP190.972 → 0.9830.878 → 0.9100.962 → 0.9861.000

FA/CF/SP = feature attribution / counterfactual / spurious feature. Few-shot helps, mostly on SP, but MEA stays significantly ahead in every modality and in every FA and CF category. With the same backbone (Qwen), demonstrations recover only ~30% of the gap between the base agent and MEA.

The x[0] example for Q4/Q10

We would like to clarify why we score Q4 and 10 this way. Q4 and Q10 are contrastive: a faithful answer must name what distinguishes the two cases, which 1−Sim(FA,FB)1 − Sim(F_A, F_B) checks directly. Causal effects are tested separately by Q1–Q3 and Q8, which mask the attributed elements and re-query the model. We agree that a feature such as x[0] can drive both predictions and would then be penalized. We therefore add two model-grounded scores, each reported minus the mean of 20 size-matched random sets (random = 0):

Contrastive Margin Collapse (CMC): mask the agent's top feature and measure the fraction of the margin P(ŷ) − P(y′) that disappears. For Q4, y′ is the other instance's prediction; for Q10, it is computed on the misclassified instance with ŷ the wrong prediction and y′ the true label. Each instance is scored on its own, so if x[0] drives both predictions, naming it earns full credit.

Occlusion precision (OP): remove each feature on its own (text: each word; vision: each cell of an 8×8 grid), rank features by the drop in P(ŷ), and report the fraction of the agent's features in the top |S|. For vision, since the returned feature is a phrase naming an image part (e.g., "the bird's red crown"), we localize the phrase in the image with CLIPSeg (Lüddecke & Ecker, 2022), and then gray-fill the top 20% of pixels in its map. The random control uses masks of the same area at shifted locations.

Results (agent − random; † = significantly below MEA, paired bootstrap):

MEABaseGemini-3.1-FLGPT-5.4-nanoClaude-Haiku
CMCTab Q40.1920.050†0.118†0.102†0.065†
Tab Q100.1590.0880.1810.1430.151
Text Q40.1020.020†0.045†0.016†0.066
Text Q100.1390.076†0.053†0.052†0.022†
Vis Q40.3460.3140.3220.3250.299†
Vis Q100.2450.2510.2280.2670.249
Occlusion precisionTab Q40.6630.091†0.368†0.179†0.172†
Tab Q100.2170.136†0.2170.1470.097†
Text Q40.3190.140†0.228†0.110†0.175†
Text Q100.2450.161†0.170†0.178†0.244
Vis Q40.1820.1780.1770.1700.176
Vis Q100.1400.1480.1280.1500.157

MEA is significantly above random in every setting. On tabular and text, MEA is significantly better in 11 of 16 CMC comparisons and 13 of 16 occlusion-precision comparisons, and best or tied-best in all occlusion-precision settings. Where a baseline is nominally higher (Tab Q10 CMC, Vis Q10), the difference is not significant. We will report both metrics for Q4/Q10.

Comparison with [1] and [2]

We thank the reviewer for the references and will cite and discuss both. A direct comparison is not feasible: neither has public code, and both target a different kind of explanation. [1] (FAX) is an inference-time verifier that filters unsupported claims from a draft explanation; it does not train the explainer. [2] (OPeX) trains behavior-level text for simulatability and recoverability; it does not select input elements, so intervention-based metrics cannot score it. Instead, we add the closest comparator: a TalkToModel-style framework [2] with an oracle parser and LIME/SHAP, what-if and DiCE operations. MEA is more faithful: 0.753 vs. 0.637 (Δ = +0.115 [0.089, 0.143]).

Human evaluation against other explainers

We agree that intervention-based faithfulness does not show whether end users find the explanations useful. We conducted a human study. 13 participants (i) rated MEA's explanations on 1–5 scales (78 ratings), and (ii) on 104 screens, each showing four anonymized explanations of the same instance in random order, picking the most and least faithful-looking one. Candidates: MEA, the base agent, Gemini Flash, Claude Haiku, GPT-5.4-nano, and the raw tool output as a ranked list.

We focus the human study on vision, where non-experts can check at a glance whether a highlighted region contains the object. Tabular instances (socio-economic and healthcare tasks) and long text passages with token-level highlights require domain expertise, which we could not recruit within the rebuttal window.

Ratings of MEA's vision explanations (29 ratings from 13 participants, 1–5 scale):

Mean [95% CI]
Consistency of text with the answer4.07 [3.66, 4.45]
Helpfulness3.90 [3.45, 4.31]
Clarity4.41 [4.10, 4.69]

Vision, blind comparison (28 screens, 13 participants):

MEAGPT-5.4-nanoGemini FlashBase agentRaw toolClaude Haiku
Net preference ↑+18+15+9−6−18−30
MEA preferred (W–L)—6–511–79–65–412–6

Net preference = % of appearances picked as most faithful − % picked as least faithful (range −100 to +100; 0 = no preference).

Pooled against the four LLM explainers, MEA is preferred in 38 of 62 decided pairs (61%). Participants also rate it 4.07/5 for consistency and 4.41/5 for clarity.

Thank you for supporting our work. We hope these clarifications and additional experiments address the reviewer's concerns, and we would be glad to discuss any remaining questions.

[1] T. Lüddecke and A. Ecker. Image Segmentation Using Text and Image Prompts. CVPR, 2022.

[2] D. Slack, S. Krishna, H. Lakkaraju, and S. Singh. Explaining Machine Learning Models with Interactive Natural Language Conversations Using TalkToModel. Nature Machine Intelligence, 2023.

[3] Agarwal, C., Tanneru, S. H., & Lakkaraju, H. Faithfulness vs. plausibility: On the (un)reliability of explanations from large language models. arXiv:2402.04614, 2024.

[4] Hase, P., & Bansal, M. Evaluating explainable AI: Which algorithmic explanations help users predict model behavior? ACL, 2020.

[5] Jacovi, A., & Goldberg, Y. Towards faithfully interpretable NLP systems: How should we define and evaluate faithfulness? ACL, 2020.

[6] Turpin, M., Michael, J., Perez, E., & Bowman, S. R. Language models don't always say what they think: Unfaithful explanations in chain-of-thought prompting. NeurIPS, 2023.

Reviewer_MPPyOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes MEA, a reward-driven multi-agent framework that uses a Proposer–Actor architecture to select explanation tools and generate structured, natural-language explanations. The framework is trained with GRPO using question-specific perturbation-based faithfulness rewards, a tool-count penalty, and a vision-specific size penalty. Experiments across tabular, text, and vision benchmarks demonstrate improved faithfulness, with additional evaluations on held-out questions, unseen datasets, and model architectures.

Critical strengths

S1: Optimizing explanations directly against a frozen target model is a sound objective. Setting up the untrained Proposer–Actor as a baseline keeps the backbone architecture and available tools identical, allowing the empirical gains to be cleanly attributed to the reward-driven training process rather than tool availability or model capacity.

Critical weaknesses

W1: The paper's strong claims regarding spurious-feature detection heavily depend on question type Q10, where the metric is defined as m10=1−sim(Fcorr,Fwrong)m_{10} = 1 - \text{sim}(F_{\text{corr}}, F_{\text{wrong}}) (Appendix Table 4). This formulation only rewards dissimilar attribution sets between correct and incorrect predictions, without verifying whether the identified discrepancies actually explain the model's erroneous behavior; arbitrary non-overlapping feature selections could trivially achieve a high score. Furthermore, on held-out question types Q8/Q9 (Appendix Table 5), which more directly evaluate causal corrective impact, MEA's faithfulness gains are modest on tabular/text data and exhibit a regression on vision compared to the Base Agent (0.034 vs. 0.069). These results raise concerns about whether the near-perfect Q10 scores reflect genuine spurious-feature identification, especially given the limited generalization to held-out Q8/Q9 questions.

W2: The reward evaluates selected features or regions through perturbation-based tests, but does not directly assess the factual accuracy of the accompanying natural-language explanation. An agent could identify the correct region while providing an incorrect rationale without being penalized for the verbal error. Although the synthetic-shortcut experiment supports the model's feature-selection ability, independent validation of the generated explanations is still lacking.

Questions for authors

Q1: What mechanisms prevent arbitrary or model-independent disjoint feature subsets from scoring near-perfectly under the current metric definitions? Could the authors benchmark Q4/Q10 performance against a random or heuristic model-independent selection control?

Q2: How is the factual faithfulness of the generated natural-language explanations evaluated independently of the selected features or regions? Could the authors provide human or quantitative evaluation of whether the verbal claims are supported by the target model's observed behavior?

5 comments

We thank the reviewer for the constructive feedback. We appreciate the recognition that our explanation optimization is a sound objective, and that the untrained agent baseline lets the gains be attributed cleanly to reward-driven training rather than to the tools or to model capacity. In response to your comments, we ran four new experiments, detailed below. We hope the response below addresses your concerns.

Model-independent controls for Q4/Q10

We thank the insightful feedback shared by the reviewer for the Q4/10 metric. However, we would like to clarify why we score Q4 and 10 this way. The questions are contrastive. Q4 asks why instances A and B receive different predictions. Q10 asks what separates a correctly from an incorrectly classified instance. A faithful answer must name features that distinguish the two cases, so 1−Sim(FA,FB)1 − Sim(F_A, F_B) checks this contrastive property directly. Hence, we have other questions to test causal effects. Interventional tests are covered by Q1–Q3 and Q8, which mask the attributed elements and re-query the target model. The two sets of questions are designed to complement each other.

We agree with the reviewer that set dissimilarity alone does not verify that the distinguishing features actually drive the model's behavior. To test this directly, we scored every Q4/Q10 explanation with a model-grounded metric and benchmarked it against the random control the reviewer asked for. Each Q4/Q10 explanation is scored by intervening on the target model, relative to the mean of 20 size-matched random sets from the same instance. Random selection therefore scores 0 by construction. We use two scores to address the reviewer’s concern:

Contrastive margin collapse (CMC). Mask the agent's top-ranked feature and measure the fraction of the margin P(ŷ) − P(y′) that disappears. For Q4, y′ is the other instance's prediction; for Q10, it’s on the misclassified instance, ŷ is the wrong prediction and y′ is the true label, so this tests whether the blamed feature drives the error. Occlusion precision (OP). Remove each feature on its own (text: each word; vision: each cell of an 8×8 grid) and rank the features by the drop in P(ŷ). The score is the fraction of the agent's features in the top |S| of this ranking. It tests whether the named features are the ones the model relies on.

For vision, since the returned feature is a phrase naming an image part (e.g., "the bird's red crown"), we localize the phrase in the image with CLIPSeg (Lüddecke & Ecker, 2022), and then gray-fill the top 20% of pixels in its map. The random control uses masks of the same area at shifted locations.

Results (agent − size-matched random; † = significantly below MEA, paired bootstrap):

MEABaseGemini-3.1-FLGPT-5.4-nanoClaude-Haiku
CMCTab Q40.1920.050†0.118†0.102†0.065†
Tab Q100.1590.0880.1810.1430.151
Text Q40.1020.020†0.045†0.016†0.066
Text Q100.1390.076†0.053†0.052†0.022†
Vis Q40.3460.3140.3220.3250.299†
Vis Q100.2450.2510.2280.2670.249
Occlusion precisionTab Q40.6630.091†0.368†0.179†0.172†
Tab Q100.2170.136†0.210.1470.097†
Text Q40.3190.140†0.228†0.110†0.175†
Text Q100.2450.161†0.170†0.178†0.244

MEA is significantly above random in every setting. On tabular and text, MEA is significantly better in 11 of 16 CMC comparisons and 13 of 16 occlusion-precision comparisons, and best or tied-best in all occlusion-precision settings. Where a baseline is nominally higher (Tab Q10 CMC, Vis Q10), the difference is not significant. We will report both metrics for Q4/Q10.

On held-out Q8, MEA successfully corrects significantly more mispredictions than the base agent (77.8% vs. 56.9% on tabular, and 44.0% vs. 28.7% on text). Although Q9 shows an isolated, statistically significant drop (p<0.0001), this is driven by a distribution shift in bounding-box sizes (such as rising from 0.383 to 0.765 for cub_resnet) rather than a general grounding degradation. Additionally, MEA demonstrates superior faithfulness, achieving a top Q4 CMC score of 0.192 and an OP score of 0.663.

We will report the grounded metrics alongside our current metric for Q4/Q10, base the spurious-feature claim on the grounded metric and on Q8, and add the random control to the appendix.

Independent evaluation of the natural-language explanation

Why the reward uses the structured output. Existing conversational explainers such as TalkToModel [2] and ECHO [3] do not quantitatively verify generated text against model behavior. Approaches that do score text rely on LLM judges [4-5], which can share the hallucinations they assess [6]. The structured output (selected features or regions) can be verified objectively by intervening on the target model. Free text cannot be scored this way without an external judge, which is expensive at RL scale and is itself open to reward hacking. Importantly, the Actor writes the text from the same tool evidence it uses for the structured output.

We agree that the text should also be validated independently and we therefore evaluated it in two ways.

Claim-level audit. We compared MEA with four agents that see the same tool outputs: the base agent, Claude Haiku, Gemini Flash and GPT-5.4-nano. We sampled 462 paired explanations per method, covering all datasets and question types. We use an LLM judge (Claude Sonnet 5.5) that splits each explanation into atomic claims. It labels each claim supported, contradicted or unverifiable against the complete tool outputs. To ensure the reliability, we introduce a second judge (Claude Opus 5.5), which agrees on ~87% of decisions and gives the same ranking.

MEABaseClaude HaikuGemini FlashGPT-5.4-nano
Evidence claims supported ↑0.590.370.360.410.48
Contradicted claims ↓0.120.180.210.160.18
Explanations with any contradicted claim ↓0.360.560.650.490.54
Fully grounded ↑0.490.250.190.300.30

MEA's text is the most grounded of all agents and is ahead of every baseline on the fully-grounded rate (paired McNemar, p < 10⁻⁹), and it has the fewest incorrect rationales.

Support from the target model's behavior. The text is tied to the target model in two steps: MEA's structured output is verified by intervening on the target model. The text agrees with that structured output. The judge rates 95% of MEA's explanations as consistent or partially consistent with it.

In addition, the evidence the claims are checked against is a direct measurement of the target model: sensitivity analysis reports the model's own probability change when each element is removed.

We also conducted a human study. 13 participants, all of whom passed attention checks, rated MEA's explanations on vision items (1–5 scale). Text consistency with the highlighted region scored 4.07 and clarity scored 4.41. We focused on vision because non-expert raters can judge it reliably. Tabular items require domain knowledge (socio-economic, healthcare), and text items are long passages with token-level highlights. A study with domain experts did not fit in the rebuttal window.

Together, these results show that MEA's natural-language explanations are grounded in the same verified evidence as its structured output, and that RL training reduces the incorrect rationales the reviewer was concerned about rather than introducing them.

We will add both evaluations to the appendix. We will also state explicitly that the perturbation-based faithfulness guarantee applies to the structured output, and report text grounding separately.

Thank you for supporting our work. We hope these clarifications and additional experiments address the reviewer's concerns, and we would be glad to discuss any remaining questions.

[1] T. Lüddecke and A. Ecker. Image Segmentation Using Text and Image Prompts. CVPR, 2022.

[2] D. Slack, S. Krishna, H. Lakkaraju, and S. Singh. Explaining Machine Learning Models with Interactive Natural Language Conversations Using TalkToModel. Nature Machine Intelligence, 2023.

[3] S. Vanbrabant, G. Eerlings, G. A. Rovelo Ruiz, and D. Vanacken. ECHO: Enhancing Conversational Explainable AI through Tool-Augmented Language Models. Proc. ACM Hum.-Comput. Interact., 9(4), 2025.

[4] J. Kim, S. Mun, S. Lee, J. Cho, and J. Ok. Towards Faithful Agentic XAI. OpenReview, 2025.

[5] K. Rawal, Z. Fu, E. Delaney, and C. Russell. Evaluating Model Explanations without Ground Truth. FAccT, 2025.

[6] N. Kroeger, D. Ley, S. Krishna, C. Agarwal, and H. Lakkaraju. In-Context Explainers: Harnessing LLMs for Explaining Black Box Models. arXiv:2310.05797, 2024.

We thank all reviewers for their insightful comments and helpful suggestions.

We have responded to each concern in detail and uploaded a revised manuscript that includes the new experiments, clarifications, and corrections.

We would appreciate it if the reviewers could take another look at our responses and let us know whether their concerns have been resolved. If so, we would respectfully appreciate their support for our work and their kind consideration of increasing their review scores accordingly.

Thank you again for your time and effort.

Comments on the paper

3 comments
AuthorsOpenReview ↗

Request to respond to the updated PDF and rebuttal responses

As the author-reviewer discussion period is coming to a close, we wanted to gently check in and see if you have had a chance to look at our rebuttal and revised PDF.

We have carefully addressed all your concerns and conducted additional experiments. If our clarifications have resolved your questions, we would be immensely grateful if you could kindly share your feedback and consider updating your score before the deadline.

If you have any questions, please let us know so we can address them. Thank you so much for your invaluable time and dedication to reviewing our work!

AuthorsOpenReview ↗

Request to respond to the updated PDF and rebuttal responses

As the author-reviewer discussion period is coming to a close, we wanted to gently check in and see if you have had a chance to look at our rebuttal and revised PDF.

We have carefully addressed all your concerns and conducted additional experiments. If our clarifications have resolved your questions, we would be immensely grateful if you could kindly share your feedback and consider updating your score before the deadline.

If you have any questions, please let us know so we can address them. Thank you so much for your invaluable time and dedication to reviewing our work!

As the discussion period is approaching its deadline, we would like to kindly follow up again on our rebuttal and revised manuscript.

We have carefully addressed all your concerns, conducted 14 additional experiments and one further analysis, and incorporated the findings and clarifications into our revised manuscript. We sincerely hope these substantial efforts have addressed the issues raised in your review.

We would greatly appreciate it if you could share your feedback before the discussion closes. If our responses have resolved your concerns, we would be grateful if you could reconsider your initial assessment and update your score accordingly.

Thank you again for your time and consideration. We remain happy to address any outstanding questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.