acceptodds
Withdrawn from ICLR 2027

A Benchmark with Known Causes: Can Language Models Explain ML and RL Decisions?

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

The quality of an explanation for an ML model or RL policy in a given situation is typically judged by readability or by alignment with another explainer. A more insightful test is whether the explanation identifies the cause of the action that the policy proposed. For such a test, we need decisions for which the cause is known. We build a benchmark of fifty such decisions: fifteen from tabular classifiers, where each feature’s contribution is exact, and thirty-five from three reinforcement learning agents (value iteration on a grid world, tabular Q-learning on cart-pole and an upper-confidence-bound contextual bandit), where we keep a case only if removing one candidate cause and re-running the agent changes the decision. Each case carries four probe situations answered by re-running the system, so the answer key is exact. A program scores faithfulness against the known cause with no language model involved, and a weaker reader from a different model family scores comprehension by predicting what the system does in situations it has not seen. The audience named in the prompt decides whether the cause survives: a plain-language request scores 0.327 on faithfulness against 0.668 for a technical one, one added instruction to name the deciding factor recovers half that gap for five points of reading ease, and across ten open-weight models from six families (1 − 42 gigabytes) the plain request never exceeds 0.42. In the model’s own next-word distribution, naming the audience alone lowers the probability of the true cause by 0.137 (p = 0.0005). A model judges how plainly a passage reads, agreeing with the formula on 97% of texts, however, it cannot count its words. Counting constraints therefore belong in code, while judgements of style can stay in the prompt. On the benchmark, a four-stage pipeline that fixes the cause before simplifying the language matches a single well-specified prompt on mean faithfulness (0.696 against 0.700); what it adds is that it checks the rules in code.

Primary area:
interpretability and explainable AI
Keywords:
LLMs, Explanations, Reinforcement Learning, Machine Learning, Agent decisions

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_o32GOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The study uses LLMs to generate explanations for decisions taking by any ML system, and how priming the model with how to generate the explanations (eg. simple vs complex/technical explanations) affects the quality of the output. To test this and make it measurable, the work uses tabular classifiers and RL agents as the ML systems whose decisions the LLM must explain, since it allows for direct interpretation of the models outputs as a metric to which the LLMs explanations can be measured. The work measures said quality using metrics such as faithfulness, predictive ability from the generated explanations (which they call comprehension), and readability and finds that asking for plain language explanations drops the faithfulness of the generated explanations from 0.668 to 0.327.

Critical strengths
  1. The idea and motivation are justified. LLMs are routinely used to interpret tool calls, many of which are themselves learned systems, which makes this line of work timely and relevant.
  2. The idea of benchmarking this explanation ability of LLMs using tasks which are natively interpretable for grounded evaluations is well posed.
Critical weaknesses
  1. The core claim depends entirely on faithfulness. The comprehension metric does not separate any of the conditions, and repeating the original decision already beats most methods, so I don't think it measures whether an explanation helps a reader at all. If anything, the plain language prompt scores slightly higher on it than the technical prompt, which goes against the core claim. Readability only reflects that the model followed the instruction to write simply, and says nothing about whether the cause survives. This leaves faithfulness as the only metric which supports the core claim, which I believe is itself flawed.

  2. The faithfulness scorer penalises paraphrasing. Since the scorer only accepts exact names or a fixed alias list, plain language explanations are penalised by design, as replacing technical names with everyday words is the point of plain language. The paper's own examples show this, where the plain explanations for the breast cancer, iris and cart-pole cases all name the correct cause in everyday words ("the cell's size", "a very long petal", "the stick is leaning") yet score 0. I believe the key finding is therefore largely an artifact of the scorer, which would also explain why the effect is much larger on tabular cases than on grid world cases. The "plain, but named" fix arguably just teaches the model to match the scorer. Without validating the scorer against human judgements, or at least an LLM judge given the known cause, I don't think the plain language results can be trusted.

  3. The benchmark is too small and too simple to support general conclusions. With only 40 main cases from logistic regression, depth three tree, Q-table, value iteration and bandit, I am not convinced the findings say much about explaining the models where explanations are actually needed. The small size also limits the statistical significance of the comparisons, which several of the null results already reflect.

Questions for authors
  • How many plain language explanations scored 0 in fact name the correct cause in different words, as judged by humans or an LLM judge given the known cause?
  • Could the probes be redesigned so that more of them impact the decision?
  • Do the findings hold on at least one deep network, for example using the planted shortcut protocol the authors cite?
  • What does FD-CR offer over a single prompt with the same rules?
Reviewer_LyLpOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper creates a benchmark to evaluate RL agents on explaining decision-making processes with exact ground-truth causes. The benchmarks include tabular classifiers and RL agents, and the exact causes are derived from the weights of the tabular classifiers and by performing interventions on the world the RL agents operate in. The benchmark computes both explanation faithfulness and readability by technical or non-technical readers. Experiments with different explanation prompts, the paper identifies pitfalls in LLM-generated explanations when the full list of rules on faithfulness and readability is not specified. The paper also proposes an attribution method, FD-CR, that explicitly checks if the explanation follows the listed rules, but finds that it does not significantly outperform just using a well-specified prompt.

Critical strengths
  1. The experiments include a diverse range of prompts that target different audiences (technical or non-technical) and either implicitly, explicitly, or comprehensively specify the requirements for faithful explanations.
Critical weaknesses
  1. The benchmark is quite small (200 samples), only includes readable systems by construction, and all systems are composed of a small number of concrete features, thus limiting the usefulness of the benchmark in testing LLM-generated explanations on realistic RL settings.
  2. The ground-truth causes are not computed exactly, but instead are subjected to arbitrary constraints, such as only the top 1 or 3 features are considered causes, only top features with a cumulative contribution of 60% are considered, or the goal state is deliberately removed as a feature.
Questions for authors
  1. [Prior work] The paper claims that existing evaluations of attribution treat one attribution method as the ground truth and only measure alignment against that method. However, prior work in feature attributions in traditional supervised learning has proposed independent metrics grounded in formal desirable properties of attributions [1,2]. In fact, in [1], the evaluation metric and the attribution method are two separate contributions. Could the authors update the Related Work section and elaborate on the value of their benchmark given these existing attribution evaluations?
  2. [Benchmark usefulness] What are the challenges in scaling the benchmark? Is it possible to make the dataset bigger and report whether the main findings in the paper stay the same on this larger dataset?
  3. [Benchmark usefulness] In Line 99, the paper states that the benchmark only includes “systems that are readable by construction”. Could the authors justify why we would be interested in explaining systems that are already readable? Similarly, in Line 431, the paper states that FD-CR should not be used when it is hard to identify the causes. Could the authors describe realistic settings where we would need an attribution method when the cause is already easy to identify?
  4. [Benchmark quality] In Line 158, could the authors justify why the true causes should be determined as the top 1 to 3 features whose total contribution reaches 60%?
  5. [Benchmark quality] In Line 172, could the authors justify why it is necessary to remove the goal state as a feature and a cause if models always accurately attribute the goal as a true cause? Is removal necessary due to the benchmark construction that only considers 1 to 3 top features as the true causes? Is it an indication that the benchmark construction should be modified, instead of removing the goal state? Similarly, in Line 419, does the fact that FD-CR chooses a single cause early on also indicate that the benchmark should not restrict the number of true causes to one?
  6. [Other clarification] In Section 3.3, could the authors briefly describe each setting in the benchmark in the main text, not just in the appendix?
  7. [Other clarification] In Line 199, could the authors provide verification that the list of trigger phrases can be considered a comprehensive list of trigger phrases that the LLMs could use or that humans would interpret as causal signals

References

[1] Jethani, Neil, et al. "Have we learned to explain?: How interpretability methods can learn to encode predictions in their interpretations." International Conference on Artificial Intelligence and Statistics. PMLR, 2021.

[2] Puli, Aahlad, Nhi Nguyen, and Rajesh Ranganath. "Explanations that reveal all through the definition of encoding." Advances in Neural Information Processing Systems 37 (2024): 99965-100006.

Reviewer_FHMXOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes a benchmark for explaining ML and RL policy outputs. The idea is to take a lot of tabular classification problems, and RL symbolic input policies, take attributions for the features, give those (and other variations) to an LLM, and ask it to parse the outputs into natural language. Results mostly indicate that plain prompting performs worst overall, with minimal differences between most other methods.

Critical strengths
  • In general, the idea of having a ground truth for causal outputs is quite nice, I know other papers have looked at similar things, but the exact fit of this paper—causal ground truths and CoT in a benchmark—hasn't been looked at before as far as I know.
  • Interesting finding: Plain-language prompting reduces measured faithfulness from 0.668 to 0.327, while explicitly instructing the LLM to preserve the decisive factor substantially improves performance.
Critical weaknesses
  • The benchmark is limited to extremely simple classification and RL problems. It is unclear if this would generalize to modern frontier models and policies where explanations are needed most.
  • The LLM is really just rehashing the prompt. This is really a benchmark for "how well can LLMs summarise explanations", I feel this would be incredibly sensitive to prompt design, which isn't explored enough I feel.
  • The evaluation uses predefined causal phrases and feature-name aliases, potentially penalizing reasonable paraphrases. For example, the paper assigns zero faithfulness to a breast-cancer explanation referring to cell size rather than the specific feature worst radius. This raises the possibility that measured faithfulness degradation reflects terminology simplification rather than genuine loss of causal information. The reason an LLM might simplify features names is because you're asking for just 120 token responses.
  • Other explanation methods like nearest neighbours or counterfactuals are not compared

I feel it would be a lot more interesting if the authors could evaluate explanation usefulness, for something like model improvement or trust calibration with humans.

Questions for authors

See weaknesses

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.