- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The study uses LLMs to generate explanations for decisions taking by any ML system, and how priming the model with how to generate the explanations (eg. simple vs complex/technical explanations) affects the quality of the output. To test this and make it measurable, the work uses tabular classifiers and RL agents as the ML systems whose decisions the LLM must explain, since it allows for direct interpretation of the models outputs as a metric to which the LLMs explanations can be measured. The work measures said quality using metrics such as faithfulness, predictive ability from the generated explanations (which they call comprehension), and readability and finds that asking for plain language explanations drops the faithfulness of the generated explanations from 0.668 to 0.327.
Critical strengths
- The idea and motivation are justified. LLMs are routinely used to interpret tool calls, many of which are themselves learned systems, which makes this line of work timely and relevant.
- The idea of benchmarking this explanation ability of LLMs using tasks which are natively interpretable for grounded evaluations is well posed.
Critical weaknesses
-
The core claim depends entirely on faithfulness. The comprehension metric does not separate any of the conditions, and repeating the original decision already beats most methods, so I don't think it measures whether an explanation helps a reader at all. If anything, the plain language prompt scores slightly higher on it than the technical prompt, which goes against the core claim. Readability only reflects that the model followed the instruction to write simply, and says nothing about whether the cause survives. This leaves faithfulness as the only metric which supports the core claim, which I believe is itself flawed.
-
The faithfulness scorer penalises paraphrasing. Since the scorer only accepts exact names or a fixed alias list, plain language explanations are penalised by design, as replacing technical names with everyday words is the point of plain language. The paper's own examples show this, where the plain explanations for the breast cancer, iris and cart-pole cases all name the correct cause in everyday words ("the cell's size", "a very long petal", "the stick is leaning") yet score 0. I believe the key finding is therefore largely an artifact of the scorer, which would also explain why the effect is much larger on tabular cases than on grid world cases. The "plain, but named" fix arguably just teaches the model to match the scorer. Without validating the scorer against human judgements, or at least an LLM judge given the known cause, I don't think the plain language results can be trusted.
-
The benchmark is too small and too simple to support general conclusions. With only 40 main cases from logistic regression, depth three tree, Q-table, value iteration and bandit, I am not convinced the findings say much about explaining the models where explanations are actually needed. The small size also limits the statistical significance of the comparisons, which several of the null results already reflect.
Questions for authors
- How many plain language explanations scored 0 in fact name the correct cause in different words, as judged by humans or an LLM judge given the known cause?
- Could the probes be redesigned so that more of them impact the decision?
- Do the findings hold on at least one deep network, for example using the planted shortcut protocol the authors cite?
- What does FD-CR offer over a single prompt with the same rules?