acceptodds
Under review as a conference paper at ICLR 2027

CALIBRATING LLM EXPLANATIONS TO HUMAN DECISIONS WITH REINFORCEMENT LEARNING

Abstract

Explanations of LLM outputs are considered an important way to make model behavior legible, helping human users make good decisions on downstream tasks. However, default training approaches for improving explanations target model task performance or human preference rather than human decision quality. To bridge this gap, we introduce HD-RL, a reinforcement learning framework for optimizing LLM explanations for human decision making. Our approach calibrates a human decision simulator on observed human decisions, applying a proxy reward model to optimize the LLM to generate explanations that maximize human decision-makers' utility. We apply this framework to AI-generated text detection. While state-of-the-art closed-source LLMs can perform this detection autonomously with an average accuracy of 76.4%, the explanations they generate for users yield near-random human accuracy (51.3%). By applying HD-RL framework to the open-source Qwen 3.6 model, we generate optimized explanations that raise human detection accuracy to over 65.6%, outperforming RLVR, RLAIF, and other baselines by over 10%.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Calibration, decision making, LLM, reinforcement learning

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_JjYUOpenReview ↗
Rating
4(+2)
Confidence
4
Informativeness
3(+1)
Presentation
2
Soundness
3(+1)
Read review
Paper summary

Explanations are usually provided to human users, but are often optimized for higher task performance by AI models or are preferred by human users. This paper shifts the target of explanations to help people make better decisions. They used an AI-human writing "Turing test" as the task to test whether LLM explanations generally help humans improve their performance on the task and whether they can train them to be better at helping humans. The authors find that humans with LLM-generated explanations can only do the task at 51.3% accuracy, while after applying training human decision simulators and also using GRPO on Qwen to optimize decision quality (which are likely to be used by humans, as indicated by the decision simulator), human performance can improve to around 65% accuracy. This suggests an actual improvement of the explanations that help humans make better decisions. But comparing different training baselines, the authors confirm the necessity of those explanations, as well as the human decision simulator.

Critical strengths

I like the general idea of improving human decision quality by training the explanations, rather than merely making AI models better at tasks or more preferred by humans. I think there are several critical strengths:

  • Real human participants are involved in the paper, not just human simulators or virtual participants.
  • A loop of the intervention. The authors show the performance of training by observing actual performance improvement by humans.
  • Several baselines and "ablations" in the training process show the necessity of using explanations and human decision simulators during training guidance.
  • The human decision simulator design is nice since it can help efficiently score the rollouts during RL post-training of Qwen rather than having human data to label online.
Critical weaknesses

The weaknesses are also significant in this paper. I will start with my overall impression and then move to specific points.

  • Overall experimental logic. When I read the paper, I found the experimental setup somewhat hard to follow. This is not because the authors did not describe the steps clearly, but because the logic connecting them has several gaps. When I try to follow a consistent axis, it is difficult to understand the designs, their rationales, and what we should expect from each step. The distinction in the introduction is already clear: an explanation that improves an LLM’s task performance or is preferred by users does not necessarily help humans make better decisions. Therefore, the project should distinguish two axes: how useful the explanations are for AI models performing the task, and how effectively the information in these explanations can be transferred to humans (quality × transferability). With these two axes, many experimental components could be placed and understood more clearly. This would also make the missing pieces in the experimental setup apparent, as discussed below.

  • Explanation quality. There are several missing baselines. For example, the authors use GPT, Gemini, and Claude to generate explanations before presenting them to humans, but do not report how well these models can use their own explanations to support decisions on this task. A related result is that Qwen achieves approximately 76% accuracy when using these explanations. However, without a text-only baseline for Qwen, we do not know whether the explanations contribute to this performance. This matters because, when humans receive these explanations and achieve only 51.3% accuracy, is it because the explanations are poor and miss the relevant signals, or because they contain useful information that humans cannot efficiently use? Without addressing this question, we do not know what the authors are “calibrating”: the quality of the explanations, their transferability, or both.

  • Explanation transferability and the human behavioral baseline. Human participants begin the experiment with explanations, so we do not know how well they perform without any explanations. This baseline would establish whether the initial explanations help humans, have no effect, or even mislead them. For example, suppose humans originally achieve 65% accuracy, the initial explanations reduce performance to 51.3%, and the RL framework brings it back to approximately 65%. These numbers come from different experimental stages in the paper, so this is a hypothetical scenario rather than an established trajectory. Nevertheless, the current design cannot rule it out. In that case, the framework improves performance relative to the initial explanations, but why do we need it in practice? Why not simply let humans perform the task without explanations? The same need for a no-explanation baseline applies to Qwen.

  • The added value of the calibration framework. The lack of a clear separation between these axes, together with the missing baselines, makes me skeptical about whether the calibration is sufficient and necessary. There may be easier ways to achieve the goal. For example, can prompting optimization help state-of-the-art LLMs generate human-friendly explanations without RL training? Could some historical human decision data be included in context to ask an LLM to adapt its explanations to the target population, or to a specific user when individual history is available? I do not require the authors to evaluate every baseline or alternative approach. The key point is to establish a reasonably clear understanding of why the method works, what it calibrates, and whether the additional training is needed to obtain the improvement. The evidence should show that the framework improves human decisions beyond reasonable baselines, rather than merely restoring performance that the initial explanations had degraded.

  • Missing related work. Several highly relevant papers from AI and behavioral/cognitive modeling are not cited or discussed. These works provide important context for both the human decision simulator and the use of RL to train LLMs with human behavioral data:

    • Binz, M., & Schulz, E. (2024). Turning large language models into cognitive models. International Conference on Learning Representations.

      This paper presents a closely related approach to adapting LLM representations to predict human behavior. The authors should discuss how their human decision simulator relates to this approach and clarify the methodological differences.

    • Binz, M., Akata, E., Bethge, M., Brändle, F., Callaway, F., Coda-Forno, J., ... & Schulz, E. (2025). A foundation model to predict and capture human cognition. Nature, 644(8078), 1002–1009.

      Although the approach is less directly similar, this work provides important context for training language models to predict human behavior and should be discussed when positioning the human decision simulator.

    • Zhu, J. Q., Xie, H., Arumugam, D., Wilson, R., & Griffiths, T. L. (2026). Using reinforcement learning to train large language models to explain human decisions. International Conference on Learning Representations.

      This paper also uses RL post-training with human behavioral data to train LLMs to explain decisions. Its objective differs from the present paper: explaining human decisions versus generating explanations that improve human decisions. Nevertheless, the methodological connection is relevant and should be discussed.

Questions for authors

I tried to make a clearer list of questions (as an extension of the weakness statement) so that the authors can potentially respond during rebuttal:

  1. Human performance without explanations. How accurately do participants perform the task without any explanations, under the same experimental conditions? This would establish whether HD-RL provides a net benefit over independent judgment, rather than improving upon explanations that may themselves reduce performance.

  2. Explanations from untrained Qwen. How well do participants perform with explanations generated by the original Qwen model using the same prompt and format? This would isolate the benefit of post-training within the final randomized experiment.

  3. Human-oriented prompting. Could an untrained model achieve comparable results when explicitly prompted to generate explanations that help non-expert humans make accurate decisions? A reasonable prompting baseline would help establish the added value of the behavioral training pipeline.

I do not expect an exhaustive comparison against all possible alternatives. These comparisons would address the main uncertainty: whether the proposed framework improves human decisions beyond independent judgment and straightforward explanation-generation approaches.

7 comments
AuthorsOpenReview ↗

Response to Reviewer JjYU (Comment 1)

We thank the reviewer for your positive feedback and insightful suggestions.

1. Overall experimental logic

Thank you for the suggestions. We agree that our initial exposition was not as clear as it should have been, and have revised since the submission deadline to clarify the motivation behind each decision we made. We have also conducted additional experiments to fill out the missing comparisons, and discuss your two proposed axes. Specifically, we evaluate Axis 1 (how useful the explanations are for AI models) in our response to Comment 2 ("Explanation quality") below, and we evaluate Axis 2 (how effectively the explanations transfer to humans) in our responses to Comment 3 ("Explanation transferability") and Comment 4 ("The added value of the calibration framework") below. We will update Section 5 in the revised manuscript.

AuthorsOpenReview ↗

Response to Reviewer JjYU (Comment 2)

2. Explanation quality

Thank you for raising this question. We ran both baselines you suggested:

(1) Self-Evaluation, where GPT, Claude, and Gemini classify texts using their own generated explanations.

ModelOverallHuman TextsAI Texts
GPT 5.490.7%97.5%83.9%
Claude Opus 4.589.1%95.8%82.4%
Gemini 3.183.0%91.9%74.2%

(2) Text-Only baseline for Qwen 3.6-27B, where the model only sees the raw text.

ModelOverallHuman TextsAI Texts
Qwen (Text-Only)70.9%82.7%59.2%
Qwen (Text + Unoptimized Explanation)76.4%86.5%66.4%

Going back to your question: "when humans receive these explanations and achieve only 51.3% accuracy, is it because the explanations are poor and miss the relevant signals, or because they contain useful information that humans cannot efficiently use?" The above results show that the explanations do contain useful signal that Qwen can use to boost it’s score. However, Qwen also starts with greater overall accuracy on the task than humans. So it is likely both humans having worse unaided performance and not extracting all the available information from the explanations.

As shown in baseline 2, on the raw text alone, Qwen already achieves 71% accuracy. Adding the unoptimized explanations increases Qwen's accuracy to 76%. However, when humans read these same explanations, they achieve only 51% accuracy, suggesting these signals are difficult to transfer to humans.

This helps motivate our method of calibrating explanations to human decisions. This process is a way to learn which signals in the unoptimized explanations actually help human readers and which signals mislead them.

AuthorsOpenReview ↗

Response to Reviewer JjYU (Comment 3)

3. Explanation transferability and the human behavioral baseline

Thank you for pointing this out. To address your concern, we evaluated two additional human baselines on the held-out test set:

  1. Text-only: Human participants see only the text without an explanation.
  2. Raw Qwen: Human participants see the text and an explanation generated by the untrained Qwen 3.6-27B model.

As shown in the Table 1 below, the Text-only baseline reaches 50.4% GLMM-estimated mean human accuracy, with 55.2% on human-written texts and 45.6% on AI-generated texts. This rules out the possibility that human participants already achieve high accuracy on this task without explanations.

One interesting observation is that providing untrained Qwen explanations barely changes aggregate accuracy (50.4% to 51.3%), but shifts human decisions toward the "Human" label (increasing accuracy on human texts from 55.2% to 60.0% while reducing accuracy on AI texts from 45.6% to 42.5%). This observation connects with our Section 6.3 that the baselines in our main draft (LLM_decision, DPO-RLAIF, and VR-GRPO) often highlight cues that imply human authorship even on AI texts. Raw Qwen exhibits this same problem.

Table 1

MethodAggregate Acc. (95% CI)Human Text Acc. (95% CI)AI Text Acc. (95% CI)
HD-RL (Ours)63.7% [60.4%, 66.8%]66.6% [62.8%, 70.1%]60.7% [56.6%, 64.6%]
Text-only50.4% [45.9%, 55.0%]55.2% [49.8%, 60.5%]45.6% [40.2%, 51.1%]
Raw Qwen51.3% [46.7%, 55.8%]60.0% [54.7%, 65.0%]42.5% [37.2%, 47.9%]
Reviewer_reyoOpenReview ↗
Rating
3(+2)
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

In this work, the authors attempt to optimize LLM generated explanations to improve human decision accuracy. To this end, they introduce HD-RL, a framework for aligning explanations with human decisions. They first train a human decision simulator, which attempts to simulate the probability distribution of human users when predicting labels given an explanation and an input sample. They use the simulator's predicted probability that a human makes the correct decision, transformed using a logarithmic surrogate, as the reward in their HD-RL framework. They evaluate their framework against a few baselines on an AI-generated text detection task, demonstrating that their framework achieves a GLMM-estimated human accuracy of 63.7%.

Critical strengths

The idea of directly optimizing explanations for their downstream utility in improving human decision-making is useful, although the general approach of directly optimizing explanations for a target notion of faithfulness or utility has been explored in prior work.

Critical weaknesses
  • Limited Novelty: The core idea of directly optimizing LLM-generated explanations is highly similar to Malladi and Srivastava (2026) [1]. In their work, the authors move beyond conventional explanation-generation objectives by directly optimizing explanations using GRPO to improve downstream utility, measured by the accuracy of users classifying samples based solely on the explanations. Both works share the motivation of making explanations more useful and actionable by optimizing them for downstream objectives, specifically measures of user accuracy in classification tasks. While this work introduces a human decision simulator trained on observed human responses and uses its predictions as a reinforcement learning reward to improve human decision accuracy, the fundamental idea of optimizing explanations based on their impact on downstream human decisions has already been explored in OPeX. Consequently, the paper's conceptual novelty appears limited.

  • Limited Experiments/Evaluation: The authors train and evaluate HD-RL exclusively on a single task, AI-generated text detection. No experiments are conducted on other tasks or domains to demonstrate how well their framework generalizes beyond this specific setting. The evaluation is narrow and provides limited evidence for the broader applicability of the approach. Given that the central claim is about improving the utility of LLM-generated explanations for human decision-making, it is important to assess whether the framework generalizes across different tasks, datasets, and decision-making contexts. The authors also experiment only with a single model. It would be useful to see how well their pipeline applies to different models across different families and sizes.

  • Train and Evaluation on the same task: The authors train and evaluate on the same task, which is acceptable. However, the usefulness of HD-RL could be more if the authors demonstrate that the explanation capabilities learned by the policy during the HD-RL procedure can transfer/generalize to new/different tasks.

  • Choice of a classifier-based human simulator: It is unclear why a classifier-based human simulator is used. A classifier cannot always faithfully replicate the decisions of human users on new unseen samples. Another concern with using classifiers as the reward model is reward hacking. All of the classifiers in the ensemble receive the same Qwen representation as input, so an explanation that exploits this shared representation can fool most of them at once in some cases. Maybe a strong language model could be used as the reward/judge model here. To prevent information leakage from the judge/reward model when making predictions, the authors could have considered using tasks not included in the training data of the reward/judge model, or synthetic classification tasks that are definitely not included in it.

  • Weak Baselines: Several baselines appear weak by construction which makes it difficult to understand the gains of HD-RL. I also believe a few simple yet important baselines are missing.

    • Flaw with DPO-RLAIF: In this baseline, the judge is only shown the text and the two explanations without the ground-truth label. In this task, an explanation is useful only if it pushes the reader toward the correct answer, so a judge that does not know the correct answer has little basis for deciding which explanation is better. The judge bias reported in Appendix C.4 (human-focused explanations preferred even on AI texts) is therefore likely a consequence of this design and not evidence of a limitation of preference-based optimization. It is very hard for a judge to understand the nuances of a classification task without seeing a few annotated samples. A fairer version would provide the label to the judge.
    • Flaw with the LLM Decision baseline: It is unclear why a simulator is trained on the synthetic LLM decisions instead of using the LLM’s decisions directly as the reward. A trained simulator can only approximate the LLM’s behavior, which adds an extra source of error. This is not ideal as the purpose of this baseline is to test LLM decisions as a substitute for human ones. If the authors had a specific reason for this design (e.g., cost or information leakage), it should be stated, as there are potential workarounds.
    • Missing baselines with task knowledge: None of the chosen baselines provide the model with labeled examples from the task. This could lead to a large improvement in scores, as this would help the model understand task-specific nuances and how the labels are specifically designed for this task. Simple training-free baselines which are missing here are: a) few-shot prompting with labeled (text, label) pairs from the training split. To be specific, these are text examples marked as AI-generated or human-generated; b) CoT-based models (e.g. DeepSeek-R1) coupled with these few-shot examples. Notably, the few-shot examples in the explanation generation prompt (Appendix B) are text-explanation pairs. These examples do not provide any information about what AI vs Human generated samples would look like. An untrained Qwen 3.6 condition with the same prompt should also be included as a reference point.

References

[1] Advaith Malladi and Shashank Srivastava. Directly Optimizing Natural Language Explanations for Behavioral Faithfulness: Simulatability and Recoverability. In Proceedings of the 43rd International Conference on Machine Learning (ICML), 2026. https://icml.cc/virtual/2026/poster/65940

Questions for authors

Please see weaknesses above and clarify any misunderstanding I may have had.

9 comments
AuthorsOpenReview ↗

Response to Reviewer reyo (Comment 1)

We thank the reviewer for the encouraging words and helpful comments.

1. Limited Novelty

Thank you for pointing us to the OPeX paper (Malladi and Srivastava, 2026) [1], which we will cite in our revised Related Work. However, we want to point out that OPeX and HD-RL address different problems and rely on opposite assumptions during training.

OPeX optimizes explanations for behavioral faithfulness using a zero-shot LLM as a proxy of a user to compute rewards during GRPO, without collecting any human decisions for training. Thus, OPeX assumes a zero-shot LLM can mimic human users during optimization. HD-RL relies on real human decision data to train the policy model in reinforcement learning.

We also want to note that a central finding of our paper is that LLM proxies are a poor substitute for real human behavior. We explicitly tested Malladi and Srivastava's assumption via our LLM_decision baseline, where we use an LLM to simulate human decisions and train Qwen via GRPO against those synthetic decisions. Our results show that this baseline fails to generate better explanations than other baselines, with assisted human accuracy at only 55.1% (95% CI: [50.9%, 59.2%]), 8 percentage points lower than HD-RL framework.

[1] Advaith Malladi and Shashank Srivastava. Directly Optimizing Natural Language Explanations for Behavioral Faithfulness: Simulatability and Recoverability. In Proceedings of the 43rd International Conference on Machine Learning (ICML), 2026. https://icml.cc/virtual/2026/poster/65940

AuthorsOpenReview ↗

Response to Reviewer reyo (Comment 2)

2. Limited Experiments/Evaluation

Thank you for raising this. HD-RL is trained on real human decisions rather than synthetic LLM proxies. Collecting human behavioral data is time-consuming and expensive, and releasing a large human decision dataset is one of our contributions.

To better answer your question, we applied our pipeline to a second task (which we didn’t include because of page limits), Political Speech Party Identification. We sampled 2,000 speeches (1,000 Democratic, 1,000 Republican) between 50 and 250 words from the Stanford Congressional Record dataset [2]. Following the pipeline from our main study, we partitioned the 2,000 speeches at the speech level into 60% (1,200 speeches) for training the human decision simulator, 25% (500 speeches) for GRPO policy optimization, and 15% (300 held-out speeches) for final human evaluation. For the simulator training split, we generated candidate explanations using GPT, Claude, and Gemini under the same template constraints ("Look for...") and collected over 15K human decisions from 2,413 participants (an average of 6.47 trials per participant). On held-out speeches, the trained decision simulator achieved 70.3% accuracy and a Log Loss of 0.52. We then used the frozen simulator to train our HD-RL policy via GRPO.

On held-out test speeches, we evaluated the HD-RL against:

  1. a Text-Only baseline, where participants read the speech without an explanation

  2. Human-Oriented Few-Shot Prompting (Human-ICL), where Qwen 3.6 27B is explicitly prompted to help non-expert human readers identify the speaker's party and is given labeled in-context speeches (Ground Truth: Republican and Ground Truth: Democrat) paired with historical explanations that led to high (100%) versus low (20%) empirical human accuracy:

Task: Generate an explanation that helps a non-expert human reader accurately decide whether the provided speech was delivered by a Democratic or Republican speaker.

Human-Centered Guidelines:
- Non-expert readers rely on the features you point out to make their final decision.
- Highlight clear, verifiable diagnostic signals that genuinely reflect the speaker's party affiliation.

Constraints:
1. Start the response IMMEDIATELY with the bulleted list. 
2. Do NOT provide an introduction, a final verdict, or a summary.
3. Do NOT group the bullet points into sub-headings (keep it a flat list).
4. Every bullet point MUST follow one of these two exact templates:
    - "Look for [linguistic category, policy focus, or type of phrasing]."
    - "Look for [linguistic category, policy focus, or type of phrasing] like '[short example]'."
5. When providing examples, you may list multiple words separated by commas (e.g., like "word 1", "word 2").
6. Do NOT use bolding, italics, or any special Markdown formatting.
7. Limit the output to exactly 3 bullet points.

[Example 1 - Ground Truth: Republican]
Speech: "{train_rep_speech_1}"
- Helpful Explanation (Human Accuracy: 100%): {good_exp_rep_1}
- Misleading Explanation (Human Accuracy: 20%): {bad_exp_rep_1}

[Example 2 - Ground Truth: Democrat]
Speech: "{train_dem_speech_1}"
- Helpful Explanation (Human Accuracy: 100%): {good_exp_dem_1}
- Misleading Explanation (Human Accuracy: 20%): {bad_exp_dem_1}

Speech:
{text}

Can you provide an explanation that tells the reader what to focus on?

Explanation:

To account for repeated decisions by participants and variation in speech difficulty, we fit the same logistic GLMM specification used in our main study:

logit⁡Pr⁡(Zij=1)=β0+βci+βRepyj+δciyj+βttijc+ui+vj\operatorname{logit}\Pr(Z_{ij}=1) = \beta_0 + \beta_{c_i} + \beta_{\mathrm{Rep}} y_j + \delta_{c_i} y_j + \beta_t t_{ij}^{c} + u_i + v_j

where Zij=1(aij=yj)Z_{ij}=\mathbb{1}(a_{ij}=y_j) indicates whether participant ii correctly classified speech jj, cic_i is the assigned condition, yjy_j is the ground-truth party, δci\delta_{c_i} is the condition-by-party interaction, tijct_{ij}^{c} is the mean-centered trial number, ui∼N(0,σp2)u_i \sim \mathcal{N}(0, \sigma_p^2) is a nested random intercept for each participant, and vj∼N(0,σs2)v_j \sim \mathcal{N}(0, \sigma_s^2) is a crossed random intercept for each of the 300 held-out speeches.

As shown in the Table 1 below, political speech identification without any explanation (Text-only baseline) is difficult for humans, and only achieved 47.2% aggregate GLMM estimated accuracy. While providing human participants with Human ICL explanations increases GLMM estimated accuracy to 54.6%, HD-RL explanation increases the accuracy to 67.2%. This result shows that HD-RL can generalize to political speech detection too. We will add this experiment result in our revised version.

Table 1

Aggregate Acc. (95% CI)Democrat Acc. (95% CI)Republican Acc. (95% CI)
HD-RL67.2% [63.2%, 70.8%]64.8% [59.5%, 69.7%]69.5% [64.5%, 74.0%]
Human ICL54.6% [49.9%, 59.1%]52.8% [46.6%, 58.8%]56.4% [50.3%, 62.3%]
Text-only47.2% [42.4%, 52.0%]48.1% [41.8%, 54.4%]46.3% [40.0%, 52.6%]
AuthorsOpenReview ↗

Response to Reviewer reyo (Comment 3)

3. Train and Evaluation on the same task

Thank you for this comment. We agree that studying how explanation policies transfer across other tasks is a valuable direction for future work. We expect that the effectiveness of HD-RL across tasks will depend on the extent to which human priors and failure cases are shared between those tasks.

For example, in AI text detection, our analysis shows that humans achieve higher accuracy on AI-generated text when its explanation contains words like "balanced" and “smooth", and achieve lower accuracy when its explanation contains words like "imperfection" and "informal". A different task, like political speech detection, will likely have different features that mislead or aid human judgment. Whether or not calibrated explanations transfer to other tasks should be approached on a task-by-task basis in future work.

Reviewer_EGuvOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a method for training an LLM to produce "explanations" which aid human decision making. The proposed method, HD-RL, trains a human decision simulator to estimate what action a human might take given an explanation. This is then used to generate rewards and train an LLM to produce explanations that result in helpful human decisions. The method is evaluated over an AI text identification task.

Critical strengths
  • Though I think the explanation framing is not particularly useful (see weakness below), the task of decision support is an important one and the proposed solution seems to be fairly effective at aiding human decision making.

  • There is a strong human participant study showing that the proposed method yields more accurate aggregate decision scores than competing baselines in the AI text identification task.

  • The ablation with the LLM synthetic decisions points at an interesting phenomenon which is further evidence that LLM-as-a-judge type schemes are not always useful in replicating human decisions, particularly when they depend on human-specific behavioral cues.

Critical weaknesses
  • I feel like there is some conceptual ambiguity around the "explanation" framing. Typically an explanation is used to explain why a particular model produced a particular prediction, even the introduction of this paper frames it this way. However, the actual methodology and experiments do not attempt to explain a particular decision. Rather it seems more like a piece of information intended to aid human decision making (i.e. decision support). So to me, it seems the explanation framing is not relevant here and potentially confusing to the reader.

  • Training a decision simulator may be difficult to on more complex decision support tasks. While it appears to work well on a binary classification task such as AI text identification, it's not immediately obvious to me that this is capable of scaling. Furthermore, this method relies on the assumption that the decision simulator is accurate and reliable. Depending on how large the training data set is, it's possible that the LLM ends up generating OOD explanations during the RL process and exploits spurious correlations in the decision simulator.

Questions for authors
  1. Can you discuss whether the explanation framing is appropriate? Maybe I am misinterpreting something with the paper. What exactly is being explained?
4 comments
AuthorsOpenReview ↗

Response to Reviewer EGuv

We thank the reviewer for the thoughtful comments and discussion.

1. I feel like there is some conceptual ambiguity around the "explanation" framing.

Thank you for raising this point. Merriam Webster defines an explanation as “a statement, piece of writing, or set of details that makes an idea, situation, or event clear and easy to understand.” Our use of the term explanation fits this definition. We respectfully disagree that we should restrict use of “explanation” in ML to only cases where the model also outputs a prediction. Explanations take many forms in practice, and we should work toward improving LLMs’ abilities to construct all of them. Our view is also in line with other views espoused in the explainable AI literature, such as Miller’s (2023) Evaluative AI framework [1], which explicitly recommends that AI explanations only point out evidence in its output and leave the final decision to the human. (Miller, 2023)

However, we also agree with you that our wording in Introduction may cause confusion. We will clarify our perspective on explanation in the revised version.

[1] Miller, Tim. "Explainable ai is dead, long live explainable ai! hypothesis-driven decision support using evaluative ai." Proceedings of the 2023 ACM conference on fairness, accountability, and transparency. 2023.

AuthorsOpenReview ↗

Response to Reviewer EGuv (Continued)

2. Training a decision simulator may be difficult to on more complex decision support tasks.

Thank you for your comment. Extending HD-RL to multi-class decision problems is an important direction for our future work. Right now, the primary bottleneck in scaling to more complex decision spaces is the cost of collecting enough human trials to cover a larger action space.

Regarding the concern about generating OOD explanations during RL, we ran several analyses as robustness checks, which we will add in the revision. We first tested whether the simulator can distinguish how different explanations shift human decisions when the source text is held constant. For each held-out text xix_i with ground-truth label yi∗y^{\ast}_i and explanations (ei,1,ei,2)(e_{i,1}, e_{i,2}), we first evaluated pairwise ranking accuracy across all held-out pairs P\mathcal{P} by comparing the empirical human accuracies (ui,1,ui,2)(u_{i,1}, u_{i,2}) against the simulator's predicted correct-decision probabilities u^i,k=Sϕ(a=yi∗∣xi,ei,k)\hat{u}_{i,k} = S_\phi(a = y^{\ast}_i \mid x_i, e_{i,k}), assigning 0.50.5 credit when observed human accuracies were tied (ui,1=ui,2u_{i,1} = u_{i,2}):

Accpair=1∣P∣∑(xi,ei,1,ei,2)∈P[I((ui,1−ui,2)(u^i,1−u^i,2)>0)+0.5⋅I(ui,1=ui,2)]\text{Acc}_{\text{pair}} = \frac{1}{|\mathcal{P}|} \sum_{(x_i, e_{i,1}, e_{i,2}) \in \mathcal{P}} \left[ \mathbb{I}\big((u_{i,1} - u_{i,2})(\hat{u}_{i,1} - \hat{u}_{i,2}) \gt 0\big) + 0.5 \cdot \mathbb{I}(u_{i,1} = u_{i,2}) \right]

Our simulator achieved a pairwise ranking accuracy of 64.1%64.1\% across all held-out pairs, rising to 72.4%72.4\% on pairs with an empirical accuracy gap of ∣ui,1−ui,2∣≥25%|u_{i,1} - u_{i,2}| \ge 25\%. We also calculated the observed difference in proportions of people choosing each decision between the two explanations, Δpi=pi,1−pi,2\Delta p_i = p_{i,1} - p_{i,2}, and compared it against the simulator-predicted difference in proportions, Δp^i=p^i,1−p^i,2\Delta \hat{p}_i = \hat{p}_{i,1} - \hat{p}_{i,2}. Across all held-out pairs, the signed error (Δpi−Δp^i\Delta p_i - \Delta \hat{p}_i) was centered near zero (mean +0.004+0.004) with a Mean Absolute Error (MAEΔ=1∣P∣∑∣Δpi−Δp^i∣\text{MAE}_\Delta = \frac{1}{|\mathcal{P}|} \sum |\Delta p_i - \Delta \hat{p}_i|) of 0.210.21. This outperforms both a baseline that always predicts equal proportions (Δp^i=0\Delta \hat{p}_i = 0; MAEΔ=0.25\text{MAE}_\Delta = 0.25) and a baseline that always predicts the most common non-zero difference across explanations (Δmode=0.20\Delta_{\text{mode}} = 0.20; MAEΔ=0.27\text{MAE}_\Delta = 0.27).

Because the decision simulator was trained on human decisions with unoptimized explanations, the explanation policy will shift its distribution after post-training. Evaluating the simulator on explanations generated by our four experimental conditions (HD-RL, LLM_decision, DPO-RLAIF, and VR-GRPO) tests whether the simulator generalizes under policy distribution shift. Across all 7,4557,455 deployment trials, we compared the frozen simulator's predictions against the observed participant decisions. Formally, let Tc,y\mathcal{T}_{c,y} denote the set of evaluation trials in condition c∈Cc \in \mathcal{C} with ground-truth label y∈{AI,Human}y \in \lbrace \text{AI}, \text{Human} \rbrace. For each of the ∣C∣×2=8|\mathcal{C}| \times 2 = 8 condition-by-class cells, we compute the observed proportion of trials in which participants selected the "Human" label, Pc,y(Human)P_{c,y}(\text{Human}), and the mean simulator-predicted probability of a "Human" selection, P^c,y(Human)\hat{P}_{c,y}(\text{Human}):

Pc,y(Human)=1∣Tc,y∣∑i∈Tc,yI(ai=Human),P^c,y(Human)=1∣Tc,y∣∑i∈Tc,ySϕ(a=Human∣xi,ei)P_{c,y}(\text{Human}) = \frac{1}{|\mathcal{T}_{c,y}|} \sum_{i \in \mathcal{T}_{c,y}} \mathbb{I}(a_i = \text{Human}), \qquad \hat{P}_{c,y}(\text{Human}) = \frac{1}{|\mathcal{T}_{c,y}|} \sum_{i \in \mathcal{T}_{c,y}} S_\phi(a = \text{Human} \mid x_i, e_i)

We calculate the discrepancy across all eight condition-by-class cells using the cell-level Mean Absolute Error (MAEcell\text{MAE}_{\text{cell}}):

MAEcell=18∑c∈C∑y∈{AI,Human}∣Pc,y(Human)−P^c,y(Human)∣\text{MAE}_{\text{cell}} = \frac{1}{8} \sum_{c \in \mathcal{C}} \sum_{y \in \lbrace \text{AI}, \text{Human} \rbrace} \left| P_{c,y}(\text{Human}) - \hat{P}_{c,y}(\text{Human}) \right|

As shown in the table below, our decision simulator closely tracks the empirical response distributions across all eight condition-by-class cells, achieving a cell-level mean absolute error of MAEcell=4.4%\text{MAE}_{\text{cell}} = 4.4\%. We will add both the within-text pairwise ranking and policy-shift evaluation to Section 4 of the revised paper to demonstrate that our simulator generalizes across explanation policies without reward hacking.

AuthorsOpenReview ↗

Table: Comparison of empirical human decisions and frozen simulator predictions across the 7,455-trial evaluation experiment, broken down by experimental condition and ground-truth class.

ConditionGround TruthTrials (NN)Empirical P(Human)P(\text{Human})Empirical P(AI)P(\text{AI})Empirical AccuracyPredicted P^(Human)\hat{P}(\text{Human})Predicted P^(AI)\hat{P}(\text{AI})Predicted Accuracy
HD-RLAI1,20037.0%63.0%63.0%30.6%69.4%69.4%
Human1,28868.0%32.0%68.0%72.9%27.1%72.9%
Overall2,48853.1%46.9%65.6%52.5%47.5%71.2%
LLM_decisionAI77050.9%49.1%49.1%53.2%46.8%46.8%
Human80261.2%38.8%61.2%63.1%36.9%63.1%
Overall1,57256.2%43.8%55.3%58.2%41.8%55.1%
DPO-RLAIFAI1,02954.3%45.7%45.7%59.7%40.3%40.3%
Human1,08457.7%42.3%57.7%61.8%38.2%61.8%
Overall2,11356.1%43.9%51.9%60.8%39.2%51.3%
VR-GRPOAI61352.7%47.3%47.3%56.9%43.1%43.1%
Human66959.0%41.0%59.0%53.4%46.6%53.4%
Overall1,28256.0%44.0%53.4%55.1%44.9%48.5%

Thanks again for your review. We are happy to answer any questions you might have on the new results we shared before the rebuttal period closes.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response to All Reviewers

We thank the reviewers for their thoughtful and constructive feedback. We are encouraged that the reviewers recognized the value of optimizing explanations to improve downstream human decision-making (Reviewers EGuv, reyo, JjYU); the strength of our closed-loop evaluation with significant amounts of data from real human participants (Reviewers EGuv, JjYU); and our ablation showing that LLM-as-a-judge proxies fail to replicate human decisions (Reviewers EGuv, JjYU).

We agree that several additional baselines should have been included in our evaluation section, and have run those since submission. Below, we clarify misunderstandings and detail how we plan to address the reviewers' suggestions into the revised manuscript.

Summary of our revisions:

  • We added four baselines on the AI Text Detection task: (1) a text-only baseline, (2) an untrained Qwen baseline, (3) an untrained Qwen baseline with a human-oriented prompt containing labeled in-context examples paired with historical explanations, and (4) a human preference RL baseline, where we collected 19K human pairwise preferences and optimized the explanation policy with a Bradley–Terry reward model.

  • We applied our framework to a new task, Political Speech Party Identification. We had previously collected 15K human decisions on congressional speeches (identifying whether the speaker is Republican or Democrat) using the same website, but neglected to include the demonstration in the paper due to lack of space. We trained and evaluated HD-RL against text-only and human-oriented prompting baselines.

  • We added within-text pairwise ranking and out-of-distribution policy shift evaluations to test our decision simulator's generalization to unseen texts and robustness against reward hacking.

  • We will revise Section 1 and Section 2 to clarify our explanation framing and discuss the related papers suggested by the reviewers.

Sincerely,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.