acceptodds
Under review as a conference paper at ICLR 2027

Should I Stay or Should I Show? Learning to Selectively Disclose Information

Abstract

In many high-stakes settings, human decision-makers can acquire support information before making a decision. However, acquiring information is costly, and disclosure may fail to improve human decisions or may even impair them. We tackle this problem by studying selective disclosure, i.e., the problem of learning when to reveal support information to a human decision-maker under a budget constraint. We first show that the optimal policy is a threshold rule on the Value of Information (VoI), i.e., the expected reduction in human decision risk induced by disclosure. Since VoI is unknown in practice, we estimate the regime-specific human risks and bound the possible degradation of the resulting plug-in policy relative to lack of disclosure, as well as its regret relative to the optimal policy. Experiments on benchmark datasets show that selective disclosure outperforms both no disclosure and full disclosure, regardless of whether the support information is beneficial or harmful. Two user studies show that human-AI team performance can improve when disclosure is led by our learned policy and not human-selected, although this advantage varies across tasks. A counterfactual benchmark, which replaces participants' predictions with a machine-learning prediction when disclosure occurs, suggests that these differences might depend on lower adherence to advice when the information is automatically provided rather than self-requested.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Human-AI Collaboration, Hybrid Decision-making

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_m13YOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a method to selectively provide support information to a human decision-maker. Theoretically, it formulates the problem as deciding whether to provide information based on the conditional value of information for reducing decision risk. As a result, the optimal disclosure rule is a threshold rule: disclosing in cases that have the largest value of information. The paper then experimentally evaluated the method on existing human judgment data and two new user studies.

Critical strengths
  • I like the paper’s formulation. Learning whether disclosure improves the final human decision seems to be a natural and the right formulation.
  • The paper combines formal analysis with experiments. I particularly appreciated the two user studies.
Critical weaknesses
  • The theoretical contribution, beyond modeling, seems to be limited. Once the objective R(d) is reached, the optimal policy follows rather straightforwardly. The novelty contribution on the theoretical side is arguably marginal. Moreover, the paper doesn’t establish how to obtain accurate value estimates, which is crucial for producing a good disclosure policy.

  • On the experimental side, the human-subject studies do not directly isolate the effect of allocation of advice. They compare human requesting advice with ClassWise automatically supplying it. As a result, the amount of advice provided in these two conditions is significantly different. In some sense, the experiments show the comparison of two different methods, but do not compare the impact of different allocation of advice. To me, a fair comparison is to keep the advice budget roughly the same and compare LSD with, for example, randomly selecting when to provide advice.

Questions for authors

See the second point of the weakness section.

4 comments
AuthorsOpenReview ↗

Rebuttal to Reviewer m13Y

We thank the reviewer for the feedback. We are glad the reviewer likes the paper’s formulation, finding that learning whether disclosure improves the final human decision seems to be a natural and the right formulation, and that the reviewer particularly appreciated the two user studies. We address both concerns below.

Novelty of the theoretical contribution. Our main theoretical contribution lies in the formalization of the problem. Once the correct formulation is provided, deriving our theoretical results does not require convoluted theoretical machinery. We see this as a strength of our proposal. Moreover, as we stressed in the main text (lines 170-184 of the newly uploaded PDF), selective disclosure differs from the canonical treatment-allocation setting and requires a dedicated causal formulation. To the best of our knowledge, LSD is the first framework that treats disclosure as a causal intervention on a human decision-maker's information and learns when to disclose based on its effect on decision loss, with identification conditions and guarantees for the resulting policy.

How to obtain accurate estimates. We thank the reviewer for this question, which allowed us to state the guarantees of the proposed estimator more precisely. Under the identification assumptions of Section 3.2, the VoI corresponds to a conditional average treatment effect (CATE), namely the difference between regime-specific conditional risks. Its estimation can therefore build on the causal inference literature, where consistency of standard meta-learners such as T- and S-learners is well studied [1, 2]. Our ClassWise estimator specializes the T-learner by exploiting the class-wise decomposition of human decision risk under the 0−10-1 loss. Since this structure departs from a standard T-learner, existing results do not apply directly. In the revised manuscript, we therefore prove that ClassWise inherits the same guarantee (new Proposition 3). Its VoI estimation error is bounded by the estimation errors of its two components, the label model ff and the class-wise error heads gDg_D. Hence, ClassWise yields a consistent VoI estimator whenever these components are consistent. Combined with our oracle regret bound (Theorem 3), this new proposition provides the sufficient conditions to make the regret of the resulting disclosure policy vanish as well. Empirically, ClassWise compares favourably against direct T- and S-learner baselines (Appendix E.1).

On the experimental side. We agree that the accuracy improvement may be driven primarily by more assistance rather than by better selecting the items for which advice would be useful. We note, however, that ClassWise is designed to address two related issues: which trials are most likely to benefit from advice and how many trials should receive it under a maximum budget. Even if the amount of advice were the main driver of the improvement in the ImageNet-16H task, the result would still have practical implications. In particular, it suggests that participants tend to request advice less often than would be beneficial, and that ClassWise can help close this gap.

More broadly, we view our comparison of human-selected and ClassWise-selected disclosure as a first step toward understanding how people behave within these two realistic workflows. Dedicated follow-up studies can build on these findings to isolate the contributions of individual factors to human-AI team accuracy, though designing such studies is far from straightforward. Indeed, we considered including a matched random-disclosure baseline like the one suggested by the reviewer. While it would have provided useful evidence on the relative contributions of selection quality and assistance quantity, we ultimately decided against including it because (i) it was less representative of realistic use cases and (ii) on its own, it would have left the mechanisms behind any accuracy difference between random and ClassWise disclosure unresolved.

For example, repeated exposure to advice perceived as unnecessary or unhelpful in the random condition could reduce participants’ willingness to follow subsequent advice. Any accuracy difference could therefore reflect how useful the advice was on the selected trials, how readily participants followed it, or both. Nevertheless, the reviewer’s comment prompted us to clarify the aim of the user studies in Sec. 4.1 and to discuss their limitations and directions for future research more explicitly in Sec. 6.

References

[1] Künzel, S. et al.. Metalearners for estimating heterogeneous treatment effects using machine learning. PNAS, 2019.

[2] Curth A. and van der Schaar M.. Nonparametric estimation of heterogeneous treatment effects: from theory to learning algorithms. AISTATS, 2021.

Dear Reviewer m13Y,

Thanks again for the feedback. As the author-discussion phase comes to an end, please let us know whether we have addressed your concerns or if there are any remaining points we can clarify.

Thank you for the response and revisions. Proposition 3 partially addresses my estimation concern, although its guarantee depends on the accuracy of the component estimators. My assessment of the limited theoretical contribution beyond the formulation is largely unchanged.

My main experimental concern also remains. Clarifying the studies’ scope is helpful, but does not resolve whether the improvement comes from better allocation or simply more advice. A matched-quantity random-disclosure baseline would provide useful evidence even if adherence differed between conditions.

Our main theoretical contribution is identifying the right formulation, which the reviewer also recognizes as natural and correct. We agree that once the problem is correctly formulated, the theoretical results do not require convoluted machinery. Simple derivations do not make the results weaker as they are rigorous. We therefore do not see this as a critical weakness.

Regarding the experimental evaluation, we respectfully disagree with the reviewer. For the purpose of the current study, i.e., identifying whether the policy can improve upon human self-selected disclosure, our experimental evaluation is sufficient to detect such an effect. We do not claim to separate better allocation from more advice. As stated in Sec. 6, we explicitly leave this to future dedicated studies. Moreover, as discussed in our previous rebuttal, a random condition would not, on its own, disentangle the drivers of the results, as other confounders (e.g., changes in adherence) would remain.

To conclude, we believe that negatively evaluating our work mainly on the theoretical machinery employed and on potential extensions of the empirical evaluation risks overlooking the actual contribution.

Reviewer_3a6AOpenReview ↗
Rating
4
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The authors introduce a new decision-theoretic framework they call learning to selectively disclose. They model the problem of deciding whether a decisionmaker should receive additional information or not prior to taking an action. They highlight that a statistic called the Value of Information (VoI) is relevant to selective disclosure and that, in fact, the optimal disclosure policy is simply a threshold rule over this statistic (Thm 1). Under some relatively standard assumptions for casual analysis, VoI is also identifiable (Prop 1). The authors then study estimation of VoI: they present results characterizing how bad plug-in estimators may be (Thms 2 and 3) and present an estimator they call ClassWise. Finally, they run experimental evaluations of ClassWise that study its performance and whether it improves human-AI collaboration.

Critical strengths

The primary positive aspects of the paper are: (1) high-quality exposition, (2) well-formed model/problem, (3) well-rounded suite of experiments. I have elaborated on each of these below:

  1. High-quality exposition. In my opinion, this paper is well-written. The introduction clearly explains the narrative of the model, all notation and equations are well-described (the elaboration of the Preliminaries Section is particularly appreciated). In my (entirely human) reading of the work's main body, there was no point at which the motivation behind results, assumptions, etc. was unclear.
  2. Well-formed model/problem. A key part of the contribution of this work is the presentation of the learning to selectively disclose problem itself. I feel that the motivations and narrative behind this problem are very natural. In general, the assumptions made are fairly standard and sufficiently explained in the context of the narrative (e.g., lines 176-185).
  3. Well-rounded suite of experiments. To my knowledge of this field (which is not exactly on this topic, but still in applied modeling, specifically strategic classification/learning), the inclusion of user studies in the experiments is unique and interesting and I appreciate their application of ClassWise in the context of real human-AI collaboration tasks.
Critical weaknesses

I do not feel totally convinced that the theoretical results for the authors' learning to selectively disclose problem are particularly mathematically interesting. My impression, from in a skim of Appendix A, is that all proof techniques are applications of relatively standard mathematical techniques. However, this is not a major problem as the model itself is relatively interesting/well-motivated and the experiments are fairly extensive.

Questions for authors
  1. Authors imagine an information budget, BB, that directly limits the expected disclosure. However, this seems to be a relatively simple way of modeling that "access to more information is costly" (line 127). For example, one could imagine that the cost of information access depends on the value of said information and then there should be a information access cost function. Could costly information access be modeled in other ways? What limitations does the assumption of simply bounding Ex[d(x)]E_{\mathbf{x}}[d(\mathbf{x})] by BB have on the narrative of information acquisition for agents?
  2. Is it the correct understanding that, because informativeness of the additional information relative to the baseline (VoI) is itself a the key statistic in your analysis, there is functionally no assumption made on the disclosed information? For example, very bad information (e.g., that which recommends the agent to make the wrong action) is fine and would simply be reflected in a very low VoI.
3 comments
AuthorsOpenReview ↗

Rebuttal to 3a6A

We thank the reviewer for the careful reading and the positive assessment of our work. We are glad the reviewer finds that this paper is well-written, that the introduction clearly explains the narrative of the model, and that all notation and equations are well-described. We also appreciate that the reviewer considers the motivations and narrative behind this problem to be very natural and describes the inclusion of user studies as unique and interesting.

We address the comments below.

Modeling costly information access. The Lagrangian formulation of our problem is equivalent to assuming a constant per-disclosure cost, as the threshold λ∗\lambda^* (see Thm. 1) is exactly the shadow price of disclosure. We opted for the budget formulation because a budget BB has an intuitive interpretation as the fraction of cases that can receive support. This choice follows the bounded-abstention formulation in learning to reject, which fixes the fraction of abstentions in advance because setting an abstention cost is harder in practice ([1, 2]). Still, LSD can be directly extended to an instance-dependent cost c(x)c(\mathbf{x}). Even when the support information SS is the same for all instances, providing it may be more costly for some instances than for others: for example, the same diagnostic test may be more expensive or more invasive for some patients. With a cost penalty, the optimal rule becomes $\mathrm{VoI}(\mathbf{x}) - \lambda, c(\mathbf{x}) \geq 0 $. With a budget on expected cost, Ex[c(x) d(x)]≤B\mathbb{E}_\mathbf{x}[c(\mathbf{x})\, d(\mathbf{x})] \leq B, it becomes VoI(x)c(x)≥λ\mathrm{VoI}(\mathbf{x}) c(\mathbf{x}) \geq \lambda, so instances are ranked by benefit per unit cost, as in treatment allocation with heterogeneous costs [3]. We have added this discussion to Section 3.1 and into Appendix C into the newly uploaded PDF. In all cases, the key estimand is still VoI(x)\mathrm{VoI}(\mathbf{x}): identification (Prop. 1) and the ClassWise estimator remain unchanged, and only the threshold built on top of the estimate changes. If costs are unknown, they must be estimated as well, and the bounds in Thms. 2-3 would need to account for this additional error.

Assumptions on the disclosed information. The reviewer is correct. LSD makes no assumption about the content or quality of the support information SS. Our assumptions (A1–A5) concern only how the data are collected, so that VoI(x)\mathrm{VoI}(\mathbf{x}) is identifiable from observational data (Prop. 1). In the provided example, misleading information increases the risk of a wrong decision, so its VoI(x)\mathrm{VoI}(\mathbf{x}) is negative. Consequently, this information would never be disclosed when using the true VoI(x)\mathrm{VoI}(\mathbf{x}) (see Theorem 1). In practice, the policy thresholds an estimate of V^oI(x)\hat VoI(\mathbf{x}), so harmful information is withheld only if the estimator is accurate. This is precisely what Thm. 3 bounds, i.e., the gap between the estimated policy and the optimal one in terms of the VoI-error estimation. We have added a specification in Sec 3 after Equation 2.

References

[1] Vojtech Franc, Daniel Prusa, Vaclav Voracek. Optimal Strategies for Reject Option Classifiers. JMLR 2023.

[2] Antonio Ferrara, Andrea Pugnana, Francesco Bonchi, Salvatore Ruggieri. Bounded-Abstention Pairwise Learning to Rank. KDD 2026.

[3] Georgy Kalashnov, Evan Munro, Hao Sun, Shuyang Du, Stefan Wager. Treatment Allocation under Uncertain Costs. arXiv:2103.11066, 2026.

[4] Adam Fisch, Shubhendu Trivedi, Fantine Huot, William W. Cohen, Michael Kaisers, Mirella Lapata, Kate Larson, Jacob Eisenstein. Pandora’s AI Model Routing Box: Efficient Allocation with Costly Value Estimation. NeurIPS 2026.

Thank you for the response to my questions and I appreciate the updates PDF that integrate explanations/justifications for my confusion about the budget choices. I continue to maintain my positive view of the paper.

Dear Reviewer 3a6A,

We are glad that the updates were appreciated. Thanks again for the meaningful feedback and the positive assessment of our work.

Reviewer_jDKeOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies when to disclose support information to a human decision maker under a limited budget. It defines the value of information as the expected reduction in human decision loss caused by disclosure and characterizes the optimal policy as a threshold on this quantity. The paper provides bounds relating policy performance to estimation error and a class-wise estimator for classification. Exerpiments include synthetic data, real datasets, and two user studies. Experiments on synthetic data and previously collected human responses show that selective disclosure can outperform both always showing advice and never showing it. In the user studies, algorithm-selected advice improved image classification but showed no significant advantage for email classification. These results support selective assistance, although the user studies do not isolate better selection from simply providing more advice. The TLDR is that assistance should be allocated according to its effect on the human’s final decision, not just predictive uncertainty alone.

Critical strengths
  1. The paper studies an important and well-motivated problem; when will showing AI advice actually help a human decision maker improve their decision.
  2. The empirical evaluation is comprehensive as it includes synthetic data, existing human-response datasets, and user studies.
  3. Empirical evidence demonstrates some practical benefit of the method; automatic advice selection improved image classification accuracy relative to when people choose when to request for advice.
Critical weaknesses
  1. The empirical observations of the method’s benefits was not consistent across tasks. The email user study found no statistically significant improvement over human-requested advice.
  2. The user studies do not separate better selection from more assistance. In the high-budget image condition, the algorithm provided advice on 14/20 trials, while people requested it on roughly six.
  3. The related work discussion could better connect to recent work on when AI assistance improves human decisions and how to design useful support. [1] examines when AI information can improve human decisions despite uncertainty about its quality. [2] constructs prediction sets tailored to downstream utility. These are relevant to the paper’s motivation and prediction-set experiments, respectively. [1] Robust Human-AI Complementarity under Uncertainty. Byun et. al (2026). [2] Decision-Focused Uncertainty Quantification. Cortes-Gomez et. al (2024).
Questions for authors

See weaknesses above.

2 comments
AuthorsOpenReview ↗

Rebuttal for Reviewer jDKe

We thank the reviewer for careful reading and constructive feedback. We are glad the reviewer finds the problem important and well-motivated, the empirical evaluation comprehensive, and the evidence of the method’s practical benefit convincing. We address each point below.

Variability across tasks. We consider the cross-task variability an informative finding, which would have been difficult to anticipate without the user studies. In the Email task, participants followed automatically provided advice significantly less than advice they requested themselves (adherence 0.850.85 vs 0.96,p<.0010.96, p < .001), whereas in ImageNet-16H adherence did not differ. Furthermore, the counterfactual benchmark, which assumes participants always follow the advice when provided by the ClassWise policy, presents significantly higher accuracy than the observed one in the Email task (p=.004p = .004, App. E.3). This suggests that the policy selected useful items, but part of the benefit was lost because participants did not follow the advice. This is, in our view, a key finding of our work: a good disclosure policy is necessary but not sufficient. Selecting the right items is only half of the problem; the other half is whether people follow advice they did not ask for. In the revised manuscript, we have presented this as one of our main empirical findings in the newly uploaded PDF and highlighted the need to understand which task characteristics affect adherence to automatically provided advice as a future-work direction.

Separating better selection from more assistance. We understand the reviewer’s concern to be that the accuracy improvement may be driven primarily by providing more assistance rather than by better selecting the items for which advice would be useful, potentially weakening the evidence for the practical benefit of our ClassWise policy. We note, however, that ClassWise is designed to address two related issues: which trials are most likely to benefit from advice and how many trials should receive it under a maximum budget. Even if the amount of advice were the main driver of the improvement in the ImageNet-16H task, the result would still have practical implications. It would suggest that participants tend to request advice less often than would be beneficial and that ClassWise can help close this gap. More broadly, we view our comparison of human-selected and ClassWise-selected disclosure as a first step toward understanding how people behave within these two realistic workflows. Dedicated follow-up studies can build on these findings to isolate the contributions of individual factors to human-AI team accuracy, though designing such studies is far from straightforward. Nevertheless, the reviewer’s comment prompted us to clarify the aim of the user studies in Sec. 4.1 and to discuss their limitations and directions for future research more explicitly in Sec. 6.

Related work. We thank the reviewer for the pointers and have added both works to Sec. 5 in the newly uploaded PDF. In [1], the authors study when AI information can improve human decisions despite uncertainty about its quality; in LSD, this quality is reflected in VoI(x)\mathrm{VoI}(\mathbf{x}), which allows the policy to withhold harmful support. In [2], the authors propose building prediction sets tailored to downstream decision utility, which shapes the form of the support, whereas LSD decides when to provide it. Therefore, we see the two approaches as complementary: decision-focused sets could be used as the support information SS in our ImageNet-16H experiments.

References

[1] Yewon Byun, Bryan Wilder. Robust Human-AI Complementarity under Uncertainty. ICML 2026.

[2] Santiago Cortes-Gomez, Carlos Patiño, Yewon Byun, Zhiwei Steven Wu, Eric Horvitz, Bryan Wilder. Utility-Directed Conformal Prediction: A Decision-Aware Framework for Actionable Uncertainty Quantification. ICLR 2025.

Dear Reviewer jDKe,

Thanks again for the positive assessment of the work. As the author-discussion phase comes to an end, please let us know whether our rebuttal has addressed your concerns or if there are any remaining points we can clarify.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General summary

We thank the reviewers for their thoughtful feedback. We are glad that all three reviewers find the problem well motivated and its formulation natural, and that they appreciate the user studies. We are also pleased that the paper was found to be clear and well-written, despite a few limitations. We first respond to the reviewers' main points, then address specific questions/comments raised by each reviewer.

Problem formulation. We are glad that all reviewers consider our problem formulation well-posed and well-motivated. Reviewer jDKe states that the paper studies an important and well-motivated problem and summarizes its key message: assistance should be allocated according to its effect on the human’s final decision, not just predictive uncertainty alone. Reviewer 3a6A notes that a key part of the contribution of this work is the presentation of the learning to selectively disclose problem itself and that the motivations and narrative behind this problem are very natural. Reviewer m13Y finds that learning whether disclosure improves the final human decision seems to be a natural and the right formulation. To the best of our knowledge, LSD is the first framework that treats disclosing support information as a budget-constrained causal intervention on the human decision-maker, with the human’s final decision as the target. This shift in perspective, from predictive uncertainty to the effect of disclosure on the final decision, is a core novelty of LSD.

Experimental evaluation and user studies. All reviewers appreciate our experimental evaluation, particularly the user studies. Reviewer jDKe finds the evaluation comprehensive, as it includes synthetic data, existing human-response datasets, and user studies, and notes that empirical evidence demonstrates some practical benefit of the method. Reviewer 3a6A describes it as a well-rounded suite of experiments and the inclusionì of user studies as unique and interesting. Reviewer m13Y appreciates that the paper combines formal analysis with experiments and particularly appreciated the two user studies. Reviewers jDKe and m13Y note that our user studies, which compare participant-requested (HP) and ClassWise-selected (MP) advice under the same budget, do not separate better selection from more assistance, since participants requested advice less often (e.g., about 6 vs. 14 out of 20 trials). We believe this concerns the scope of the studies rather than the validity of their results. Our goal was to compare two realistic workflows, including differences in how much advice they provide, rather than isolate the factors underlying any performance differences. Identifying these factors requires dedicated experiments whose design presents additional challenges, as we discuss in our responses to Reviewers jDKe and m13Y. We also note that ClassWise is designed to determine both which trials receive advice and how much advice to provide. Thus, even if the observed benefit were driven primarily by receiving more advice, the finding would retain practical relevance. It would suggest that ClassWise can help address users’ tendency to request advice less often than would be beneficial.

Clarity and soundness. Reviewer 3a6A states that this paper is well-written, that there was no point at which the motivation behind results, assumptions, etc. was unclear, and that the assumptions made are fairly standard and sufficiently explained. Reviewer jDKe also rates the presentation as clear and easy to follow. No reviewer questioned the correctness of our theoretical results.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.