- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes a method to selectively provide support information to a human decision-maker. Theoretically, it formulates the problem as deciding whether to provide information based on the conditional value of information for reducing decision risk. As a result, the optimal disclosure rule is a threshold rule: disclosing in cases that have the largest value of information. The paper then experimentally evaluated the method on existing human judgment data and two new user studies.
Critical strengths
- I like the paper’s formulation. Learning whether disclosure improves the final human decision seems to be a natural and the right formulation.
- The paper combines formal analysis with experiments. I particularly appreciated the two user studies.
Critical weaknesses
-
The theoretical contribution, beyond modeling, seems to be limited. Once the objective R(d) is reached, the optimal policy follows rather straightforwardly. The novelty contribution on the theoretical side is arguably marginal. Moreover, the paper doesn’t establish how to obtain accurate value estimates, which is crucial for producing a good disclosure policy.
-
On the experimental side, the human-subject studies do not directly isolate the effect of allocation of advice. They compare human requesting advice with ClassWise automatically supplying it. As a result, the amount of advice provided in these two conditions is significantly different. In some sense, the experiments show the comparison of two different methods, but do not compare the impact of different allocation of advice. To me, a fair comparison is to keep the advice budget roughly the same and compare LSD with, for example, randomly selecting when to provide advice.
Questions for authors
See the second point of the weakness section.
4 comments
Dear Reviewer m13Y,
Thanks again for the feedback. As the author-discussion phase comes to an end, please let us know whether we have addressed your concerns or if there are any remaining points we can clarify.
Thank you for the response and revisions. Proposition 3 partially addresses my estimation concern, although its guarantee depends on the accuracy of the component estimators. My assessment of the limited theoretical contribution beyond the formulation is largely unchanged.
My main experimental concern also remains. Clarifying the studies’ scope is helpful, but does not resolve whether the improvement comes from better allocation or simply more advice. A matched-quantity random-disclosure baseline would provide useful evidence even if adherence differed between conditions.
Our main theoretical contribution is identifying the right formulation, which the reviewer also recognizes as natural and correct. We agree that once the problem is correctly formulated, the theoretical results do not require convoluted machinery. Simple derivations do not make the results weaker as they are rigorous. We therefore do not see this as a critical weakness.
Regarding the experimental evaluation, we respectfully disagree with the reviewer. For the purpose of the current study, i.e., identifying whether the policy can improve upon human self-selected disclosure, our experimental evaluation is sufficient to detect such an effect. We do not claim to separate better allocation from more advice. As stated in Sec. 6, we explicitly leave this to future dedicated studies. Moreover, as discussed in our previous rebuttal, a random condition would not, on its own, disentangle the drivers of the results, as other confounders (e.g., changes in adherence) would remain.
To conclude, we believe that negatively evaluating our work mainly on the theoretical machinery employed and on potential extensions of the empirical evaluation risks overlooking the actual contribution.
Rebuttal to Reviewer m13Y
We thank the reviewer for the feedback. We are glad the reviewer likes the paper’s formulation, finding that learning whether disclosure improves the final human decision seems to be a natural and the right formulation, and that the reviewer particularly appreciated the two user studies. We address both concerns below.
Novelty of the theoretical contribution. Our main theoretical contribution lies in the formalization of the problem. Once the correct formulation is provided, deriving our theoretical results does not require convoluted theoretical machinery. We see this as a strength of our proposal. Moreover, as we stressed in the main text (lines 170-184 of the newly uploaded PDF), selective disclosure differs from the canonical treatment-allocation setting and requires a dedicated causal formulation. To the best of our knowledge, LSD is the first framework that treats disclosure as a causal intervention on a human decision-maker's information and learns when to disclose based on its effect on decision loss, with identification conditions and guarantees for the resulting policy.
How to obtain accurate estimates. We thank the reviewer for this question, which allowed us to state the guarantees of the proposed estimator more precisely. Under the identification assumptions of Section 3.2, the VoI corresponds to a conditional average treatment effect (CATE), namely the difference between regime-specific conditional risks. Its estimation can therefore build on the causal inference literature, where consistency of standard meta-learners such as T- and S-learners is well studied [1, 2]. Our ClassWise estimator specializes the T-learner by exploiting the class-wise decomposition of human decision risk under the 0−1 loss. Since this structure departs from a standard T-learner, existing results do not apply directly. In the revised manuscript, we therefore prove that ClassWise inherits the same guarantee (new Proposition 3). Its VoI estimation error is bounded by the estimation errors of its two components, the label model f and the class-wise error heads gD. Hence, ClassWise yields a consistent VoI estimator whenever these components are consistent. Combined with our oracle regret bound (Theorem 3), this new proposition provides the sufficient conditions to make the regret of the resulting disclosure policy vanish as well. Empirically, ClassWise compares favourably against direct T- and S-learner baselines (Appendix E.1).
On the experimental side. We agree that the accuracy improvement may be driven primarily by more assistance rather than by better selecting the items for which advice would be useful. We note, however, that ClassWise is designed to address two related issues: which trials are most likely to benefit from advice and how many trials should receive it under a maximum budget. Even if the amount of advice were the main driver of the improvement in the ImageNet-16H task, the result would still have practical implications. In particular, it suggests that participants tend to request advice less often than would be beneficial, and that ClassWise can help close this gap.
More broadly, we view our comparison of human-selected and ClassWise-selected disclosure as a first step toward understanding how people behave within these two realistic workflows. Dedicated follow-up studies can build on these findings to isolate the contributions of individual factors to human-AI team accuracy, though designing such studies is far from straightforward. Indeed, we considered including a matched random-disclosure baseline like the one suggested by the reviewer. While it would have provided useful evidence on the relative contributions of selection quality and assistance quantity, we ultimately decided against including it because (i) it was less representative of realistic use cases and (ii) on its own, it would have left the mechanisms behind any accuracy difference between random and ClassWise disclosure unresolved.
For example, repeated exposure to advice perceived as unnecessary or unhelpful in the random condition could reduce participants’ willingness to follow subsequent advice. Any accuracy difference could therefore reflect how useful the advice was on the selected trials, how readily participants followed it, or both. Nevertheless, the reviewer’s comment prompted us to clarify the aim of the user studies in Sec. 4.1 and to discuss their limitations and directions for future research more explicitly in Sec. 6.
References
[1] Künzel, S. et al.. Metalearners for estimating heterogeneous treatment effects using machine learning. PNAS, 2019.
[2] Curth A. and van der Schaar M.. Nonparametric estimation of heterogeneous treatment effects: from theory to learning algorithms. AISTATS, 2021.