- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The authors regard medical diagnosis as a sequential and cost-aware process and formulate it as a reinforcement learning problem. This paper proposes a counterfactual reinforcement learning approach called Counterfactual Step Pricing (CoSteP). CoSteP focuses on decision-critical states where the policy is uncertain. At each selected state, it compares the chosen action with plausible counterfactual alternatives using short rollouts and a utility that considers both diagnostic accuracy and examination cost. A persistent rollout cache reuses compatible trajectories accumulated across training batches, reducing computation without expert step annotations or a learned critic. Experiments on MIMIC-IV, ClinicalBench, and a private hospital dataset show that CoSteP improves diagnostic accuracy while reducing both the number and cost of examinations.
Critical strengths
-
This paper presents an important and interesting problem based on real medical applications. The motivation behind this paper is clearly presented.
-
The proposed method is technically sound and core ideas are well presented.
-
The empirical experiments are conducted in various benchmarks. The proposed method works better than baseline algorithms.
-
The source code is attached as a supplementary material. I feel it is nicely written and seems reproducible (though I did not run it on my env).
Critical weaknesses
-
Equation (11) is closely related to a Monte Carlo estimate of an action advantage relative to a candidate-set baseline. It is unclear what is the novelty beyond uncertainty-based state selection and rollout caching.
-
Low-entropy states can still contain confidently incorrect decisions. It would be useful to evaluate whether errors concentrate in such low-entropy states.
Questions for authors
Please see the Critical Weaknesses.
2 comments
Thank you for the responses. I read other reviews and rebuttals. I will keep my original rating.
Response to Reviewer w1gV
We sincerely appreciate the reviewer’s careful reading of our manuscript and the feedback. Our detailed responses are provided below.
A1. Compared with standard Monte Carlo estimation, our method uses uncertainty-guided state selection to identify decision-critical states and performs counterfactual rollouts only at these selected states to construct process supervision. It further incorporates a cost-aware utility to jointly consider diagnostic correctness and examination efficiency, while rollout caching reuses compatible continuations to reduce redundant computation. In contrast, standard Monte Carlo estimation performs rollouts uniformly across intermediate states. CoSteP therefore concentrates computation on more informative decisions, providing targeted process supervision with substantially lower rollout cost.
A2. As suggested, we evaluated whether low-entropy states contain confidently incorrect decisions. As shown in Table K, we conducted a clinician-based evaluation and found that only 4% of low-entropy actions were judged to be incorrect. This suggests that confidently incorrect decisions are relatively uncommon in low-entropy states. As for the evaluation procedure, since intermediate actions do not have ground-truth labels, we randomly sampled 100 low-entropy states and asked three clinicians to independently judge whether each proposed action was clinically reasonable given the available evidence. An action was considered correct if at least two clinicians supported it.
Table K. Clinician assessment of actions at low-entropy states.