acceptodds
Under review as a conference paper at ICLR 2027

The Road Not Ordered: Counterfactual Step Pricing for Reinforcement Learning in Sequential Medical Diagnosis

Abstract

Medical diagnosis is a sequential and cost-aware process: physicians order examinations, update their diagnostic reasoning as evidence accumulates, and make a final diagnosis once the evidence is sufficient. Reinforcement learning provides a natural framework for training sequential diagnostic agents. However, reinforcement learning in this setting faces two key challenges: reward blindness, where diagnostic paths with different levels of efficiency can receive the same score if they reach the same conclusion; and credit dilution, where one final score is shared across many steps, making the pivotal examination difficult to distinguish from routine or unnecessary ones. These challenges arise from assigning a single final-outcome reward to the entire trajectory. To address these issues, we propose a counterfactual reinforcement learning approach called Counterfactual Step Pricing (CoSteP). CoSteP focuses on decision-critical states where the policy is uncertain. At each selected state, it compares the chosen action with plausible counterfactual alternatives using short rollouts and a utility that considers both diagnostic accuracy and examination cost. A persistent rollout cache reuses compatible trajectories accumulated across training batches, reducing computation without expert step annotations or a learned critic. Experiments on MIMIC-IV, ClinicalBench, and a private hospital dataset show that CoSteP improves diagnostic accuracy while reducing both the number and cost of examinations.

Primary area:
reinforcement learning
Keywords:
Sequential Medical Diagnosis, Counterfactual Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_w1gVOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The authors regard medical diagnosis as a sequential and cost-aware process and formulate it as a reinforcement learning problem. This paper proposes a counterfactual reinforcement learning approach called Counterfactual Step Pricing (CoSteP). CoSteP focuses on decision-critical states where the policy is uncertain. At each selected state, it compares the chosen action with plausible counterfactual alternatives using short rollouts and a utility that considers both diagnostic accuracy and examination cost. A persistent rollout cache reuses compatible trajectories accumulated across training batches, reducing computation without expert step annotations or a learned critic. Experiments on MIMIC-IV, ClinicalBench, and a private hospital dataset show that CoSteP improves diagnostic accuracy while reducing both the number and cost of examinations.

Critical strengths
  1. This paper presents an important and interesting problem based on real medical applications. The motivation behind this paper is clearly presented.

  2. The proposed method is technically sound and core ideas are well presented.

  3. The empirical experiments are conducted in various benchmarks. The proposed method works better than baseline algorithms.

  4. The source code is attached as a supplementary material. I feel it is nicely written and seems reproducible (though I did not run it on my env).

Critical weaknesses
  1. Equation (11) is closely related to a Monte Carlo estimate of an action advantage relative to a candidate-set baseline. It is unclear what is the novelty beyond uncertainty-based state selection and rollout caching.

  2. Low-entropy states can still contain confidently incorrect decisions. It would be useful to evaluate whether errors concentrate in such low-entropy states.

Questions for authors

Please see the Critical Weaknesses.

2 comments
AuthorsOpenReview ↗

Response to Reviewer w1gV

We sincerely appreciate the reviewer’s careful reading of our manuscript and the feedback. Our detailed responses are provided below.

Q1. Equation (11) is closely related to a Monte Carlo estimate of an action advantage relative to a candidate-set baseline. It is unclear what is the novelty beyond uncertainty-based state selection and rollout caching.

A1. Compared with standard Monte Carlo estimation, our method uses uncertainty-guided state selection to identify decision-critical states and performs counterfactual rollouts only at these selected states to construct process supervision. It further incorporates a cost-aware utility to jointly consider diagnostic correctness and examination efficiency, while rollout caching reuses compatible continuations to reduce redundant computation. In contrast, standard Monte Carlo estimation performs rollouts uniformly across intermediate states. CoSteP therefore concentrates computation on more informative decisions, providing targeted process supervision with substantially lower rollout cost.

Q2. Low-entropy states can still contain confidently incorrect decisions. It would be useful to evaluate whether errors concentrate in such low-entropy states.

A2. As suggested, we evaluated whether low-entropy states contain confidently incorrect decisions. As shown in Table K, we conducted a clinician-based evaluation and found that only 4% of low-entropy actions were judged to be incorrect. This suggests that confidently incorrect decisions are relatively uncommon in low-entropy states. As for the evaluation procedure, since intermediate actions do not have ground-truth labels, we randomly sampled 100 low-entropy states and asked three clinicians to independently judge whether each proposed action was clinically reasonable given the available evidence. An action was considered correct if at least two clinicians supported it.

Table K. Clinician assessment of actions at low-entropy states.

Clinician AssessmentProportion of Low-Entropy States
Supported by at least two clinicians96%
Not supported by a majority of clinicians4%

Thank you for the responses. I read other reviews and rebuttals. I will keep my original rating.

Reviewer_bXR6OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes Counterfactual Step Pricing (CoSteP), a reinforcement learning framework for sequential medical diagnosis. The method is intuitively motivated, and the experiments report improvements over outcome-only GRPO and several medical-agent baselines. However, the paper currently has substantial concerns regarding the validity of its counterfactual estimates, the construction of the offline diagnostic environment, the interpretation of the cached rollouts, and the statistical support for the reported gains.

Critical strengths
  1. Sequential diagnosis under examination-cost constraints is a meaningful healthcare decision-making problem, and the paper correctly identifies the limitations of outcome-only rewards.
  2. Clear high-level motivation. Comparing the selected examination with plausible alternatives is a natural way to construct action-sensitive reward signals.
Critical weaknesses
  1. The proposed "counterfactual” rollouts do not establish causal counterfactual validity.

The paper describes the utility of an alternative examination as a counterfactual outcome. However, the proposed procedure appears to replace the original action in a recorded patient trajectory and then generate a continuation using the current policy. This does not by itself provide an estimate of the outcome that would have occurred under the alternative examination. Eq. (6) should not be interpreted as a causal counterfactual value without additional assumptions. At most, it is a policy-relative, simulator-dependent utility comparison. The paper should explicitly define the transition model and state the assumptions required for the estimated U^(s,a)\hat{U}(s,a) to approximate QUπ(s,a)Q_{U}^{\pi}(s,a).

  1. The bounded-horizon rollout and forced-diagnosis rule can distort the reward.

In Section 3.3.2, the authors claim that each branch is truncated after at most HH steps. If no diagnosis is reached within this horizon, the model is required to produce a diagnosis using the currently available evidence. In the experiments (Appendix A.3), it seems like HH is set to 3. This creates a horizon-dependent reward that may favor actions producing rapidly available evidence, even when a longer trajectory would eventually yield a more accurate or safer diagnosis. The reward may also be inconsistent with the full-horizon objective in Eq. (5), since the practical estimate in Eq. (10) is not an unbiased estimate of the full-horizon utility. The paper does not quantify how often branches are truncated or forcibly terminated, nor does it compare the process reward with different horizon values beyond the accuracy-only sensitivity plot in Figure 4.

Questions for authors
  1. What exactly supplies the examination result for a counterfactual action that was not taken in the original trajectory? Are counterfactual branches generated from a learned environment model, retrieved from the same patient record, copied from other patients, or treated as unavailable? Please explicitly define the transition model and state the assumptions required for the estimated U^(s,a)\hat{U}(s,a) to approximate QUπ(s,a)Q_{U}^{\pi}(s,a).
  2. What fraction of candidate actions are classified as unavailable, and how much of this is due to genuine unavailability versus parsing or matching failure?
  3. How often are branches truncated at H=3H=3, and how often is a forced diagnosis required?
  4. Is the cost-aware training reward computed from ordinal cost levels or from monetary costs? Why is the reported utility in Eq. (4) not the exact training utility?
  5. The main results are averaged over only three random seeds. No confidence intervals, standard errors, paired comparisons, or significance tests are provided for the accuracy, AEN, or AEC results. This is particularly problematic because several reported differences are modest, and the out-of-domain results have different dataset sizes and distributions. Are the reported gains statistically significant across seeds and per patient case?
  6. How is the rollout cache invalidated or corrected as the policy changes during training?
4 comments
AuthorsOpenReview ↗

Response to Reviewer bXR6 (1/3)

We thank the reviewer for the thoughtful comments and helpful suggestions on our manuscript. We address each point in detail below.

Q1. The paper describes the utility of an alternative examination as a counterfactual outcome. However, the proposed procedure appears to replace the original action in a recorded patient trajectory and then generate a continuation using the current policy. This does not by itself provide an estimate of the outcome that would have occurred under the alternative examination. What exactly supplies the examination result for a counterfactual action that was not taken in the original trajectory? Please explicitly define the transition model and state the assumptions required for the estimated U^(s,a)\hat{U}(s,a) to approximate QUπ(s,a)Q_U^\pi(s,a).

A1. We first want to clarify that the alternative examination is generated by the model, rather than directly obtained by replacing an action in the recorded patient trajectory. It can provide an estimate of the outcome under the alternative examination by continuing the rollout from each alternative examination until a diagnostic outcome is reached. For each counterfactual action request, the environment retrieves the matching examination result from the same patient’s record. If no matching result is found, it returns unavailable. This ensures that all returned examination results are factual and traceable, as the environment only uses results available in the corresponding patient record.

As suggested, we explicitly define the transition model as follows: “The state consists of the patient’s initial information and interaction history. For an examination request, the environment matches the request to the patient record, returns the corresponding result if available, and appends the action–observation pair to the history; otherwise, it returns unavailable. A final diagnosis terminates the trajectory.”

The estimation relies on the following assumptions: “Each patient record is treated as a fixed clinical snapshot: examination results are independent of request order, and the environment does not model disease progression or treatment effects. All candidate actions are compared under the same patient and diagnostic history. Under this setting, QUπ(s,a)Q_U^\pi(s,a) is the expected utility of taking aa at ss and following the current policy to diagnosis, while U^(s,a)\hat{U}(s,a) approximates it with bounded-horizon rollouts.” We will add these definitions and assumptions to the revised manuscript.

Q2. What fraction of candidate actions are classified as unavailable, and how much of this is due to genuine unavailability versus parsing or matching failure?

A2. The fraction of candidate actions classified as unavailable is 4.85%, based on all candidate actions involved in the counterfactual evaluation. We further grouped these unavailable actions into three categories:

  • No matching examination record (91.6%): a valid examination request was generated, but no corresponding result was matched in the patient record. Among these cases, 94.2% were due to genuine unavailability, while 5.8% were due to matching failure.
  • Invalid action output (7.9%): the model did not follow the required action format, such as omitting the ACTION field.
  • Parsing failure (0.5%): the output followed the expected format, but the parser failed to extract the required action information.
AuthorsOpenReview ↗

Response to Reviewer bXR6 (2/3)

Q3. The bounded-horizon rollout (H=3H=3) and forced-diagnosis rule can distort the reward. This creates a horizon-dependent reward that may favor actions producing rapidly available evidence, even when a longer trajectory would eventually yield a more accurate or safer diagnosis. How often are branches truncated at H=3H=3, and how often is a forced diagnosis required? How does the process reward compare across different horizon values?

A3. The bounded-horizon rollout (H=3H=3) provides a stable approximation of the reward in our setting. To verify this, we trained our method with H=(1,2,...,6)H=(1,2,...,6) and report the final accuracy and total training time in Table F. The results show that H>3H>3 yields only marginal changes in accuracy, while substantially increasing training time. A larger horizon increases training time because a small number of long-running branches delay the completion of the corresponding batch of parallel rollouts.

As suggested by the reviewer, we further conducted two analyses. First, we examined how often branches are truncated at H=3H=3. As shown in Table G, the average rollout length is 2.27 steps, and only 2.61% of branches require forced diagnosis, indicating that most branches naturally terminate within three steps. In our implementation, truncation at H=3H=3 and forced diagnosis are equivalent: if a branch reaches the horizon without producing a diagnosis, the forced-diagnosis rule is triggered. Second, we compared the average process reward across H=(1,2,...,6)H=(1,2,...,6). As shown in Table F, the process reward becomes stable around H=3H=3, with only marginal changes for larger horizons. Together, these results suggest that H=3H=3 provides a favorable trade-off between stable reward estimation and computational efficiency.

Table F. Effect of rollout horizon HH on process reward, accuracy, and training time.

HHAvg. Process RewardFinal Acc. (%)Total Training Time (h)
10.031550.0083.219
20.048651.82101.071
30.058553.07117.360
40.057753.02133.582
50.057152.93145.248
60.058052.98154.921

Table G. Actual rollout behavior with H=3H=3.

Avg. Rollout Length (steps)Forced-Diagnosis Rate (%)
2.272.61

Q4. Is the cost-aware training reward computed from ordinal cost levels or from monetary costs? Why is the reported utility in Eq. (4) not the exact training utility?

A4. The cost-aware training reward is computed from ordinal cost levels. This bounded ordinal representation controls the numerical scale of examination costs, keeps the cost penalty comparable to the other reward components, and reduces the risk that a few very expensive examinations dominate the utility. Similar scale-control strategies have also been used in prior reinforcement learning work [1][2]. Specifically, we map examination types to different cost levels, as summarized in Table H. The corresponding implementation code is also provided in the supplementary materials. Thus, CtotalC_{\mathrm{total}} in Eq. (4) denotes the sum of ordinal cost levels. We will clarify this definition explicitly in the revised manuscript.

Table H. Ordinal cost levels used in training.

Cost LevelExamination Category and Examples
0Physical examination and already-available information
1Basic laboratory tests, e.g., complete blood count, basic metabolic panel, and urinalysis
2Specialized laboratory tests, e.g., cultures, PCR, viral loads, and serology
3Basic imaging, e.g., X-rays and plain radiographs
4Advanced imaging and electrophysiological examinations, e.g., CT, MRI, echocardiography, ECG, and EEG
5Invasive or high-cost procedures, e.g., angiography, catheterization, bronchoscopy, biopsy, and endoscopy

[1]Mnih V, Kavukcuoglu K, Silver D, et al. Human-level control through deep reinforcement learning[J]. nature, 2015, 518(7540): 529-533.
[2]Van Hasselt H P, Guez A, Hessel M, et al. Learning values across many orders of magnitude[J]. Advances in neural information processing systems, 2016, 29.

AuthorsOpenReview ↗

Response to Reviewer bXR6 (3/3)

Q5. The main results are averaged over only three random seeds. No confidence intervals, standard errors, paired comparisons, or significance tests. Are the reported gains statistically significant across seeds and per patient case?

A5. As suggested, we conducted additional analyses of both cross-seed variability and patient-level paired statistical significance.

  • First, we re-aggregated the results from the three independent training seeds and report the mean ±\pm standard deviation of Accuracy for each method, as shown in Table I. The small standard deviations indicate that the performance gains of CoSteP are consistent across different random seeds.

  • Second, we performed per-patient-case analyses, as shown in Table J. The results show that CoSteP achieves statistically significant gains over most baselines across datasets and metrics, which supports the robustness and statistical reliability of the reported improvements. For each patient, we first averaged the predictions or metric values across the three seeds for CoSteP and each baseline, and then computed the corresponding metric differences. We subsequently performed 10,000 paired bootstrap resamples over patients to estimate 95% confidence intervals for these differences.

Table I. Performance reported as mean ±\pm standard deviation.

MethodMIMIC-IV Acc. (%)ClinicalBench Acc. (%)Private Acc. (%)
GPT-5.442.93 ± 1.0756.00 ± 1.1036.97 ± 0.90
Qwen3-32B37.73 ± 1.0455.60 ± 1.0333.40 ± 1.03
Med-Gemma-27B35.60 ± 0.9654.80 ± 1.1032.25 ± 1.11
DiagAgent-14B41.33 ± 1.0150.73 ± 0.9434.24 ± 1.11
GRPO-4B47.33 ± 1.0951.67 ± 0.8729.41 ± 1.00
CoSteP-4B (Ours)53.07 ± 0.8159.13 ± 0.9030.06 ± 1.07

Table J. Patient-level paired comparison of CoSteP against representative baselines.

DatasetBaselineAcc. Improvement (pp) [95% CI]AEN Reduction [95% CI]AEC Reduction (USD) [95% CI]
MIMIC-IVQwen3-4B20.67* [18.27, 23.20]-0.72* [-0.83, -0.61]3.15 [-0.03, 6.31]
MIMIC-IVMed-Gemma-27B17.47* [15.16, 19.78]1.25* [1.07, 1.44]69.76* [62.23, 77.72]
MIMIC-IVDiagAgent-14B11.73* [9.42, 14.04]4.06* [3.76, 4.38]155.16* [142.33, 168.50]
ClinicalBenchQwen3-4B14.27* [12.60, 15.93]-1.89* [-2.00, -1.78]2.99* [1.68, 4.29]
ClinicalBenchMed-Gemma-27B4.33* [2.64, 5.96]-1.22* [-1.33, -1.11]11.26* [9.72, 12.83]
ClinicalBenchDiagAgent-14B8.40* [6.71, 10.11]2.50* [2.32, 2.68]174.78* [164.93, 185.02]
PrivateQwen3-4B4.11* [2.24, 6.06]-0.16* [-0.23, -0.09]18.76* [15.90, 21.79]
PrivateMed-Gemma-27B-2.19 [-4.13, -0.18]0.68* [0.58, 0.78]43.10* [38.88, 47.72]
PrivateDiagAgent-14B-4.18* [-6.13, -2.14]4.08* [3.82, 4.34]246.59* [228.22, 266.40]

Q6. How is the rollout cache invalidated or corrected as the policy changes during training?

A6. We have considered this issue when designing the rollout cache and therefore introduced a continuously updated cache mechanism. The cache is a stack-like buffer that continuously incorporates newly trajectories, allowing older entries to be replaced. This mechanism is explicitly described in Appendix A.3 (Page 19, Lines 972–973): “The continuation cache is maintained across training batches with a capacity of 8192 entries and is continuously updated as new trajectories are generated.”

Reviewer_bXR6OpenReview ↗

Response to Rebuttal

Thank the authors for the rebuttal. It helped me better understand the paper, so I will change my score from 2 to 3.

Reviewer_KapBOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes CoSteP for sequential diagnosis that balances diagnostic accuracy and examination costs. At high-uncertainty states, it compares candidate actions through short rollouts to derive counterfactual step rewards, combining trajectory caching with GRPO to train the diagnostic policy. Across three datasets, the method improves overall accuracy while reducing examination costs.

Critical strengths
  1. The problem has practical value, addressing diagnostic correctness, examination selection, and when to stop.
  2. The method is well motivated.
Critical weaknesses
  1. Step rewards are still aggregated into trajectory-level rewards, leaving insufficient evidence of precise step-level credit assignment.
  2. Utility estimates may be biased. Three-step rollouts may underestimate long-term benefits, while cached trajectories from older policies may no longer reflect the current policy.
  3. Evidence of generalization and reliability is limited. Accuracy on the private dataset trails some strong baselines, variability is not reported, and training and evaluation rely on the same AI judge.
Questions for authors
  1. How do you distinguish unrecorded from inappropriate examinations? What happens if unavailable requests are not penalized?
  2. How does CoSteP compare with random state selection under equal compute and with direct step-level rewards?
4 comments
AuthorsOpenReview ↗

Response to Reviewer KapB (1/3)

We sincerely thank the reviewer for the time reviewing our paper and providing useful feedback, and we address each question in detail below.

Q1. Step rewards are still aggregated into trajectory-level rewards, leaving insufficient evidence of precise step-level credit assignment.

A1. Aggregating step-level rewards into the trajectory-level rewards can still alleviate the step-level credit assignment problem, because trajectories containing higher-value intermediate decisions will receive higher relative rewards, making the policy more likely to reinforce such decision patterns during optimization. The reason why we aggregate step rewards into trajectory-level rewards is that our method provides process rewards at selected decision-critical states, where these rewards are sparse and may occur at different numbers of steps across trajectories. Aggregating them into trajectory-level reward provides a unified optimization signal for all trajectories.

Q2. Utility estimates may be biased. Three-step rollouts may underestimate long-term benefits.

A2. A three-step rollout sufficiently captures long-term benefits in our setting. To verify this, we conducted additional experiments with different values of HH. We trained our method with H=(1,2,...,6)H=(1,2,...,6) and report the accuracy and training time. As shown in Table A, increasing HH beyond 3 yields only marginal changes in accuracy while substantially increasing training time. In addition, we collected selected states during the final training epoch and measured the average rollout length, the percentage of rollouts that terminated within three steps, and the average time required to complete all counterfactual rollouts for one selected state, which are also shown in Table A. Even when a larger HH is allowed, most counterfactual continuations naturally terminate within three steps. However, a larger horizon increases training time because a small number of long-running branches delay the completion of the corresponding batch of parallel rollouts. Overall, H=3H=3 achieves a favorable trade-off between utility estimation and computational efficiency.

Table A. Effect of the counterfactual rollout horizon.

HHFinal Acc. (%)Total Training Time (h)Avg. Rollout Length (steps)Naturally Terminated within 3 Steps (%)Avg. Rollout Time per Selected State (s)
150.0083.2191.00-2.79
251.82101.0711.92-4.36
353.07117.3602.2797.395.47
453.02133.5822.2997.205.84
552.93145.2482.2997.545.86
652.98154.9212.3097.126.03

Q3. Cached trajectories from older policies may no longer reflect the current policy.

A3. We have considered this issue when designing the cache and therefore introduced a continuously updated cache mechanism. The cache is a stack-like buffer that continuously incorporates newly trajectories, allowing older entries to be replaced. This mechanism is explicitly described in Appendix A.3 (Page 19, Lines 972–973): “The continuation cache is maintained across training batches with a capacity of 8192 entries and is continuously updated as new trajectories are generated.”

AuthorsOpenReview ↗

Response to Reviewer KapB (2/3)

Q4. Evidence of generalization and reliability is limited. Accuracy on the private dataset trails some strong baselines, variability is not reported, and training and evaluation rely on the same AI judge.

A4. As for the accuracy on the private dataset, our method achieves the highest accuracy among all training variants using the same backbone, while also obtaining the lowest examination cost. Since the private dataset is not used during training, these results support the out-of-domain generalization and reliability of our method under a controlled backbone setting.

As for the variability, we conducted an additional variability analysis, as shown in Table B. We report the mean ±\pm standard deviation of Accuracy over the original three independent training runs. The results show that our method maintains consistent performance across random seeds.

As for the AI judgement, we conducted an additional re-evaluation using Claude Sonnet 5.5 and DeepSeek V4.1-Flash-1017 under the same judging criteria. As shown in Table C1~C2, our method remains competitive under both independent judges, providing additional evidence that the main conclusions are robust to the choice of evaluation model. Due to space constraints, we report representative strong baselines here; the complete results and evaluation settings will be added to the revised manuscript.

Table B. Results over three independent runs, reported as mean ±\pm standard deviation.

MethodMIMIC-IV Acc. (%)ClinicalBench Acc. (%)Private Acc. (%)
GPT-5.442.93 ± 1.0756.00 ± 1.1036.97 ± 0.90
Qwen3-32B37.73 ± 1.0455.60 ± 1.0333.40 ± 1.03
Med-Gemma-27B35.60 ± 0.9654.80 ± 1.1032.25 ± 1.11
DiagAgent-14B41.33 ± 1.0150.73 ± 0.9434.24 ± 1.11
GRPO-4B47.33 ± 1.0951.67 ± 0.8729.41 ± 1.00
CoSteP-4B (Ours)53.07 ± 0.8159.13 ± 0.9030.06 ± 1.07

Table C1. Accuracy (%) under the Claude judge

MethodMIMIC-IV Acc. (%)ClinicalBench Acc. (%)Private Acc. (%)
GPT-5.442.53 ± 1.1455.73 ± 1.1337.43 ± 0.89
Qwen3-32B37.20 ± 1.0455.47 ± 1.0733.58 ± 1.07
Med-Gemma-27B35.16 ± 0.9554.62 ± 1.0932.49 ± 1.01
DiagAgent-14B41.02 ± 0.9550.44 ± 0.9734.56 ± 1.01
GRPO-4B47.02 ± 1.1551.42 ± 0.9129.48 ± 1.06
CoSteP-4B (Ours)52.71 ± 0.9159.02 ± 0.8930.08 ± 1.09

Table C2. Accuracy (%) under the DeepSeek judge

MethodMIMIC-IV Acc. (%)ClinicalBench Acc. (%)Private Acc. (%)
GPT-5.443.07 ± 1.1455.91 ± 1.0736.76 ± 0.86
Qwen3-32B38.04 ± 1.0155.42 ± 0.9933.23 ± 1.05
Med-Gemma-27B35.96 ± 0.9154.49 ± 1.0932.14 ± 1.11
DiagAgent-14B41.47 ± 1.0950.38 ± 0.9834.17 ± 1.09
GRPO-4B48.09 ± 1.1551.38 ± 0.8929.38 ± 1.07
CoSteP-4B (Ours)53.42 ± 0.8658.60 ± 0.8230.18 ± 1.07

Q5. How do you distinguish unrecorded from inappropriate examinations? What happens if unavailable requests are not penalized?

A5. We do not explicitly distinguish unrecorded from inappropriate examinations in our setting, because the dataset contains comprehensive records of examinations performed during clinical care and covers nearly all clinically relevant examinations. To verify this, we randomly sampled 100 examination requests with no matching result in the patient’s record and asked three clinicians to classify each case as either unrecorded (clinically reasonable but missing from the record) or inappropriate (irrelevant, redundant, or unjustified). The final label was determined by majority vote (i.e., agreement of at least two of the three clinicians). Of the 100 requests, 93 were judged inappropriate. Further analysis showed that all 7 remaining cases were matching failure, where the requested examination results were present in the patients’ records. Thus, among the 100 requests reviewed, we found no case in which a clinically reasonable examination request was unrecorded in the patient’s record. If unavailable requests are not penalized, the model is more likely to generate unexecutable actions, resulting in more unavailable requests and retries. This effect is already reported in Section 4.3, Table 2, and reproduced below.

Table D. Effect of removing the unavailable-examination penalty.

SettingAcc. (%) ↑AEN ↓AEC (USD) ↓
CoSteP53.072.1036.65
w/o λna\lambda_{\mathrm{na}}50.624.8562.50
AuthorsOpenReview ↗

Response to Reviewer KapB (3/3)

Q6. How does CoSteP compare with random state selection under equal compute and with direct step-level rewards?

A6. Following the reviewer’s suggestion, we added two additional experiments: random state selection and direct step-level reward updates, shown in Table E. CoSteP outperforms both random state selection with the same number of selected states and direct step-level updates in accuracy and examination efficiency, supporting the effectiveness of uncertainty-guided state selection and trajectory-level integration of sparse process rewards.

Table E. Comparison with random state selection and direct step-level updates.

MethodAcc. (%) ↑AEN ↓AEC (USD) ↓
CoSteP53.072.1036.65
CoSteP + Random State Selection50.762.2842.91
CoSteP + Direct Step-Level Update51.912.1939.78
Reviewer_KapBOpenReview ↗

Official Comment by Reviewer

Thank you for the response, which addresses most of my concerns. I recommend acceptance and maintain my score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.