- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper formulates sequential diagnostic testing as a planning problem under partial observability, on the premise that a test's value should be judged by the downstream diagnostic trajectory it enables rather than by immediate information gain. It contributes FISC, a dataset of 156 longitudinal inpatient records re-encoded as HL7 FHIR resources across ten clinical domains, with a standardized test catalog carrying codes and prices and a designated set of time-critical conditions (TCCs); SPARC, a receding-horizon planner whose State, Dreamer, and Explore modules perform finite-depth stochastic AND–OR search over imagined outcome branches, scoring actions by Q(S,a) = C(a) + Σ_x P(x|S,a) V(T(S,a,x)) and stopping under the safety constraints b(d̂) ≥ τ_B and L_extra_TCC ≤ τ_M; and an evaluation against four baselines across planning depths D0–D5 and two result environments, reporting Top-k accuracy, Top-k TCC miss rate, mean test cost, and mean test count. All experiments are run on FISC alone.
Critical strengths
1. The problem formulation is well-motivated.
Casting inpatient diagnosis as a POMDP in which an agent iteratively selects tests, observes results, and updates beliefs — then evaluating candidate actions through their anticipated downstream consequences rather than immediate information gain — is a sound and well-argued framing. Making the risk of missing a time-critical condition an explicit asymmetric terminal loss rather than absorbing it into diagnostic accuracy matches clinical reality. This matters because the formulation is the part most likely to outlive the specific results, and it is worth building on.
2. The paper contributes real clinical data and a relatively complete evaluation system.
FISC is built from 156 real longitudinal inpatient records re-encoded as HL7 FHIR, spanning ten clinical domains, with a standardized test catalog carrying codes and prices and a designated TCC set. The evaluation covers diagnostic accuracy, TCC miss rate, resource cost, test count, and stopping statistics across planning depths and two result environments, which is a more complete evaluation protocol than most work in this area. The evaluation substrate — FHIR-grounded action space, real cost table, TCC-aware metrics — is reusable and is a genuine contribution independent of whether this particular method succeeds. The paper is also clearly written and well organized, and the running example in Figure 1 makes the central motivation immediately legible.
Critical weaknesses
1. The method does not constitute a substantial advance over standard receding-horizon planning.
SPARC combines three established ingredients — a belief update, LLM-simulated test outcomes, and finite-depth search over imagined branches — under a receding-horizon scheme. The paper does not articulate what is new relative to generic active feature acquisition or Bayesian experimental design, the latter being the principle behind ACTMED, one of its own baselines. The apparatus that would carry the novelty, namely the objective Q(S,a) and the stopping constraints b(d̂) ≥ τ_B and L_extra_TCC ≤ τ_M, is presented without any analysis: there is no guarantee, no characterization of when lookahead helps, and some quantities it depends on are not fully specified. As written, the contribution reads as a careful engineering assembly rather than a methodological advance.
2. The experiments do not establish that lookahead improves diagnostic decisions.
Section 6.2 states that one-step planning improves diagnostic accuracy while reducing the TCC miss rate, testing cost, and number of tests. The reported depth sweep does not support this: accuracy across D0–D5 is 60.4, 63.5, 57.7, 59.6, 61.5, 60.9, and TCC miss rate is 41.5, 40.4, 44.2, 50.0, 46.2, 40.4, so the trend is non-monotonic and D2 and D3 are worse than D0 on both measures. Nor can the evaluation compensate: all experiments are run on FISC alone, and the LLM driving the State, Dreamer, and Explore modules is never identified, so the numbers cannot be separated from a particular unnamed model and cross-model or cross-dataset generalization cannot be assessed. Since the title, abstract, and RQ2 rest on this claim, the central proposition is left unsupported.
Taken together, the formulation and the evaluation substrate are worth keeping, but the method is a standard receding-horizon planner and the experiments do not show that it improves diagnostic decisions significantly.
Questions for authors
-
Model and evaluation scope. Which LLM drives the State, Dreamer, and Explore modules — model family, version, and scale? Given that the paper reports no model identity and evaluates on FISC alone, can the authors report results for at least one additional backbone and, if available, on a second dataset, so that the findings can be separated from the specific model and data used?
-
Reconciling §6.2 with Appendix Table 2. Table 2 reports paired 95% CIs of +3.0 [−2.2, +8.3] for accuracy, −1.1 [−10.0, +7.8] for TCC miss rate, and −6 [−25, +13] CNY for cost — all containing zero — while §6.2 states these as improvements. Could the authors reconcile the main-text claim with their own paired analysis and state plainly which effects the experiments establish?