- Rating
- 3
- Confidence
- 1
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper investigates which responsibilities in interactive diagnosis should remain within an LLM and which should be delegated to an external execution harness. It decomposes diagnosis into evidence-state maintenance, search policy, and terminal control, then progressively externalizes these responsibilities to a deterministic runtime executing structured diagnostic programs. The LLM primarily handles language-to-symbol grounding and residual reasoning when the programs cannot resolve a case.
Critical strengths
-
Providing the same diagnostic programs as either prompt content or executable runtime instructions helps distinguish access to procedural knowledge from reliable execution. This perspective is relevant beyond medicine to agents that must maintain state and follow procedures over extended interactions.
-
The system distinguishes unknown, ambiguous, resolved, and unavailable evidence. Its three-valued logic avoids treating missing information as negative evidence. The interfaces between state updates, question selection, program routing, and diagnostic authorization are clearly specified.
-
The paper includes abstention accounting, jointly answered comparisons, matched-coverage analysis, scoring-path checks, grounding audits, simulator sensitivity, and interaction costs.
-
The authors acknowledge that ClinDiag programs lack individual clinical review and show that Hybrid fallback can increase incorrect diagnoses when critical imaging evidence is unavailable.
Critical weaknesses
-
Components are externalized in a fixed sequence, leaving component interactions and ordering effects unresolved. Section 3.6 dismisses runtime search combined with LLM terminal control as structurally redundant. However, selecting evidence-acquisition actions and authorizing a final diagnosis are conceptually separable decisions. Additional feasible combinations, particularly matched search policies with different terminal controllers, would strengthen the boundary claim.
-
The reported 24.7–49.7 percentage-point gains are measured against Program Prompt, which sometimes performs substantially worse than the simpler Candidate Set baseline. For example, on Cardiovascular with Flash, correctness is 53.3% for Program Prompt, 88.3% for Candidate Set, and 92.2% for Full Harness. The improvement over the stronger baseline is therefore 3.9 percentage points. The paper should foreground these comparisons and include a stronger LLM-controlled agent with matched program access, structured memory, tool interfaces, and inference budgets.
-
On ClinDiag with Flash, rule-supported correctness contributes 17.7 percentage points, while fallback contributes 50.0 percentage points to the final 67.7%. Fallback also contributes more correct answers than rules on Cardiovascular. Moreover, Hybrid accepts an LLM diagnosis based partly on self-reported confidence.
-
All three benchmarks use record-driven patient simulation and predefined disease catalogs. ClinDiag contains only one case per disease, and its programs lack external guideline provenance and individual clinical review. These settings provide limited evidence about out-of-catalog diseases, multimorbidity, or real patient communication. Similar correctness across model sizes also conceals substantial differences in risk. On ClinDiag, Qwen3-1.7B achieves 47.0% correctness with 49.0% wrong answers, whereas Qwen3-32B achieves 45.0% correctness with only 5.0% wrong answers.
Questions for authors
See above.
6 comments
Response to Reviewer (Part II)
Response to Reviewer WFvR (2/4): Candidate Set and Stronger LLM-Agent Baselines
W2. "The reported 24.7–49.7 percentage-point gains are measured against Program Prompt, which sometimes performs substantially worse than the simpler Candidate Set baseline. … The improvement over the stronger baseline is therefore 3.9 percentage points." "…include a stronger LLM-controlled agent with matched program access, structured memory, tool interfaces, and inference budgets."
Thank you for the careful cross-check. To match our controlled experimental design—holding the diagnostic programs fixed while varying only who executes them—Program Prompt receives the exact same program text as the runtime and thus serves as our direct baseline (App. D.3, Table 11 (formerly Table 7)). By contrast, Candidate Set follows no diagnostic process; it is instead given a list of candidate diseases (Table 11), so it does not have matched program access.
Additionally, even in the setting with the smallest gap you pointed out (Flash · Cardiovascular), where correctness improves by 3.9 points, the wrong-answer rates differ greatly:
| Method (Flash · Cardiovascular) | Correct (%) | Wrong (%) | Abstain (%) |
|---|---|---|---|
| Program Prompt | 53.3 | 0.6 | 46.1 |
| Candidate Set | 88.3 | 11.1 | 0.6 |
| Full Harness (Hybrid) | 92.2 | 1.7 | 6.1 |
Stronger LLM-controlled agents with matched information and tools.
- The appendix already includes Guideline Retrieval, which is equipped with a guideline retrieval tool (App. E.1.1, Table 12 (formerly Table 8)), and the multi-agent framework MDAgents (App. E.1.2, Table 13 (formerly Table 9)).
- For the agent you describe, with "matched program access, structured memory, tool interfaces, and inference budgets" but with decisions controlled by the LLM, +Runtime State and +Runtime Terminal are controlled variants built to this standard: their programs, state, and interfaces match Full Harness, and only the control allocation differs. Their comparison with Full Harness (Table 2 and the matrix in W1) directly quantifies the net effect of handing control to the runtime.
Response to Reviewer (Part III)
Response to Reviewer WFvR (3/4): Fallback and Self-Reported Confidence
W3. "On ClinDiag with Flash, rule-supported correctness contributes 17.7 percentage points, while fallback contributes 50.0 percentage points to the final 67.7%. Fallback also contributes more correct answers than rules on Cardiovascular." "Moreover, Hybrid accepts an LLM diagnosis based partly on self-reported confidence."
Your decomposition is entirely correct: on ClinDiag, fallback contributes more correct diagnoses than rules. This is precisely the boundary our experiments are designed to reveal in §4.3. Rule mode measures how many cases deterministic rules can close on their own under strict evidence requirements, while Hybrid mode measures how much coverage LLM reasoning recovers when rules cannot close a case due to missing evidence.
As Table 23 (formerly Table 14) shows, rule-supported diagnoses are correct in 89.9%–100% of closed cases; for cases beyond rule coverage, incorporating LLM fallback substantially boosts overall diagnostic coverage, though at the cost of introducing unverified guesses and reduced interpretability:
| Backbone · Dataset | Rule-supported (correct / wrong %) | Fallback (correct / wrong %) | Hybrid correctness = rule correct + fallback correct |
|---|---|---|---|
| Qwen3-8B · Abdominal | 68.9 / 7.8 | 8.9 / 0.6 | 68.9 + 8.9 = 77.8% |
| Qwen3-8B · Cardiovascular | 40.0 / 0.0 | 48.3 / 3.3 | 40.0 + 48.3 = 88.3% |
| Qwen3-8B · ClinDiag | 17.7 / 1.3 | 28.3 / 10.7 | 17.7 + 28.3 = 46.0% |
| Flash · Abdominal | 61.1 / 1.1 | 22.2 / 0.0 | 61.1 + 22.2 = 83.3% |
| Flash · Cardiovascular | 30.6 / 0.6 | 61.7 / 1.1 | 30.6 + 61.7 = 92.2% |
| Flash · ClinDiag | 17.7 / 0.7 | 50.0 / 7.0 | 17.7 + 50.0 = 67.7% |
Self-reported confidence. We agree that self-reported confidence cannot be equated with rule-level formal guarantees. Rather, we use it as a gating constraint to prevent the LLM from drawing over-confident conclusions on insufficient evidence: a fallback diagnosis is output only if its confidence reaches at least . Empirically, as increases, the model's abstention rate consistently rises while its wrong-answer rate falls or stays flat (Table 19). Thus, serves as a practical control knob that lets deployers tailor the system to their setting: selecting Rule mode () for high-safety scenarios, Hybrid mode () for high-coverage scenarios, or adjusting continuously in between. Entries are abstain / wrong (%):
| Backbone · Dataset | (Hybrid) | (Rule) | |||
|---|---|---|---|---|---|
| Qwen3-8B · Abdominal | 13.9 / 8.3 | 14.4 / 7.8 | 15.0 / 7.8 | 21.7 / 7.8 | 23.3 / 7.8 |
| Qwen3-8B · Cardiovascular | 8.3 / 3.3 | 8.3 / 3.3 | 10.6 / 1.7 | 32.8 / 0.6 | 60.0 / 0.0 |
| Qwen3-8B · ClinDiag | 42.0 / 12.0 | 45.0 / 10.3 | 52.3 / 6.7 | 75.7 / 1.3 | 81.0 / 1.3 |
| Flash · Abdominal | 15.6 / 1.1 | 21.1 / 1.1 | 28.3 / 1.1 | 35.0 / 1.1 | 37.8 / 1.1 |
| Flash · Cardiovascular | 6.1 / 1.7 | 11.7 / 1.1 | 22.8 / 0.6 | 38.3 / 0.6 | 68.9 / 0.6 |
| Flash · ClinDiag | 24.7 / 7.7 | 50.0 / 1.0 | 65.0 / 0.7 | 74.0 / 0.7 | 81.7 / 0.7 |
Response to Reviewer (Part I)
Thank you for recognizing our controlled design. Rather than claiming superiority for a specific diagnostic system, our goal is to pinpoint how reliability bottlenecks shift as control responsibilities move between the LLM and the harness. Consequently, some limitations you highlighted are precisely the empirical phenomena we aim to quantify and analyze. Our point-by-point responses follow below:
Response to Reviewer WFvR (1/4): Decoupling Search and Terminal Control
We agree that search and terminal control are conceptually separable decisions. They are coupled in our implementation because runtime search checks the terminal rules after every answer to decide whether to continue, but "structurally redundant" overstated this; §3.6 now describes the coupling instead.
Following your suggestion, we add Runtime Search + LLM Terminal: state and search are runtime, and only terminal control returns to the LLM, which decides after every turn whether to complete, reject, or continue. This gives matched search policies with different terminal controllers, a full 2×2 design (§4.2, Table 3; App. E.3, Table 22). Entries are correct / wrong / abstain (%):
The 2×2 matrix experiment separates these effects: search guides the diagnostic direction to gather relevant evidence, whereas terminal control regulates diagnostic convergence. Without runtime terminal control (Col. 2), Qwen3-8B often commits on partial evidence, and its wrong answers rise to 23.3–30.0%; with runtime terminal control (Col. 4), a diagnosis is issued only after a rule holds.