acceptodds
Under review as a conference paper at ICLR 2027

Locating the Model-Harness Boundary in Interactive Diagnosis

Abstract

Interactive diagnosis requires an agent to recognize diseases, maintain evidence state, decide next steps, and determine when evidence is sufficient for a diagnosis. Existing LLM-based medical agents typically place this entire sequential control loop inside the model. We study a fundamental architectural question: which responsibilities should remain inside the LLM, and which should be executed by the surrounding harness? We decompose interactive diagnosis into evidence-state maintenance, search strategy, and terminal control, progressively externalizing them to a rule-bound runtime that executes machine-readable diagnostic programs compiled from clinical knowledge. At the fully externalized endpoint, the full harness directly takes over these responsibilities, restricting the LLM primarily to language–symbol grounding and residual reasoning for cases not resolved by the programs. Across three interactive medical benchmarks, the full harness improves diagnostic correctness by 24.7–49.7 percentage points over prompting with the same programs, exposes explicit execution provenance, and narrows performance variation across model scales. These results highlight a critical distinction between merely accessing procedural knowledge and reliably executing it. The effectiveness of an interactive agent depends not just on the underlying model's capability, but on how sequential control is strategically divided between the model and its execution harness during diagnosis.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM, Harness, Doctor Agent, Clinical Inquiry

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_WFvROpenReview ↗
Rating
3
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper investigates which responsibilities in interactive diagnosis should remain within an LLM and which should be delegated to an external execution harness. It decomposes diagnosis into evidence-state maintenance, search policy, and terminal control, then progressively externalizes these responsibilities to a deterministic runtime executing structured diagnostic programs. The LLM primarily handles language-to-symbol grounding and residual reasoning when the programs cannot resolve a case.

Critical strengths
  1. Providing the same diagnostic programs as either prompt content or executable runtime instructions helps distinguish access to procedural knowledge from reliable execution. This perspective is relevant beyond medicine to agents that must maintain state and follow procedures over extended interactions.

  2. The system distinguishes unknown, ambiguous, resolved, and unavailable evidence. Its three-valued logic avoids treating missing information as negative evidence. The interfaces between state updates, question selection, program routing, and diagnostic authorization are clearly specified.

  3. The paper includes abstention accounting, jointly answered comparisons, matched-coverage analysis, scoring-path checks, grounding audits, simulator sensitivity, and interaction costs.

  4. The authors acknowledge that ClinDiag programs lack individual clinical review and show that Hybrid fallback can increase incorrect diagnoses when critical imaging evidence is unavailable.

Critical weaknesses
  1. Components are externalized in a fixed sequence, leaving component interactions and ordering effects unresolved. Section 3.6 dismisses runtime search combined with LLM terminal control as structurally redundant. However, selecting evidence-acquisition actions and authorizing a final diagnosis are conceptually separable decisions. Additional feasible combinations, particularly matched search policies with different terminal controllers, would strengthen the boundary claim.

  2. The reported 24.7–49.7 percentage-point gains are measured against Program Prompt, which sometimes performs substantially worse than the simpler Candidate Set baseline. For example, on Cardiovascular with Flash, correctness is 53.3% for Program Prompt, 88.3% for Candidate Set, and 92.2% for Full Harness. The improvement over the stronger baseline is therefore 3.9 percentage points. The paper should foreground these comparisons and include a stronger LLM-controlled agent with matched program access, structured memory, tool interfaces, and inference budgets.

  3. On ClinDiag with Flash, rule-supported correctness contributes 17.7 percentage points, while fallback contributes 50.0 percentage points to the final 67.7%. Fallback also contributes more correct answers than rules on Cardiovascular. Moreover, Hybrid accepts an LLM diagnosis based partly on self-reported confidence.

  4. All three benchmarks use record-driven patient simulation and predefined disease catalogs. ClinDiag contains only one case per disease, and its programs lack external guideline provenance and individual clinical review. These settings provide limited evidence about out-of-catalog diseases, multimorbidity, or real patient communication. Similar correctness across model sizes also conceals substantial differences in risk. On ClinDiag, Qwen3-1.7B achieves 47.0% correctness with 49.0% wrong answers, whereas Qwen3-32B achieves 45.0% correctness with only 5.0% wrong answers.

Questions for authors

See above.

6 comments
AuthorsOpenReview ↗

Response to Reviewer (Part I)

Thank you for recognizing our controlled design. Rather than claiming superiority for a specific diagnostic system, our goal is to pinpoint how reliability bottlenecks shift as control responsibilities move between the LLM and the harness. Consequently, some limitations you highlighted are precisely the empirical phenomena we aim to quantify and analyze. Our point-by-point responses follow below:

Response to Reviewer WFvR (1/4): Decoupling Search and Terminal Control

"W1. Components are externalized in a fixed sequence, leaving component interactions and ordering ... would strengthen the boundary claim."

We agree that search and terminal control are conceptually separable decisions. They are coupled in our implementation because runtime search checks the terminal rules after every answer to decide whether to continue, but "structurally redundant" overstated this; §3.6 now describes the coupling instead.

Following your suggestion, we add Runtime Search + LLM Terminal: state and search are runtime, and only terminal control returns to the LLM, which decides after every turn whether to complete, reject, or continue. This gives matched search policies with different terminal controllers, a full 2×22\times2 design (§4.2, Table 3; App. E.3, Table 22). Entries are correct / wrong / abstain (%):

Backbone · DatasetSearch LLM / Term. LLMSearch Runtime / Term. LLM (new)Search LLM / Term. RuntimeSearch Runtime / Term. Runtime
Qwen3-8B · Abd.53.9 / 26.1 / 20.076.7 / 23.3 / 0.050.0 / 5.0 / 45.068.9 / 7.8 / 23.3
Qwen3-8B · Cardio.60.0 / 11.7 / 28.373.3 / 23.3 / 3.335.6 / 2.8 / 61.740.0 / 0.0 / 60.0
Qwen3-8B · ClinDiag15.3 / 7.0 / 77.764.0 / 30.0 / 6.04.0 / 1.0 / 95.017.7 / 1.3 / 81.0
Flash · Abd.56.1 / 6.1 / 37.866.7 / 5.0 / 28.356.7 / 4.4 / 38.961.1 / 1.1 / 37.8
Flash · Cardio.48.3 / 1.1 / 50.650.0 / 1.7 / 48.333.9 / 1.7 / 64.430.6 / 0.6 / 68.9
Flash · ClinDiag18.7 / 6.0 / 75.320.0 / 2.0 / 78.06.3 / 1.7 / 92.017.7 / 0.7 / 81.7
  • Externalizing search mainly raises correctness (columns 1→2 and 3→4): it rises in all six settings with an LLM terminal and in five of six with a runtime terminal.
  • Externalizing terminal control mainly lowers wrong answers and raises abstention (columns 2→4 and 1→3): wrong answers fall in all six settings under runtime search and in five of six under LLM search.

The 2×22\times2 matrix experiment separates these effects: search guides the diagnostic direction to gather relevant evidence, whereas terminal control regulates diagnostic convergence. Without runtime terminal control (Col. 2), Qwen3-8B often commits on partial evidence, and its wrong answers rise to 23.3–30.0%; with runtime terminal control (Col. 4), a diagnosis is issued only after a rule holds.

AuthorsOpenReview ↗

Response to Reviewer (Part II)

Response to Reviewer WFvR (2/4): Candidate Set and Stronger LLM-Agent Baselines

W2. "The reported 24.7–49.7 percentage-point gains are measured against Program Prompt, which sometimes performs substantially worse than the simpler Candidate Set baseline. … The improvement over the stronger baseline is therefore 3.9 percentage points." "…include a stronger LLM-controlled agent with matched program access, structured memory, tool interfaces, and inference budgets."

Thank you for the careful cross-check. To match our controlled experimental design—holding the diagnostic programs fixed while varying only who executes them—Program Prompt receives the exact same program text as the runtime and thus serves as our direct baseline (App. D.3, Table 11 (formerly Table 7)). By contrast, Candidate Set follows no diagnostic process; it is instead given a list of candidate diseases (Table 11), so it does not have matched program access.

Additionally, even in the setting with the smallest gap you pointed out (Flash · Cardiovascular), where correctness improves by 3.9 points, the wrong-answer rates differ greatly:

Method (Flash · Cardiovascular)Correct (%)Wrong (%)Abstain (%)
Program Prompt53.30.646.1
Candidate Set88.311.10.6
Full Harness (Hybrid)92.21.76.1

Stronger LLM-controlled agents with matched information and tools.

  • The appendix already includes Guideline Retrieval, which is equipped with a guideline retrieval tool (App. E.1.1, Table 12 (formerly Table 8)), and the multi-agent framework MDAgents (App. E.1.2, Table 13 (formerly Table 9)).
  • For the agent you describe, with "matched program access, structured memory, tool interfaces, and inference budgets" but with decisions controlled by the LLM, +Runtime State and +Runtime Terminal are controlled variants built to this standard: their programs, state, and interfaces match Full Harness, and only the control allocation differs. Their comparison with Full Harness (Table 2 and the 2×22\times2 matrix in W1) directly quantifies the net effect of handing control to the runtime.
AuthorsOpenReview ↗

Response to Reviewer (Part III)

Response to Reviewer WFvR (3/4): Fallback and Self-Reported Confidence

W3. "On ClinDiag with Flash, rule-supported correctness contributes 17.7 percentage points, while fallback contributes 50.0 percentage points to the final 67.7%. Fallback also contributes more correct answers than rules on Cardiovascular." "Moreover, Hybrid accepts an LLM diagnosis based partly on self-reported confidence."

Your decomposition is entirely correct: on ClinDiag, fallback contributes more correct diagnoses than rules. This is precisely the boundary our experiments are designed to reveal in §4.3. Rule mode measures how many cases deterministic rules can close on their own under strict evidence requirements, while Hybrid mode measures how much coverage LLM reasoning recovers when rules cannot close a case due to missing evidence.

As Table 23 (formerly Table 14) shows, rule-supported diagnoses are correct in 89.9%–100% of closed cases; for cases beyond rule coverage, incorporating LLM fallback substantially boosts overall diagnostic coverage, though at the cost of introducing unverified guesses and reduced interpretability:

Backbone · DatasetRule-supported (correct / wrong %)Fallback (correct / wrong %)Hybrid correctness = rule correct + fallback correct
Qwen3-8B · Abdominal68.9 / 7.88.9 / 0.668.9 + 8.9 = 77.8%
Qwen3-8B · Cardiovascular40.0 / 0.048.3 / 3.340.0 + 48.3 = 88.3%
Qwen3-8B · ClinDiag17.7 / 1.328.3 / 10.717.7 + 28.3 = 46.0%
Flash · Abdominal61.1 / 1.122.2 / 0.061.1 + 22.2 = 83.3%
Flash · Cardiovascular30.6 / 0.661.7 / 1.130.6 + 61.7 = 92.2%
Flash · ClinDiag17.7 / 0.750.0 / 7.017.7 + 50.0 = 67.7%

Self-reported confidence. We agree that self-reported confidence cannot be equated with rule-level formal guarantees. Rather, we use it as a gating constraint to prevent the LLM from drawing over-confident conclusions on insufficient evidence: a fallback diagnosis is output only if its confidence reaches at least τF\tau_F. Empirically, as τF\tau_F increases, the model's abstention rate consistently rises while its wrong-answer rate falls or stays flat (Table 19). Thus, τF\tau_F serves as a practical control knob that lets deployers tailor the system to their setting: selecting Rule mode (τF>1\tau_F > 1) for high-safety scenarios, Hybrid mode (τF=0.35\tau_F = 0.35) for high-coverage scenarios, or adjusting τF\tau_F continuously in between. Entries are abstain / wrong (%):

Backbone · DatasetτF=0.35\tau_F=0.35 (Hybrid)τF=0.70\tau_F=0.70τF=0.80\tau_F=0.80τF=0.90\tau_F=0.90τF>1\tau_F>1 (Rule)
Qwen3-8B · Abdominal13.9 / 8.314.4 / 7.815.0 / 7.821.7 / 7.823.3 / 7.8
Qwen3-8B · Cardiovascular8.3 / 3.38.3 / 3.310.6 / 1.732.8 / 0.660.0 / 0.0
Qwen3-8B · ClinDiag42.0 / 12.045.0 / 10.352.3 / 6.775.7 / 1.381.0 / 1.3
Flash · Abdominal15.6 / 1.121.1 / 1.128.3 / 1.135.0 / 1.137.8 / 1.1
Flash · Cardiovascular6.1 / 1.711.7 / 1.122.8 / 0.638.3 / 0.668.9 / 0.6
Flash · ClinDiag24.7 / 7.750.0 / 1.065.0 / 0.774.0 / 0.781.7 / 0.7
Reviewer_KEwXOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how sequential control should be divided between an LLM and its execution harness in interactive diagnosis. The authors decompose control into three responsibilities: evidence-state maintenance, search, and terminal control. Clinical knowledge is encoded as executable diagnostic programs, consisting of typed evidence slots and prioritized COMPLETE/REJECT rules evaluated under three-valued logic. Holding these programs fixed, the study progressively moves the three responsibilities from the LLM to a rule-bound runtime. In the resulting Full Harness, the LLM only grounds patient replies, phrases runtime-selected questions, and, in the Hybrid variant, provides a fallback diagnosis when no rule fires. Experiments use three benchmarks (Abdominal, Cardiovascular, and ClinDiag), Qwen3 and DeepSeek-V4 backbones, and an LLM patient simulator. Full Harness (Hybrid) improves correctness over Program Prompt by 24.7–49.7 points, lowers wrong-answer rates, and narrows the gap across model scales. The authors conclude that the model–harness boundary should be treated as a first-class design choice.

Critical strengths
  1. The paper addresses an important architectural question by systematically externalizing evidence-state maintenance, search, and terminal control from the LLM to an executable harness. This provides valuable insights into how procedural control can improve the reliability and interpretability of interactive medical agents.

  2. The experiments demonstrate substantial improvements in diagnostic correctness across three medical benchmarks and multiple model scales. The progressive ablation study further highlights the contributions of different harness components, supporting the practical effectiveness of the proposed architecture.

Critical weaknesses
  1. Some mathematical formulas, such as Eq. 14, appear to have been generated by AI and require further verification of their validity.

  2. The proposed evidence-acquisition policy relies on a handcrafted greedy utility without establishing its relationship to expected diagnostic information gain or long-horizon decision value. Consequently, its optimality and efficiency remain theoretically unclear.

  3. The search and graph-routing procedures depend on several manually specified parameters. Their effects on diagnostic accuracy, abstention, interaction cost, and robustness across benchmarks remain insufficiently characterized. A systematic sensitivity analysis would help determine whether the reported improvements generalize beyond the selected hyperparameter configuration.

  4. I hope the author can include the following experiments: Precision, recall, and F1 score for evidence extraction. Sensitivity of anchoring error to the final diagnosis. Using manually annotated correct “evidence states” as an “oracle.” Comparing the diagnostic performance of “oracle states” with that of states extracted by the LLM.

  5. The core state representation is not clearly defined. Eq (2) represents MtM_t as a set of state values, which loses the correspondence between slots and their associated states; it should instead be defined as a mapping. The aggregation SR=⋃tMtS_{\mathrm{R}} = \bigcup_t M_t conflates outdated states with current states. Furthermore, the paper is inconsistent about whether states are maintained separately for each program or shared across programs. The paper does not explicitly specify how slot states are mapped to {⊤,⊥,?}\{\top, \bot, ?\}, nor does it define the three-valued semantics of \texttt{any}, \texttt{not}, and \texttt{atleast}. Additionally, the rule grammar in Eq (8) is inconsistent with the schema presented in Appendix F.1.1. Finally, several critical parameters, including τF\tau_F, ρ\rho, α\alpha, β\beta, ϵ\epsilon, and c(e)c(e), are not reported, which limits the reproducibility of the results.

Questions for authors

Could the authors clarify how the utility weights and routing hyperparameters were selected, and whether the results are robust to these choices? Additionally, given that only 35.4% of case-runs successfully terminate through diagnostic rules, how much of the Full Harness (Hybrid) improvement is attributable to executable control versus LLM-based fallback? Finally, could the authors clarify whether the partial-information monotonicity guarantee extends beyond rule evaluation to the overall diagnostic process, particularly under imperfect evidence grounding?

6 comments
AuthorsOpenReview ↗

Response to Reviewer (Part I)

Thank you for your detailed review. We clarify that our goal is to quantify the impact of allocating the three control responsibilities to the LLM versus the runtime harness. Even though Full Harness achieves the highest correctness, it is not a proposed end-system, but rather the fully externalized baseline serving as an instrument for our controlled comparison. Consequently, we froze all harness parameters prior to evaluation and kept their discussion brief in the main text, which inadvertently left their rationale and exact values under-reported. We detail these specifications and address your comments below:

Response to Reviewer KEwX (1/4): Eq. 14 and Parameters

W1: "Some mathematical formulas, such as Eq. 14, appear to have been generated by AI and require further verification of their validity."

All constants were set by hand on development cases disjoint from the test sets and frozen before evaluation; none were tuned on test accuracy. The revision adds their rationale and a sensitivity analysis (App. B.2, App. C.5).

1. Rationale for Eq. 14. wij=clip[ϵ,1−ϵ](btype+αmin⁡(nshared,k)+β(nscenarios−1))w_{ij} = \mathrm{clip}_{[\epsilon,1-\epsilon]}\big(b_{\rm type} + \alpha\min(n_{\rm shared},k) + \beta(n_{\rm scenarios}-1)\big) This formula is designed based on the thinking of clinical differential diagnosis:

  • Clinical relationship prior btypeb_{\text{type}} (primary tiering): When physicians shift their differential diagnosis, they first consider how the two diseases are clinically related. High-overlap mimics (0.800.80) and must-not-miss critical conditions (0.750.75) take top priority, followed by standard differentials (0.600.60), complications (0.450.45), and shared risk factors (0.300.30).
  • Shared-feature & co-occurrence refinement αmin⁡(nshared,k)+β(nscenarios−1)\alpha\min(n_{\text{shared}},k) + \beta(n_{\text{scenarios}}-1) : Within the same relationship tier, candidates sharing more symptoms/signs (nsharedn_{\text{shared}}) or clinical scenarios (nscenariosn_{\text{scenarios}}) are prioritized. Using small weights (α=0.03,β=0.02\alpha=0.03, \beta=0.02) and saturation clipping (k=5k=5) caps the shared-symptom bonus at αk=0.15\alpha k = 0.15, so that the relation type remains the main factor and non-specific shared symptoms cannot accumulate without bound.
  • Boundary clipping clip[ϵ,1−ϵ]\mathrm{clip}_{[\epsilon,1-\epsilon]} ([0.05,0.95][0.05, 0.95]): Preserves valid low-frequency diagnostic paths (avoiding zero-weight dead ends) while preventing any single transition from monopolizing Personalized PageRank (PPR) probabilities.

Empirical verification in constructed graphs: Across the actual graphs built for all three datasets, the resulting edge-weight intervals for the five relationship tiers remain strictly non-overlapping (mimics: 0.92–0.950.92\text{--}0.95; must-not-miss: 0.81–0.900.81\text{--}0.90; standard differentials: 0.63–0.770.63\text{--}0.77; complications: 0.54–0.600.54\text{--}0.60; shared risk factors: 0.420.42).

AuthorsOpenReview ↗

Response to Reviewer (Part II)

Response to Reviewer KEwX (2/4): Sensitivity of Routing and Utility Weights

W3 / Q1: "… selected, and whether the results are robust to these choices?"

Although we do not aim to find a set of optimal parameters, we have supplemented the sensitivity analysis of relevant parameters as you suggested:

1. Sensitivity (new analyses, App. C.5, Tables 7–8).

Routing weights. We tested alternative Semantic / PPR weight ratios to assess sensitivity. At 0.80/0.200.80 / 0.20, Hit@1\text{Hit@1} and Hit@3\text{Hit@3} vary by within ∼6\sim 6 points of the 0.65/0.350.65 / 0.35 default, whereas dropping semantic weight to 0.50/0.500.50 / 0.50 reduces Abdominal Hit@1\text{Hit@1} by ∼15\sim 15 points and Cardiovascular Hit@3\text{Hit@3} by 9–159\text{--}15 points. Thus, the default ratio reflects a stable regime rather than a tuned optimum.

Semantic / PPRBackboneAbd. Hit@1\text{Hit@1}Cardio. Hit@1\text{Hit@1} / Hit@3\text{Hit@3}ClinDiag Hit@1\text{Hit@1} / Hit@3\text{Hit@3}
0.80 / 0.20Qwen3-8B46.338.0 / 74.413.4 / 30.6
Flash52.243.0 / 74.813.3 / 28.1
0.65 / 0.35 (default)Qwen3-8B46.338.0 / 80.514.0 / 32.4
Flash52.243.0 / 76.514.5 / 28.1
0.50 / 0.50Qwen3-8B30.538.0 / 66.013.5 / 26.6
Flash37.638.9 / 67.110.6 / 24.5

Utility weights. To test sensitivity, we sample 200 weight configurations by independently scaling each weight within [0.5,1.5][0.5, 1.5] (±50%\pm 50\%) across all main-table case-runs (App. C.5, Table 8). We track the share of individual turns selecting the exact same question, and the share of full consultations where every turn remains identical—yielding the exact same dialogue and final diagnosis:

DatasetConsultationsQuestion ChoicesSame Question (%)Same Dialogue (%)
Abdominal3603,75898.688.7
Cardiovascular3603,79498.793.5
ClinDiag6009,46399.292.5

Even under ±50%\pm 50\% weight perturbations, ∼99%\sim 99\% of per-turn question choices and 88.7–93.5%88.7\text{--}93.5\% of entire consultation trajectories remain completely invariant, confirming that our findings do not hinge on exact utility weight tuning.

AuthorsOpenReview ↗

Response to Reviewer (Part III)

Response to Reviewer KEwX (3/4): Greedy Utility and Evidence Extraction

W2: "The proposed evidence-acquisition policy relies on a handcrafted greedy utility without establishing its relationship to expected diagnostic information gain or long-horizon decision value. Consequently, its optimality and efficiency remain theoretically unclear."

We agree that our policy is a handcrafted greedy heuristic, and we do not claim theoretical optimality. Rather, its design is dictated by the structure of clinical SOP programs [1] and follows classic trade-offs in diagnostic expert systems [2,3]:

  • Why not EIG or long-horizon planning: Exact expected information gain (EIG) requires full conditional probability tables P(finding∣disease)P(\text{finding}\mid\text{disease}), and long-horizon value requires a patient-response transition model [4]—neither of which exists in deterministic clinical guidelines. Given logical rules rather than priors, scoring a slot by how many pending rules it helps resolve serves as a natural combinatorial proxy for information gain.
  • Role in our controlled study: Because our goal is to isolate the model–harness boundary, the runtime requires a transparent, deterministic policy to ensure experimental reproducibility.

We have conducted some additional analyses as you suggested: Empirically, even this greedy policy, executed by the runtime, outperforms LLM-controlled search. Moving only search to the runtime for Qwen3-8B roughly halves the number of turns and raises correctness on all three datasets (Table 2; App. E.5, Table 25 (formerly Table 16)):

DatasetTurns (LLM → Utility)Correctness % (LLM → Utility)
Abdominal22 → 1270.0 → 77.8
Cardiovascular23 → 1474.4 → 88.3
ClinDiag38 → 1918.7 → 46.0

Replacing the utility with a fixed or random order lengthens consultations by 47–61% (39.2 → 63.3 / 57.5 messages), because the utility prefers findings whose answer advances several rules at once.

Evidence-acquisition policyMean messages
Utility (ours)39.2
Fixed order63.3
Random order57.5
  • [1] Liu, Yujia, et al. "A multi-agent framework combining large language models with medical flowcharts for self-triage." Nature Health (2026): 1-10.
  • [2] Gorry & Barnett. Sequential diagnosis by computer. JAMA, 1968.
  • [3] Miller. INTERNIST-1/CADUCEUS: problems facing expert consultant programs. Methods Inf. Med., 1984.
  • [4] Golovin & Krause. Adaptive submodularity. JAIR, 2011.

W4: "…Precision, recall, and F1 score for evidence extraction. Sensitivity of anchoring error to the final diagnosis. Using manually annotated correct 'evidence states' as an 'oracle.' Comparing the diagnostic performance of 'oracle states' with that of states extracted by the LLM."

1. Precision / recall / F1 of evidence extraction (new experiment, App. E.2.4, Table 17). Because this analysis requires labeling thousands of slots, we first generated reference labels using DeepSeek-V4-Pro (which is strictly external to our diagnostic pipeline) for immediate evaluation; manual annotation is currently underway, and we will post the human-verified results during the discussion period. Across all 18 main-table runs, we evaluate two slot sets: all 1,494 slots used by rule-supported diagnoses and 1,000 randomly sampled other slots:

BackboneRule-used slots (P / R / F1)Random slots (P / R / F1)
DeepSeek-V4-Flash0.86 / 0.94 / 0.900.85 / 0.88 / 0.86
Qwen3-8B0.70 / 0.92 / 0.800.53 / 0.68 / 0.60

Recall is high on rule-used slots (0.92–0.94); errors lie mainly in precision: a recorded value that the patient's reply contradicts or does not contain, which is the failure mode that can falsely trigger a rule. However, extraction quality improves substantially with a stronger backbone.

2. Sensitivity of anchoring error and oracle states (App. E.2.4, Table 16 (formerly Table 12)). We replace the misread slots of rule-supported diagnoses with the reference labels and re-execute the rules on the same dialogues. For Qwen3-8B, 15 of 18 wrong rule-supported diagnoses are corrected; for Flash, 1 of 5. While rule-supported errors in smaller models come mainly from grounding, larger models are much less affected. Nevertheless, as the interface fitting patient semantics to symbolic slots, evidence grounding remains a core pain point in clinical diagnosis.

Reviewer_7oNAOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper investigates how responsibilities should be divided between an LLM and an external execution harness in interactive medical diagnosis. The authors decompose the diagnostic process into three components: evidence-state maintenance, search strategy, and terminal decision control. They progressively move these responsibilities from the LLM to a rule-based runtime that executes predefined diagnostic programs.

Experiments are conducted on three medical diagnosis benchmarks using different LLM backbones. The results show substantial improvements in diagnostic correctness when the full harness is used, particularly for smaller models. The paper also analyzes abstention, rule coverage, interaction costs, and the effects of missing clinical evidence.

Critical strengths

The paper examines an important aspect of LLM agent design: which responsibilities should be handled by the model and which should be delegated to an external system. This question is relevant beyond medical diagnosis. Separating evidence-state maintenance, search strategy, and terminal control provides a useful way to study the architecture of interactive diagnostic agents.

Critical weaknesses

The methodological novelty is somewhat limited. The central idea of externalizing state tracking, workflow execution, and decision rules from an LLM is not entirely new. Clinical decision-support systems and executable guideline frameworks have explored similar ideas for many years. More recent LLM agent systems also use external state management, structured workflows, and tool-based execution. Although the authors position the contribution as an empirical study of the model–harness boundary, I am not convinced that the proposed decomposition or implementation offers sufficient technical novelty. The paper would benefit from a clearer explanation of what is fundamentally new beyond integrating existing executable clinical workflows with LLMs.The improvements may depend heavily on task-specific diagnostic programs. The full harness relies on predefined diagnostic programs containing evidence requirements, decision criteria, and routing information. This introduces considerable structured domain knowledge into the system. Although the Program Prompt baseline receives the same procedural content, it does not necessarily have the same ability to enforce rules or maintain structured state. Therefore, the experiments demonstrate the benefit of executable rule enforcement, but it is less clear whether they establish a broader conclusion about the optimal model–harness boundary. I would like to see stronger baselines using conventional workflow engines or simpler deterministic controllers. The progressive ablation does not fully isolate the three responsibilities. The authors progressively externalize state, terminal control, and search, but these components are not evaluated independently. For example, runtime search is only introduced after state and terminal control have already been externalized. I understand that there are implementation dependencies, but this makes it difficult to determine the individual contributions of the three components. The results support the effectiveness of the complete configuration more clearly than they support the proposed functional decomposition. The strong performance of the hybrid harness relies substantially on LLM fallback. Table 2 shows a large gap between Full Harness (Rule) and Full Harness (Hybrid), especially on Cardiovascular and ClinDiag. For example, with DeepSeek-V4-Flash on Cardiovascular, correctness increases from 30.6% to 92.2% when hybrid fallback is enabled. This raises an important question about how much of the final performance is actually attributable to executable clinical control. The paper also reports that only 35.4% of execution trajectories terminate through a reference rule, while many reach the correct program without satisfying its terminal criteria. These results suggest that the predefined programs have limited coverage and that residual LLM reasoning remains essential. The safety and reliability claims need more careful interpretation. One motivation for moving control into the harness is to improve diagnostic reliability. However, the hybrid fallback can still produce unsupported diagnoses when clinical evidence is incomplete. In Table 3, removing rule-supporting imaging evidence increases the wrong-answer rate for Full Harness (Hybrid) on Cardiovascular by 11.1 percentage points. This is a meaningful limitation, especially in a medical setting. I would like to see a more systematic evaluation of the accuracy–coverage trade-off and whether the proposed system actually improves reliability under realistic missing or noisy evidence. Generalization to new clinical tasks and diagnostic programs remains unclear. The evaluation uses three benchmarks with predefined program libraries. In particular, the ClinDiag programs are elicited from model knowledge rather than externally validated clinical guidelines. This raises concerns about the quality and consistency of the generated programs. It is also unclear how much manual effort is required to construct, verify, and maintain these programs, or how the system would behave when deployed on diseases not covered by the existing library. The dependence on predefined diagnostic programs may limit scalability.

Questions for authors

What is the main technical distinction between the proposed harness and existing executable clinical guideline systems or conventional workflow engines? Is the primary contribution a new methodology, or an empirical comparison of where control is implemented?

6 comments
AuthorsOpenReview ↗

Response to Reviewer (Part I)

Thank you for the detailed review. We agree that executable clinical workflows are established; introducing another workflow engine is not our claimed contribution. Our question is what changes when specific control responsibilities are transferred from an LLM to the runtime while diagnostic programs are held fixed. (§1 §4)

From this perspective, limited rule coverage and continued dependence on LLM fallback are precisely the boundaries we aim to quantify. In response to your concerns, we have analyzed these issues more closely through an explicit rule/fallback decomposition and additional grounding and coverage analyses. We also add a search × terminal-control experiment to better isolate the contributions of the two responsibilities. We address each point below.

Response to Reviewer 7oNA (1/6): Novelty and Contribution

"What is the main technical distinction from executable clinical guideline systems or conventional workflow engines? Is the primary contribution a new methodology, or an empirical comparison?"

Our primary contribution is a controlled empirical comparison rather than a new workflow methodology. By holding the underlying diagnostic programs constant, our framework systematically isolates what happens as control shifts across the model–runtime boundary. We adopt the three-part decomposition from standard agent design not as a methodological novelty, but as an analytical lens to examine the spectrum between two existing paradigms:

  • Executable guideline systems and workflow engines (e.g., PROforma, GLIF) execute criteria as deterministic programs. They offer strong interpretability but assume structured inputs, struggling with ambiguous, open-ended patient dialogue.
  • LLM-based consultation agents (e.g., MedDM, AIDA-SEAT, CPGPrompt, GuideSkill) interact fluently, yet keep guideline logic implicit inside the model—leaving state tracking, question selection, and diagnostic commitment entirely to the model's internal decisions.

These two paradigms represent the two dominant—yet largely disconnected—ends of the clinical diagnostic spectrum. Bridging these two paradigms—using LLMs for natural dialogue and deterministic harnesses for interpretable execution—represents a compelling frontier in medical AI. Crucially, however, this integration is not a binary switch: control can be transferred incrementally. Prior work has yet to identify which specific responsibility acts as the primary bottleneck, or to quantify the exact gains and trade-offs of each handoff. In this work, we conduct comprehensive experiments to isolate each responsibility and quantify its distinct trade-offs and also release a complete pipeline from disease programs to deployable harnesses for clinical care.

Response to Reviewer 7oNA (2/6): Controls and Baselines

"The improvements may depend heavily on task-specific diagnostic programs… . I would like to see stronger baselines using conventional workflow engines or simpler deterministic controllers."

1. Dependence on task-specific diagnostic programs. We fully agree that much of the performance gain stems from incorporating prior procedural clinical knowledge. In fact, this is the premise of our study rather than its conclusion. To separate the value of the program content itself from how it is executed, our ablations compare providing the exact same program elements in the LLM prompt versus enforcing them in the runtime:

  • +evidence requirements: correctness changes by −6.0 to +8.9 points, with no consistent gain.
  • +decision criteria: wrong answers fall in five of six settings, and abstention rises.
  • +routing information: correctness rises substantially and consistently.

These results show more than "executable rule enforcement helps": rather than claiming a single universal boundary, they reveal which specific metric each responsibility governs and at what cost.

2. Workflow engines and simpler deterministic controllers. As noted above, a pure workflow engine cannot conduct multi-turn natural-language dialogue without an LLM interface. Once paired with an LLM purely for language grounding and verbalization, a conventional deterministic controller is precisely our Full Harness (Rule) baseline: the runtime deterministically selects every question and evaluates all rules, while the LLM only maps patient replies to slots and paraphrases the chosen questions. Its rule-supported answers are correct in 89.9%–100% of cases, but it abstains whenever evidence is incomplete, resulting in 17.7%–68.9% overall correctness (App. E.4, Table 23 [formerly Table 14]). Thus, a simpler deterministic controller is highly precise but suffers from limited coverage.

AuthorsOpenReview ↗

Response to Reviewer (Part II)

Response to Reviewer 7oNA (3/6): Individual Contributions of Search and Terminal Control

"Runtime search is only introduced after state and terminal control have already been externalized… difficult to determine the individual contributions."

1. Individual contributions of search and terminal control (new experiment). The main sequence externalizes state, then terminal control, then search, because state is a prerequisite for both, and runtime search checks the terminal rules after every answer to decide whether to continue.

But we agree that the effect of runtime search with terminal control left to the LLM should be tested directly. Therefore, we add Runtime Search + LLM Terminal: the runtime selects every question, and after each turn the LLM reads the full program and the current slot table and decides whether to complete, reject, or continue. With state externalized, this completes a 2×22\times2 matrix experiment (§4.2, Table 3; App. E.3, Table 22). Entries are correct / wrong / abstain (%):

Backbone · DatasetSearch: LLM / Terminal: LLM (+Runtime State)Search: Runtime / Terminal: LLM (new)Search: LLM / Terminal:Runtime (+Runtime Terminal, Rule)Search: Runtime / Terminal: Runtime (Full Harness, Rule)
Qwen3-8B · Abd.53.9 / 26.1 / 20.076.7 / 23.3 / 0.050.0 / 5.0 / 45.068.9 / 7.8 / 23.3
Qwen3-8B · Cardio.60.0 / 11.7 / 28.373.3 / 23.3 / 3.335.6 / 2.8 / 61.740.0 / 0.0 / 60.0
Qwen3-8B · ClinDiag15.3 / 7.0 / 77.764.0 / 30.0 / 6.04.0 / 1.0 / 95.017.7 / 1.3 / 81.0
Flash · Abd.56.1 / 6.1 / 37.866.7 / 5.0 / 28.356.7 / 4.4 / 38.961.1 / 1.1 / 37.8
Flash · Cardio.48.3 / 1.1 / 50.650.0 / 1.7 / 48.333.9 / 1.7 / 64.430.6 / 0.6 / 68.9
Flash · ClinDiag18.7 / 6.0 / 75.320.0 / 2.0 / 78.06.3 / 1.7 / 92.017.7 / 0.7 / 81.7
  • Externalizing search mainly raises correctness (columns 1→2 and 3→4): it rises in all six settings with an LLM terminal and in five of six with a runtime terminal.
  • Externalizing terminal control mainly lowers wrong answers and raises abstention (columns 2→4 and 1→3): wrong answers fall in all six settings under runtime search and in five of six under LLM search.

The 2×22\times2 matrix experiment separates these effects: search guides the diagnostic direction to gather relevant evidence, whereas terminal control regulates diagnostic convergence. Without runtime terminal control (Col. 2), Qwen3-8B often commits on partial evidence, and its wrong answers rise to 23.3–30.0%; with runtime terminal control (Col. 4), a diagnosis is issued only after a rule holds.

AuthorsOpenReview ↗

Response to Reviewer (Part III)

Response to Reviewer 7oNA (4/6): Fallback and Safety

"The strong performance of the hybrid harness relies substantially on LLM fallback… correctness increases from 30.6% to 92.2% when hybrid fallback is enabled. This raises an important question about how much of the final performance is actually attributable to executable clinical control. … only 35.4% of execution trajectories terminate through a reference rule… residual LLM reasoning remains essential."

1. Rule vs. Hybrid, and the contribution of executable clinical control. We do not use Hybrid's correctness to argue that the harness itself is stronger. Rule and Hybrid are the same harness and differ only in whether LLM fallback is enabled: Rule measures how many cases deterministic programs can close; Hybrid measures what is gained and risked when the remaining cases are passed to the LLM. The 30.6% → 92.2% jump is therefore the main contribution of the fallback mechanism, not the clinical control.

Consequently, to isolate the gain attributable to executable clinical control, one must hold the fallback setting fixed. For instance, when moving search from the LLM to the runtime (+Runtime Terminal → Full Harness) under a fixed fallback mode, correctness (%) on Flash changes as follows:

Mode (fallback fixed), FlashAbdominalCardiovascularClinDiag
Rule (no fallback)56.7 → 61.133.9 → 30.66.3 → 17.7
Hybrid (fallback)69.4 → 83.388.9 → 92.243.0 → 67.7

Across all 12 matched comparisons in Table 2, externalizing search raises correctness in 11 out of 12 cases.

2. The 35.4% rule-closure rate. We agree that the programs alone close only a subset of cases, and the original submission reports this as a finding: "while rule-supported decisions achieve high precision, their coverage remains incomplete and setting-dependent. Therefore, the harness does not obsolete the model" (§4.4). Guidelines do not fully match the information available in real cases; the gap is largest on ClinDiag, whose programs were built without clinical experts, and it is a known limitation of symbolic diagnostic systems. Rule measures this gap, and Hybrid passes the unclosed cases to the LLM fallback, which raises correctness but adds unsupported answers. Table 4 (formerly Table 3) was designed to measure exactly this difference, and the original §4.3 concluded that fallback "is essentially a compromise. Finding a balance between safety and accuracy remains a challenge."

"The safety and reliability claims need more careful interpretation. … the hybrid fallback can still produce unsupported diagnoses when clinical evidence is incomplete. In Table 3, … increases the wrong-answer rate … by 11.1 percentage points."

When critical evidence is missing, Hybrid's wrong-answer rate rises by 11.1 points on Cardiovascular, while Rule's falls (Table 4 (formerly Table 3)). We do not claim that either mode is better in every setting; rather, this comparison is intended to illustrate that although fallback answers substantially extend coverage, they also reintroduce safety risks by exposing the system to ungrounded LLM guesses. In the revision, we have narrowed the safety claims in §4.3 to state this trade-off clearly.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response: Core Framing and Summary of Revisions

We thank all reviewers for their constructive feedback. Several comments evaluate our work as proposing a superior end-to-end diagnostic system. We would like to clarify our core framing:

As the title Locating the Model–Harness Boundary in Interactive Diagnosis states, our paper is a controlled boundary study: holding diagnostic programs fixed, what changes as state maintenance, search, and terminal control are progressively moved from an LLM to an external harness? Among our findings, externalizing search generally improves correctness, while externalizing terminal control reduces wrong diagnoses and increases abstention. Together with the grounding and fallback analyses, these results reveal how reliability bottlenecks shift as control moves from the model to the harness.

Limited rule coverage and continued dependence on fallback are precisely the boundaries we aim to discuss and quantify. Building on the analyses in the original submission, the revision adds a search × terminal-control experiment to strengthen component attribution, alongside additional grounding evaluation, a fallback-threshold sweep, and random removal of rule-supporting evidence to examine these boundaries more closely. These analyses further clarify the roles of individual control responsibilities, the coverage–error trade-off, and the effects of missing evidence. We have also revised the manuscript to state this boundary-study framing consistently.

Summary of Revisions (Marked in Blue in the Revised PDF)

  • Reframing & Calibration: We revised the Abstract, Introduction, §3.6, §4.4, and Conclusion to keep the paper strictly centered on this boundary study, removed the "structurally redundant" claim in §3.6, and clarified in §4.4 that while the harness compresses correctness gaps across model scales, risk differences persist.
  • Formal Definitions (§3.3, App. A.2): Completed formalisms for the slot-state mapping, three-valued logic semantics (any, not, at_least), and rule grammar (between, unavailable).
  • New Experiments & Analyses:
ContentLocationReviewers
2×22\times2 design over search and terminal control (new variant: Runtime Search + LLM Terminal)§4.2, Table 3; App. E.3, Table 227oNA, WFvR
Fallback-threshold sweep: accuracy–coverage trade-off§4.3, Fig. 4; App. E.2.6, Fig. 14, Table 197oNA, WFvR
Out-of-catalog testApp. E.2.8, Table 217oNA, WFvR
Random removal of rule-supporting evidence (Abdominal, 2 backbones)App. E.2.7, Table 207oNA
Precision / recall / F1 of evidence extraction§4.3; App. E.2.4, Table 17KEwX
Full parameter tableApp. C.5, Table 9KEwX
Sensitivity to routing weights and utility weightsApp. C.5, Tables 7–8KEwX

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.