acceptodds
Under review as a conference paper at ICLR 2027

AnesTRACE: Benchmarking Intraoperative Anesthesia from Multimodal Perception to Multi-step Decision-Making

Abstract

Intraoperative anesthesia requires systems to interpret evolving multimodal evidence, recommend timely management, and revise decisions as patient states change, yet existing benchmarks usually isolate perception or single-point reasoning. We introduce AnesTRACE, an evaluation suite comprising AnesTRACE-Bench and AnesTRACE-Eval. Built from public perioperative datasets with anesthesiologist annotation, AnesTRACE-Bench evaluates Intraoperative Perception, Single-point Anesthesia Decision-Making, and Multi-step Anesthesia Decision-Making. AnesTRACE-Eval assesses open-ended responses through anesthesiologist-defined criteria for Clinical Correctness, Evidence Grounding, Task Completeness, and Safety, with Temporal Consistency for multi-step decisions; its domain-specific evaluator is trained by supervised fine-tuning and preference alignment on expert-reviewed judgments. Across more than 30 models, fine-grained visual grounding and intervention selection remain difficult: the leading model reaches only 32.2 mIoU for TEE visual grounding and retains a 17.5% Major/Critical Safety Error Rate in multi-step management. Evaluator alignment with anesthesiologists improves across both training stages, while the best decision quality is accompanied by a 74.3-second P95 Latency. These results show that aggregate performance alone does not establish safe, timely longitudinal decision-making. We release our datasets and code at https://anonymous.4open.science/r/AnesTRACE-027C/.

Primary area:
datasets and benchmarks
Keywords:
Multimodal Large Language Models, Clinical Decision-Making, AI Agents, Datasets and Benchmarks, Healthcare AI, Safety Evaluation, Intraoperative Anesthesia

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_JaCtOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces AnesTRACE, an anesthesia-specific benchmark covering multimodal perception, single-point management, and multi-step decision updating through recorded clinical trajectories. It also develops an expert-guided evaluator for clinical correctness, evidence grounding, completeness, safety, and temporal consistency. Evaluation across more than 30 models, physician-agreement analysis, and input, history, and tool ablations reveal persistent limitations in perception and management, while highlighting the importance of clinical context and information access.

Critical strengths
  1. The clinically motivated task hierarchy provides useful coverage of perception, management reasoning, and longitudinal decision updating. Its focus on intraoperative anesthesia makes it a valuable complement to general medical question-answering benchmarks.

  2. Reporting serious safety errors separately from average clinical quality makes the evaluation more informative. The broad model comparison and history, memory, and tool ablations provide useful insights into the capabilities needed for this setting.

Critical weaknesses
  1. The main model comparisons and safety findings rely on one learned evaluator. Although agreement with clinicians is reported, the robustness of the headline conclusions and ablation effects to evaluator choice remains unclear.

  2. The history and memory ablations are informative, but the temporal evaluation lacks a targeted control demonstrating that it distinguishes necessary management updates from simply maintaining earlier recommendations.

Questions for authors
  1. Using existing responses from representative models and major ablations, do the main quality and safety conclusions remain stable under an independent evaluator using the same clinical criteria and decision-time evidence?

  2. On episodes where the existing reference criteria require a management change, how does the actual agent compare with a baseline that repeats its previous plan? Reporting temporal consistency alongside clinical correctness and safety would help establish what the temporal metric captures.

  3. Could you clarify the discrepancy between the TEE Factual-F1 results in Tables 3 and 15? Under the equal-weight aggregation specified in Appendix B.2, GPT-6-Astra’s scores of 13.0 and 15.7 yield 14.35, whereas Table 3 reports 27.5. Were different evaluation settings used, and does resolving this discrepancy affect the model rankings? I would reconsider my rating if the authors address these concerns.

6 comments

W1 and Q1: Robustness to Evaluator Choice

We appreciate your suggestion to examine whether our main model comparisons and ablation effects remain stable under an independent evaluator. To address this concern, we re-evaluated existing responses using Gemini 4 Carbon as an independent evaluator, keeping the clinical rubrics, reference criteria, decision-time evidence, and score aggregation unchanged. We conducted two comparisons covering representative models and major agent ablations.

1. Robustness of main model comparisons. We selected six models spanning leading general-purpose models, different open-weight model scales, and a medical-domain model. We compared their L2/L3 clinical quality and safety, together with L3 Temporal Consistency, under both evaluators.

Table R1: Robustness of L2 and L3 Model Comparisons Across Evaluators.

(a) L2: Single-Point Decision-Making

ModelAvg. ↑ (Ours)Avg. ↑ (Gemini)Safety Error ↓ (Ours)Safety Error ↓ (Gemini)
GPT-6-Astra92.293.13.46.7
Claude-Fable-590.889.99.415.3
DeepSeek-V4-Pro83.681.814.817.9
Qwen3.5-27B73.076.228.631.3
Qwen3.5-9B66.167.227.731.5
Fleming-R1-32B60.357.037.644.2

(b) L3: Multi-Step Decision-Making

ModelAvg. ↑ (Ours)Avg. ↑ (Gemini)Temporal Consistency ↑ (Ours)Temporal Consistency ↑ (Gemini)Safety Error ↓ (Ours)Safety Error ↓ (Gemini)
GPT-6-Astra82.783.987.681.517.519.4
Claude-Fable-579.482.182.576.024.325.1
DeepSeek-V4-Pro80.078.981.978.519.220.4
Qwen3.5-27B69.563.281.678.332.540.8
Qwen3.5-9B55.149.664.259.661.964.1
Fleming-R1-32B52.847.466.958.056.559.6

The results show that, although the two evaluators produce differences in absolute scores and minor ranking changes among closely performing models, the overall performance trends and key model rankings remain consistent. In particular, the L2 Avg. and L3 Safety Error rankings are identical across all six models, and GPT-6-Astra remains the top-performing model in L2/L3 Avg. and L3 Temporal Consistency (e.g., its L3 Avg. is 82.7 vs. 83.9, while its Safety Error is 17.5% vs. 19.4% under AnesTRACE-Eval and Gemini, respectively). These consistent findings support the robustness of our main model comparisons and confirm that stronger overall clinical quality does not eliminate serious safety errors. Importantly, AnesTRACE-Eval further provides a lightweight, locally deployable 9B evaluator aligned with anesthesiologists. As shown in Table 5, our evaluator achieves strong correlations with anesthesiologist judgments (L3 Spearman's ρ: 0.74 for Overall Quality and 0.81 for Safety Severity).

2. Robustness of major ablation effects. We also re-evaluated the existing Qwen3.5-27B outputs for Full Agent and four ablated configurations. The results are presented below.

Table R2: Robustness of L3 Agent Ablation Results Across Evaluators (Qwen3.5-27B).

MethodAvg. ↑ (Ours)Avg. ↑ (Gemini)Temporal Consistency ↑ (Ours)Temporal Consistency ↑ (Gemini)Safety Error ↓ (Ours)Safety Error ↓ (Gemini)
Full Agent69.563.281.678.332.540.8
w/o Memory63.859.467.561.735.341.1
w/o Knowledge Tools63.256.780.578.143.848.6
w/o Clinical Requests48.747.279.276.254.359.9
w/o All Tools37.938.164.865.157.460.7

The results show that both evaluators report lower Avg. and Temporal Consistency and higher Safety Error than Full Agent (e.g., removing memory reduces Temporal Consistency from 81.6 to 67.5 under AnesTRACE-Eval and from 78.3 to 61.7 under Gemini). Removing All Tools yields the lowest Avg. and highest Safety Error under both evaluators. These results support the roles of cross-turn memory in temporal updating and information access in clinical quality and safety.

In addition, we conducted independent anesthesiologist validation with adjudicated safety labels, as detailed in our response to Reviewer 8TTv, providing complementary clinical evidence for the reliability of our evaluator.

Together, these results demonstrate that our main quality and safety findings, as well as the key ablation trends, remain largely consistent across independent evaluators. We will incorporate these additional experiments into the revised manuscript to further strengthen the robustness and clinical reliability of AnesTRACE-Eval.

W2 and Q2: Distinguishing Management Updating from Plan Repetition

We appreciate your suggestion to compare the actual agent with a baseline that repeats its preceding plan. To address your concerns, we conducted a Repeat-Previous-Plan control experiment on four representative models: Qwen3.5-9B, Qwen3.5-27B, GPT-6-Astra, and Claude-Fable-5, spanning different model scales, performance levels, and deployment settings. We evaluated this control across all 167 L3 episodes, each requiring at least one management change at a subsequent decision point. The Full Agent uses its original recommendations, whereas the baseline repeats the preceding management plan at subsequent turns. The following table represents the comparison results:

ModelMethodClinical Correctness ↑Evidence Grounding ↑Task Completeness ↑Temporal Consistency ↑Safety Error ↓
Qwen3.5-9BFull Agent50.9245.6368.6964.2261.94%
Repeat-Previous-Plan44.2137.8873.1731.3878.16%
Qwen3.5-27BFull Agent66.1661.8080.6181.5932.50%
Repeat-Previous-Plan52.8247.1078.1356.4453.05%
GPT-6-AstraFull Agent80.6580.5486.9287.6017.54%
Repeat-Previous-Plan68.1170.9886.2260.2128.73%
Claude-Fable-5Full Agent76.1579.3182.6482.5024.30%
Repeat-Previous-Plan57.2659.1883.5159.6233.80%

Across all four models, Repeat-Previous-Plan consistently reduces Temporal Consistency, Clinical Correctness, and Evidence Grounding while increasing the Safety Error Rate (e.g., for GPT-6-Astra, Temporal Consistency drops from 87.60 to 60.21, Clinical Correctness from 80.65 to 68.11, and Safety Error rises from 17.54% to 28.73%).

These results indicate that the temporal evaluation does not simply reward repetition of earlier recommendations. The consistent pattern across four models provides evidence that AnesTRACE-Eval distinguishes actual agent trajectories from mechanical plan repetition, with Temporal Consistency capturing failures to update management alongside corresponding declines in Clinical Correctness and Safety. We will include this comparison and clarify its scope in the revised manuscript.

Reviewer_vdP8OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper presents AnesTRACE, an intraoperative anesthesia benchmark suite that evaluates models across three levels: multimodal perception (L1), single-point decision-making (L2), and multi-step longitudinal decision-making (L3). This benchmark utilizes expert-aligned criteria to evaluate clinical correctness, evidence grounding, completeness, safety, and temporal consistency. 30+ models are evaluated on this dataset, exposing persistent bottlenecks in fine-grained TEE/waveform perception and intervention selection

Critical strengths
  1. This paper unifies perception, single-point decision making and multi-step patient management into one framework, and constructs a respective benchmark dataset, allowing model failures to be localized to specific stages.
  2. The paper evaluates 30+ models on this benchmark, and utilizes evaluation metrics beyond aggregated accuracy, e.g., safety-error severity grading, evidence grounding checks, and a 30-second response-time bound.
Critical weaknesses
  1. While the three-level framework may be new to anesthesia, there are many benchmark datasets for individual/combination of tasks on perception, single-point decision making and multi-step patient management. The authors need to put more emphasis on justifying what is new in respect to the existing benchmarks in literature.
  2. The other critical limitation is that the benchmark is relatively small. Quote from the paper: "After expert review, L1 contains 817 TEE and 1,000 waveform or trend question–answer instances; L2 contains 493 clinical decision anchors from 467 surgical cases; and L3 contains 167 episodes comprising 517 turns from 124 surgical cases." Evaluation on this relatively small dataset may not lead to any solid conclusions.
  3. Regarding the most valuable stage, multi-step decision making, the current benchmark only provides trajectory replay, but without intervention actions (multiple decision-making points) that may alter the outcomes. Hence, the current benchmark could not properly evaluate multi-step decision making from the given trajectories.
  4. In addition, this paper did not generate new findings beyond the current literature. For example, perception bottlenecks, safety errors, and latency-quality tradeoffs largely echo already-known findings in medical AI literature.
Questions for authors
  1. Could the authors more clearly state their novel designs beyond a task-specific adaptation to anesthesiology?
  2. How do the authors justify the robustness of their main conclusions given the relatively small dataset?
  3. Could the authors clarify how multi-step decision-making differs from repeated single-step reasoning? If L3 is intended to evaluate the former, each decision should generate an outcome that affects the subsequent decision. The authors should either provide further justification or revise their claims.
  4. What are the most significant and new findings that go beyond the study of anesthesiology?
9 comments

W1 and Q1: Novelty and Differences from Existing Benchmarks

We appreciate your suggestion to clarify the novelty of AnesTRACE. We have revised the contributions in Section 1 and the comparison with existing benchmarks in Section 2 (see also Table 12). Specifically, AnesTRACE differs from existing benchmarks in four aspects:

  1. Coverage of intraoperative perception and decision-making. AnesTRACE evaluates raw TEE and physiological-waveform perception, management reasoning from standardized patient-state evidence, and recommendation updating across recorded intraoperative trajectories. These complementary settings assess the capabilities required to interpret monitoring evidence, select interventions, and reassess management as patient states evolve.

  2. Longitudinal assessment of clinical quality and safety. AnesTRACE assesses both the quality of individual recommendations and whether management is appropriately maintained or revised as new evidence becomes available. It evaluates Clinical Correctness, Evidence Grounding, Task Completeness, and Temporal Consistency, while reporting Safety separately so that serious clinical errors remain visible alongside average performance.

  3. Explicit assessment of intraoperative response-time constraints. Intraoperative decision support requires timely recommendations as well as clinical quality. AnesTRACE therefore reports P95 Response Latency against a clinician-informed 30-second response-time target, enabling joint assessment of quality, safety, and timeliness.

  4. Reusable designs for clinical evaluation beyond anesthesiology. AnesTRACE controls evidence representation, information access, and temporal scope, supporting targeted comparisons of input formats, history, memory, and tool access. Patient information is restricted to what is available at each decision point, and recorded interventions are distinguished from the agent's previous recommendations. Evaluation combines turn-level clinical quality and episode-level decision updating, with safety reported separately. These designs could be adapted to other clinical specialties by changing the clinical data and expert criteria while retaining the rules for evidence access, intervention tracking, and longitudinal assessment.

BenchmarkIntraoperative DecisionPhysiological WaveformTEEVisual/Signal InputTemporal UpdatingOpen-ended ManagementSafety EvaluationResponse Latency
General medical perception
GMAI-MMBench [1]×××✓××××
OmniMedVQA [2]×××✓××××
ECG-QA [3]×✓×✓××××
General clinical evaluation and decision-making
HealthBench [4]×××××✓✓×
LLMEval-Med [5]×××××△✓×
MIMIC-CDM [6]××××△✓△×
LiveClin [7]×××✓△×××
AgentClinic [8]×××✓△△××
MedAgentBench [9]×××××△××
LongMedBench [10]××××✓×××
Surgical visual benchmarks
SurgVLM-Bench [11]×××✓××××
SurgCoTBench [12]×××✓××××
Anesthesiology benchmarks
CAB [13]△××××△✓×
Hypnos benchmark [14]△××××△✓×
AnesBench / AnesSuite [15]△××××△××
AnesTRACE (ours)✓✓✓✓✓✓✓✓

Note. ✓ indicates explicit coverage; △ indicates partial coverage; × indicates that the capability is not an explicit evaluation target.

References

[1] Chen P, Ye J, Wang G, et al. Gmai-mmbench: A comprehensive multimodal evaluation benchmark towards general medical ai[J]. Advances in Neural Information Processing Systems, 2024, 37: 94327-94427.

[2] Hu Y, Li T, Lu Q, et al. Omnimedvqa: A new large-scale comprehensive evaluation benchmark for medical lvlm[C]//2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2024: 22170-22183.

[3] Oh J, Lee G, Bae S, et al. Ecg-qa: A comprehensive question answering dataset combined with electrocardiogram[J]. Advances in Neural Information Processing Systems, 2023, 36: 66277-66288.

[4] Arora R K, Wei J, Hicks R S, et al. Healthbench: Evaluating large language models towards improved human health[J]. arXiv preprint arXiv:2505.08775, 2025.

[5] Zhang M, Shen Y, Li Z, et al. Llmeval-med: A real-world clinical benchmark for medical llms with physician validation[J]. arXiv preprint arXiv:2506.04078, 2025.

[6] Hager P, Jungmann F, Holland R, et al. Evaluation and mitigation of the limitations of large language models in clinical decision-making[J]. Nature medicine, 2024, 30(9): 2613-2622.

[7] Wang X, Shen Y, Chen J, et al. Liveclin: A live clinical benchmark without leakage[C]//International Conference on Learning Representations. 2026, 2026: 97981-98011.

[8] Schmidgall S, Ziaei R, Harris C, et al. Agentclinic: a multimodal agent benchmark to evaluate ai in simulated clinical environments[J]. arXiv preprint arXiv:2405.07960, 2024.

[9] Jiang Y, Black K C, Geng G, et al. MedAgentBench: a virtual EHR environment to benchmark medical LLM agents[J]. Nejm Ai, 2025, 2(9): AIdbp2500144.

[10] Chen Y, Xu Z, Zhang X, et al. Longmedbench: Benchmarking medical agents for long-horizon clinical decision-making[C]//International Conference on Medical Image Computing and Computer-Assisted Intervention. Cham: Springer Nature Switzerland, 2026: 152-161.

[11] Zeng Z, Zhuo Z, Jia X, et al. Surgvlm: A large vision-language model and systematic evaluation benchmark for surgical intelligence[J]. arXiv preprint arXiv:2506.02555, 2025.

[12] Low C H, Wang Z, Zhang T, et al. Surgraw: Multi-agent workflow with chain of thought reasoning for robotic surgical video analysis[J]. IEEE Robotics and Automation Letters, 2026.

[13] Zhou B, Zhan Y, Wang Z, et al. Benchmarking medical LLMs on anesthesiology: A comprehensive dataset in Chinese[J]. IEEE Transactions on Emerging Topics in Computational Intelligence, 2025, 9(4): 3057-3071.

[14] Wang Z, Jiang J, Zhan Y, et al. Hypnos: A domain-specific large language model for anesthesiology[J]. Neurocomputing, 2025, 624: 129389.

[15] Feng X, Jiang W, Wang Z, et al. AnesSuite: a comprehensive benchmark and dataset suite for anesthesiology reasoning in LLMs[C]//International Conference on Learning Representations. 2026, 2026: 141654-141700.

Reviewer_8TTvOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

AnesTRACE-Bench/Eval provides a three-level benchmark for anesthesia AI: TEE and waveform perception, single-point clinical anchors, and multi-step episodes with memory, evidence requests, and tool use. It combines 817 TEE questions, 1,000 waveform/trend questions, 493 anchors, and 167 episodes/517 turns from de-identified retrospective datasets. Models are scored for clinical correctness, evidence grounding, completeness, temporal consistency, safety, and latency. GPT-6-Astra leads the reported L3 average (82.7) but still has 17.5% safety errors and 74.3s P95 latency; local models have substantially higher error rates.

Critical strengths
  1. This is a clinically motivated task design with explicit information boundaries. The three levels assess distinct capabilities: extracting findings from raw evidence, selecting management given a patient state, and updating recommendations as evidence evolves. L3 restricts information access and distinguishes documented interventions from previous model proposals, providing a carefully scoped setting for studying longitudinal reasoning.
  2. The paper provide separate evaluation of clinical quality, safety, and computational cost. Reporting safety independently prevents serious errors from being obscured by stronger completeness or average quality scores. Ablations on history, memory, patient-information requests, and knowledge tools provide useful evidence beyond a single aggregate leaderboard.
Critical weaknesses
  1. The automatic evaluator underlying the central safety conclusions lacks sufficient independent human validation. Appendix C.3 states that three anesthesiologists divide the validation workload, with each item receiving only one annotation. Consequently, the study does not establish inter-rater agreement or provide adjudicated reference labels. This matters because the abstract emphasizes a precise 17.5% safety error rate, and safety comparisons are central to the paper.
  2. Important aggregate scores are inconsistent with the supplementary component results. Appendix B.2 specifies that aggregate TEE Factual-F1 equally averages functional assessment and abnormality detection. However, GPT-6-Astra receives component scores of 13.0 and 15.7 in Table 15, implying an aggregate of 14.35, while Table 3 reports 27.5. Similarly, Qwen3.8-Max receives 12.5 and 10.1, implying 11.3, while Table 3 reports 29.9.
Questions for authors
  1. Can at least two anesthesiologists independently assess a sample including leading-model outputs, with adjudication of disagreements? Please report Major/Critical precision, recall, confusion matrices, and confidence intervals accounting for observations from the same surgical case. Also provide applicable sample counts, valid safety-label counts, and missing or failed evaluations for each model.
  2. How are the Factual-F1 values in Table 3 derived from Table 15 under the aggregation rule in Appendix B.2? Do the tables reflect different experimental versions, and would corrections change the model rankings?
  3. L2 and L3 differ in case sources, information interfaces, and safety-evaluation units. Can you compare single-point and multi-step settings on matched cases and decision anchors? Such a comparison would help determine how much of the increase from 3.4% to 17.5% reflects longitudinal decision-making rather than differences in task composition.
4 comments

W1 and Q1: Inter-Rater Agreement and Independent Human Validation

We appreciate your concern regarding the limited independent human validation of AnesTRACE-Eval. To address this concern, we expanded the original validation set of Fleming-R1-32B, GPT-OSS-20B, HuatuoGPT-3-32B, MedGemma-27B-text-it, and Qwen3.5-27B (100 L2 responses and 117 L3 decision turns from 39 episodes) by adding 100 L2 responses and 87 L3 decision turns from 29 episodes, generated by GPT-6-Astra and Claude-Fable-5. And we conducted additional annotation. Each response was independently assessed by at least two anesthesiologists, and one senior anesthesiologist is in charge of aligning disagreements.

  1. Inter-Rater Agreement. We report inter-rater agreement across all evaluation dimensions using Cohen’s κ, calculated from independent anesthesiologist ratings before adjudication.

Table R1: Inter-Anesthesiologist Agreement Across Evaluation Dimensions (Cohen’s κ, Before Adjudication)

LevelNClinical CorrectnessEvidence GroundingTask CompletenessSafety (Major/Critical)Temporal Consistency
L2200 responses0.7230.8460.9010.860—
L3204 decision turns0.6270.7620.8010.7350.740
  1. Major/Critical Safety Detection. We report the precision, recall, and confusion matrices of AnesTRACE-Eval in detecting Major/Critical safety errors, using the final decisions made by a senior anesthesiologist to resolve disagreements as the human reference, with surgery-clustered 95% confidence intervals. The results show that AnesTRACE-Eval achieves 85.1% Precision and 84.1% Recall on L2, and 84.7% Precision and 85.7% Recall on L3 for Major/Critical safety errors. These results support its ability to identify serious safety errors, including in leading-model outputs.

Table R2: Major/Critical Safety Detection Against Adjudicated Human Labels.

LevelValidation groupReviewed responses/turnsValid Safety pairsTP / FP / FN / TNPrecision (95% CI)Recall (95% CI)
L2Original five models10010036 / 6 / 7 / 5185.7% (74.3–95.3%)83.7% (72.1–94.1%)
L2GPT-6-Astra505019 / 4 / 4 / 2382.6% (65.2–95.8%)82.6% (65.2–96.0%)
L2Claude-Fable-5505019 / 3 / 3 / 2586.4% (70.4–100.0%)86.4% (70.6–100.0%)
L2All seven models20020074 / 13 / 14 / 9985.1% (77.2–92.1%)84.1% (76.0–91.4%)
L3Original five models11711742 / 9 / 8 / 5882.4% (69.6–93.0%)84.0% (73.2–93.9%)
L3GPT-6-Astra454516 / 2 / 2 / 2588.9% (71.4–100.0%)88.9% (72.2–100.0%)
L3Claude-Fable-5424214 / 2 / 2 / 2487.5% (66.7–100.0%)87.5% (68.8–100.0%)
L3All seven models20420472 / 13 / 12 / 10784.7% (75.6–92.6%)85.7% (77.8–92.8%)
  1. Evaluation Coverage. All 404 validation responses and decision turns received valid evaluations, with zero missing or failed evaluations. Furthermore, across the full benchmark, all applicable Safety evaluations for every model produced valid labels, with no missing or failed evaluations.

W2 and Q2: Consistency of TEE Factual-F1 Results

We sincerely thank you for identifying the inconsistency between the aggregate Factual-F1 scores in Table 3 and the component scores in Table 18, Appendix E.1.

We mistakenly included results from an earlier experimental run in Appendix Table 18. The aggregate TEE Factual-F1 scores in Table 3 are based on the final evaluation run and remain correct. Using the unrounded component scores from that run, we reproduced the TEE Factual-F1 values for all 14 models in Table 3, confirming that the reported scores and model rankings remain unchanged.

As specified in Appendix B.2, the aggregate TEE Factual-F1 is the equally weighted average of the functional-assessment and abnormality-detection scores:

Factual-F1 (TEE)=Factual-F1 (Function)+Factual-F1 (Abnormality)2.\displaystyle \text{Factual-F1 (TEE)} = \frac{\text{Factual-F1 (Function)} + \text{Factual-F1 (Abnormality)}}{2}.

Using the corrected component scores, GPT-6-Astra yields (30.1 + 24.8)/2 = 27.45, reported as 27.5, and Qwen3.8-MAX yields (28.5 + 31.3)/2 = 29.9.

The table below provides the corrected Factual-F1 entries for Table 18. Bold indicates the corrected component values.

ModelProbe ACCAxis ACCCFD ACCPlane ACCVisibility ACCLocalization mIoUAssessability ACCFunction ACCFunction Factual-F1Abnormality ACCAbnormality Factual-F1Value ACCTrend Set-F1Anomaly Metric-F1
GPT-6-Astra58.570.993.372.762.932.268.830.130.123.224.891.561.164.8
Gemini 3 Pro Preview69.567.193.356.646.124.646.327.724.529.322.977.062.654.8
Claude-Fable-565.166.793.372.942.723.439.239.129.335.124.580.568.759.6
Qwen3.8-MAX68.364.693.366.751.722.862.557.828.524.431.372.069.642.8
InternVL3.5-38B61.027.893.332.337.116.847.547.021.124.410.157.069.623.0
Qwen3-VL-32B52.449.793.346.832.318.223.645.97.641.56.455.071.728.6
Qwen3.5-27B36.659.593.344.443.818.943.856.617.929.315.156.065.336.5
Qwen3.8-27B58.548.193.334.333.714.842.551.814.539.013.557.572.531.3
Qwen3.5-9B59.830.493.333.352.815.540.057.89.828.07.654.560.733.1
Qwen3-VL-8B58.529.190.028.323.614.445.037.35.124.46.345.058.223.5
InternVL3.5-8B59.831.693.331.348.313.941.233.77.129.35.541.070.117.3
LLaVA-OneVision-2-8B54.930.490.024.250.614.332.547.010.831.78.649.560.623.6
Fleming-VL-38B46.319.093.342.437.114.836.228.911.619.510.841.559.424.0
Lingshu-32B51.239.293.336.446.112.141.233.719.730.515.549.564.620.0
Lingshu-I-8B58.549.476.729.356.213.533.841.09.925.68.539.059.523.3
Fleming-VL-8B34.132.983.338.446.18.72.524.18.012.26.638.541.117.6

Q3: Matched Single-Step and Multi-Step Evaluation

We appreciate your suggestion to compare the two settings on matched cases and decision anchors. The key distinction lies in the use of cross-turn clinical history and episode memory. Single-Step processes each decision point independently, removing preceding patient states, model responses, and cross-turn memory while retaining within-turn tool use and reasoning. To address this concern, we conducted a matched comparison across 14 models using the same L3 cases and decision anchors. We kept the model, patient information available at the current decision point, tool inventory, and per-decision tool budget identical between conditions. Both conditions use the same AnesTRACE-Eval protocol, with Safety assessed at the same decision-turn level. The table below provides detailed results:

ModelMethodClinical Correctness ↑Evidence Grounding ↑Task Completeness ↑Avg. ↑Safety Error ↓
Qwen3-8BSingle-Step14.5915.4446.8025.6179.93%
Multi-Step23.0919.5848.0230.2364.05%
Qwen3.5-9BSingle-Step43.1743.4064.7050.4269.29%
Multi-Step50.9245.6368.6955.0861.94%
HuatuoGPT-3-8BSingle-Step16.2012.7049.5026.1388.20%
Multi-Step19.0214.1668.1933.7986.58%
Fleming-R1-7BSingle-Step13.5011.8029.6018.3091.70%
Multi-Step36.7631.9264.6744.4569.37%
Morpheus-7BSingle-Step15.8012.2045.1024.3789.40%
Multi-Step23.7217.5864.0535.1278.19%
Qwen3.5-27BSingle-Step61.9059.3075.3065.5040.80%
Multi-Step66.1661.8080.6169.5232.50%
Qwen3.8-27BSingle-Step57.1151.8274.2261.0540.11%
Multi-Step62.8160.5274.9166.0834.05%
HuatuoGPT-3-32BSingle-Step40.5835.9967.1947.9279.29%
Multi-Step43.2537.9268.5949.9264.69%
Fleming-R1-32BSingle-Step45.7043.0059.9049.5360.45%
Multi-Step48.7743.5366.2052.8356.52%
GPT-6-AstraSingle-Step69.9972.8385.9176.2422.01%
Multi-Step80.6580.5486.9282.7017.54%
Claude-Fable-5Single-Step75.2271.5080.3375.6830.92%
Multi-Step76.1579.3182.6479.3724.30%
Gemini 3 Pro PreviewSingle-Step62.9855.3173.9564.0844.34%
Multi-Step63.2360.7074.3166.0838.52%
DeepSeek-V4-ProSingle-Step79.6865.3989.3378.1332.32%
Multi-Step77.5274.8587.6580.0119.17%
Qwen3.8-MaxSingle-Step60.1356.9272.5063.1845.09%
Multi-Step64.6358.4178.3467.1340.09%

Our analysis indicates that Multi-Step setting achieves higher average clinical quality, and exhibits lower Safety Error Rates (e.g., for GPT-6-Astra, Clinical Correctness increases from 69.99 to 80.65, Avg. from 76.24 to 82.70, and Safety Error decreases from 22.01% to 17.54%). These results suggest that cross-turn history and memory help models track evolving patient states and update management. In contrast, without accumulated clinical history and episode memory, Single-Step cannot directly build on earlier assessments and recommendations across decision points.

Dear Reviewer 8TTv,

Thank you for your time and effort in reviewing our work. We would like to know if our additional results and explanations has addressed your concerns and if it would be possible to consider raising the score. If there be any remaining issues, we would be more than willing to resolve them.

Best regards, the Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.