- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces AnesTRACE, an anesthesia-specific benchmark covering multimodal perception, single-point management, and multi-step decision updating through recorded clinical trajectories. It also develops an expert-guided evaluator for clinical correctness, evidence grounding, completeness, safety, and temporal consistency. Evaluation across more than 30 models, physician-agreement analysis, and input, history, and tool ablations reveal persistent limitations in perception and management, while highlighting the importance of clinical context and information access.
Critical strengths
-
The clinically motivated task hierarchy provides useful coverage of perception, management reasoning, and longitudinal decision updating. Its focus on intraoperative anesthesia makes it a valuable complement to general medical question-answering benchmarks.
-
Reporting serious safety errors separately from average clinical quality makes the evaluation more informative. The broad model comparison and history, memory, and tool ablations provide useful insights into the capabilities needed for this setting.
Critical weaknesses
-
The main model comparisons and safety findings rely on one learned evaluator. Although agreement with clinicians is reported, the robustness of the headline conclusions and ablation effects to evaluator choice remains unclear.
-
The history and memory ablations are informative, but the temporal evaluation lacks a targeted control demonstrating that it distinguishes necessary management updates from simply maintaining earlier recommendations.
Questions for authors
-
Using existing responses from representative models and major ablations, do the main quality and safety conclusions remain stable under an independent evaluator using the same clinical criteria and decision-time evidence?
-
On episodes where the existing reference criteria require a management change, how does the actual agent compare with a baseline that repeats its previous plan? Reporting temporal consistency alongside clinical correctness and safety would help establish what the temporal metric captures.
-
Could you clarify the discrepancy between the TEE Factual-F1 results in Tables 3 and 15? Under the equal-weight aggregation specified in Appendix B.2, GPT-6-Astra’s scores of 13.0 and 15.7 yield 14.35, whereas Table 3 reports 27.5. Were different evaluation settings used, and does resolving this discrepancy affect the model rankings? I would reconsider my rating if the authors address these concerns.
6 comments
2. Robustness of major ablation effects. We also re-evaluated the existing Qwen3.5-27B outputs for Full Agent and four ablated configurations. The results are presented below.
Table R2: Robustness of L3 Agent Ablation Results Across Evaluators (Qwen3.5-27B).
| Method | Avg. ↑ (Ours) | Avg. ↑ (Gemini) | Temporal Consistency ↑ (Ours) | Temporal Consistency ↑ (Gemini) | Safety Error ↓ (Ours) | Safety Error ↓ (Gemini) |
|---|---|---|---|---|---|---|
| Full Agent | 69.5 | 63.2 | 81.6 | 78.3 | 32.5 | 40.8 |
| w/o Memory | 63.8 | 59.4 | 67.5 | 61.7 | 35.3 | 41.1 |
| w/o Knowledge Tools | 63.2 | 56.7 | 80.5 | 78.1 | 43.8 | 48.6 |
| w/o Clinical Requests | 48.7 | 47.2 | 79.2 | 76.2 | 54.3 | 59.9 |
| w/o All Tools | 37.9 | 38.1 | 64.8 | 65.1 | 57.4 | 60.7 |
The results show that both evaluators report lower Avg. and Temporal Consistency and higher Safety Error than Full Agent (e.g., removing memory reduces Temporal Consistency from 81.6 to 67.5 under AnesTRACE-Eval and from 78.3 to 61.7 under Gemini). Removing All Tools yields the lowest Avg. and highest Safety Error under both evaluators. These results support the roles of cross-turn memory in temporal updating and information access in clinical quality and safety.
In addition, we conducted independent anesthesiologist validation with adjudicated safety labels, as detailed in our response to Reviewer 8TTv, providing complementary clinical evidence for the reliability of our evaluator.
Together, these results demonstrate that our main quality and safety findings, as well as the key ablation trends, remain largely consistent across independent evaluators. We will incorporate these additional experiments into the revised manuscript to further strengthen the robustness and clinical reliability of AnesTRACE-Eval.
W2 and Q2: Distinguishing Management Updating from Plan Repetition
We appreciate your suggestion to compare the actual agent with a baseline that repeats its preceding plan. To address your concerns, we conducted a Repeat-Previous-Plan control experiment on four representative models: Qwen3.5-9B, Qwen3.5-27B, GPT-6-Astra, and Claude-Fable-5, spanning different model scales, performance levels, and deployment settings. We evaluated this control across all 167 L3 episodes, each requiring at least one management change at a subsequent decision point. The Full Agent uses its original recommendations, whereas the baseline repeats the preceding management plan at subsequent turns. The following table represents the comparison results:
| Model | Method | Clinical Correctness ↑ | Evidence Grounding ↑ | Task Completeness ↑ | Temporal Consistency ↑ | Safety Error ↓ |
|---|---|---|---|---|---|---|
| Qwen3.5-9B | Full Agent | 50.92 | 45.63 | 68.69 | 64.22 | 61.94% |
| Repeat-Previous-Plan | 44.21 | 37.88 | 73.17 | 31.38 | 78.16% | |
| Qwen3.5-27B | Full Agent | 66.16 | 61.80 | 80.61 | 81.59 | 32.50% |
| Repeat-Previous-Plan | 52.82 | 47.10 | 78.13 | 56.44 | 53.05% | |
| GPT-6-Astra | Full Agent | 80.65 | 80.54 | 86.92 | 87.60 | 17.54% |
| Repeat-Previous-Plan | 68.11 | 70.98 | 86.22 | 60.21 | 28.73% | |
| Claude-Fable-5 | Full Agent | 76.15 | 79.31 | 82.64 | 82.50 | 24.30% |
| Repeat-Previous-Plan | 57.26 | 59.18 | 83.51 | 59.62 | 33.80% |
Across all four models, Repeat-Previous-Plan consistently reduces Temporal Consistency, Clinical Correctness, and Evidence Grounding while increasing the Safety Error Rate (e.g., for GPT-6-Astra, Temporal Consistency drops from 87.60 to 60.21, Clinical Correctness from 80.65 to 68.11, and Safety Error rises from 17.54% to 28.73%).
These results indicate that the temporal evaluation does not simply reward repetition of earlier recommendations. The consistent pattern across four models provides evidence that AnesTRACE-Eval distinguishes actual agent trajectories from mechanical plan repetition, with Temporal Consistency capturing failures to update management alongside corresponding declines in Clinical Correctness and Safety. We will include this comparison and clarify its scope in the revised manuscript.
W1 and Q1: Robustness to Evaluator Choice
We appreciate your suggestion to examine whether our main model comparisons and ablation effects remain stable under an independent evaluator. To address this concern, we re-evaluated existing responses using Gemini 4 Carbon as an independent evaluator, keeping the clinical rubrics, reference criteria, decision-time evidence, and score aggregation unchanged. We conducted two comparisons covering representative models and major agent ablations.
1. Robustness of main model comparisons. We selected six models spanning leading general-purpose models, different open-weight model scales, and a medical-domain model. We compared their L2/L3 clinical quality and safety, together with L3 Temporal Consistency, under both evaluators.
Table R1: Robustness of L2 and L3 Model Comparisons Across Evaluators.
(a) L2: Single-Point Decision-Making
(b) L3: Multi-Step Decision-Making
The results show that, although the two evaluators produce differences in absolute scores and minor ranking changes among closely performing models, the overall performance trends and key model rankings remain consistent. In particular, the L2 Avg. and L3 Safety Error rankings are identical across all six models, and GPT-6-Astra remains the top-performing model in L2/L3 Avg. and L3 Temporal Consistency (e.g., its L3 Avg. is 82.7 vs. 83.9, while its Safety Error is 17.5% vs. 19.4% under AnesTRACE-Eval and Gemini, respectively). These consistent findings support the robustness of our main model comparisons and confirm that stronger overall clinical quality does not eliminate serious safety errors. Importantly, AnesTRACE-Eval further provides a lightweight, locally deployable 9B evaluator aligned with anesthesiologists. As shown in Table 5, our evaluator achieves strong correlations with anesthesiologist judgments (L3 Spearman's ρ: 0.74 for Overall Quality and 0.81 for Safety Severity).