- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces TraceScope, a report-grounded multimodal benchmark for upper-GI endoscopy, organized into six task families under a unified clinical specification. The authors also conduct extensive experiments to analyze the capability boundaries of current models.
Critical strengths
-
The motivation is important. Decomposing the overall workflow into finer-grained tasks enables more detailed assessment of model capabilities at different stages.
-
Beyond introducing the benchmark, the authors conduct extensive experiments to investigate the limitations of current models, providing useful insights for the community.
Critical weaknesses
-
The overall novelty is limited, and the necessity has not been fully demonstrated yet. Similar evaluation settings have been explored in prior endoscopy benchmarks.
-
The benchmark evaluates models using task-specific visual evidence, but does not assess whether models can autonomously identify and retrieve relevant evidence from full examinations. This limits its ability to characterize end-to-end clinical workflow capabilities.
-
As the benchmark is the paper's core contribution, its construction relies heavily on LLMs, while the extent of independent expert validation appears limited. This raises concerns about the quality and reliability of the benchmark.
-
The prior-only baseline achieves very high accuracy on some tasks, particularly T2 single-finding presence. This may indicate issues with the data distribution or benchmark construction, potentially undermining the validity and interpretability of the evaluation results.
Questions for authors
-
How many LLM-generated samples were independently reviewed by independent experts, and what quantitative validation results support the overall quality of the benchmark?
-
Have the authors investigated the reasons behind the high prior-only accuracy? What are its implications for benchmark validity and the interpretation of the reported model performance?
-
What unique value does the task–evidence alignment design offer beyond existing multi-stage evaluation settings?
5 comments
Response to Reviewer 26oi (Part 2/3)
(Continued from above)
3. How does TraceScope evaluate evidence identification and retrieval? (W2)
T4 and T6 assess evidence identification and integration within supplied examination contexts. In T4, models must identify and integrate relevant findings from redundant examination-level images; site coverage, finding recovery, and the multidimensional SEFR rubric assess the completeness and clinical consistency of the resulting reports. In T6, models must identify guideline-relevant signs, cite supporting frames, and connect those observations to clinical conclusions. The ELRC rubric further evaluates factual fidelity, reasoning consistency, and evidence-calibrated conclusions against the fixed references.
We additionally evaluated automatic evidence selection using a question-and-option-guided BiomedCLIP + MMR selector, which retrieves frames from the retained examination image pool without reference impressions or ground-truth labels. Under matched case-specific frame budgets, automatic selection achieves 47.21% mean diagnostic accuracy, compared with 53.20% for Selected across nine models—a 5.99 pp gap. Selected also outperforms equal-count uniform video sampling by 8.8 pp.
These evaluations address complementary aspects of evidence handling: task-level identification and integration within supplied contexts, and automatic selection from retained images. Autonomous retrieval over uncurated procedural streams is a broader setting. The controlled Presentation axis provides a fixed downstream diagnostic endpoint for comparing evidence-selection methods.
We have clarified the evaluation rationale for task–evidence alignment in the Introduction and Section 3 and sharpened the comparison with existing endoscopic benchmarks in Related Work and the discussion accompanying Table 1. The automatic-selection protocol and results are included in Sections 4–5.3 and Appendices B.2/C.3. Appendix E clarifies the scope of evidence identification and retrieval, and Appendix F discusses the corresponding implications for model design.
[W3 & Q1] Independent Expert Validation and Benchmark Reliability
We thank the reviewer for requesting quantitative expert validation. We report independent clinician assessments of released references, terminology mappings, and automatic semantic scoring.
To quantitatively assess reference quality, two clinicians independently reviewed 275 released benchmark items across T2–T6, evaluating reference correctness and visual evidence support under task-specific clinical criteria. The results are summarized below:
| Task | Reviewed | Reference Correctness | Visual Evidence Support |
|---|---|---|---|
| T2 · Findings | 80 | 97.5% | 96.2% |
| T3 · Attributes | 60 | 98.3% | 96.7% |
| T4 · Report | 20 | 100% | 100% |
| T5 · Impression | 55 | 98.2% | 98.2% |
| T6 · Guideline | 60 | 96.7% | 96.7% |
| Overall | 275 | 97.8% | 97.1% |
Percentages denote both-clinician approval based on independent judgments before reconciliation; visual support is assessed according to each task's clinical requirements.
Beyond these reference-level assessments, 70 clinical terminology mappings were independently evaluated by two clinicians, achieving 98.6% both-clinician approval and 98.6% exact agreement. For T6, the audit additionally confirmed 96.7% key-sign support and 98.3% guideline consistency. These independent assessments complement the construction pipeline's visual evidence verification and deterministic rule checks.
We also separately calibrated LLM-based evaluation against clinician judgments on 10 T4 questions and 24 T6 questions, covering 160 question–dimension scoring units across their four and five scoring dimensions, respectively. Judge–clinician exact agreement reached 85.0% and 85.4%, respectively, providing empirical calibration evidence for the automatic rubric scores on this sampled panel.
These assessments cover reference correctness, visual evidence support, terminology mappings, and semantic scoring. Appendix A.5 reports the protocols and results together, including task-specific review criteria, independent both-clinician approval rates, and agreement statistics.
Response to Reviewer 26oi (Part 3/3)
[W4 & Q2] Class Imbalance, Prior-only Performance, and Visual Dependence
We thank the reviewer for asking us to explain the high prior-only score. We examined answer-frequency structure, class-balanced finding recognition, and sensitivity to image content on the Presence task.
1. Why does the prior-only baseline achieve high accuracy?
“very high accuracy ... particularly T2 single-finding presence” (W4)
Among the 1,558 Presence items shared by all 22 models, 204 (13.09%) are Visible and 1,354 (86.91%) are Not Visible. A constant Not Visible response therefore achieves 86.91% accuracy and 50.0% balanced accuracy (BA). These frequencies describe the visibility of the queried finding in the supplied evidence. We interpret accuracy alongside BA and class-wise recall to distinguish overall correctness from recognition of both classes.
2. Does this affect benchmark validity and model-performance interpretation?
“benchmark validity and the interpretation of ... model performance” (Q2)
This evaluation issue has precedents in visual benchmarks. TDIUC [3] evaluates most-frequent-answer and question-only baselines and averages accuracy across unique answers to account for answer imbalance. Following the same principle of distinguishing frequent-answer success from performance across answer classes, we evaluated all 22 models on the same 1,558-item panel, reporting balanced accuracy (BA) and class-wise recall.
| Model | Acc. | BA | Visible Recall | Not-visible Recall |
|---|---|---|---|---|
| Majority-class prior | 86.91% | 50.00% | 0.00% | 100.00% |
| Qwen3.5-27B | 93.52% | 93.56% | 93.63% | 93.50% |
| Hulu-Med-4B | 94.54% | 87.70% | 78.43% | 96.97% |
| Qwen3-VL-8B | 92.75% | 83.34% | 70.59% | 96.09% |
| Lingshu-I-8B | 90.12% | 62.67% | 25.49% | 99.85% |
The class-wise results reveal different finding-recognition profiles. Qwen3.5-27B achieves 93.56% BA, with Visible and Not Visible recalls of 93.63% and 93.50%, respectively. Lingshu-I-8B combines 90.12% overall accuracy with 25.49% Visible recall. Reporting both recalls and their balanced average makes these differences explicit. Appendix C.1 provides the complete 22-model results.
3. How does image content affect finding recognition?
We conducted paired image interventions using Hulu-Med-4B and Qwen3-VL-8B on the 1,555 items with eligible donor examinations, drawn from the original 1,558-item panel. We compared the original images with (i) images from another examination at the same anatomical site and frame count and (ii) no images. Questions, options, and reference answers remain fixed across conditions.
| Model | Original BA | Swapped BA | No-image BA |
|---|---|---|---|
| Hulu-Med-4B | 88.04% | 65.37% | 52.50% |
| Qwen3-VL-8B | 83.37% | 64.44% | 51.49% |
Replacing case-matched images reduces BA by 22.67 pp and 18.93 pp, respectively; removing images reduces it by 35.54 pp and 31.88 pp (all paired 95% CIs exclude zero). The substitution comparison holds anatomical site and frame count constant, showing that finding-recognition performance in both models responds to case-specific visual content.
These analyses connect answer-frequency structure, class-balanced finding recognition, and the effect of case-specific images. Section 5.2 summarizes the class-balanced Presence result, Appendix B.4 defines BA and the class labels, and Appendix C.1 reports the complete 22-model evaluation and paired image interventions. The tables separately identify the 1,558-item recognition panel and its 1,555-item intervention subset.
References
[1] Liu, S., et al. EndoBench: A comprehensive evaluation of multi-modal large language models for endoscopy analysis. NeurIPS, 2025.
[2] Zhu, Y., et al. GI-Bench: A panoramic benchmark revealing the knowledge-experience dissociation of multimodal large language models in gastrointestinal endoscopy against clinical standards. arXiv:2601.08183, 2026.
[3] Kafle, K., & Kanan, C. An analysis of visual question answering algorithms. ICCV, 2017.
Response to Reviewer 26oi (Part 1/3)
Thank you for reviewing our work and for the thoughtful feedback. We appreciate your acknowledgment of the importance of fine-grained clinical evaluation in endoscopy. Your comments have helped us present the motivation, scope, and distinctive value of TraceScope more clearly, and better explain its evaluation design and findings. Below, we respond to each of your concerns.
[W1, W2 & Q3] Motivation and Distinctive Value of Task–Evidence Alignment
We thank the reviewer for encouraging us to clarify the motivation and distinctive value of TraceScope. Notably, reviewers ERy8 and bBeb explicitly recognized the evaluative value of task–evidence alignment, and we appreciate your own acknowledgment of the importance of fine-grained clinical evaluation. We also value your concerns regarding novelty and autonomous evidence retrieval. Our central contribution is to align clinical tasks with their required evidence and examine how diagnostic decisions respond to controlled changes in visual and clinical information. Below, we clarify why this alignment is necessary, how TraceScope advances beyond existing benchmarks, and how its evaluation incorporates evidence identification and selection.
1. Why does task–evidence alignment matter? (Q3)
Different clinical tasks require different evidence scopes. Recognizing a local finding may require only a few relevant frames, whereas generating a complete examination report requires integrating observations across anatomical sites. Guideline-based decisions further require identifying relevant clinical signs and applying diagnostic criteria.
TraceScope explicitly aligns these requirements: T2–T3 assess findings and attributes using compact, target-relevant evidence; T4 evaluates anatomical coverage and finding recovery across redundant examination-level images; and T5–T6 assess diagnostic impressions and guideline-based decisions using matched examination-level evidence. A shared clinical specification connects task targets, evidence requirements, reference provenance, clinical concepts, and scoring criteria. This alignment makes the evidence demands of each task explicit, supporting more interpretable profiles of local recognition, cross-image integration, and guideline-based reasoning. The paired axes then hold the case and diagnostic target fixed to examine responses to controlled information changes.
2. What distinguishes TraceScope from existing multi-stage benchmarks? (W1)
The distinction extends beyond task coverage. Among the most closely related endoscopic benchmarks, EndoBench [1] evaluates diverse endoscopic tasks across five visual prompting granularities, including region-level annotations, while GI-Bench [2] assesses five clinical workflow dimensions through image-based questions. Both provide valuable assessments of task-level competence.
TraceScope connects local recognition, examination-level synthesis, and guideline-based decisions through explicitly matched evidence requirements. Crucially, it also introduces two complementary within-case diagnostic axes:
These paired comparisons extend conventional stage-wise evaluation by revealing how evidence organization and clinical-state information influence the same downstream decision. TraceScope therefore provides both task-specific clinical capability profiles and controlled analyses of diagnostic information dependencies.