acceptodds
Under review as a conference paper at ICLR 2027

TraceScope: Tracing Failures from Evidence to Decision in Upper-GI Endoscopy

Abstract

Upper-gastrointestinal (upper-GI) endoscopy spans clinical tasks from local visual assessment to examination-level interpretation and decision making. During an examination, clinicians localize anatomy, identify abnormalities, and characterize lesion attributes, while integrating distributed observations to formulate reports, diagnostic impressions, and guideline-guided decisions. These tasks require different scopes of visual evidence: local judgments rely on compact, target-relevant views, whereas examination-level interpretation requires integrating distributed and often redundant evidence across the examination. Recent endoscopic benchmarks have substantially broadened clinical task coverage, yet task definition and evidence scope are still commonly evaluated without systematic alignment. To bridge this gap, we introduce TraceScope, a report-grounded multimodal benchmark for upper-GI endoscopy. Its main board contains 9,106 task-specific items from 4,893 examinations and 70,386 item–frame references. A shared clinical specification organizes six task families spanning site recognition, abnormal findings, finding attributes, report generation, diagnostic impression, and guideline reasoning. For each task, the delivered visual context is matched to its evidence requirements in clinical practice, ranging from compact, target-relevant views to redundant examination-level evidence. Across 22 multimodal models, the main board reveals a stage-selective capability profile: performance on local readouts does not extend uniformly to clinical composition, while reference-consistent reasoning remains separable from exact guideline decisions. Two complementary within-case analyses further show that selected multi-image presentation consistently outperforms equal-count uniformly sampled video input, while making guideline-sign polarity explicit yields the largest positive fleet-level change along the clinical-state cascade. Together, TraceScope aligns stage-resolved clinical evaluation with task-appropriate evidence context, enabling unified analysis of model capability across the upper-GI workflow and of downstream decision sensitivity to visual and clinical information.

Primary area:
datasets and benchmarks
Keywords:
medical benchmark, multimodal large language model, endoscopy

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_26oiOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces TraceScope, a report-grounded multimodal benchmark for upper-GI endoscopy, organized into six task families under a unified clinical specification. The authors also conduct extensive experiments to analyze the capability boundaries of current models.

Critical strengths
  1. The motivation is important. Decomposing the overall workflow into finer-grained tasks enables more detailed assessment of model capabilities at different stages.

  2. Beyond introducing the benchmark, the authors conduct extensive experiments to investigate the limitations of current models, providing useful insights for the community.

Critical weaknesses
  1. The overall novelty is limited, and the necessity has not been fully demonstrated yet. Similar evaluation settings have been explored in prior endoscopy benchmarks.

  2. The benchmark evaluates models using task-specific visual evidence, but does not assess whether models can autonomously identify and retrieve relevant evidence from full examinations. This limits its ability to characterize end-to-end clinical workflow capabilities.

  3. As the benchmark is the paper's core contribution, its construction relies heavily on LLMs, while the extent of independent expert validation appears limited. This raises concerns about the quality and reliability of the benchmark.

  4. The prior-only baseline achieves very high accuracy on some tasks, particularly T2 single-finding presence. This may indicate issues with the data distribution or benchmark construction, potentially undermining the validity and interpretability of the evaluation results.

Questions for authors
  1. How many LLM-generated samples were independently reviewed by independent experts, and what quantitative validation results support the overall quality of the benchmark?

  2. Have the authors investigated the reasons behind the high prior-only accuracy? What are its implications for benchmark validity and the interpretation of the reported model performance?

  3. What unique value does the task–evidence alignment design offer beyond existing multi-stage evaluation settings?

5 comments
AuthorsOpenReview ↗

Response to Reviewer 26oi (Part 1/3)

Thank you for reviewing our work and for the thoughtful feedback. We appreciate your acknowledgment of the importance of fine-grained clinical evaluation in endoscopy. Your comments have helped us present the motivation, scope, and distinctive value of TraceScope more clearly, and better explain its evaluation design and findings. Below, we respond to each of your concerns.


[W1, W2 & Q3] Motivation and Distinctive Value of Task–Evidence Alignment

We thank the reviewer for encouraging us to clarify the motivation and distinctive value of TraceScope. Notably, reviewers ERy8 and bBeb explicitly recognized the evaluative value of task–evidence alignment, and we appreciate your own acknowledgment of the importance of fine-grained clinical evaluation. We also value your concerns regarding novelty and autonomous evidence retrieval. Our central contribution is to align clinical tasks with their required evidence and examine how diagnostic decisions respond to controlled changes in visual and clinical information. Below, we clarify why this alignment is necessary, how TraceScope advances beyond existing benchmarks, and how its evaluation incorporates evidence identification and selection.

1. Why does task–evidence alignment matter? (Q3)

Different clinical tasks require different evidence scopes. Recognizing a local finding may require only a few relevant frames, whereas generating a complete examination report requires integrating observations across anatomical sites. Guideline-based decisions further require identifying relevant clinical signs and applying diagnostic criteria.

TraceScope explicitly aligns these requirements: T2–T3 assess findings and attributes using compact, target-relevant evidence; T4 evaluates anatomical coverage and finding recovery across redundant examination-level images; and T5–T6 assess diagnostic impressions and guideline-based decisions using matched examination-level evidence. A shared clinical specification connects task targets, evidence requirements, reference provenance, clinical concepts, and scoring criteria. This alignment makes the evidence demands of each task explicit, supporting more interpretable profiles of local recognition, cross-image integration, and guideline-based reasoning. The paired axes then hold the case and diagnostic target fixed to examine responses to controlled information changes.

2. What distinguishes TraceScope from existing multi-stage benchmarks? (W1)

The distinction extends beyond task coverage. Among the most closely related endoscopic benchmarks, EndoBench [1] evaluates diverse endoscopic tasks across five visual prompting granularities, including region-level annotations, while GI-Bench [2] assesses five clinical workflow dimensions through image-based questions. Both provide valuable assessments of task-level competence.

TraceScope connects local recognition, examination-level synthesis, and guideline-based decisions through explicitly matched evidence requirements. Crucially, it also introduces two complementary within-case diagnostic axes:

  • Evidence Presentation varies visual evidence selection and presentation while holding the examination and diagnostic target fixed.
  • Diagnostic Cascade varies supplied clinical observations, sign polarity, and decision rules for the same diagnostic target.

These paired comparisons extend conventional stage-wise evaluation by revealing how evidence organization and clinical-state information influence the same downstream decision. TraceScope therefore provides both task-specific clinical capability profiles and controlled analyses of diagnostic information dependencies.

AuthorsOpenReview ↗

Response to Reviewer 26oi (Part 2/3)

(Continued from above)

3. How does TraceScope evaluate evidence identification and retrieval? (W2)

T4 and T6 assess evidence identification and integration within supplied examination contexts. In T4, models must identify and integrate relevant findings from redundant examination-level images; site coverage, finding recovery, and the multidimensional SEFR rubric assess the completeness and clinical consistency of the resulting reports. In T6, models must identify guideline-relevant signs, cite supporting frames, and connect those observations to clinical conclusions. The ELRC rubric further evaluates factual fidelity, reasoning consistency, and evidence-calibrated conclusions against the fixed references.

We additionally evaluated automatic evidence selection using a question-and-option-guided BiomedCLIP + MMR selector, which retrieves frames from the retained examination image pool without reference impressions or ground-truth labels. Under matched case-specific frame budgets, automatic selection achieves 47.21% mean diagnostic accuracy, compared with 53.20% for Selected across nine models—a 5.99 pp gap. Selected also outperforms equal-count uniform video sampling by 8.8 pp.

These evaluations address complementary aspects of evidence handling: task-level identification and integration within supplied contexts, and automatic selection from retained images. Autonomous retrieval over uncurated procedural streams is a broader setting. The controlled Presentation axis provides a fixed downstream diagnostic endpoint for comparing evidence-selection methods.

We have clarified the evaluation rationale for task–evidence alignment in the Introduction and Section 3 and sharpened the comparison with existing endoscopic benchmarks in Related Work and the discussion accompanying Table 1. The automatic-selection protocol and results are included in Sections 4–5.3 and Appendices B.2/C.3. Appendix E clarifies the scope of evidence identification and retrieval, and Appendix F discusses the corresponding implications for model design.


[W3 & Q1] Independent Expert Validation and Benchmark Reliability

We thank the reviewer for requesting quantitative expert validation. We report independent clinician assessments of released references, terminology mappings, and automatic semantic scoring.

To quantitatively assess reference quality, two clinicians independently reviewed 275 released benchmark items across T2–T6, evaluating reference correctness and visual evidence support under task-specific clinical criteria. The results are summarized below:

TaskReviewedReference CorrectnessVisual Evidence Support
T2 · Findings8097.5%96.2%
T3 · Attributes6098.3%96.7%
T4 · Report20100%100%
T5 · Impression5598.2%98.2%
T6 · Guideline6096.7%96.7%
Overall27597.8%97.1%

Percentages denote both-clinician approval based on independent judgments before reconciliation; visual support is assessed according to each task's clinical requirements.

Beyond these reference-level assessments, 70 clinical terminology mappings were independently evaluated by two clinicians, achieving 98.6% both-clinician approval and 98.6% exact agreement. For T6, the audit additionally confirmed 96.7% key-sign support and 98.3% guideline consistency. These independent assessments complement the construction pipeline's visual evidence verification and deterministic rule checks.

We also separately calibrated LLM-based evaluation against clinician judgments on 10 T4 questions and 24 T6 questions, covering 160 question–dimension scoring units across their four and five scoring dimensions, respectively. Judge–clinician exact agreement reached 85.0% and 85.4%, respectively, providing empirical calibration evidence for the automatic rubric scores on this sampled panel.

These assessments cover reference correctness, visual evidence support, terminology mappings, and semantic scoring. Appendix A.5 reports the protocols and results together, including task-specific review criteria, independent both-clinician approval rates, and agreement statistics.

AuthorsOpenReview ↗

Response to Reviewer 26oi (Part 3/3)

[W4 & Q2] Class Imbalance, Prior-only Performance, and Visual Dependence

We thank the reviewer for asking us to explain the high prior-only score. We examined answer-frequency structure, class-balanced finding recognition, and sensitivity to image content on the Presence task.

1. Why does the prior-only baseline achieve high accuracy?

“very high accuracy ... particularly T2 single-finding presence” (W4)

Among the 1,558 Presence items shared by all 22 models, 204 (13.09%) are Visible and 1,354 (86.91%) are Not Visible. A constant Not Visible response therefore achieves 86.91% accuracy and 50.0% balanced accuracy (BA). These frequencies describe the visibility of the queried finding in the supplied evidence. We interpret accuracy alongside BA and class-wise recall to distinguish overall correctness from recognition of both classes.

2. Does this affect benchmark validity and model-performance interpretation?

“benchmark validity and the interpretation of ... model performance” (Q2)

This evaluation issue has precedents in visual benchmarks. TDIUC [3] evaluates most-frequent-answer and question-only baselines and averages accuracy across unique answers to account for answer imbalance. Following the same principle of distinguishing frequent-answer success from performance across answer classes, we evaluated all 22 models on the same 1,558-item panel, reporting balanced accuracy (BA) and class-wise recall.

ModelAcc.BAVisible RecallNot-visible Recall
Majority-class prior86.91%50.00%0.00%100.00%
Qwen3.5-27B93.52%93.56%93.63%93.50%
Hulu-Med-4B94.54%87.70%78.43%96.97%
Qwen3-VL-8B92.75%83.34%70.59%96.09%
Lingshu-I-8B90.12%62.67%25.49%99.85%

The class-wise results reveal different finding-recognition profiles. Qwen3.5-27B achieves 93.56% BA, with Visible and Not Visible recalls of 93.63% and 93.50%, respectively. Lingshu-I-8B combines 90.12% overall accuracy with 25.49% Visible recall. Reporting both recalls and their balanced average makes these differences explicit. Appendix C.1 provides the complete 22-model results.

3. How does image content affect finding recognition?

We conducted paired image interventions using Hulu-Med-4B and Qwen3-VL-8B on the 1,555 items with eligible donor examinations, drawn from the original 1,558-item panel. We compared the original images with (i) images from another examination at the same anatomical site and frame count and (ii) no images. Questions, options, and reference answers remain fixed across conditions.

ModelOriginal BASwapped BANo-image BA
Hulu-Med-4B88.04%65.37%52.50%
Qwen3-VL-8B83.37%64.44%51.49%

Replacing case-matched images reduces BA by 22.67 pp and 18.93 pp, respectively; removing images reduces it by 35.54 pp and 31.88 pp (all paired 95% CIs exclude zero). The substitution comparison holds anatomical site and frame count constant, showing that finding-recognition performance in both models responds to case-specific visual content.

These analyses connect answer-frequency structure, class-balanced finding recognition, and the effect of case-specific images. Section 5.2 summarizes the class-balanced Presence result, Appendix B.4 defines BA and the class labels, and Appendix C.1 reports the complete 22-model evaluation and paired image interventions. The tables separately identify the 1,558-item recognition panel and its 1,555-item intervention subset.


References

[1] Liu, S., et al. EndoBench: A comprehensive evaluation of multi-modal large language models for endoscopy analysis. NeurIPS, 2025.

[2] Zhu, Y., et al. GI-Bench: A panoramic benchmark revealing the knowledge-experience dissociation of multimodal large language models in gastrointestinal endoscopy against clinical standards. arXiv:2601.08183, 2026.

[3] Kafle, K., & Kanan, C. An analysis of visual question answering algorithms. ICCV, 2017.

Reviewer_bBebOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents TraceScope, a multimodal benchmark for upper gastrointestinal endoscopy comprising 9,106 task entries derived from 4,893 examinations. It covers six task categories—anatomical recognition, abnormality detection, attribute assessment, report generation, diagnostic impression, and guideline reasoning—with visual evidence scopes tailored to each task type. The authors evaluated 22 models and conducted within-case experiments that altered the presentation of visual evidence or incrementally provided anatomy, findings, sign polarity, and decision rules. Results indicate that curated images generally outperform an equal number of uniformly sampled frames, and that, under a fixed information accumulation order, providing sign polarity yields the largest average improvement.

Critical strengths

Strength 1: The alignment between tasks and evidence scope offers clear evaluative value.

The benchmark distinguishes among local recognition, cross-image report integration, and guideline-based decision making, linking construction and scoring through a unified clinical vocabulary. This enables the benchmark to expose capability gaps where models perform well on localized questions yet still fail at comprehensive reporting or diagnosis.

Strength 2: The within-case controlled experiments are highly informative.

The Evidence Presentation design holds the case, question, options, and target constant while including an equal-frame-count control, ensuring that the advantage of curated images cannot be attributed simply to input quantity. The Diagnostic Cascade reports both sensitivity and specificity, revealing that incorporating findings may increase sensitivity while decreasing specificity.

Critical weaknesses

Weakness 1: The construction of T6 reference reasoning chains is susceptible to conclusion-conditioned bias, and independent evidence of reliability remains lacking.

Generators are given ground-truth answers and asked to construct supporting reasoning; adjudicators likewise know the answers and are instructed not to re-evaluate conclusions. This setup risks two biases: seeking supportive signs for known conclusions and retaining only cases that yield consistent reasoning chains, thereby excluding difficult samples where visual evidence conflicts with final outcomes. Section A.5 mentions expert sampling review but does not report its scale, error rate, or candidate case exclusion rates.

Weakness 2: Quantitative attribute metrics contain scoring loopholes that weaken claims about visual measurement ability.

The paper states that predictions receive zero error as long as the predicted interval overlaps the ground-truth interval, and redundant predictions are not penalized. Consequently, outputting sufficiently wide intervals or enumerating many candidate values can achieve perfect scores without precise visual judgment. Table 9 shows that a fixed-prior baseline attains a Count MRA of 0.794 while the best model reaches only 0.809, further indicating that high absolute scores should not be interpreted as reliable counting ability.

Questions for authors

Question 1: The final model achieves a balanced accuracy of 0.619 versus 0.762 for deterministic rules, yet both are scored against pathology labels. I believe it is necessary to directly report the model’s rule-adherence rate and to stratify cases by whether the rule agrees or disagrees with the pathology label, so that the performance gap can be properly attributed to failures in rule execution.

Question 2: The current Cascade uses only a fixed accumulation order, so the largest gain from +Sign Polarity does not necessarily imply that semantic mapping of signs is the primary bottleneck. Could you add ablations such as “images + rules,” “sign text only ± rules,” and “model-predicted signs → deterministic rules”? If time is limited, even a modest set of additional results sufficient to support the claim would be adequate. I would reconsider my rating if the authors address these concerns.

6 comments
AuthorsOpenReview ↗

Response to Reviewer bBeb (Part 1/4)

We sincerely thank you for your careful reading of our manuscript and for the constructive feedback. We appreciate your attention to both the benchmark design and the interpretation of its results. Your comments have helped us clarify the evaluation framework and strengthen the empirical support for our findings. We address your specific concerns below.


[W1] Reliability and Selection of T6 Reference Reasoning Chains

We appreciate the reviewer's concerns regarding conclusion-conditioned reasoning and potential selection bias. We provide further clarification of the construction rationale, selection criteria, and independent clinical validation.

1. Conclusion-conditioned construction and evidence safeguards. In T6 reference construction, generators reconstruct a clinically verifiable reasoning reference for an established clinical target, following a paradigm also explored in M3CoTBench [1] and TumorChain [2]. Crucially, the supplied conclusion determines the reference target, not the supporting visual evidence. Each guideline-defining sign must be grounded in cited frames, and its relationship to the conclusion must satisfy executable guideline rules. The Rule Gate checks this derivation, while the Chain Adjudicator examines disputed visual observations, demoting uncertain signs or discarding unsupported claims. Consequently, a chain cannot be accepted merely because it agrees with the known conclusion; its visual justification must also be established. At evaluation, models infer the decision and reasoning from the supplied images and task prompt.

2. Reference admissibility and the task objective. T6 evaluates whether a model can perform multi-step, guideline-grounded clinical reasoning when the supplied visual evidence supports the required observations. A recorded clinical conclusion may be valid even when the available images cannot establish its defining signs. Requiring the model to recover such an outcome would conflate missing evidence with reasoning failure. We therefore require visually supported, guideline-consistent reasoning for T6 admission, so that the reference connects observable signs to an assessable clinical derivation. Retrospectively examining all available T6 cases (1,029 eligible cases after preprocessing and extraction), we found that 67% initially passed the Rule Gate, increasing to 84% proceeding to adjudication after regeneration. Among adjudicated cases, 63% passed directly, 23% were flagged for targeted clinician review, and 14% were excluded by downstream checks; approximately 44% of flagged cases passed subsequent validation after clinician intervention. Ultimately, 635 T6 chains (61.7%) were retained.

Difficult rule–pathology disagreements remain explicitly evaluated. The H. pylori Diagnostic Cascade includes all 1,220 eligible examinations, without requiring an admissible T6 reference chain. It retains all 286 cases in which the deterministic rule disagrees with pathology (R≠YR \neq Y). In Q1, we separately quantify model performance in this conflict stratum: InternVL3.5-8B follows the rule in 260 cases and reaches the pathology-correct diagnosis in 26. T6 results characterize guideline reasoning on evidence-admissible references; the Cascade separately evaluates pathology-based decisions across the broader cohort, including all rule–pathology conflicts.

3. Independent clinical validation. To further establish the reliability of retained references, two clinicians independently reviewed 60 released T6 chains. Both-clinician approval reached 96.7% for reference correctness, visual evidence support, and key-sign support, and 98.3% for guideline consistency. These findings independently support the quality of retained reasoning references, complementing the construction safeguards and selection statistics. Appendix A.3 describes targeted clinician intervention and recorded rejection or demotion reasons; Appendix A.5 reports the independent clinical audit, including the T6-specific assessments alongside the task-level results.

AuthorsOpenReview ↗

Response to Reviewer bBeb (Part 2/4)

[W2] Robust Quantitative Scoring and Prior-aware Measurement Assessment

We appreciate the reviewer's identification of numerical scoring issues. We revised the matching rules and re-evaluated all 22 models on the same 435 T3 numerical items (316 size; 119 count), alongside fixed-prior controls and physician responses. The revised evaluation retains the original relative-error thresholds while introducing stricter interval matching, enumeration penalties, and count-balanced evaluation.

1. Strict interval scoring and complete matching.

The revised Size MRA evaluates the entire predicted interval, instead of awarding zero error for any overlap with the reference. For a prediction p=[pl,pu]p=[p_l,p_u] and reference g=[gl,gu]g=[g_l,g_u] with positive midpoint, we define:

e(p,g)=max⁡{0, gl−pl, pu−gu}(gl+gu)/2.e(p,g)=\frac{\max\lbrace 0,\ g_l-p_l,\ p_u-g_u\rbrace}{(g_l+g_u)/2}.

For point references, this reduces to the maximum relative error of the predicted endpoints. Thus, reasonable intervals retain tolerance-based credit, while widening a prediction cannot improve its score. We additionally require equal prediction/reference cardinality and complete one-to-one matching, preventing redundant candidate enumeration from receiving free credit. Measurements are matched according to the requested clinical quantity, distinguishing multidimensional sizes from alternative guesses. Missing or invalid responses are scored as incorrect. For Size, we report Acc0.3\mathrm{Acc}_{0.3} and MRA averaged over the original thresholds {0.1,0.2,0.3,0.5}\lbrace 0.1,0.2,0.3,0.5\rbrace. For Count, we report MRA together with Exact and Macro Accuracy.

2. Size estimation under the updated criterion.

Size EvaluationOriginal MRARevised MRAAcc@30%
Fixed prior0.4760.4760.475
InternVL3.5-14B0.5040.4770.579
Lingshu-I-8B0.4720.4670.563
GPT-5.40.5560.2740.228
Gemini 3.8 Flash0.5690.1780.133

Under the updated criterion, InternVL3.5-14B leads Size MRA at 0.477 and achieves 0.579 accuracy at the 30% tolerance. The table also shows that permissive interval matching affected some scores and rankings, particularly for GPT-5.4 and Gemini. We therefore use the updated scores throughout the numerical tables and associated model comparisons. The two readouts distinguish performance at a specific tolerance from average performance across tolerances. On the separate physician reader-study subset, Size MRA changes from 0.650 to 0.638 (20 items).

3. Count accuracy reveals the influence of answer-frequency priors.

Answer-frequency effects are well recognized in counting evaluation. TDIUC [3] averages accuracy across unique answers within each question type to account for answer imbalance. CountVideoBench [4] documents concentrated count distributions across nine video benchmarks and motivates cardinality-balanced evaluation. Accordingly, we supplement Count MRA with Exact Accuracy and Macro Accuracy: Exact measures overall counting correctness, while Macro gives equal weight to each reference count value.

Count EvaluationRevised MRAExact Acc.Macro Acc.
Fixed prior0.7940.7560.143
Hulu-Med-4B0.8000.7560.156
HealthGPT-Pro-8B0.6260.5880.245
ColonR10.1280.1010.288

The fixed prediction of one matches 90/119 references, yielding 0.756 Exact Accuracy; accepting nearby counts at the larger tolerances gives an MRA of 0.794. Hulu-Med-4B achieves a +0.006 MRA difference over the prior and identical Exact Accuracy. Macro Accuracy reveals a different capability profile: ColonR1 achieves the highest Macro Accuracy (0.288) among the displayed models, while Hulu-Med-4B achieves 0.156, close to the prior (0.143). Reporting both Exact and Macro makes performance across count values visible alongside overall correctness. The physician reader-study Count MRA remains 0.836 on its separate 35-item subset.

These matching rules remove free credit from broad interval overlap and redundant enumeration. Appendix B.4 gives the formal definitions, and the main and expanded result tables use the updated scores for all 22 models, fixed priors, and physician subsets where applicable. The expanded table reports Size Acc@30% and MRA, and Count MRA, Exact and Macro Accuracy, making tolerance-based, exact and class-balanced results available together.

AuthorsOpenReview ↗

Response to Reviewer bBeb (Part 3/4)

[Q1] Rule Adherence and Pathology-stratified Performance Attribution

We appreciate the reviewer's suggestion to distinguish rule adherence from pathology-based diagnostic accuracy. We analyzed 19 models on 1,220 H. pylori examinations (23,180 predictions), directly measuring decision-level rule adherence and the diagnostic consequences of model–rule disagreements.

Direct rule adherence. Adherence is agreement between the model's final diagnosis and the deterministic rule applied to the same supplied sign evidence. The rule agrees with pathology in 934/1,220 cases (76.56%) and disagrees in 286 (23.44%), achieving 76.24% balanced accuracy (BA) against pathology. Across 19 models, mean adherence is 67.23% (95% CI [66.20%, 68.26%]), with mean pathology BA of 61.95% (95% CI [60.77%, 63.13%]).

Fleet-level pathology-stratified attribution. When the rule is correct (R=YR=Y), a wrong or invalid model decision introduces an error. When the rule is incorrect (R≠YR\neq Y), a pathology-correct model decision corrects a rule error. We quantify both effects within each pathology class and average them with equal class weights, then average across all 19 models:

Fleet 19-model mean attributionBA / Contribution
Deterministic rule baseline76.24%
Loss from breaking correct rule judgments−22.64 pp
Gain from correcting wrong rule judgments+8.35 pp
Fleet mean model BA61.95%

Class-specific rates use the full 459 pathology-positive and 761 pathology-negative denominators. Invalid replies remain incorrect; those on rule-correct cases contribute to the loss term.

Thus, 76.24 − 22.64 + 8.35 = 61.95: corrections recover part of the loss from breaking correct rule judgments, leaving the 14.29 pp fleet BA gap. Losses are concentrated in pathology-negative examinations, which contribute 16.52 of the 22.64 BA percentage points lost. This decomposition distinguishes harmful departures from departures that correctly override a rule–pathology conflict.

Representative model. InternVL3.5-8B has the highest adherence (92.38%) and achieves 74.19% pathology BA, approaching the deterministic rule's diagnostic performance. Its breakdown illustrates the same distinction:

Rule–Pathology RelationCasesFollows RuleDeviates from RuleDiagnostic Effect
R=YR = Y9348676767 losses
R≠YR \neq Y2862602626 gains

Relative to the rule, it gains 3.27 pp in sensitivity and loses 7.36 pp in specificity, yielding a net −2.05 pp BA difference (74.19% vs. 76.24%). Its 26 pathology-correct departures partially offset the 67 errors introduced when the rule was correct.

Following your suggestion, we have made the distinction between rule adherence and diagnostic correctness more explicit in the revised manuscript. Appendix B.4 defines rule adherence, and Appendix C.4 presents the full 19-model results, the class-balanced gain–loss decomposition, and the representative-model analysis. In Section 5.4, we discuss these findings alongside the Cascade results to clarify how consistent use of the supplied rule relates to pathology-based diagnostic performance.

Reviewer_ERy8OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces TRACESCOPE, a report-grounded, multimodal benchmark for upper-GI endoscopy that explicitly aligns the scope of visual evidence with clinically distinct task families across the endoscopic workflow. The main board comprises 9,106 items from 4,893 examinations spanning six task families (T1–T6: anatomy recognition, abnormal findings, finding attributes, report generation, diagnostic impression, and guideline-grounded reasoning), and two within-case diagnostic axes probe how fixed downstream decisions change with (i) alternative visual presentations and (ii) progressively explicit clinical state/guideline information. Evaluating 22 MLLMs and a blinded physician panel reveals stage-selective capabilities, a strong advantage for selected multi-image presentations over uniformly sampled video beyond mere input count, and the largest diagnostic gain when guideline sign polarity is made explicit.

Critical strengths
  • The benchmark is designed around a shared clinical specification that aligns task targets, admissible evidence, response forms, and evaluation, uncommon in prior endoscopic VQA-style datasets that decouple task definition from evidence scope.
  • Inclusion of a blinded physician reader study spanning all task families provides a valuable clinical reference, with clear contrasts to model performance.
  • The paper clearly motivates the need to match evidence scope to clinical task, and presents an understandable taxonomy (T1–T6) aligned to real-world endoscopic workflow.
  • Upper-GI endoscopy is a critical procedural domain where real clinical deployment requires evidence-aware evaluation. This study fills a gap by integrating reporting, impression, and guideline steps rather than isolated image questions.
Critical weaknesses
  • LLMs are used in several construction/assessment steps (report extraction, evidence verification assistance, T6 chain generation, and rubric-based scoring), which can introduce judge-model bias or circularity. While deterministic rule-gating and expert spot checks mitigate the issue, the degree of human adjudication coverage is not fully quantified.
  • The Anatomy Router is trained on physician-labeled frames from the same cohort; while used for routing rather than scoring, details on cross-validation and accuracy bounds are limited, leaving potential concerns about data coupling.
  • The Evidence Presentation analysis contrasts selected images with uniform video sampling but does not include stronger selection baselines (e.g., learned or query-guided selectors) that could quantify how much of the “selection advantage” is achievable algorithmically.
Questions for authors
  • What proportion of T6 guideline chains required human adjudication beyond the Rule Gate, and what were the most common reasons for rejection/demotion (e.g., sign misclassification, anatomical mismatch)?
  • Could you please report inter-rater agreement for clinician validation of vocabularies/cross-walks and for sampled evidence adjudication?
  • How accurate is the Anatomy Router on a held-out subset, and how sensitive are downstream tasks (especially T2–T3 routing) to router errors?
  • For T4 and T6 semantic/judge-based metrics, did you perform any clinician calibration or spot-audits quantifying agreement between LLM judges and experts? If so, what was the agreement?
  • Could you please include a baseline in the Evidence Presentation axis using a learned/query-guided selector (e.g., CLIP-based or domain-adapted), to bound how much of the selection advantage can be approximated algorithmically?
  • Are the source examinations single-institution? If so, do you plan a multi-center extension or domain shift split to assess generalization across devices and clinical practices?
5 comments
AuthorsOpenReview ↗

Response to Reviewer ERy8 (Part 1/3)

Thank you for your careful and thorough review of our work. We appreciate your recognition of the value of the clinical task design and physician study, as well as your thoughtful questions about the benchmark. Your feedback has helped us clarify the methodology and strengthen the presentation of the clinical validation. Below, we respond to each of your comments.


[W1] Reliability of LLM-assisted Construction and Evaluation

We appreciate the reviewer's concern regarding potential LLM-induced bias and the quality of the constructed benchmark data. To mitigate potential bias and support data quality, TraceScope incorporates complementary quality safeguards across construction and evaluation, combining deterministic rule constraints, clinician-guided refinement, and independent clinical validation. Importantly, we distinguish automated quality control from independent expert verification, and further calibrate LLM-based scoring against clinician judgments. We provide detailed procedures and quantitative evidence below, covering T6 construction and human intervention (Q1), clinical agreement on terminology and visual evidence (Q2), Anatomy Router Reliability (Q3 & W2), and judge–clinician calibration (Q4).


[Q1] Human Adjudication and T6 Quality Control

We appreciate the reviewer's concern regarding the reliability of T6 guideline-chain construction. TraceScope adopts an iterative, clinician-guided construction framework, in which clinical experts contribute to guideline formulation, terminology harmonization, and refinement of construction policies through feedback on sampled outputs. Building upon these specifications, multiple chain generators independently produce candidate reasoning chains under standardized prompts and clinical guidelines. The resulting chains undergo a multi-stage verification process: a deterministic Rule Gate checks anatomical compatibility, frame references, and guideline consistency, followed by an LLM-based Chain Adjudicator (Claude Opus 4.8) that reconciles candidate chains at the claim level, retaining supported findings while demoting or rejecting ambiguous or unsupported ones. Clinicians further inspect selected cases and provide feedback to improve the construction pipeline.

To quantify this process, we retrospectively examined all available T6 cases (1,029 eligible cases after preprocessing and extraction), with the principal construction outcomes summarized below.

Checkpoint / DispositionOutcomeRecorded Reasons / Actions
Initial Rule Gate67% passed
After regeneration84% proceeded to adjudicationRejections: frame-reference inconsistencies, reference leakage, formatting violations
Adjudication panel: direct pass63%
Adjudication panel: flagged for clinician review23%Disputed key signs, ambiguous visual support
Adjudication panel: excluded by downstream checks14%Structural and consistency violations

The first two percentages use all 1,029 eligible cases as the denominator. The last three describe dispositions within the adjudication panel, using cases entering adjudication as their common denominator; they are not successive pass rates.

Clinicians revisited flagged evidence and refined sign assignments or frame references, enabling approximately 44% of flagged cases to pass subsequent validation. Ultimately, 635 T6 chains (61.7%) were retained.

Beyond these construction-stage safeguards, we further assessed the released references through an independent dual-clinician audit of 60 sampled T6 items. Both-clinician pass rates reached 96.7% for reference correctness and visual evidence support, 96.7% for key-sign support, and 98.3% for guideline consistency. Exact inter-rater agreement ranged from 96.7% to 100%. Appendix A.3 describes targeted clinician intervention and recorded rejection or demotion reasons; Appendix A.5 reports the independent clinical validation of released references.

AuthorsOpenReview ↗

Response to Reviewer ERy8 (Part 2/3)

[Q2] Inter-rater Agreement for Clinical Terminology and Evidence Validation

We thank the reviewer for this suggestion. The clinical vocabularies and terminology cross-walks were originally developed through iterative clinician consultation and consensus, rather than independent pre-consensus annotations. To independently assess their reliability, we subsequently sampled 70 mappings from the complete registry, with two clinicians evaluating each mapping for clinical appropriateness and semantic preservation (Appropriate / Inappropriate / Uncertain). In parallel, two clinicians independently evaluated the visual evidence support of randomly sampled T2–T3 and T5–T6 benchmark references, allowing us to assess agreement on both semantic normalization and image-grounded evidence.

The results are summarized below, reporting both the proportion independently approved by both clinicians (Both Approved/Pass) and their agreement across the full rating categories (Exact Agreement).

Validation TargetReviewedBoth Approved/PassExact Agreement
Terminology cross-walks7098.6%98.6%
T2 visual evidence8096.2%98.8%
T3 visual evidence6096.7%98.3%
T5 visual compatibility5598.2%98.2%
T6 visual evidence6096.7%98.3%

The high agreement across these complementary assessments supports the reproducibility of both clinical terminology normalization and visual evidence validation. Importantly, all agreement rates are calculated from independent, pre-reconciliation clinician ratings, preserving a clear distinction between inter-rater consistency and subsequent quality-control refinements. The protocols and task-specific results are reported together in Appendix A.5.


[Q3 & W2] Anatomy Router Reliability and Downstream Robustness

We appreciate the reviewer's concern regarding Anatomy Router reliability and the potential propagation of routing errors into downstream benchmark construction. As documented in Appendix A.3 of the original submission, the Anatomy Router achieves 88.72% overall accuracy (95% CI: 86.34–91.06%) and 89.71% macro recall on an independent held-out set of 833 frames from 38 examinations.

Importantly, downstream T2–T3 construction incorporates several safeguards against routing errors. First, multiple frames from different views provide redundant visual candidates, reducing dependence on individual routing predictions. Second, the Evidence Grounder independently evaluates finding support and anatomical compatibility, rather than directly accepting Router predictions. Third, confidence-aware verification allows unsupported or ambiguous finding–image pairs to be rejected or flagged for review, avoiding forced evidence assignments.

These safeguards are reflected in actual construction records. For example, the Grounder rejected a candidate pair despite high-confidence esophageal routing when the reported lesion was not visible. In another case, it rejected anatomically incompatible evidence when the provided frames failed to support the reported finding. Such cases illustrate how evidence verification can prevent certain routing or candidate-selection errors from becoming final references.

Beyond these intermediate safeguards, the independent clinician audit reported in Q2 further confirms the quality of the released T2–T3 evidence, with over 96% both-clinician approval for visual support across both tasks. Together, the held-out Router evaluation, evidence-verification mechanisms, and downstream clinical audit provide complementary support for the reliability of released T2–T3 references. The annotation partitions and held-out Router evaluation are described in Appendices A.1 and A.3; the independent reference audit is reported in Appendix A.5.

AuthorsOpenReview ↗

Response to Reviewer ERy8 (Part 3/3)

[Q4] Clinician Calibration of LLM-based Evaluation

Thank you for this suggestion. We conducted an independent clinician calibration study, sampling 10 T4 questions and 24 T6 questions. Their model outputs were evaluated on four T4 scoring dimensions and five T6 scoring dimensions, respectively, yielding 40 T4 and 120 T6 question–dimension scoring units (160 in total across 34 questions). Two clinicians independently scored the anonymized outputs using the same references, guideline cards (where applicable), and scoring rubrics as the LLM judge, without access to model identities or judge scores. We then compared clinician–clinician (H–H) and judge–clinician (J–H) agreement on the original five-level rating scale.

TaskScoring Units (Questions ×\times Dimensions)H–H ExactJ–H ExactJ–H Within ±1\pm 1
T4 Report Generation40 (10×410 \times 4)90.0%85.0%97.2%
T6 Guideline Reasoning120 (24×524 \times 5)90.8%85.4%96.6%

Each scoring unit corresponds to one question–dimension pair. Agreement is macro-averaged across scoring dimensions; J–H results average comparisons against both clinicians.

The LLM judge achieves 85.0–85.4% exact agreement with clinicians, compared with 90.0–90.8% agreement between clinicians, while over 96% of judge ratings fall within one scoring band. These results provide empirical clinician calibration of the automatic rubric scores on the sampled T4/T6 panel. Appendix A.5 reports the calibration protocol and task-level results.


[Q5 & W3] Automatic and Query-guided Evidence Selection

To assess whether automatic selection can approximate curated evidence presentation, we introduced a question-and-option-guided BiomedCLIP selector with maximal marginal relevance (MMR). It retrieves frames from the retained examination image pool under a case-matched frame budget (Ki=∣Selectedi∣K_i=|\mathrm{Selected}_i|), without accessing reference impressions or ground-truth labels.

PresentationSelectedBiomedCLIP + MMRRetained
Mean accuracy (9 models)53.20%47.21%46.40%
Mean frames15.815.8~44

Selected outperforms BiomedCLIP across all nine models, with a mean advantage of +5.99 pp (95% CI: +2.88 to +8.98 pp) from paired examination-level bootstrap analysis over 193 examinations. This comparison provides an algorithmic selection baseline under the same frame budget as Selected. Figure 4(a)–(b) includes this baseline; Appendix B.2 describes the selection protocol, and Appendix C.3 reports the full nine-model results and paired comparison.


[Q6] Multi-center Data Collection and Domain Generalization

We appreciate the reviewer's interest in cross-center generalizability. We clarify that TraceScope already comprises 4,893 examinations collected across multiple clinical centers, as documented in Appendix A.1.

For multi-center benchmark construction, TraceScope establishes a shared clinical specification through clinician-guided terminology cross-walks, normalization of 1,079 surface aliases into 150 canonical finding terms, and structured report extraction preserving anatomical locations, negation, certainty, and quantitative attributes. Together, these mechanisms provide a consistent clinical semantic framework for constructing and evaluating T1–T6 across heterogeneous clinical sources.

Our planned expansion pairs a larger, multi-center SFT corpus with independent evaluation cohorts to study transfer across centers and devices under the same clinical specification. Appendices A.1 and E describe the current multi-center provenance and clinical scope.

Comments on the paper

3 comments
AuthorsOpenReview ↗

General Response to the Reviewers and the Area Chair (1/3): Clinical Motivation and Distinctive Contributions

We sincerely thank the reviewers ERy8, bBeb, 26oi and Area Chair for their careful engagement. Their questions on clinical grounding, reliability, and novelty have substantially strengthened TraceScope. To clarify how our work advances multimodal evaluation beyond existing benchmarks, we summarize our contributions (1/3), scientific insights (2/3), and validation & revisions (3/3).

The clinical need: From isolated lesions to examination-level understanding.

Upper-GI examinations involve coexisting abnormalities across anatomical sites, whose attributes and implications must be integrated into reports, diagnoses, and guideline decisions. Yet multi-image clinical synthesis and evidence-grounded guideline reasoning remain under-evaluated under examination-level image/video evidence and verifiable references.

1. Making clinical heterogeneity assessable (Figs. 2, 5–7).

TraceScope links eight anatomical stations, 150 canonical findings (from 1,079 report expressions), 12 attribute axes, and 176 diagnostic terms (128 active for upper-GI; 96 realized T5 labels), alongside six guideline families. Clinician-validated mappings connect shared concepts across recognition (T2), attributes (T3), reports (T4), and impressions (T5), extending evaluation well beyond predefined lesion classes.

2. Matching clinical capabilities to their evidence (Figs. 1–2).

Building on this semantic foundation, six tasks span local perception, multi-image examination synthesis, diagnosis, and guideline reasoning. T4 integrates distributed findings into site-organized reports; T6 links frame-cited observations → sign states → guideline criteria → decisions and recommendations. A shared task–evidence–reference specification aligns inputs, clinical targets, responses, and scoring.

3. Preserving clinical provenance (Fig. 3).

To ground these tasks in reliable references, construction follows report normalization → anatomical routing → visual evidence verification → constrained task compilation. References retain their task-specific origins in physician labels, reports, diagnostic impressions, pathology, and guidelines. T6 further combines multiple generators, executable Rule Gates, and evidence-level adjudication, while clinician input across annotation, terminology, guidelines, and audits establishes reusable construction principles.

4. Revealing diagnostic dependencies (Figs. 1, 4).

Beyond stage-wise scores, TraceScope introduces two controlled within-case axes. Evidence Presentation holds T5 diagnostic targets fixed while varying selected/retained images, video sampling, and frame budgets. Diagnostic Cascade holds images and pathology-backed H. pylori targets fixed while varying anatomy, findings, sign polarity, and rules. Together, these interventions isolate visual and clinical information dependencies within the same decision.

What existing benchmarks leave untested (Table 1).

Benchmark and evaluation boundaryWhat TraceScope makes measurable
EndoBench: Pan-endoscopy single-image VQA assembled from public datasets, without multi-image or video examination integrationReport/pathology-linked upper-GI examinations with multi-image/video evidence, site-structured reports, and task-matched references
GI-Bench: Image-centered assessment restricted to 20 lesion categories, rather than examination-wide synthesis150 findings and 96 realized diagnoses connecting coexisting abnormalities, attributes, sites, and examination conclusions
CE-Bench: Capsule-endoscopy VQA, CoT, and reporting, without executable guideline-decision validationFrame-cited signs, rule-checked reasoning, adjudicated references, and separable reasoning vs. decision evaluation
MedHorizon: Full-procedure video retrieval and multi-hop MCQ, lacking controlled clinical-state interventionsFixed-target, within-case experiments isolating visual presentation from supplied clinical-state effects
M3CoTBench: General medical-image CoT evaluation, not anchored to guideline-specific rule executionSign polarity, rule adherence, and pathology-backed outcomes testing clinical decision validity

The distinction is not task count alone. While related benchmarks examine valuable individual components, they do not jointly connect heterogeneous examination-level findings, task-appropriate image/video evidence, traceable references, and controlled diagnostic dependencies. Across 4,893 examinations, 9,106 items, and 70,386 item–frame references evaluated on 22 MLLMs and a blinded physician panel, TraceScope provides a clinically grounded resource and methodology for measuring these capabilities. Part (2/3) presents the resulting scientific insights.

AuthorsOpenReview ↗

General Response to the Reviewers and the Area Chair (2/3): Scientific Insights and Broader Significance

A central question in the discussion is what we learn from TraceScope beyond model rankings. The following findings reveal clinical capability gaps and diagnostic information dependencies, with concrete implications for model development:

Workflow QuestionEmpirical Profiles and Controlled ComparisonsCapability Insight & Implication
1. How do local recognition and examination-level capabilities differ?Main board (Table 2): best models reach 0.945 single-finding presence, versus 0.200 exact-set multi-finding enumeration, 0.516 report finding recall, and 0.378 guideline-grade agreement (across task-specific populations/metrics). T4 exposes site-coverage gaps and unsupported statements; T6 separates reference-consistent reasoning from exact decisions.Task-specific profiles expose limits in complete finding recovery and exact clinical decisions that local-recognition scores alone leave unmeasured.
2. Does adding more visual frames replace evidence selection?Across 193 fixed T5 questions (9 models), selected images (53.20%, 15.8 frames) outperform equal-count uniform video sampling by 8.8 pp. Full retained pools (~44 frames) reach 46.40%, and matched-budget BiomedCLIP + MMR reaches 47.21%, a 5.99 pp gap to Selected (95% CI: 2.88–8.98 pp).Curated selection retains an advantage over uniform sampling and the tested automatic selector, highlighting task-relevant evidence selection and integration.
3. Is supplying clinical rules enough for guideline reasoning?Across 1,220 pathology-backed H. pylori cases (19 models), polarity yields the largest adjacent Cascade gain in Cohen's kappa. Factorial ablations show BA gains of +4.53 pp without rules and +7.22 pp with rules. Images + rules yield 50.72% BA vs. 54.05% for images alone, whereas structured anatomy + findings + polarity + rules reach 62.70%. Rule-adherence analyses further separate breaking correct rules (−22.64 pp) from valid conflict overrides (+8.35 pp).Providing a rule is distinct from observing its signs, representing their diagnostic polarity, and executing it reliably; these steps must be evaluated separately.
4. How do we separate genuine capability from dataset shortcuts?On single-finding Presence, the majority prior achieves 86.91% accuracy but 50.0% BA, while class-wise recall and paired cross-case image swaps (18.93–22.67 pp BA drops) verify visual dependence. Stricter T3 metrics remove free credit from partial interval overlap and enumeration.Clinical benchmarks require prior-aware baselines, strict scoring criteria, and controlled visual interventions alongside standard accuracy.

Why this matters beyond the individual experiments

  • For clinical AI: Moving from isolated lesion labels to examination-level clinical reality. Real examinations involve coexisting abnormalities across anatomical sites rather than single predefined lesions. By linking 150 standardized findings and 96 realized diagnostic labels across recognition, attributes, reports, and guideline decisions, TraceScope establishes a broad, clinically interpretable test space where cross-image synthesis and decision correctness can be evaluated alongside local perception.

  • For evaluation methodology: Making diagnostic information dependencies testable. Shared clinical targets, evidence requirements, provenance, and paired axes make capability boundaries and information dependencies assessable. With prior-aware baselines, strict scoring, and image interventions, this turns broad performance gaps into testable questions about evidence selection, clinical-state representation, and rule use.

  • For model development: Defining concrete technical priorities beyond local perception. The controlled comparisons highlight the value of selected evidence and explicit clinical state. They point toward adaptive evidence selection, multi-image synthesis, structured clinical-state modeling, and verifiable guideline execution—research challenges with relevance well beyond endoscopy.

  • For the community: Providing reusable building blocks for domain benchmarks. Our validated terminology cross-walks, report-to-image verification pipeline, traceable reasoning references, and clinician-calibrated evaluation protocols offer an adaptable foundation for subsequent benchmark and model development across report-linked clinical settings.

Together, these findings clarify why TraceScope is more than a comprehensive dataset or model leaderboard: it makes clinical capability gaps and diagnostic information dependencies measurable, offering a foundation for evidence-aware medical MLLMs. In (3/3), we summarize the additional checks that strengthen these conclusions.

AuthorsOpenReview ↗

General Response to the Reviewers and the Area Chair (3/3): Clinical Reliability, Additional Evidence, and Revisions

We appreciate the reviewers' insistence on rigorous validation. Their questions prompted additional experiments that made our clinical references, evaluation metrics, and diagnostic interpretations more reliable and transparent.

1. Clinical expertise and reference provenance throughout construction.

TraceScope draws on task-appropriate sources: physician anatomical labels, contemporaneous endoscopy reports, visually supported findings and impressions, pathology outcomes, and clinician-specified guidelines. It preserves provenance instead of treating every generated answer as interchangeable ground truth. Clinicians contribute to annotation and adjudication, vocabulary cross-walks, guideline criteria, targeted evidence review, and independent reference audits.

For guideline-reasoning references, multiple generators, frame citations, executable Rule Gates, and evidence-level adjudication constrain T6 chains. For complex outputs, T4/T6 use task-specific scoring dimensions, with text-based semantic judges calibrated against clinicians; independent audits separately assess visual evidence support. These elements make the construction and assessment of high-value clinical reasoning data auditable.

2. Additional evidence addressing the reviews.

ConcernAdditional validation or analysisRevision
T6 reliability and selection (ERy8, bBeb)1,029 eligible → 635 retained (61.7%). In 60 T6 references, dual-clinician approval: 96.7% correctness/visual/key-sign support; 98.3% guideline consistency.App. A.3, A.5
Clinical reference quality (ERy8, 26oi)Dual-clinician audit of 275 T2–T6 items, including those 60 T6 items: 97.8% correctness and 97.1% visual-support approval. 70 terminology mappings: 98.6% approval/exact agreement.App. A.5
Router and semantic judges (ERy8)Held-out Anatomy Router: 88.72% / 833 frames. Judge–clinician exact agreement: 85.0–85.4% / 160 T4/T6 scoring units.App. A.1, A.3, A.5
Numerical scoring (bBeb)Strict interval and complete one-to-one matching; 22 models re-evaluated on 435 items; added size/count metrics and updated rankings.App. B.4; tables
Class imbalance/visual dependence (26oi)Presence majority prior: 86.91% accuracy vs. 50.0% BA on 1,558 items; class-wise results for 22 models and image-swap BA drops of 18.93–22.67 pp.§5.2; App. B.4, C.1
Automatic selection (ERy8, 26oi)On 193 cases, nine models, matched-budget BiomedCLIP + MMR trails curated images by 5.99 pp (95% CI 2.88–8.98); original equal-count video gap 8.8 pp.§§4, 5.3; App. B.2, C.3
Rule adherence/pathology (bBeb)19 models, 1,220 cases; deterministic-rule BA 76.24% vs. model mean 61.95%; breaking correct rules −22.64 pp, correcting rule errors +8.35 pp; all 286 conflicts retained.§5.4; App. B.4, C.4
Clinical-state ablations (bBeb)Factorial polarity × rule and alternative inputs across 19 models; polarity gains +4.53 pp without rules and +7.22 pp with rules.§§4–5; App. B.2, C.4

3. Where the manuscript has changed.

  • Motivation/positioning: Introduction, Related Work, Table 1, and §3 clarify the clinical problem, task–evidence alignment, and distinctions from prior benchmarks.

  • Construction/clinical audit: Appendices A.1, A.3, A.5 document provenance, T6 selection, terminology audits, clinician reviews, Router and judge checks.

  • Protocols/scoring: Appendices B.2/B.4 specify matched-budget selection, alternative clinical states, stricter quantitative scoring, prior-aware metrics, and rule adherence.

  • Results/implications: §§4–5, Appendix C, and Appendices E–F present expanded controls, corrected metrics, pathology-stratified findings, scope, and future applications.

In summary, these additions strengthen clinical validity (audited and source-grounded references), evaluation reliability (stricter metrics and calibrated scoring), and scientific interpretability (matched diagnostic comparisons and rule analyses). We believe this supports TraceScope as a substantive datasets-and-benchmarks contribution, through the clinically important capabilities it makes measurable and the scientific questions it enables.

Finally, we sincerely thank ERy8, bBeb, and 26oi for their time, careful questions, and the improvements they prompted. We also extend our sincere gratitude to the Area Chair for overseeing the review process. We hope this discussion conveys both the clinical value and scientific contribution of TraceScope.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.