- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper argues that automating medical VLM benchmark construction should cover the benchmark specification itself—what to evaluate, which annotation evidence supports each task, and how references and protocols are defined—not only item generation. It frames construction as constrained compilation and proposes MedBenchAgent, a five-agent pipeline built around a Benchmark Intermediate Representation (BIR). Three planning agents derive scope, task tree, evidence mappings, and item designs under verification checkpoints. The BIR is then locked, and items are instantiated with LLM-written queries but deterministically derived reference answers. Validation includes task-space agreement on three breast-imaging datasets (16,248 items, 20 dataset–task pairs), achieving F1 90.9 versus 79.2–80.0 for single-call induction and 85.1 with the same task prior; an author audit of 994/1,000 sampled items; evaluation of twelve VLMs with ranking-reversal analysis and a blurred-image control; and an ISIC 2018 portability case study.
Critical strengths
1/The paper is well written and clearly presented, making the proposed framework and its contributions easy to follow.Separating LLM-generated wording from deterministically derived references under a locked, provenance-carrying specification. This removes hallucinated references by construction (0/1,000 reference-groundedness failures) and makes each item traceable to a dataset field and BIR state. This is the most convincing and reusable contribution.
2/Transparent, stage-wise reporting. Per-proposal discrepancy tables, raw planner outputs, residual-error analysis, and cost traces let readers verify what the headline numbers mean. The observation that deciding what to evaluate is harder than evidence mapping is useful, though preliminary.
Critical weaknesses
-
The central claim is not adequately attributed. The gain over the prior-guided single-call baseline is only three fewer false positives (FP 3 vs. 6, identical TP/FN) on 21 reference targets, from one run, with no variance estimate. There is no ablation isolating the BIR, locking, RAG/MedicalAlignAgent, or review/audit stages, and no adapted agentic baseline. Thus the improvement cannot be attributed to constrained compilation rather than multi-step prompting.
-
The benchmark’s validity as a measurement instrument is not established. Many models are near random-guess Macro-F1; ranking-reversal statistics lack confidence intervals on small tasks; and in the blurred-image control several task scores rise (e.g., Lingshu-7B AC 25.28→32.85; Hulu-Med-7B AC 22.75→31.07; BreastGPT FT 35.73→42.01). This suggests label priors or class imbalance may drive some tasks, contradicting the stated conclusion.
Questions for authors
Can you report the task-space comparison with a multi-call baseline without BIR locking, using the same number of LLM calls, and with independent adjudication by at least two raters with agreement? If decoding is deterministic, please state so and report backend variance rather than seed variance, so the gain can be attributed to constrained compilation rather than multi-step prompting?
What does the fully automated pipeline (no human checkpoint edits) produce end to end, and how do item quality and model rankings differ from the human-reviewed version? How many edits and how much reviewer time did checkpoints require?
Please provide per-task blurred-vs-original results with majority-class and text-only baselines, plus bootstrap confidence intervals for Tables 3–4 and the reversal counts. How many reversals remain significant?
T* has 21 targets but the realized benchmark has 20 dataset–task pairs. Was the missed VinDr finding-level assessment task added? Were the faulty finding-type items removed before model evaluation?
Why is mask-derived localization invalid for BrEaST but used for ISIC/BUS-BRA? Which decisions are made in the manual normalization step before agents run? How are multi-finding VinDr images handled in single-box detection? Also, please confirm that releasing normalized VinDr-Mammo annotations and using the ACR BI-RADS manual for retrieval comply with the respective licenses.
I thank the authors for the detailed response and additional experiments. I have also carefully read the other reviewers’ comments. Reviewer FC56 and I both appreciate the separation of question generation from reference answer construction. This makes the reference answers easier to trace and check, and remains a clear strength of the paper.
Reviewer Lk7o and I raised similar concerns about model selection and dataset coverage. The additional model evaluations, alternative-backbone experiment, heterogeneous verification, and broader planning evaluation provide useful evidence addressing these concerns. The distinction between planning level validation across domains and end-to-end construction in breast imaging and dermoscopy is also important. These additions have largely addressed my concerns about model and domain coverage.
The hardware details and baseline cost comparisons are helpful for researchers seeking to reproduce or use the framework. The explanation of task-prior expansion also clarifies the intended capability: the framework selects and refines predefined task families based on annotation evidence, rather than claiming unrestricted discovery of new top-level task families.
Overall, the response has strengthened the experimental support and clarified the framework’s scope. Most of my concerns are resolved. I have updated my assessment accordingly.