acceptodds
Under review as a conference paper at ICLR 2027

MedBenchAgent: Towards Systematic Automation of Medical VLM Benchmark Construction

14 views

Abstract

Large-scale construction of medical vision-language model (VLM) benchmarks is increasingly feasible with richly annotated imaging datasets and large language models (LLMs), yet existing automation largely focuses on generating evaluation items within predefined benchmark specifications. We study the broader problem of automatically deriving the specification itself: what to evaluate, which annotations support each task, and how to translate this evidence into reliable evaluation items. We formulate benchmark construction as constrained compilation, in which the benchmark specification is progressively derived from evaluation requirements, heterogeneous annotations, and medical knowledge. Based on this formulation, we introduce MedBenchAgent, a multi-agent framework with a Benchmark Intermediate Representation (BIR) that encodes task definitions, evidence mappings, evaluation protocols, and item specifications across construction stages. MedBenchAgent separates planning, which derives and verifies the specification, from instantiation, which constructs and audits items under the locked specification. MedBenchAgent achieves a Task-Space F1 of 90.9%, outperforming direct task induction (79.2-80.0%) and prior-guided induction (85.1%); 994 of 1,000 sampled items from correctly identified tasks pass human audit. We further demonstrate portability to a specialized medical domain and evaluate twelve VLMs, revealing task- and setting-specific variation obscured by aggregate scores. These results establish constrained compilation as a scalable and auditable framework for medical VLM benchmark construction beyond question generation.

Primary area:
datasets and benchmarks
Keywords:
automated benchmark construction, vision-language models, medical AI, LLM agents, multimodal evaluation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_FC56OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper argues that automating medical VLM benchmark construction should cover the benchmark specification itself—what to evaluate, which annotation evidence supports each task, and how references and protocols are defined—not only item generation. It frames construction as constrained compilation and proposes MedBenchAgent, a five-agent pipeline built around a Benchmark Intermediate Representation (BIR). Three planning agents derive scope, task tree, evidence mappings, and item designs under verification checkpoints. The BIR is then locked, and items are instantiated with LLM-written queries but deterministically derived reference answers. Validation includes task-space agreement on three breast-imaging datasets (16,248 items, 20 dataset–task pairs), achieving F1 90.9 versus 79.2–80.0 for single-call induction and 85.1 with the same task prior; an author audit of 994/1,000 sampled items; evaluation of twelve VLMs with ranking-reversal analysis and a blurred-image control; and an ISIC 2018 portability case study.

Critical strengths

1/The paper is well written and clearly presented, making the proposed framework and its contributions easy to follow.Separating LLM-generated wording from deterministically derived references under a locked, provenance-carrying specification. This removes hallucinated references by construction (0/1,000 reference-groundedness failures) and makes each item traceable to a dataset field and BIR state. This is the most convincing and reusable contribution.

2/Transparent, stage-wise reporting. Per-proposal discrepancy tables, raw planner outputs, residual-error analysis, and cost traces let readers verify what the headline numbers mean. The observation that deciding what to evaluate is harder than evidence mapping is useful, though preliminary.

Critical weaknesses
  1. The central claim is not adequately attributed. The gain over the prior-guided single-call baseline is only three fewer false positives (FP 3 vs. 6, identical TP/FN) on 21 reference targets, from one run, with no variance estimate. There is no ablation isolating the BIR, locking, RAG/MedicalAlignAgent, or review/audit stages, and no adapted agentic baseline. Thus the improvement cannot be attributed to constrained compilation rather than multi-step prompting.

  2. The benchmark’s validity as a measurement instrument is not established. Many models are near random-guess Macro-F1; ranking-reversal statistics lack confidence intervals on small tasks; and in the blurred-image control several task scores rise (e.g., Lingshu-7B AC 25.28→32.85; Hulu-Med-7B AC 22.75→31.07; BreastGPT FT 35.73→42.01). This suggests label priors or class imbalance may drive some tasks, contradicting the stated conclusion.

Questions for authors

Can you report the task-space comparison with a multi-call baseline without BIR locking, using the same number of LLM calls, and with independent adjudication by at least two raters with agreement? If decoding is deterministic, please state so and report backend variance rather than seed variance, so the gain can be attributed to constrained compilation rather than multi-step prompting?

What does the fully automated pipeline (no human checkpoint edits) produce end to end, and how do item quality and model rankings differ from the human-reviewed version? How many edits and how much reviewer time did checkpoints require?

Please provide per-task blurred-vs-original results with majority-class and text-only baselines, plus bootstrap confidence intervals for Tables 3–4 and the reversal counts. How many reversals remain significant?

T* has 21 targets but the realized benchmark has 20 dataset–task pairs. Was the missed VinDr finding-level assessment task added? Were the faulty finding-type items removed before model evaluation?

Why is mask-derived localization invalid for BrEaST but used for ISIC/BUS-BRA? Which decisions are made in the manual normalization step before agents run? How are multi-finding VinDr images handled in single-box detection? Also, please confirm that releasing normalized VinDr-Mammo annotations and using the ACR BI-RADS manual for retrieval comply with the respective licenses.

Reviewer_2fKTOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

MedBenchAgent builds medical VLM benchmarks from existing annotated imaging datasets. Five agents share a Benchmark Intermediate Representation (BIR). They use it to decide which tasks the annotations can support, map annotations to evidence, and then generate and audit the evaluation items. The pipeline is tested mainly on breast imaging, with dermoscopy as a transfer case. The authors report task-selection accuracy, item consistency, and results for twelve VLMs.

Critical strengths
  1. It looks beyond question generation. Most work on automated benchmarks stops at writing questions. This paper also covers the earlier decisions: which tasks the annotations actually support, how reference answers are derived, and how outputs are scored. These decisions usually take most of the effort and are rarely written down, so building them into the workflow is worthwhile.

  2. References and questions are produced separately. Reference answers come deterministically from the annotations, and the LLM only writes the question text within the task specification. This seems the right split. Because the BIR records the evidence mappings, individual items can be traced back and checked.

  3. The construction analysis is informative. Task-selection errors are reported separately from item-generation errors, and the remaining failures are described concretely rather than folded into one pass rate. The paper also makes clear that task-space numbers are measured before human confirmation. The per-dataset model results show differences that the aggregate scores hide.

Critical weaknesses
  1. The claimed scope is wider than what is tested. After reading the introduction, the reviewer was still unsure which data and task types the framework is meant to handle. The experiments are mostly breast imaging, plus a dermoscopy case study. The tasks are mostly multiple-choice and single-box localization. That is a reasonable demonstration, but it doesn’t support the broader framing of “medical VLM benchmark construction.” The paper should state its scope early and separate what the framework is intended for from what has actually been shown.

  2. Every role uses the same model. All five agents run on DeepSeek-V4-Flash. If the model misreads an annotation or task definition during planning, the same misreading can carry through generation and go unnoticed in review. A reviewer role played by the same model is not really an independent check. The reviewer would like to see the workflow run with a few different backbones, and mixed setups, especially one with a separate verifier model. The key measurement is whether the verifier catches errors the generator made, not just the final pass rate.

  3. The BIR’s own contribution is hard to isolate. In Table 1, the comparison changes both the workflow structure and the number of model calls. The same-prior baseline helps, but it can’t separate three possible sources of gain: the BIR itself, specification locking, or simply more chances to inspect and revise task proposals. Two additions would settle this: a staged baseline with the same model, task prior, and roughly the same inference budget, and an ablation that removes specification locking.

  4. It is unclear how much is actually discovered. TaskPlanningAgent keeps, drops, or expands entries in a predefined task catalogue. Most of what is shown therefore looks like selection and refinement within a given structure. The paper doesn’t show that the system can propose valid tasks outside the catalogue. It also doesn’t say how much expert work is needed to adapt task definitions and support conditions to a new domain. Listing what was changed for the dermoscopy experiment would show what is really automated.

  5. Locking can spread upstream errors, and recovery isn’t described. Locking the BIR prevents task drift. It also means a single wrong task definition or evidence mapping is copied into many items. Researcher checkpoints help somewhat, but the reviewer could not find how a specification-level problem found during generation or auditing leads to a revision. The paper should state clearly how fixing one item differs from reopening the specification.

  6. The audit checks consistency, not answerability. A reference answer can agree with its annotation even when the image doesn’t show enough to answer the question. This matters when an examination-level assessment or a lesion-specific label is paired with a single image. It isn’t clear that clinicians examined the actual image–question pairs for input sufficiency and target ambiguity. The blurred-image experiment shows that models use the image, but it doesn’t address this concern.

  7. Compute reporting is incomplete. The paper names NVIDIA B200 GPUs and gives construction time, token counts, and latency in Tables 7–8. That isn’t enough to tell what is needed to reproduce the workflow or how its cost compares with the baselines. Please report GPU count, model placement and parallelism, peak GPU memory, and wall-clock time per stage. Please give the same figures for the baselines, so readers can judge whether the quality gain is worth the extra compute.

Questions for authors
  1. What had to change in the task prior, preprocessing, and human review for the dermoscopy experiment? Can the framework find tasks that are not in its initial catalogue?

  2. Does a separate verifier model catch errors that same-model review misses? How does the framework compare with a staged baseline that uses a similar inference budget?

  3. How are errors in the locked BIR detected and fixed? Did the human auditors look at the actual images to judge whether the questions could be answered?

  4. Could the authors report GPU count, peak memory, and runtime per stage, with the same figures for the baselines?

1 comment

I thank the authors for the detailed response and additional experiments. I have also carefully read the other reviewers’ comments. Reviewer FC56 and I both appreciate the separation of question generation from reference answer construction. This makes the reference answers easier to trace and check, and remains a clear strength of the paper.

Reviewer Lk7o and I raised similar concerns about model selection and dataset coverage. The additional model evaluations, alternative-backbone experiment, heterogeneous verification, and broader planning evaluation provide useful evidence addressing these concerns. The distinction between planning level validation across domains and end-to-end construction in breast imaging and dermoscopy is also important. These additions have largely addressed my concerns about model and domain coverage.

The hardware details and baseline cost comparisons are helpful for researchers seeking to reproduce or use the framework. The explanation of task-prior expansion also clarifies the intended capability: the framework selects and refines predefined task families based on annotation evidence, rather than claiming unrestricted discovery of new top-level task families.

Overall, the response has strengthened the experimental support and clarified the framework’s scope. Most of my concerns are resolved. I have updated my assessment accordingly.

Reviewer_Lk7oOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper formulates medical VLM benchmark construction as constrained compilation, in which the benchmark specification (task space, evidence mappings, evaluation protocols and item-construction specifications) is derived from the evaluation requirement, heterogeneous annotations and medical knowledge before any items are built. MedBenchAgent implements this with five agents that share a Benchmark Intermediate Representation (BIR), separating planning, which derives and locks the specification, from instantiation, which constructs and audits items under the locked BIR. On three breast-imaging datasets it reports a Task-Space F1 of 90.9% against a reference task space, 994 of 1,000 items passing a human consistency audit, an evaluation of twelve VLMs, and a cross-domain case study on ISIC 2018.

Critical strengths
  • Clear problem framing and a transparent, well-documented system. The paper separates planning (deciding what to evaluate and which annotation fields support it) from instantiation (building the items), locks the specification before any item is built, and derives every reference answer from the annotation fields with fixed code. Hallucinated reference answers and answer leakage are therefore ruled out by design. The appendix reports the configurations, token usage, every raw task proposal, every counted mismatch and the audit rules, which makes the work easy to check.
  • The 16k-item breast-imaging benchmark with task-level metrics, the blurred-image intervention and the ISIC 2018 case study give the community something to build on, and Tables 10 and 11 show concretely why direct task induction fails (non-visual metadata, laterality and view markers burned into the image, masks treated as boxes).
Critical weaknesses
  • The headline validation is self-referential and small. Task-Space F1 is agreement with a 21-target reference task space T* designed by the same team (with informal clinical input), matched by one researcher without inter-rater agreement, in a single run; the gain over the strongest baseline (DeepSeek-V4-Flash + T0) is three false positives (FP 6 to 3, Table 1). The task-category prior T0 was also written by the authors, so T0 and T* likely encode the same mental model, and the multi-call planning stage (about 100 LLM calls) is compared only with direct task induction in a single call, not with a baseline of comparable effort (for example one self-review pass). The 99.4% human consistency audit measures agreement between query, locked specification and annotation evidence, judged by the authors; reference groundedness is 100% by construction. Neither number says whether the items are clinically valid or answerable from the image, which is what Desideratum D2 (medical alignment) promises.
  • Several produced tasks are not answerable from the input, the trivial baseline is set too low, and contamination is not checked. Examples in Figures 4 and 5: histologic diagnosis from one ultrasound frame with options such as invasive carcinoma NST vs invasive lobular carcinoma (not separable on imaging; BI-RADS assigns no histology); visual grounding of "Global Asymmetry" in a single view (defined by comparison with the contralateral breast); breast tissue composition, a per-case whole-breast descriptor, from one lesion-centered frame; study-level BI-RADS from a single view. Such items test label priors, not perception, and the numbers agree: "Random Guess" is uniform over options, but the source labels are skewed (VinDr-Mammo: density C 76.5%, BI-RADS 1 67.0%, Mass about 50% of findings; BrEaST: skin thickening absent 94%, calcifications absent 89%, halo absent 78%), and the best accuracies on these tasks (AA 70.3%, AC 63.4%, FT 54.6%, skin thickening 89.5%, calcifications 89.3%, halo 72.0%) are at or below the majority-class rate with Macro-F1 of 0.1 to 0.3. Several "task-specific differences" in Tables 3, 15 and 17 are thus differences in which label a model defaults to; the blurred-image intervention points the same way (Hulu-Med drops 5.9 pp). Finally, BreastGPT-8B very likely saw BUS-BRA and/or VinDr-Mammo in training (82.8 [email protected] on BUS-BRA detection vs 0.9 on VinDr), and the paper has no overlap check.
  • The model evaluation does not support the third claim (D3). All twelve evaluated models are open-weight models of 32B parameters or less, and most were released in 2025 (InternVL3-8B, Qwen2.5-VL, MedGemma-4B, Lingshu, Hulu-Med, QoQ-Med). No closed model was tested, for example, no latest GPT, Claude, and Gemini models which were available at submission time, and the newest open models were skipped too: Qwen3.5 (February 2026) and Qwen3.8 (August 2026) were available, and Qwen3.5-9B is even used as a baseline for direct task induction in Table 1, yet neither was evaluated on the benchmark. For a benchmark paper this is more than a missing row. With the strongest available models absent, we cannot tell whether the low scores reflect hard tasks, weak models, or a benchmark that frontier models already saturate, and we cannot check the answerability concern above (a frontier model stuck at the majority-class rate would confirm it). Running two or three API models over 16k items costs little next to the construction effort reported in Table 7.
Questions for authors
  • Who adjudicated the Task-Space matching and the 1,000-item audit, how many people, what was the inter-rater agreement, and how many radiologists are involved?
  • Can the authors run MedBenchAgent with the other two backbones and several seeds to show that 90.9% is stable, and add a direct baseline that receives T0 plus one self-review pass?
  • How does MedicalAlignAgent decide that a task is answerable from the presented image, and why were histologic diagnosis from ultrasound, global-asymmetry grounding from a single view and tissue composition from a lesion frame retained? Can a radiologist audit a sample for image-answerability?
  • Please add a majority-class baseline row next to "Random Guess" and report the item-level class distribution per task. How many results in Tables 3, 15, and 17 stay above majority-class accuracy?
  • Which evaluated models were trained on VinDr-Mammo, BUS-BRA, BrEaST or ISIC 2018? Please report the overlap or mark affected results.

Comments on the paper

2 comments

Dear Reviewers,

As we are now halfway through the discussion period, I would like to kindly remind everyone to check the authors' responses to your reviews. If the authors have addressed your questions or concerns, please take a moment to follow up and engage in the discussion as appropriate.

Thank you all for your time and efforts throughout the review process!

Sincerely, Your AC

Dear Reviewers,

As we are approaching the end of the discussion period, I would like to kindly remind everyone to review the authors' responses and follow up on any outstanding questions or concerns. If you have not yet engaged in the discussion, please take a moment to do so before the deadline.

Thank you all for your time and efforts throughout the review process! Your AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.