acceptodds
Under review as a conference paper at ICLR 2027

MedBenchAgent: An Agentic Framework for Automated Medical Benchmark Construction

Abstract

The growing use of large language models (LLMs) in the medical domain drives demand for benchmarks to evaluate their capabilities. Constructing these benchmarks manually is labor-intensive, motivating the use of LLM-based agents to automate the process. However, general-purpose agents often lack sufficient medical expertise for benchmark construction, while existing medical benchmark agents typically target specific tasks or modalities and struggle to consistently produce high-quality benchmarks, limiting both generalizability and reliability. To address these limitations, we introduce MedBenchAgent, an agentic framework for reliable medical benchmark construction across multiple tasks and modalities. MedBenchAgent follows a two-stage workflow supported by an extensible medical tool library. The benchmark design stage decomposes the user request into subtasks, integrates subtask responses into a benchmark specification, and uses this specification to generate template items. To improve design reliability, a field-level quality model guides specification refinement. The data construction stage retrieves relevant medical samples for each template item, performs fine-grained annotation on the retrieved samples, and instantiates final benchmark items from the templates and annotated data. To improve annotation reliability, an evidence-guided Thought-Tool-Observation loop verifies and refines annotations. The medical tool library combines diverse medical resources and specialized tools with dynamic tool expansion to support both stages and improve generalizability. Experiments on five benchmark construction tasks spanning CT, pathology, medical video, electrocardiography (ECG), and electronic health records (EHR) show that MedBenchAgent outperforms existing methods overall across LLM jury evaluation, downstream fine-tuning, and human evaluation.

Primary area:
datasets and benchmarks
Keywords:
LLM Agents, Multi-Agent System, Automated Benchmark Construction, Medical AI

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_c5dZOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Since building benchmarks and QA datasets is a lengthy and resourceful process, this paper proposes an automatic approach to do the same in the medical field. MedAgentBench is a system of LLM agents that takes as an input a user request and a raw medical dataset and produces a set of questions and answers relevant to the query and dataset.

The framework works in two stages: benchmark design and data construction. Benchmark design includes 4 agents (planner, solver, architect, and builder) and aims to provide template items. Data construction takes the template items, retrieves data samples from the existing datasets, annotates the goal of the template if it doesn’t exist in the dataset, and then provides a list of QA. Both building blocks are supported with a tool manager that includes existing tools and is backed by external tool search.

The framework works in pathology, CT, ECG, colonoscopy videos, and EHR. It is evaluated with a mixture of human input and LLM input.

Critical strengths

The scope of the paper covers five modalities under one framework and includes many tools as evidence that agents can use. The results show that MedBenchAgent provides considerable improvements to all five modality benchmarks.

Evaluation is rigorous and comprehensive. The combination of 3 LLMs, finetuning-based metrics, and human evaluation make the benchmark design more stable. The iterations that the annotator agent takes and training a field-level quality model encourage less mistakes coming from automation.

Critical weaknesses

The main concern is medical accuracy and reliability, which is claimed to be addressed in Fig.1 in the introduction. Since the framework is evaluated using design-related rubric, more focus on medical correctness is needed. This is apparent in the low focus on the clinical relevance in A.3: validation of templates and Table 8: LLM jury evaluation, compared to general design metrics. The usage of questions that are not validated based on their clinical reliability decreases the strength of the paper, especially that it is introduced to fill the gap in medical benchmark construction. Considering the sample size of the testing and training datasets, it is challenging to take human input on every sample. However, uncertainty metrics would be an asset and can help exclude any clinically inaccurate QA generation.

Moreover, it would be good to include the specialty of these medical experts, especially that the 5 modalities are in different fields, and some of them require more depth of expertise (cancer, colonoscopy).

It is also recommended to clarify the motivation of the paper as a one that builds benchmarks rather than evaluates models on QA. It gets misleading in some sections of the paper (for example: finetuning metric and A.6). Additionally, since all methods draw on the same existing datasets, what is the claimed gain: more clinically important questions, or better executed ones?

Questions for authors
  1. In 3.1 - Input: Please give more examples of the user requests. How do the generated template items differ with high level user requests compared to detailed ones?

  2. 3.2 - Benchmark Design: How does the planner decide the relevant subtasks when the user request is high level? Does it know anything about the dataset?

  3. 3.2 - Benchmark Design: Does the builder know any information about the source data before generating the template items? For example, the builder could generate template items that need clinical data to get a correct answer, while the dataset only contains images.

  4. If the quality model scores a specification before any template items exist, why do you think this improves correctness compared to evaluating the quality after the builder step?

  5. In A.3 - Semantic validation: Where does the medical evidence come from? Also, the data dependence states that the item should not require annotation and should be answerable from the template alone. Since this needs to be true, why is the annotator then needed?

  6. 3.2 - Data Construction: How do you guarantee that the tools used for annotation are accurate? These models have inherent errors, even when they are trained on the data. How do you then create reliable questions without ensuring the tool accuracy? This is even riskier when the agent reaches the tool expansion step, where there is no supervision on the choice of the tool in the first place.

  7. 3.3 - Evidence-guided annotation: How is the relevance of the medical evidence validated? Even if it was relevant, it is good to keep track of the resources that instantiated items rely on, since medical resources can sometimes vary across regions.

  8. Table 1: Why are the human approval percentages in the same range despite how challenging a domain is? For example, the FT accuracy is the lowest for CT analysis (expected), but the human approval percentage remains similar to the ECG understanding task and even higher than colonoscopy QA (0.68 accuracy)

  9. Figure 5 – Please clarify why the example provided by SpatialMed is poor. The liver volume is of high clinical priority when assessing liver cancer, and the task design involves reasoning over anatomical structures (A. 1). Additionally, a generic medical reference is valid as long as it refers to normal liver volume.

  10. Why is fine-tuning a valid approach for benchmark quality? It measures usefulness as training data and depends on similarity to the test set format.

  11. What motivated your choice of the chosen retrieval models?

Reviewer_LW17OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces MedBenchAgent, a multi-agent framework for automated medical benchmark construction across diverse tasks and modalities. The framework addresses two challenges in existing approaches: limited generalizability across medical domains and insufficient reliability in benchmark design and data annotation.

The proposed framework consists of two stages. In Benchmark Design, specialized agents decompose user requests, develop benchmark specifications, and generate item templates. A learned field-level quality model provides feedback to refine 11 specification fields. In Data Construction, agents retrieve relevant medical samples, perform evidence-guided annotation through an iterative Thought–Tool–Observation process, and instantiate final benchmark items. A shared Medical Tool Library supports both stages through specialized resources and dynamic tool expansion.

The authors evaluate the framework on five tasks covering ECG, 3D CT, colonoscopy videos, EHR, and breast pathology. Compared with general-purpose and specialized baselines, MedBenchAgent achieves the highest downstream fine-tuning accuracy and human approval rate across all five tasks. Ablation studies further examine the contributions of specification refinement, evidence-guided iterative annotation, and dynamic tool expansion.

Critical strengths

S1. Comprehensive empirical evaluation across diverse medical modalities.

The evaluation is a major strength of this work. The authors consider five substantially different medical tasks and compare against both general-purpose agents and specialized medical benchmark construction methods. Importantly, benchmark quality is assessed through complementary perspectives: LLM jury evaluation, downstream fine-tuning effectiveness, and independent medical expert review.

S2. A well-structured framework with targeted reliability mechanisms and supporting ablations.

The separation of benchmark design from data construction is sensible, as these stages involve distinct sources of errors. The learned field-level quality model provides targeted feedback for specification refinement, while evidence-guided iterative annotation attempts to ground reference answers in medical criteria and tool observations.

The ablation studies in Tables 2–4 and Appendix Tables 9–11 provide useful evidence for these design decisions.

Critical weaknesses

W1. The evidence for generalizability is narrower than the paper's key claims suggest.

Although the framework is evaluated on five distinct modalities, each evaluation uses an established task formulation, task-specific datasets, and relevant pretrained retrieval models or specialized tools. The source data also frequently include existing annotations or structured metadata, such as ECG labels, CT segmentation masks, and EHR records. Consequently, the experiments primarily demonstrate the framework's effectiveness across five selected, relatively well-supported medical benchmark construction settings. They do not fully establish how reliably the framework transfers to genuinely new tasks, unfamiliar modalities, or settings with limited annotations and unavailable specialized tools.

Dynamic tool expansion is presented as a key mechanism supporting generalizability. However, the paper provides limited characterization of what tools are actually discovered and integrated, how often this mechanism succeeds or fails, and how its effectiveness changes under unfamiliar task requirements. While Table 4 and Appendix Table 11 show improvements from enabling dynamic expansion, they do not directly establish open-ended generalization.

This limitation matters because broad generalizability is one of the paper's central motivations and claimed contributions. A more direct evaluation of previously unseen tasks or tool-availability constraints would substantially strengthen this claim.

W2. The benefits of the overall agent architecture are not sufficiently disentangled from additional tools and computational resources.

MedBenchAgent introduces a relatively complex pipeline involving multiple specialized agents, a separately trained quality model, iterative specification refinement, retrieval, repeated tool calls, and dynamic tool acquisition. While the authors control the underlying agent backbone, source data, and target benchmark sizes, it is unclear whether the competing methods have comparable access to specialized medical tools, external knowledge, and inference or execution budgets. For example, the component ablations demonstrate that the individual mechanisms help within MedBenchAgent, but do not fully address whether a simpler pipeline with comparable tools and computational resources could achieve similar results.

Additionally, the paper does not report sufficient end-to-end cost and efficiency measurements, such as token usage, tool calls, execution time, or cost per accepted benchmark item. These measurements are particularly relevant given that automation and scalability are primary motivations for the work.

Without these comparisons, it remains difficult to assess the practical efficiency and distinct contribution of the proposed architecture relative to simpler alternatives.

Questions for authors

Q1. Generalization and dynamic tool expansion (W1).

Could the authors provide a breakdown of the tools dynamically discovered and integrated for each task, including their frequency of use, successful integration rate, and contribution to the final benchmark items? More importantly, is there evidence that MedBenchAgent can handle previously unseen benchmark construction tasks or situations where suitable specialized tools are not initially available?

This would help determine whether the reported generalizability primarily comes from broad coverage of existing tools or from genuinely transferable tool-selection and adaptation capabilities.

Q2. Resource-matched comparison and efficiency (W2).

How do MedBenchAgent and the competing methods compare in terms of specialized tool access, LLM calls, inference tokens, total execution time, and computational cost?

Could the authors provide a cost-matched comparison or a simpler LLM-based construction pipeline equipped with the same medical tool resources? Such a comparison would help isolate the benefits of the proposed multi-agent architecture from those of additional resources and computation.

Q3. Reliability and statistical uncertainty.

The reported human approval rates are based on 100 sampled items per method per task, and the downstream fine-tuning results are presented without uncertainty estimates.

Could the authors report inter-rater agreement, confidence intervals for human approval, and variability across fine-tuning runs where feasible? This would be especially valuable for tasks where the improvements over the strongest baseline are relatively modest, such as EHR clinical reasoning and 3D CT spatial analysis.

Q4. Reproducibility.

As a benchmark paper, it matters a lot if you could provide link to a valid huggingface or github repo.

Reviewer_mko6OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes MedBenchAgent, a multi-agent framework for automatically constructing medical benchmarks across different tasks and modalities. It uses a two-stage pipeline for benchmark design and data construction, with a field-level quality model, evidence-guided iterative annotation, and an extensible medical tool library.

Experiments across five medical tasks (ECG, CT, colonoscopy video, EHR, and pathology) show improvements over general-purpose and specialized baselines in LLM-based evaluation, downstream fine-tuning, and human expert approval.

Critical strengths
  • The framework addresses a practical problem in medical AI evaluation. Supporting five distinct modalities within one benchmark construction pipeline is a meaningful contribution compared with existing task-specific approaches.
  • The experimental evaluation is fairly comprehensive. Improvements are consistent across all five tasks, and the ablation studies provide useful evidence for the individual contributions of the quality model, iterative annotation, and dynamic tool expansion.
Critical weaknesses
  • The methodological novelty is somewhat limited. Most components build on established agentic workflows, retrieval, tool use, and iterative verification. The main contribution appears to be their integration rather than a fundamentally new approach.
  • The evaluation does not sufficiently establish the framework's efficiency and robustness. There is no clear comparison of computational cost, tool calls, latency, or construction effort against baselines. This is important given the complexity of the proposed pipeline and its motivation of reducing manual benchmark construction effort.
Questions for authors
  1. How does MedBenchAgent compare with the baselines in terms of construction time, computational cost, and tool usage for the same number of benchmark items?
  2. How well does the framework generalize to a new medical modality or task without manually configuring additional tools or modifying the pipeline?
  3. How sensitive is the annotation quality to errors from external medical tools, esp dynamically integrated tools that are only checked for successful execution?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.