acceptodds
Withdrawn from ICLR 2027

MedHarness: A Unified Budgeted Runtime for Reliable Medical Imaging Agents

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Medical imaging agents still lack a shared runtime reliability layer: under a limited verification budget, deployment must decide when to stop and accept, when to open additional evidence by failure mode, when to repair, and when to reject. Tool-using workflows often freeze a fixed evidence path or a private always-on stack per dataset; those pipelines are costly, still leave many wrong accepts, and cannot share one controller across heterogeneous imaging modalities. We present MedHarness, a registered multi-surface runtime reliability layer that attaches after a proposer's initial prediction: a shared control policy with surface recipe adapters under one packet-route-decision-trace contract, so a new imaging surface attaches by recipe registration to the same stop / open-evidence / repair / reject controller. We evaluate on four registered surfaces (4848 cases) under two complementary tasks: live-pool decision quality and failure-enriched stress efficiency. On a 512-case live pool, MedHarness cuts wrong accepts versus always-on by about 47% and raises accepted accuracy (83.2% vs. 72.9%). On a failure-enriched four-surface stress ranking, the same controller cuts estimated API calls/case under matched context-load accounting by about 71% versus always-on. A held-out three-surface check and an alternate LLM transfer keep the same control ordering. The code is available at https://anonymous.4open.science/r/MedHarness-paper-release-A7F3/.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Medical imaging agents, Visual question answering, Budgeted verification, Runtime reliability, Multimodal evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_PRMfOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The authors present MedHarness as a runtime for medical imaging agents. The runtime is designed to work for multiple imaging modalities using a shared protocol. It specifically targets reliability (not deciding if unsure / not enough evidence) as well as budget efficiency (only using tools until enough evidence is present to make a decision). The runtime is evaluated in two experimental settings (Live512, Mix4), each of which is composed of diagnostic data from four modalities (skin lesion, fundus image, clinical image QA, endoscopy). In the experiments, MedHarness consistently improved correct accepts and reduced wrong accepts, while using fewer module calls than the “always-open” baseline where every applicable tool is used. The authors provide further evidence for the utility of the components of their runtime through various ablations.

Critical strengths
  • (S1) Performance and reliability gains. The runtime seems well-composed and appears to yield remarkable gains in performance and reliability, especially since it abstains in many cases where baseline systems predict incorrectly.
  • (S2) Applicability across modalities. MedHarness seems to consistently outperform baselines across various modalities, and its shared protocol seems to apply well to all of them.
Critical weaknesses

(W1) Presentation. The paper heavily relies on field-specific jargon that makes it hard to understand for readers who are not experts in the design of agents and harnesses. Much of the core terminology used is never introduced or explained, leaving it to the reader to try to find out what it means. This includes terminology in the description of the runtime itself, the baseline methods, as well as the evaluation. For example:

  • What are a “fixed evidence path” and a “private always-on stack” (l. 39)? This is crucial to understand the contribution of MedHarness as well as the baselines.
  • What are “dispatch features” and “stop/open/repair/lock predicates” (l. 129)?
  • How does the Mix4 stress test work? How is the dataset composed, what does it measure, and how is it different from the Live512 setup? Are the two setups contributions of this work or benchmarks from previous work? Without this information, I find it particularly hard to understand Tables 4 and 5.
  • The frequent use of abbreviations defined in Table 1 makes it laborious to read the text and the results tables. One often has to refer back to the table to parse results.
  • What are the “Rescue / Harm” counts?
  • What is an “agree/conf cascade”? Although mentioned in Section A.3, I could not find an intuitive explanation of how it works and why it is compared to MedHarness.

There are many more such examples. Overall, I struggled to understand many important details of the paper, especially in the evaluation section, even after multiple reads. My impression is that the designed harness and the experimental results are interesting and useful, but I find that the presentation makes the contributions of the paper quite inaccessible to the wide audience of a conference like ICLR.

Questions for authors

(Q1) Extension to other modalities. How could the runtime be extended to unseen medical imaging modalities? Would this require adaptation of the runtime (e.g., registration of a new recipe)? Or could this even work out-of-the-box?

Reviewer_MmPEOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

Medical imaging agents still lack a shared runtime reliability layer: under a limited verification budget, deployment must decide when to stop and accept, when to open additional evidence by failure mode, when to repair, and when to reject. Tool-using workflows often freeze a fixed evidence path or a private always-on stack per dataset; those pipelines are costly, still leave many wrong accepts, and cannot share one controller across heterogeneous imaging modalities. We present MedHarness, a registered multi-surface runtime reliability layer that attaches after a proposer's initial prediction: a shared control policy with surface recipe adapters under one packet-route-decision-trace contract, so a new imaging surface attaches by recipe registration to the same stop / open-evidence / repair / reject controller. We evaluate on four registered surfaces (4848 cases) under two complementary tasks: live-pool decision quality and failure-enriched stress efficiency. On a 512-case live pool, MedHarness cuts wrong accepts versus always-on by about 47% and raises accepted accuracy (83.2% vs. 72.9%). On a failure-enriched four-surface stress ranking, the same controller cuts estimated API calls/case under matched context-load accounting by about 71% versus always-on.

Critical strengths
  1. It's a very good research topic, but the work is very rough.
Critical weaknesses

The experimental data failed to demonstrate the novelty of this paper. Is this the first paper to use a agent-based shared runtime reliability layer in medical agents? Are there any other similar methods available for comparison?

Questions for authors
  1. All the figures in this paper were generated by AI, which reduced my interest in reading this paper.
  2. The experimental data failed to demonstrate the novelty of this paper. Is this the first paper to use a agent-based shared runtime reliability layer in medical agents? Are there any other similar methods available for comparison?
  3. I understand the author's motivation, but I think it's too rough. Please polish it further before submitting.
Reviewer_UXZ9OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents MedHarness, a unified runtime controller for medical imaging agents. It decides when to accept a prediction, collect additional evidence, attempt repair, or abstain. The framework is evaluated across several medical imaging tasks and reports fewer incorrect acceptances and lower estimated inference costs.

Critical strengths
  • The problem is practically important, especially for medical applications where incorrect predictions can be costly.
  • The experiments provide useful comparisons between different verification and abstention policies.
Critical weaknesses
  • The methodological novelty is limited. The framework mainly integrates existing verification, repair, and abstention mechanisms into a shared rule-based controller.
  • The reported accuracy improvements are partly driven by abstention. On Live512, the number of correct acceptances decreases from 373 to 367, while 71 cases are rejected. Also, the large API cost reduction is estimated on a separate failure-enriched stress set rather than the primary live evaluation.
Questions for authors
  • How does MedHarness compare with simpler confidence-based selective prediction methods at matched coverage on Live512?
  • Can the authors provide measured end-to-end latency and API costs on the primary evaluation set?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.