- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The authors present MedHarness as a runtime for medical imaging agents. The runtime is designed to work for multiple imaging modalities using a shared protocol. It specifically targets reliability (not deciding if unsure / not enough evidence) as well as budget efficiency (only using tools until enough evidence is present to make a decision). The runtime is evaluated in two experimental settings (Live512, Mix4), each of which is composed of diagnostic data from four modalities (skin lesion, fundus image, clinical image QA, endoscopy). In the experiments, MedHarness consistently improved correct accepts and reduced wrong accepts, while using fewer module calls than the “always-open” baseline where every applicable tool is used. The authors provide further evidence for the utility of the components of their runtime through various ablations.
Critical strengths
- (S1) Performance and reliability gains. The runtime seems well-composed and appears to yield remarkable gains in performance and reliability, especially since it abstains in many cases where baseline systems predict incorrectly.
- (S2) Applicability across modalities. MedHarness seems to consistently outperform baselines across various modalities, and its shared protocol seems to apply well to all of them.
Critical weaknesses
(W1) Presentation. The paper heavily relies on field-specific jargon that makes it hard to understand for readers who are not experts in the design of agents and harnesses. Much of the core terminology used is never introduced or explained, leaving it to the reader to try to find out what it means. This includes terminology in the description of the runtime itself, the baseline methods, as well as the evaluation. For example:
- What are a “fixed evidence path” and a “private always-on stack” (l. 39)? This is crucial to understand the contribution of MedHarness as well as the baselines.
- What are “dispatch features” and “stop/open/repair/lock predicates” (l. 129)?
- How does the Mix4 stress test work? How is the dataset composed, what does it measure, and how is it different from the Live512 setup? Are the two setups contributions of this work or benchmarks from previous work? Without this information, I find it particularly hard to understand Tables 4 and 5.
- The frequent use of abbreviations defined in Table 1 makes it laborious to read the text and the results tables. One often has to refer back to the table to parse results.
- What are the “Rescue / Harm” counts?
- What is an “agree/conf cascade”? Although mentioned in Section A.3, I could not find an intuitive explanation of how it works and why it is compared to MedHarness.
There are many more such examples. Overall, I struggled to understand many important details of the paper, especially in the evaluation section, even after multiple reads. My impression is that the designed harness and the experimental results are interesting and useful, but I find that the presentation makes the contributions of the paper quite inaccessible to the wide audience of a conference like ICLR.
Questions for authors
(Q1) Extension to other modalities. How could the runtime be extended to unseen medical imaging modalities? Would this require adaptation of the runtime (e.g., registration of a new recipe)? Or could this even work out-of-the-box?