- Rating
- 3
- Confidence
- 1
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This work presents MedGauge, a small commit layer that decides which LLM agent values may be signed as trusted and which action would make a rejected value acceptable. The results show that MedGauge improved performance of language-model agents on a benchmark of 1,068 cases spanning cardiac MRI, abdominal CT and chest radiography.
Critical strengths
An important problem with a practical solution.
Critical weaknesses
(1) Difficult to follow as it lacks details for clinical concepts (2) This study does not cite some relevant work, making it hard to assess the innovations and contributions, including ShieldAgent, FormalJudge, and RvLLM.
Questions for authors
can you compare MedGauge with a deterministic controller? does MedGauge reverse naturally occurred failures?
Response to Reviewer pmtY
Thank you for reviewing our paper and for finding the solution practical. To make the paper accessible to readers outside medical imaging, we added clinical background and a step-by-step example, discussed the three related works you pointed out, and ran new experiments for both of your questions.
We added Appendix A0 specifically for readers without a clinical background. This section explains the clinical quantities we measure, including heart chamber volumes, abdominal organ volumes and the heart-to-chest width ratio, where pixel-to-millimetre scaling comes from, and why a scaling error produces a wrong number that still looks physiologically normal. It concludes with a glossary of all key terms, and Table A1 collects the mathematical notation.
In the main text, Section 3.1 now walks through one CT case from start to finish: a calculation fault reports a liver volume that is too low, an independent check flags the liver, the agent recomputes the volume, and MedGauge signs the corrected value once the flag is cleared. Section 3 also defines terms such as handles inline where they first appear.
Section 2 and Appendix N now cite and compare all three.
All three verify whether an agent trajectory or model output satisfies a written specification. MedGauge addresses a different problem: deciding which continuous measurements from a tool pipeline may be reported as trusted. Its checks compute evidence without running the pipeline step under audit, and each alarm names the exact tool call that clears it. This executable repair path is what turns a blocked measurement into a recovered one.
Yes. Table 1 now includes a deterministic rule controller that runs every check without a language model and withholds a value whenever a blocking alarm fires.
The comparison shows what a fixed rule can and cannot do. The controller reaches high safety by withholding every flagged value, with zero silent errors (wrong values signed as trusted) on cardiac MRI and abdominal CT. Because it never investigates, however, it repairs zero faults and needlessly withholds at least half of the correct cases where a check fires. MedGauge is at least as safe on every task, yet its reasoning agent follows each alarm's repair instructions to fix faulty measurements and clear false alarms, raising usable output (cases signed as trusted with correct values) by 24 to 31 percentage points across the three tasks (Table 1). Putting a reasoning agent under a programmatic commit check thus preserves safety while recovering usable measurements.
It keeps them from being reported as trusted, and it repairs those that a parameter can fix. For any natural pipeline error inside its coverage, MedGauge prevents a plausible-looking wrong number from being signed as trusted and routes it to human review. Tested against expert annotations on abdominal CT, MedGauge signs none of the covered natural errors under any tested language model, including severe failures where an organ is measured at a tiny fraction of its true volume (Section 4.2, Appendix M.2). The same checks without our contract do not achieve this: a ReAct agent driven by a weaker model still signs most of the covered errors on the main benchmark.
Repair applies when an error comes from a recoverable parameter such as a wrong slice thickness or unit scale: the alarm names the corrective call, and the agent runs it to restore the correct value. On injected CT faults, where the true value is known, MedGauge repairs nearly half (Table 1). When a natural error comes from two segmentation masks disagreeing on an organ's boundary, no parameter change can tell which mask is right, so MedGauge flags the value for review without guessing.
Thank you again for your helpful feedback. We hope these revisions make both the clinical context and the contributions easy to follow.