- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper seems to propose an improvement over a prior LLM-agent that has tool use capabilities. The tools in question are VLMs or image classification/segmentation/detection models, working on medical imaging datasets. The proposed LLM-multiagent uses these tools to come to verdicts which are then evaluated against ground truth values to compute verdict accuracy metrics.
The paper is extremely hard to read and most likely it was written by an LLM.
The novelty or claimed contributions are unclear. There seems to be an extension to the number of tasks from the prior work, VERITAS. There also seems to be some kind of variance measurement (wrongly) framed as uncertainty.
Methods are extremely unclear due to, probably, the LLM prose of the paper. The proposed system seems to consist of agents capable of image processing tasks (i.e. the tools mentioned above), and some LLM-agents that conduct statistical testing (probably by using statistics libraries) and some user-facing LLM outputting human-readable strings as verdicts.
The reported empirical improvements lack context. The tasks are not clearly defined. There seems to be improvement in metric values but judging the benefits are difficult.
Critical strengths
I like to think the authors extended a prior multi-agent platform for hypothesis testing in medical imaging tasks with uncertainty-aware methods. This might have been a contribution if it was true but it is extremely difficult for me to say whether it is true or not.
Critical weaknesses
The paper is quite impossible to read for a human.
The contributions are unclear. Novelty is unclear. Methods are unclear. Datasets, tasks and experimental setup are unclear unless one goes and reads the prior VERITAS paper.
Questions for authors
None.