acceptodds
Under review as a conference paper at ICLR 2027

CT-AGENT: AN EVIDENCE-GROUNDED MULTI-TOOL AGENT FOR HETEROGENEOUS CT ANALYSIS

Abstract

Clinical CT interpretation comprises a sequence of evidence-gathering steps. A radiologist moves across slices, localizes findings, measures lesions, compares examinations, and reconciles these observations before producing a report. Current 3D vision-language models usually compress this process into a label, answer, or generated report. Specialist imaging models expose useful intermediate results and usually operate as separate tools. We present CT-Agent, an evidence-grounded agent that organizes image-only, text-only, and image-text requests as structured tasks, invokes expert models through the Model Context Protocol, and retains masks, bounding boxes, measurements, text records, and execution provenance in a case-level evidence memory. The same evidence supports quantitative reasoning and output verification. Each capability is evaluated with a task-specific benchmark. On LiTS, CT-Agent improves mean Dice from 0.715 to 0.731 and reduces measurement errors relative to 3DMedAgent. On 40 BronAtlas cases, coordinating ATM22 and AeroPath raises Dice to 0.750 and precision to 0.993. For abdominal screening, organ-level inputs, a lightweight adapter, and calibrated thresholds recover the zero sensitivity observed under direct CT-CLIP transfer. In a 40-case CT-RATE workflow, every case completes tool execution and evidence export; verification corrects four report assertions and introduces one error. These results characterize the effects of tool orchestration and the current limits of calibration, sample size, and automatic verification.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Medical imaging agents, Computed tomography, Tool-augmented LLMs, Evidence-grounded reasoning, Output verification

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_jHtdOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents a CT-AGENT that accepts image-only, text-only, and image-text inputs. It utilizes the Model Context Protocol to invoke tools to extract masks, bounding boxes, biomarkers, and text records. It also does an evidence support and output verification.

Critical strengths

The paper is organized and contains many illustrative figures. The framework applies output verification to its output for revision.

Critical weaknesses

The paper contains one main contribution: an executive framework that integrates multiple tools into a single agentic framework. This is a common feature for generally agentic tools, which does not make it unique.

The segmentation results, compared with 3D-MEDAGENT, do not indicate whether the performance difference is due to the different tools (models) used or to the agentic framework.

The adaptation of CT-CLIP improved results through fine-tuning, but it is not related to the framework and therefore not a relevant contribution.

The framework is supposed to work with different inputs (text-only, image-only, and image-text pairs); however, the experiments do not show which data type is being used.

Questions for authors

What is the main contribution that distinguishes this work from 3D-MEDAGENT?

1 comment

Thank you for asking us to clarify how CT-Agent differs from 3DMedAgent. CT-Agent’s main contribution is a mechanism for verifying and revising CT analysis outputs against evidence from specialist tools, while recording the evidence behind each decision.

3DMedAgent’s Think-with-1-Slice loop selects and examines additional CT slices to verify evidence and refine its answers. CT-Agent verifies the analysis output itself. For example, in Section 4.4, it checks a pulmonary nodule assertion in a draft report against the bounding boxes and confidence scores produced by a separate detector. If the outputs disagree, a calibrated rule determines whether to revise the assertion or retain it and flag the case for review. The case record links the decision to the underlying detection and its source, making the discrepancy and any resulting revision traceable. This output-level verification and revision is the distinction we draw from general multi-tool integration. 3DMedAgent methodology

Separating workflow effects from tool choice. Section 4.1 compares the two complete systems on the same 50 MedSeg records, using the same liver-segment questions. CT-Agent’s mean macro Dice was 2.36 percentage points higher than 3DMedAgent’s, while both systems achieved the same overall question-answering accuracy. Sections 4.2.1 and 4.3 compare adaptive and fixed workflows with access to the same tools, allowing us to assess the effect of workflow decisions. On LiTS, CT-Agent achieved 60.10% mean Dice, compared with 54.30% for fixed fusion using SynTumor and VISTA3D. On HCC-TACE-Seg, CT-Agent could stop after reviewing the initial mask, reducing mean runtime from 86.4 to 34.2 seconds while maintaining the same mean Dice of 80.74%. These comparisons quantify how using intermediate evidence to select, refine, or accept an output affects segmentation quality and runtime.

CT-CLIP and input modalities. We removed the CT-CLIP adaptation experiment and its associated claims. All revised evaluations use CT images as input. The MedSeg comparison also gives both systems the same liver-segment questions, while the integrated chest workflow uses a comprehensive analysis request and clinical information when available. Text-only input is described as a supported interface setting, but our experimental conclusions concern CT-based tasks.

I tightened the phrasing and clarified the distinction between the verification mechanism, workflow comparisons, and input modalities. All claims, numbers, and the citation were preserved.

Reviewer_NzWJOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper presents CT-Agent, an evidence-grounded agent in which an LLM controller converts heterogeneous CT requests into structured task objects, invokes expert tools through MCP in a ReAct loop, retains their outputs in a case-level evidence memory, and verifies the final output against that evidence. Each capability is evaluated with a separate task-specific benchmark: lesion segmentation on LiTS against 3DMedAgent, multi-tool airway segmentation on BronAtlas, CT-CLIP transfer to abdominal screening, and evidence verification within a 40-case CT-RATE workflow. The reported gains are small (for example Dice 0.731 vs 0.715 on 22 LiTS cases), and the authors acknowledge limits of calibration, sample size and automatic verification.

Critical strengths
  • Sensible engineering goal and also discuss limits. Keeping tool outputs as source-linked evidence (mask to volume to report statement) and checking the report against that evidence is a reasonable design for CT, and the limitations paragraph openly lists the small samples, the in-sample calibration, and the verifier's sensitivity/specificity trade-off.
  • The CT-RATE verification experiment tests the central claim with case-level counts (18 conflicts, 5 revisions, 4 correct, 1 wrong) instead of aggregates only.
Critical weaknesses
  • The experiments do not separate what the agent adds from what the tools it calls already do, so the main claims are not supported. (a) LiTS: the paper never says which segmentation tool CT-Agent called, and there is no row for the tool on its own, so this may be a comparison between two tools instead of between two agents; 3DMedAgent’s paper reports no LiTS result, so the 0.715 comes from the authors’ own run of it. The LiTS training volumes are the same data as MSD Task03 Liver, which common tools were trained on (VISTA3D, the tool used by 3DMedAgent, reports MSD03 hepatic tumor Dice of 0.59 to 0.70 in its own paper); Figure 5 even labels the cases liver_13, liver_17 and liver_33, the MSD Task03 names. A mean Dice of 0.731 on 22 such cases almost equals the best LiTS challenge result on the held-out test set (0.739), which is quite interesting to believe for an off-the-shelf tool on unseen cases. (b) BronAtlas: the "case-level quality assessment, candidate fusion, and precision control" rules are never written down, and there is no simple fusion baseline (union, intersection, or picking the better tool per case). A 0.01 Dice gain over AeroPath on 40 cases with no error bars cannot be judged, and UNets trained on BronAtlas itself reach 0.82 to 0.88 [1], against 0.68 to 0.75 here. (c) CT-CLIP: the "direct transfer" baseline is set up to fail: a fixed 0.5 threshold on cosine similarity, and AUCs of 0.26 and 0.31 mean the score points the wrong way, so flipping its sign alone would give better results: 0.74 and 0.69. The adapted numbers are measured on the same cases used to fit the thresholds, and nothing in this experiment needs an agent. (d) CT-RATE: the only comparison is CT-CHAT without and with evidence verification and feedback refinement; the MONAI lung-nodule detection bundle on its own is not reported. CT-CHAT gets 16 of 40 right on a balanced set, which is worse than guessing, so a rule that simply trusts the detector may beat the whole verification step.

  • The evidence is too thin and too under-specified for the claims. Sample sizes are so small (e.g., 22/40/200), with no confidence intervals or tests anywhere, while numbers are given to five decimals (Table 3). The abdominal screening dataset, its size and label source are never stated; the decimals (sensitivity 0.600, 0.667, 0.833) suggest a handful of positives per organ. Prompts, tool calls per case, tool-selection failures, and runtime are not reported; code is "to be released". The presentation does not help: all eight figures in the main paper look like with AI-generated contents in large part: the report icon in Figure 1 contains non-word text, a typical sign of a generated image; Figures 1 to 4 are generic icon diagrams that repeat Sections 3.1 to 3.3; and Figures 5 to 7 are look like some AI style & low-resolution figures, and the numbers in Figure 6(b) cannot be read. Also, the CTs in Figure A1 are flipped.

Overall, I do not think this paper is ready for ICLR. The paper presentation, experiments, and figures do not meet the bar for the venue.

[1] Intensity Confusion Matters: An Intensity-Distance Guided Loss for Bronchus Segmentation

Questions for authors
  • Which tools did CT-Agent and 3DMedAgent call for LiTS, were any trained on LiTS / MSD Task03 Liver, and what does each tool score alone on the same 22 cases? How were the 22 cases chosen, and which 40 of the 100 BronAtlas scans were used?
  • What exactly are the airway quality assessment, candidate fusion and precision control rules, and how do they compare with union, intersection, and a per-case pick of ATM22 vs AeroPath (quality score or oracle)?
  • Which dataset, how many cases and positives per organ were used for abdominal screening, where do the labels come from, and were the ExtraTrees adapter and thresholds fit and evaluated on the same cases?
  • On the 40 CT-RATE cases, what does the detection bundle alone score, and a rule that overrides CT-CHAT whenever the detector fires? Why is CT-CHAT below chance, and how was the assertion extracted from its report?
  • In the CheXpert experiment, how is a macro AUC computed from LLM outputs when the classifier probabilities are identical in both arms, and what is the classifier?
1 comment

Thank you for your comments on case selection, tool baselines, and figure presentation. In the revised manuscript, we expanded the LiTS evaluation, added standalone-tool and fixed-fusion baselines on the same scans, and compared adaptive and fixed workflows with access to the same tools across three additional tasks.

1. Liver tumor segmentation and comparison with 3DMedAgent. Section 4.2.1 replaces the original 22-case LiTS comparison with an evaluation of all 126 scans in the labeled LiTS training archive that passed the geometry check. SynTumor, VISTA3D, the fixed-fusion workflow, and CT-Agent achieved mean Dice scores of 53.00%, 54.10%, 54.30%, and 60.10%, respectively. The fixed workflow starts with the VISTA3D mask and adds only adjacent voxels predicted by SynTumor but absent from VISTA3D. Added voxels are capped at 5% of the original VISTA3D voxel count. CT-Agent uses mask statistics and CT overlays to inspect disagreements between the predictions. It can also invoke VISTA3D with point prompts for local refinement before returning the final mask. Compared with fixed fusion, CT-Agent increased mean Dice by 5.80 percentage points and voxel recall by 4.50 percentage points. We report this as a paired workflow comparison on the evaluated LiTS scans.

Section 4.1 separately compares CT-Agent and 3DMedAgent on 50 MedSeg CT scans, using the same questions about liver segment location and volume. CT-Agent can use TotalSegmentator or AtlasNet and combine their predictions by segment. The 3DMedAgent workflow uses TotalSegmentator to generate segment masks, then passes the derived measurements and structured report to its published Final_Test module. CT-Agent achieved a 2.36-percentage-point improvement in mean macro Dice (95% CI: 0.62 to 4.71). Both systems achieved the same overall question-answering accuracy.

2. Airway fusion rules and baselines. Section 4.2.2 specifies that the evaluation included all 40 hospital-contributed BronAtlas scans, with case IDs newcase-01 through newcase-40. Residual fusion starts with the AeroPath mask and adds ATM22-only regions that pass connectivity, size, and CT intensity checks. Agreement-band fusion retains voxels in the union that lie within 1.6 mm of the intersection. The revised manuscript reports how often each candidate was selected. In paired comparisons on the same scans, the archived outputs had mean Dice scores 1.36 percentage points higher than simple union (95% CI: 0.36 to 3.03) and 7.34 percentage points higher than simple intersection (95% CI: 4.74 to 11.57). The cited study trained models on a BronAtlas training split, whereas our evaluation applies existing tools and their combinations to the 40 hospital-contributed scans. The two studies therefore use different training and evaluation settings. Gong et al.

3. CT-CLIP and CheXpert. We removed these experiments and their associated claims. Section 4.3 now directly compares adaptive and fixed workflows with access to the same tools. On COCA, CT-Agent inspected regions of disagreement before selecting an output, increasing mean Dice from 60.50% to 75.17%. On HCC-TACE-Seg, CT-Agent could stop after reviewing the initial mask. This reduced mean runtime from 86.4 to 34.2 seconds while maintaining the same mean Dice of 80.74%.

4. CT-RATE report verification. Section 4.4 distinguishes the 20 calibration cases from the separate set of 40 evaluation cases. The workflow links each nodule assertion under review to its corresponding detections and confidence scores. The original CT-CHAT outputs correctly classified eight positive and eight negative cases. The complete verification workflow corrected four assertions, increasing the number of correct judgments from 16/40 to 19/40. This evaluation compares the original CT-CHAT outputs with the complete verification workflow. Detector-only prediction and direct detector override were not evaluated as separate comparison conditions.

5. Statistical reporting and figures. The revised manuscript reports paired differences and confidence intervals for MedSeg and BronAtlas, as well as end-to-end runtimes in Section 4.3. Airway segmentation percentages are reported to two decimal places, and Figure 6(b) was revised to make the values easier to read. We consolidated the four method diagrams into three. The integrated chest CT example now appears in Section 4.4 as Figure 7; its caption specifies that the overlays correspond to the same axial slice.

I tightened several long sentences and clarified the comparison conditions and scope of the claims. All numbers, results, and the citation were preserved.

Reviewer_7ZFbOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

My understanding is that CT-Agent aims to coordinate specialist CT tools through a shared case record. A language-model controller selects tools and stores their masks, measurements, text, and sources. Rules and a language-model judge check candidate outputs against this record. The experiments cover liver tumors, airways, abdominal findings, chest report correction, and complete chest workflows. A separate chest X-ray experiment tests structured handling of classifier outputs. The authors report segmentation gains and report corrections, including one harmful revision.

Critical strengths
  1. The case record is useful for inspecting quantitative results. Section 3.2 links measurements to their masks, scan spacing, and producing tools. This gives readers a way to check where a number came from. Appendix A also demonstrates integration of segmentation, detection, measurement, and reporting on 40 cases.
  2. The verification study reports both gains and failures. It uses 20 calibration cases separate from 40 test cases. Five revisions fix four errors and introduce one error. The paper also reports the loss in specificity. This gives a useful, bounded account of the correction rule's behavior.
Critical weaknesses
  1. The steps producing the segmentation gains are not specified well enough to assess the central claim. I could not tell from Section 4.1 how either CT-Agent or 3DMedAgent produces the liver tumor masks. Section 4.2 names airway quality checks, fusion, and precision control, but does not define these operations. The Dice gain over 3DMedAgent on LiTS is 0.016. The gain over AeroPath on BronAtlas is about 0.010. Without the processing rules and matched tool access, these results do not establish what the controller contributes.
  2. I do not find the demonstrated contribution of the execution framework sufficient. Appendix A shows completion of one type of chest request with four named tools. It does not compare a fixed workflow using the same tools. This shows that the components can run together, but does not establish a benefit from adaptive control. The report test raises correct nodule assertions from 16/40 to 19/40. However, it does not compare the verifier with directly applying the same calibrated detector rule. The value added by the language-model verifier is therefore also unclear. These gaps limit the new knowledge I can identify from the experiments.
Questions for authors
  1. What exact steps produce the liver tumor masks for both systems and the fused airway mask? Which choices does the controller make?
  2. How does a fixed workflow using the same tools perform on the complete-workflow cases in Appendix A? Please show a case-level trace that explains any difference.
  3. On the CT-RATE test cases, what do detector-only predictions and the same calibrated detector rule without a language-model verifier achieve?
1 comment

Thank you for pointing out the need to clarify which parts of CT-Agent contribute to the reported results. We revised the segmentation experiments, added comparisons with fixed workflows using the same tools, and narrowed the conclusions drawn from the CT-RATE study.

  1. How the segmentation masks are produced

We removed the original 22-case LiTS comparison with 3DMedAgent. The revised paper reports two separate experiments.

  • LiTS liver tumor segmentation (Section 4.2.1). We evaluated SynTumor, VISTA3D, a fixed workflow, and CT-Agent on the same 126 scans that passed the geometry check. The fixed workflow starts with the VISTA3D mask and adds adjacent regions detected by SynTumor but not by VISTA3D, capped at 5% of the original VISTA3D mask volume. If VISTA3D returns an empty mask, the fixed workflow retains it. CT-Agent reviews mask statistics and CT overlays. When both predictions are available, it also inspects regions where they disagree and may use point prompts to refine the VISTA3D prediction before selecting the final mask. Mean Dice was 60.10% for CT-Agent and 54.30% for the fixed workflow. This comparison measures the complete adaptive procedure, including optional refinement; the separate contributions of tool selection and refinement have not been isolated. We also discuss potential overlap with the tools’ training data, since the scans came from the labeled LiTS training archive.
  • 3DMedAgent comparison (Section 4.1). On 50 MedSeg records, both systems analyzed the same CT scans and answered the same questions about liver segment location and volume. CT-Agent could use TotalSegmentator, AtlasNet, or combine their predictions by segment. The 3DMedAgent workflow used TotalSegmentator’s liver_segments task and passed measurements from its masks, together with the structured report, to the published Final_Test module. CT-Agent’s mean macro Dice was 2.36 percentage points higher (95% CI: 0.62 to 4.71). Both systems answered 103 of 150 questions correctly. For BronAtlas (Section 4.2.2), residual fusion adds regions detected only by ATM22 to the AeroPath mask after connectivity, size, and CT intensity checks. Agreement-band fusion retains voxels in the union that lie within 1.6 mm of the intersection. The 40 archived final masks comprised 4 AeroPath outputs, 12 residual-fusion outputs, 21 agreement-band-fusion outputs, and 3 union outputs. Their mean Dice was 1.36 percentage points higher than that of the simple union (95% paired bootstrap CI: 0.36 to 3.03). The archive records which candidate was selected for each scan, but the selection rationale and thresholds were unavailable. Independence from the reference annotations therefore remains unverified. We report this as a comparison of archived outputs.
  1. Fixed-workflow comparisons and case-level trace

Section 4.3 adds same-tool comparisons on COCA, HCC-TACE-Seg, and LUNA16. On COCA, adaptive control increased mean Dice from 60.50% to 75.17%, at an additional 8.8 seconds per scan. On HCC-TACE-Seg, mean Dice was unchanged while mean runtime fell from 86.4 to 34.2 seconds. On LUNA16, the fixed consensus output matched CT-Agent’s sensitivity. These results show that the benefit of adaptive control varies with the task and the fixed workflow used for comparison.

The fixed workflow has not been evaluated on the same 40 integrated CT-RATE cases. Section 4.4 and Figure 7 document workflow completion and present the evidence record for one case. A paired execution trace comparing the two workflows is unavailable. We therefore limit our conclusions on these cases to workflow feasibility and traceability.

  1. Detector-only and direct-rule baselines

Section 4.4 compares the original CT-CHAT assertions with the assertions after verification and revision. Across the 40 evaluation cases, revisions corrected four errors and introduced one. The number of correct assertions increased from 16/40 to 19/40, with higher sensitivity and lower specificity. A constant prediction would also have been correct in 20/40 cases on this balanced set. Detector-only performance and performance from directly applying the same calibrated rule have not been evaluated on these cases. The incremental contribution of the language-model verifier therefore remains undetermined.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.