acceptodds
Under review as a conference paper at ICLR 2027

Med3DNavi: Agentic Visual Exploration of 3D CT Volumes via a Structured Volumetric Reading Protocol

Abstract

Three-dimensional computed tomography (CT) is central to diagnosing abdominal, thoracic, and neurological conditions, yet vision-language models (VLMs) natively perceive only 2D images. Prior work bridges this gap in two ways, each with a characteristic failure mode. Fixed 2D conversions, whether uniformly sampled slices or a trained 3D encoder, commit what the model sees before it knows what it needs, yielding **vision without agency**. Text-based medical agents instead reason over tool reports without ever inspecting the pixels, yielding **agency without vision**. We argue that both limitations trace to a single root cause: the VLM is never placed in a closed-loop visual dialogue with the volume, unlike a radiologist who actively navigates and cross-checks across planes and windows before concluding. Consequently, we present **Med3DNavi**, a training-free agent that couples a programmable CT viewer, which renders on-demand multi-planar slices, grids, HU windowing, ROI zoom, and segmentation-mask overlays, with a Structured Volumetric Reading Protocol (SVRP). SVRP follows an orient–investigate–adjudicate workflow, routing each case either to an audit path for verifying an available tool prior or to an autonomous path for proactive and independent visual exploration. Experiments on DeepTumorVQA and 3D-RAD show that Med3DNavi improves over competitive baselines, with benefits that generalize across backbones.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
Medical, CT, 3D, VLM, Agent

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_Zw9JOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Med3DNavi, a training-free framework that enables VLMs to actively explore 3D CT volumes. The framework develops a programmable viewer and a Structured Volumetric Reading Protocol (SVRP). It introduces audit and autonomous exploration routes to guide evidence collection and reasoning. Experiments on DeepTumorVQA and 3D-RAD demonstrate improvements over existing VLMs and agentic approaches.

Critical strengths
  1. The paper addresses an important limitation of existing agentic frameworks such as 3DMedAgent, where inaccurate statistics derived from upstream visual tools may affect subsequent reasoning. The proposed audit mechanism enables dynamic visual inspection and potential correction of these measurements.

  2. The programmable viewer integrates multi-planar viewing, HU windowing, ROI zooming, and mask overlays, providing flexible visual access to 3D CT volumes without requiring additional model training.

  3. SVRP organizes visual exploration into orientation, investigation, and adjudication, with dedicated audit and autonomous routes. The independent verification paths offer a reasonable approach to collecting and cross-checking visual evidence.

  4. The paper evaluates two benchmarks and multiple VLM backbones, with additional analyses of routing behavior, viewer capabilities, and verification paths. The results provide encouraging evidence for the proposed framework.

Critical weaknesses
  1. Although the audit route is a valuable contribution, the experiments mainly demonstrate improvements in final task accuracy. A more direct evaluation of error detection, measurement correction, and the risk of incorrectly modifying accurate tool outputs would better establish the reliability of this mechanism.

  2. The multi-path reasoning design is interesting, but the experiments do not clearly distinguish the benefits of context isolation from those of additional visual observations and reasoning steps. A comparison with shared-context multi-path verification would help clarify its contribution.

  3. The framework requires multiple VLM and viewer interactions, but its computational overhead is not fully characterized.

Questions for authors

See weakness.

2 comments
AuthorsOpenReview ↗

Reply to Reviewer Zw9J

We sincerely thank the Reviewer for the positive assessment, high confidence in our work, and insightful suggestions. We are delighted that you found our audit mechanism, programmable viewer, and SVRP protocol well-motivated and effective. Below, we address each of your three concerns with direct empirical evaluations.


1. Direct Evaluation of Audit Reliability (W1)

We report the audit transition matrix:

Transition Type (Prior-Only →\to Post-Audit)Share of All Audit Cases (%)Conditional Rate (%)
Incorrect →\to Correct (Error Recovery)24%24\%63%63\% (of Prior-Incorrect)
Incorrect →\to Incorrect (Unrecovered)14%14\%37%37\% (of Prior-Incorrect)
Correct →\to Correct (Valid Retention)54%54\%88%88\% (of Prior-Correct)
Correct →\to Incorrect (False Override)8%8\%12%12\% (of Prior-Correct)

On the Measurement subtasks, visual auditing reduces the Mean Absolute Percentage Error (MAPE) from 7.77% to 3.08%. We acknowledge that VLM-based visual auditing provides directional and heuristic calibration rather than exact voxel-level re-measurement. Consequently, residual errors or false overrides typically occur when candidate options are closely spaced and the agent slightly over-estimates the magnitude of a minor boundary discrepancy, shifting the adjusted estimate past the true option toward an adjacent distractor.

Nevertheless, this behavior mirrors real-world radiological reading: a radiologist inspecting a 3D CT scan cannot mentally integrate exact voxel counts to tell whether a kidney is 178.3178.3 or 188.3 cm3188.3\text{ cm}^3 by naked eye alone, and likewise relies on segmentation tools for precise quantification. However, when an automated segmentation mask is imperfect, the clinician excels at identifying whether the mask suffers from under-segmentation or boundary leakage, and determining the direction and rough magnitude of the bias to avoid trusting a flawed report blindly (such as the 158.0 cm3158.0\text{ cm}^3 distractor in Option C).

Med3DNavi inherits this exact division of labor. Furthermore, this insight points to an exciting future extension: closing the loop not only at the visual inspection level, but also at the tool execution level—allowing the VLM to issue spatial prompts (e.g., positive clicks on unsegmented voids or negative clicks on leaked boundaries) to interactive 3D segmentation models to trigger mask refinement and re-computation.


2. Isolating Context Isolation from Observation/Reasoning Budget in Multi-Path Verification (W2)

Following your exact suggestion, we conducted a controlled ablation comparing Isolated-Context Multi-Path against Shared-Context Multi-Path under the exact same planner strategies:

Multi-Path VariantMeasurementMed. ReasoningRecognitionVis. ReasoningOverall Avg.# Model Calls# Cumul. Images†# TokensLatency (s)
Shared-Context64.871.763.555.964.010.831.145K40.0 s
Isolated-Context (Ours)67.070.766.456.965.312.320.040K37.7 s (24.8 s*)

(†\dagger# Cumul. Images counts cumulative image inputs across multi-turn VLM invocations; *denotes parallel execution of the K isolated verification paths.)

Enforcing context isolation provides three advantages:

  1. Preventing early visual misinterpretations from biasing whole inference trajectory;
  2. Reducing the repeated token and visual-input overhead;
  3. (Potential) Provide native parallelizability: Because our KK verification paths are context-isolated by construction, they are mutually independent and can be optimized to execute concurrently, further cutting end-to-end wall-clock latency.

3. Computational Overhead Characterization (W3)

As noted in our Limitations (Appendix D), multi-stage agentic frameworks inevitably incur higher token consumption and latency than single-pass 2D VLM inference. However, when compared against peer agentic methods, Med3DNavi achieves the best accuracy under a comparable computational budget.

Method / VariantMeasurementMed. ReasoningRecognitionVis. ReasoningOverall Avg.# Model Calls# Cumul. Images †^\dagger# Total TokensLatency (s/case)
2D VLM35.235.248.348.357.857.845.845.846.846.81.0±0.01.0 \pm 0.01.0±0.01.0 \pm 0.01.5K±0.4K1.5\text{K} \pm 0.4\text{K}3.2±0.5 s3.2 \pm 0.5\text{ s}
3DMedAgent63.163.162.662.650.450.449.649.656.456.47.6±0.47.6 \pm 0.42.6±0.42.6 \pm 0.438.5K±5.6K38.5\text{K} \pm 5.6\text{K}35.2±3.4 s35.2 \pm 3.4\text{ s}
ReAct Agent58.558.561.861.860.560.554.454.458.858.812.6±5.212.6 \pm 5.234.1±16.534.1 \pm 16.547.8K±21.3K47.8\text{K} \pm 21.3\text{K}42.1±18.6 s42.1 \pm 18.6\text{ s}
Med3DNavi (Ours)67.067.070.770.766.466.456.956.965.365.312.3±5.812.3 \pm 5.820.0±9.220.0 \pm 9.240.0K±15.6K40.0\text{K} \pm 15.6\text{K}37.7±14.2 s37.7 \pm 14.2\text{ s}

(†\dagger# Cumul. Images counts cumulative image inputs across multi-turn VLM invocations)


We hope these additional experiments fully address your concerns.

Thanks authors for their effort in rebuttal. Most of my concerns have been addressed and I would keep my score as 3.

Reviewer_HQ2zOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces Med3DNavi, a training-free framework that enables an off-the-shelf vision-language model to interactively explore 3D CT volumes. It combines a programmable CT viewer with multi-planar rendering, HU windowing, mask overlays, and ROI zoom with a Structured Volumetric Reading Protocol (SVRP). Depending on the availability of segmentation-derived priors, the agent either audits an existing answer or conducts autonomous exploration through separate inspection paths followed by adjudication.

The framework is evaluated on DeepTumorVQA and 3D-RAD using Qwen and Gemini backbones. With Qwen, average DeepTumorVQA accuracy increases from 46.5% to 64.9%, compared with 56.0% for 3DMedAgent. Route-level and viewer-component ablations provide additional evidence for the practical value of the protocol and its visual tools. However, the experiments establish the effectiveness of the complete system more clearly than the specific benefits of closed-loop navigation or the reliability of quantitative visual correction.

Critical strengths
  1. A concrete and practical framework for interactive volumetric inspection. The programmable viewer and bounded execution protocol turn a general-purpose VLM into a system that can select views, adjust visualization settings, and revisit evidence. The distinction between auditing available measurements and independently exploring the volume is sensible. This is consequential because it offers a usable approach to 3D image interaction without task-specific VLM training.

  2. Substantial improvements supported by informative component analyses. Results across two benchmarks and two backbones strengthen the empirical contribution. The same-backbone comparison with 3DMedAgent is particularly useful. Beyond aggregate scores, the ablations show that removing mask overlays particularly harms measurement, removing window adjustment harms medical reasoning, and forcing audit cases into autonomous exploration reduces performance despite requiring more viewer calls. These findings support the practical design choices.

Critical weaknesses
  1. The controls do not fully isolate the claimed mechanism or establish its efficiency. The primary comparisons change several factors simultaneously: segmentation-derived priors, visual access, evidence quantity, and multi-step inference. Table 2 provides a valuable ReAct control with approximately matched viewer-call counts, but this does not match the number of rendered slices, visual/text tokens, or total model computation, including planning and adjudication. Similarly, the path-count analysis in Appendix C.2 changes the available viewing budget between one-path and multi-path settings, and uses path count rather than measured inference cost to support its efficiency claim. Consequently, the results support the full framework more strongly than the attribution to closed-loop navigation or independent verification specifically.

  2. The claimed visual verification of quantitative priors is insufficiently validated, and the central example contains inconsistencies. In Figure 4, the agent assumes approximately 10% undersegmentation and converts 160.5 cm³ into an estimated 178.3 cm³ without showing a corrected 3D mask, voxel-based remeasurement, or independent reference supporting that percentage. It then selects option D, which is actually 188.3 cm³, while Section 4.5 describes D as approximately 178.3 cm³. D is the nearest listed option, so this does not by itself establish an incorrect answer; it does expose a discrepancy between the estimated measurement and the reported result. The autonomous example also initially places the kidneys at axial slices 9–125 but subsequently requests slices 160–220, without explaining an indexing transformation. More broadly, the reported recovery of 63% of prior-only errors omits the complementary correct-to-incorrect transition rate. These limitations matter because reliable, verifiable correction is a central motivation of the method.

Questions for authors
  1. For Figure 4, can you provide the complete execution traces and explain both the 178.3-versus-188.3 cm³ discrepancy and the non-overlapping axial slice ranges? What quantitative evidence supports the assumed 10% volume correction, and what reference segmentation convention determines whether the displayed internal voids should count toward kidney volume?

  2. Across the evaluated audit cases, how often does auditing change an incorrect answer to a correct one, and how often does it change a correct answer to an incorrect one? For measurement questions, does auditing reduce continuous measurement error against a reference, beyond improving multiple-choice accuracy?

  3. Do the gains persist against a baseline with the same priors and comparable visual and total inference budgets, but fixed or question-conditioned view selection? Can you report actual model calls, visual/text tokens, and latency for SVRP, ReAct, and the path-count variants? This would materially clarify whether the advantage comes from adaptive inspection and protocol structure or additional inference resources.

  4. Please specify the evaluated question and scan counts, splits, and subset relationships for the main results and auxiliary analyses. In Appendix C.3, 193 disagreements together with 80.7% agreement imply approximately 1,000 questions. The directional disagreement counts, 178 versus 15, imply an audit-fraction difference of approximately 0.163, whereas the reported fractions, 0.578 and 0.384, differ by 0.194. Do these statistics use different populations or averaging conventions?

4 comments
AuthorsOpenReview ↗

Reply to Reviewer HQ2z (1/3)

We sincerely thank the Reviewer for the thorough reading and constructive critique. We are encouraged that you found Med3DNavi "concrete and practical" with "substantial improvements supported by informative component analyses." We sincerely apologize for the presentation oversights in Figure 4 and the phrasing ambiguities in the text that caused confusion, and we deeply appreciate your sharp eye in helping us catch and resolve them. Below, we clarify the exact execution trace of Fig. 4, report fine-grained token/latency profiles alongside controlled baselines, and detail the dataset statistics.


Inference Cost Profiling and Controlled Visual/Budget Baselines (W1 & Q3)

We appreciate your thoughtful point that viewer-call counts alone do not capture full test-time computation. Since our initial evaluation logger only recorded tool-call counts, we instrumented our evaluation pipeline with fine-grained cost telemetry and re-ran the profiling benchmark (N=1,000N=1,000) to report exact model calls, cumulative image inputs, total tokens, and latencies below:

Method / VariantMeasurementMed. ReasoningRecognitionVis. ReasoningOverall Avg.# Model Calls# Cumul. Images †^\dagger# Total TokensLatency (s/case)
2D VLM35.235.248.348.357.857.845.845.846.846.81.0±0.01.0 \pm 0.01.0±0.01.0 \pm 0.01.5K±0.4K1.5\text{K} \pm 0.4\text{K}3.2±0.5 s3.2 \pm 0.5\text{ s}
3DMedAgent63.163.162.662.650.450.449.649.656.456.47.6±0.47.6 \pm 0.42.6±0.42.6 \pm 0.438.5K±5.6K38.5\text{K} \pm 5.6\text{K}35.2±3.4 s35.2 \pm 3.4\text{ s}
ReAct Agent58.558.561.861.860.560.554.454.458.858.812.6±5.212.6 \pm 5.234.1±16.534.1 \pm 16.547.8K±21.3K47.8\text{K} \pm 21.3\text{K}42.1±18.6 s42.1 \pm 18.6\text{ s}
Med3DNavi (Ours)67.067.070.770.766.466.456.956.965.365.312.3±5.812.3 \pm 5.820.0±9.220.0 \pm 9.240.0K±15.6K40.0\text{K} \pm 15.6\text{K}37.7±14.2 s37.7 \pm 14.2\text{ s}
Route / SubsetProtocolMeasurementMed. ReasoningRecognitionVis. Reasoning# Model Calls# Viewer Calls# Cumul. Images# Total TokensLatency (s)
Autonomous--47.447.457.657.648.248.21.0±0.01.0 \pm 0.00.0±0.00.0 \pm 0.00.0±0.00.0 \pm 0.01.2K±0.2K1.2\text{K} \pm 0.2\text{K}1.8±0.3 s1.8 \pm 0.3\text{ s}
AutonomousReAct (-)-51.251.256.456.453.553.519.2±4.619.2 \pm 4.613.1±4.213.1 \pm 4.257.2±18.457.2 \pm 18.469.4K±22.1K69.4\text{K} \pm 22.1\text{K}59.0±19.5 s59.0 \pm 19.5\text{ s}
AutonomousSVRP-58.658.663.863.857.657.618.9±2.118.9 \pm 2.112.8±1.512.8 \pm 1.528.5±4.828.5 \pm 4.854.5K±8.2K54.5\text{K} \pm 8.2\text{K}51.3±7.9 s51.3 \pm 7.9\text{ s}
Audit-62.262.267.867.878.578.552.052.01.0±0.01.0 \pm 0.00.0±0.00.0 \pm 0.00.0±0.00.0 \pm 0.01.2K±0.2K1.2\text{K} \pm 0.2\text{K}1.7±0.3 s1.7 \pm 0.3\text{ s}
AuditReAct (-)65.565.589.489.486.286.266.066.07.3±1.87.3 \pm 1.84.6±1.14.6 \pm 1.115.4±5.215.4 \pm 5.230.3K±9.4K30.3\text{K} \pm 9.4\text{K}28.4±8.8 s28.4 \pm 8.8\text{ s}
AuditForced Auto67.267.283.583.584.484.465.265.218.1±2.218.1 \pm 2.212.2±1.512.2 \pm 1.526.4±4.626.4 \pm 4.652.1K±8.5K52.1\text{K} \pm 8.5\text{K}49.2±8.1 s49.2 \pm 8.1\text{ s}
AuditSVRP71.071.093.893.890.590.575.175.17.0±1.67.0 \pm 1.64.7±1.54.7 \pm 1.513.0±3.813.0 \pm 3.828.4K±7.1K28.4\text{K} \pm 7.1\text{K}26.7±6.8 s26.7 \pm 6.8\text{ s}

Note: Route accuracies are compured on exclusive-route subsets.

Path-Count VariantMed. ReasoningRecognitionVis. Reasoning# Model Calls# Cumul. Images# Total TokensSeq. Latency
Single Path (K=1K = 1)53.853.856.256.252.652.616.8±3.916.8 \pm 3.926.2±8.426.2 \pm 8.450.4K±14.6K50.4\text{K} \pm 14.6\text{K}48.7±13.8 s48.7 \pm 13.8\text{ s}
Two Paths (K=2K = 2)56.956.960.160.154.254.218.3±2.818.3 \pm 2.828.1±6.128.1 \pm 6.152.2K±10.5K52.2\text{K} \pm 10.5\text{K}50.2±10.1 s50.2 \pm 10.1\text{ s}
Three Paths (K=3K = 3)57.057.063.163.155.855.819.4±1.819.4 \pm 1.829.1±4.229.1 \pm 4.255.2K±7.3K55.2\text{K} \pm 7.3\text{K}52.9±7.1 s52.9 \pm 7.1\text{ s}
Adaptive K∈{2,3}K \in \{2,3\}58.658.663.863.857.657.618.9±2.118.9 \pm 2.128.5±4.828.5 \pm 4.854.5K±8.2K54.5\text{K} \pm 8.2\text{K}51.3±7.9 s51.3 \pm 7.9\text{ s}

These results clarify three key aspects of our framework's efficiency and fairness:

  • Why SVRP Consumes Fewer Tokens and Less Latency than Unstructured ReAct at Matched Calls: The Context Isolation prevents earlier rendered images from accumulating in a single long conversation history. In contrast, unstructured ReAct and Shared-Context repeatedly re-encode all historical images at every subsequent turn.
  • Controlled Path-Count Scaling (K=1,2,3K = 1, 2, 3): Under the same total viewer budget (BV=12B_V=12), increasing KK from 11 to Adaptive K∈{2,3}K \in \{2,3\} results in only a modest increase in sequential tokens and latency because the budget is partitioned across paths rather than multiplied. Moreover, partitioning the budget into smaller per-path caps reduces run-to-run cost variance, and allows the KK isolated paths to be executed in parallel (∼24.8 s\sim 24.8\text{ s} overall latency).
  • Comparison with Prior Baselines: As noted in our Limitations (App. D), agentic visual exploration naturally incurs higher test-time cost than single-shot 2D VLM inference, but yields an +18.4%+18.4\% overall accuracy gain. Compared with 3DMedAgent, Med3DNavi inspects richer multi-slice grids and mask overlays at comparable latency and improves overall accuracy by +8.9%+8.9\%.
AuthorsOpenReview ↗

Reply to Reviewer HQ2z (2/3)

Figure 4 Trace, Coordinate Indexing, and Quantitative Verification (W2 & Q1)

The 178.3 vs. 188.3 cm3\text{cm}^3 Notation in Fig. 4 & Sec. 4.5: Thank you for pointing out this phrasing ambiguity. In Sec. 4.5, our notation "shifts the answer to D (≈178.3 cm3\approx 178.3\text{ cm}^3)" was intended as shorthand for "shifts the answer to D (188.3 cm3188.3\text{ cm}^3), as it is the closest option to the agent's adjusted estimate (≈178.3 cm3\approx 178.3\text{ cm}^3)"—not that Option D itself equals 178.3 cm3178.3\text{ cm}^3. Specifically, the prior reports 160.5 cm3160.5\text{ cm}^3 (which matches distractor Option C). Upon inspecting the mask overlay, the agent identifies unfilled internal parenchymal voids, estimates a ∼10%\sim 10\% under-segmentation (160.5/0.9≈178.3 cm3160.5 / 0.9 \approx 178.3\text{ cm}^3) via visual reasoning ability. This rules out Option C (158.0 cm3158.0\text{ cm}^3) as an underestimate, and selects the nearest higher candidate. We will revise Sec. 4.5 to explicitly state both the estimated value (178.3 cm3178.3\text{ cm}^3) and the literal option text (188.3 cm3188.3\text{ cm}^3) to prevent any confusion.

Non-Overlapping Axial Slice Ranges (9–125 vs. 160–220): We sincerely thank you for catching this detail, which prompted us to re-examine the raw execution log and NIfTI header for this CT volume (D=247D = 247 axial slices). We found that this specific CT scan is stored in LAI (Left-Anterior-Inferior) orientation, causing the zz-axis indexing of the offline segmentation prior generator and the canonicalized 3D viewer to be inverted relative to each other (zviewer=247−zpriorz_{\text{viewer}} = 247 - z_{\text{prior}}). Under the viewer's canonicalized zz-axis, the prior's reported range of 9–125 actually maps to slices 247−125=122247 - 125 = 122 through 247−9=238247 - 9 = 238 (122–238), which aligns directly with the 160–220 region inspected by the agent.

Notably, this case inadvertently illustrates the core strength of our framework: even when the prior contains unusable information, the agent is not misled by the raw index numbers (9–125), but instead relies on its active visual tools to locate the true anatomical span (160–220) and recover the correct answer. We also inspect the entire benchmark for LAI-oriented volumes and found that they account for only 0.7% of DeepTumorVQA and 0% of 3D-RAD. While this has a negligible impact on overall benchmark metrics, we have fixed the NIfTI orientation canonicalization in our codebase, corrected the misleading figure presentation in the revised PDF, and included the complete execution traces for Figure 4 in the Appendix F (with file paths anonymized to prevent patient/dataset identity leakage).


Audit Transition Matrix, Continuous Error, and the Role of Visual Auditing. (Q2)

We report the audit transition matrix:

Transition Type (Prior-Only →\to Post-Audit)Share of All Audit Cases (%)Conditional Rate (%)
Incorrect →\to Correct (Error Recovery)24%24\%63%63\% (of Prior-Incorrect)
Incorrect →\to Incorrect (Unrecovered)14%14\%37%37\% (of Prior-Incorrect)
Correct →\to Correct (Valid Retention)54%54\%87%87\% (of Prior-Correct)
Correct →\to Incorrect (False Override)8%8\%13%13\% (of Prior-Correct)

On the Measurement subtasks, visual auditing reduces the Mean Absolute Percentage Error (MAPE) from 7.77% to 3.08%. We acknowledge that VLM-based visual auditing provides directional and heuristic calibration rather than exact voxel-level re-measurement. Consequently, residual errors or false overrides typically occur when candidate options are closely spaced and the agent slightly over-estimates the magnitude of a minor boundary discrepancy, shifting the adjusted estimate past the true option toward an adjacent distractor.

Nevertheless, this behavior mirrors real-world radiological reading: a radiologist inspecting a 3D CT scan cannot mentally integrate exact voxel counts to tell whether a kidney is 178.3178.3 or 188.3 cm3188.3\text{ cm}^3 by naked eye alone, and likewise relies on segmentation tools for precise quantification. However, when an automated segmentation mask is imperfect, the clinician excels at identifying whether the mask suffers from under-segmentation or boundary leakage, and determining the direction and rough magnitude of the bias to avoid trusting a flawed report blindly (such as the 158.0 cm3158.0\text{ cm}^3 distractor in Option C).

Med3DNavi inherits this exact division of labor. Furthermore, this insight points to an exciting future extension: closing the loop not only at the visual inspection level, but also at the tool execution level—allowing the VLM to issue spatial prompts (e.g., positive clicks on unsegmented voids or negative clicks on leaked boundaries) to interactive 3D segmentation models to trigger mask refinement and re-computation.

AuthorsOpenReview ↗

Reply to Reviewer HQ2z (3/3)

Dataset Splits, Subset Counts, and Appendix C.3 Statistics (Q4)

Clarifying the 0.163 vs. 0.194 Discrepancy in Appendix C.3: We sincerely thank you for your meticulous reading and arithmetic check, which caught a critical reporting inconsistency that we had completely overlooked during manuscript preparation.

As you rightly deduced from the 193193 disagreements and 80.7%80.7\% agreement rate, the cross-backbone routing analysis in App. C.3 was conducted on our 1,0001,000-question evaluation subset. It is actually a scout study held before scaling to full benchmark in order to analyze the idea while keeping the API costs manageable. While drafting App. C.3, we inadvertently plugged in Qwen's audit fraction from the full 10,000-question benchmark (0.5780.578). On this 1,0001,000-question subset, Qwen's actual audit fraction is 0.5470.547. Replacing 0.5780.578 with the subset-matched 0.5470.547 yields 0.547−0.384=0.1630.547 - 0.384 = \mathbf{0.163}, which reconciles every number in App. C.3 to the single question.

We deeply apologize for mixing statistics from two different sample sizes in the same sentence, which caused this confusing discrepancy. We are truly grateful to you for catching this mistake so we can correct it and make our manuscript rigorous and self-consistent.

Complete Specification of Scans, Questions, and Subset Relationships:

  • Table 1 & Table 2 & Fig. 3 & Appendix C.2: Evaluated on the full official test splits of DeepTumorVQA (991 CT volumes, 10,000 questions) and 3D-RAD (2,696 CT volumes, 33,910 questions across 6 subtasks).
  • Ablation & Analyses Table 3 & Appendix C.3: Evaluated on the fixed, stratified random subset of 1,000 questions (563 CT volumes) from DeepTumorVQA.

We hope these clarifications and additional experiments fully address your concerns.

Thank you for the rebuttal. I will raise my score.

Reviewer_nTirOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The reviewer uses "human" to denote my purely hand-written opinions in the review and the reviewer uses "GPT" to denote GPT 6 Medium. The reviewer uses the incognito mode so no data will be shared to Claude. The input to the LLM is simply "please review this paper for me".

Human: this paper proposes a new inference-only VLM-based 3D CT diagnosis paradigm. Different than previous agency only or vision only approaches, this approach proposes a closed-loop and direct visual dialogue w/ the 3D volume. This paper proposes a SVRP method, which includes a strcutured volumetric reading protocol consists of "orient phase", "investigation phase" and "audit route".

Critical strengths

Human:

  1. This paper proposes a domain-specific harness for 3D CT diagnosis. This could have real-world impact and may impact other works in agent harness designs.

  2. The empirical results seem to be better than previous 2D VLM and 3D encoder methods.

  3. The visualizations and ablation studies seem to be solid.

Critical weaknesses

Human:

  1. The reviewer isn't familar with relevent works in the field. The proposed method isn't compared against other agentic designs. In the main comparation table Table 1, no costs are listed, so it's not sure whether this paper's gain comes from more compute.

  2. It's hard to say if the proposed method is novel or not. This design is pretty straight forward, and is common in agentic field. The difference against [1] 3DMedAgent isn't so significant.

[1] Ziyue Wang, Linghan Cai, Chang Han Low, Haofeng Liu, Junde Wu, Jingyu Wang, Rui Wang, Lei Song, Jiang Bian, Jingjing Fu, et al. 3DMedAgent: Unified perception-to-understanding for 3D medical analysis. arXiv preprint arXiv:2602.18064, 2026.

GPT:

  1. It remains unclear whether the audit mechanism genuinely corrects unreliable upstream evidence. The paper argues that text-based agents blindly trust specialized tools, whereas Med3DNavi can detect and correct their mistakes by inspecting segmentation overlays. This is an important claim, but the experiments do not directly characterize the system's ability to distinguish correct from incorrect tool outputs. The paper reports that 63% of cases answered incorrectly using prior-only inference are recovered after visual auditing. However, these cases need not all correspond to incorrect upstream tool predictions. Some errors could arise from the VLM incorrectly interpreting an accurate report, whereas others could be caused by actual segmentation failures.

Minor:

The captions in this paper are too simple and not informative.

Questions for authors

Human:

  1. Can the authors provide controlled compute experiments, especially compare against 3DMedAgent?

  2. Can the authors show a diagram of comparing across datasetes, such as Figure 4 in 3DMedAgent?

  3. Can the authors explain which part or parts contribute to the gain beyond 3DMedAgent?

3 comments
AuthorsOpenReview ↗

Reply to Reviewer nTir (1/2)

We sincerely thank the Reviewer for the positive assessment and for recognizing the real-world impact of our domain-specific harness design. Below, we address your questions regarding controlled compute comparisons, the core distinctions over 3DMedAgent, and the source of prior-only errors.

Controlled Compute Comparison with 3DMedAgent and Other Agentic Designs (W1 & Q1)

As noted in our Limitations (Appendix D), multi-stage agentic frameworks inevitably incur higher token consumption and latency than single-pass 2D VLM inference. However, when compared against peer agentic methods, Med3DNavi achieves the best accuracy under a comparable computational budget.

Method / VariantMeasurementMed. ReasoningRecognitionVis. ReasoningOverall Avg.# Model Calls# Cumul. Images †^\dagger# Total TokensLatency (s/case)
2D VLM35.235.248.348.357.857.845.845.846.846.81.0±0.01.0 \pm 0.01.0±0.01.0 \pm 0.01.5K±0.4K1.5\text{K} \pm 0.4\text{K}3.2±0.5 s3.2 \pm 0.5\text{ s}
3DMedAgent63.163.162.662.650.450.449.649.656.456.47.6±0.47.6 \pm 0.42.6±0.42.6 \pm 0.438.5K±5.6K38.5\text{K} \pm 5.6\text{K}35.2±3.4 s35.2 \pm 3.4\text{ s}
ReAct Agent58.558.561.861.860.560.554.454.458.858.812.6±5.212.6 \pm 5.234.1±16.534.1 \pm 16.547.8K±21.3K47.8\text{K} \pm 21.3\text{K}42.1±18.6 s42.1 \pm 18.6\text{ s}
Med3DNavi (Ours)67.067.070.770.766.466.456.956.965.365.312.3±5.812.3 \pm 5.820.0±9.220.0 \pm 9.240.0K±15.6K40.0\text{K} \pm 15.6\text{K}37.7±14.2 s37.7 \pm 14.2\text{ s}

(†\dagger# Cumul. Images counts cumulative image inputs across multi-turn VLM invocations)

Moreover, since the KK verification paths in our Autonomous route are context-isolated, they can run in parallel, further reducing latency.


Core Differences and Component-Wise Gains Beyond 3DMedAgent (W2, Q2 & Q3)

While both works adopt an agentic perspective, 3DMedAgent remains tool-driven and open-loop in visual perception, whereas Med3DNavi enables VLM-directed, closed-loop volumetric exploration:

Dimension3DMedAgent (Wang et al., 2026)Med3DNavi (Ours)
Slice Selection MechanismTool-ranked: Relies on pretrained CLIP-style encoders to score and propose candidate slices.VLM-directed closed loop: The VLM actively navigates depths (grid/slice) based on what it just saw.
Visual ManipulationSingle-slice inspection per step; fixed anatomical plane and default HU windowing.Multi-planar (axial/coronal/sagittal), dynamic HU windowing, and ROI zoom.
Treatment of Tool PriorsBlind trust: Accepts upstream segmentation statistics as static text facts.Visual Auditing: Renders mask overlays onto slices to verify and correct flawed priors.
Verification StructureSingle-thread sequential reasoning history.SVRP: Context-isolated multi-path verification followed by multimodal adjudication.

Furthermore, combining Tables 1, 2, and 3 allows us to trace exactly which components drive the improvement:

  1. Visual Auditing of Mask Overlays: In Table 3, removing Mask Overlay drops Measurement accuracy from 67.6%67.6\% to 60.4%60.4\%, and in Table 2, adding visual auditing over text priors boosts Audit-route accuracy from 66.8%66.8\% to 82.9%82.9\%.
  2. Dynamic Viewer Controls — Windowing, Zoom & Multi-planar: In Table 3, stripping the viewer down to basic axial slices without windowing/zoom/overlay degrades average accuracy from 65.8%65.8\% to 57.9%57.9\%. Specifically, dynamic HU Windowing drives the gain on Medical Reasoning (+8.1%+8.1\%), while Grid + Slice synergy drives Visual Reasoning (+5.4%+5.4\%).
  3. Structured Protocol (SVRP) over Unstructured Exploration: Even with the full viewer enabled, replacing unstructured ReAct with SVRP (Orient + Isolated Verify + Adjudicate) improves accuracy under the same number of viewer calls (Table 2).

We have also added a plot showing the typical trace of 3DMedAgent in the Appendix E to illustrate the difference between two methods.

AuthorsOpenReview ↗

Reply to Reviewer nTir (2/2)

Does Auditing Correct Upstream Tool Failures or VLM Text Misinterpretations? (W3)

Because the structured prior report presents explicit pre-computed numbers, a modern VLM (e.g. Qwen3.6-27B in our experiment) rarely misreads the text itself. In fact the VLM faithfully selects the multiple-choice option matching the report's number—which is wrong solely because the upstream VISTA3D segmentation mask suffered from under-segmentation, boundary leakage, or missed lesions. By rendering the mask onto the CT slice, the Audit route directly exposes these segmentation defects and shifts the prediction toward the ground truth.

We also report the audit transition matrix:

Transition Type (Prior-Only →\to Post-Audit)Share of All Audit Cases (%)Conditional Rate (%)
Incorrect →\to Correct (Error Recovery)24%24\%63%63\% (of Prior-Incorrect)
Incorrect →\to Incorrect (Unrecovered)14%14\%37%37\% (of Prior-Incorrect)
Correct →\to Correct (Valid Retention)54%54\%87%87\% (of Prior-Correct)
Correct →\to Incorrect (False Override)8%8\%13%13\% (of Prior-Correct)

On the Measurement subtasks, visual auditing reduces the Mean Absolute Percentage Error (MAPE) from 7.77% to 3.08%. This directly proves that audit can distinguish correct from incorrect tool outputs.


We hope these additional experiments fully address your concerns.

The reviewer's concerns are resolved. However, the reviewer isn't an expert in the medical domain, so the decision will be made through the AC and other reviewers.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.