- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper introduces ReVision3D, an attribution-guided recursive self-improvement framework for 3D medical perception. Annotated CT volumes serve as replay environments for rendering previously unvisited views and verifying localized predictions against reference masks. The framework distinguishes failures in acquisition, perception, training, and inference, and uses a frozen LLM designer to propose targeted revisions. A fixed external objective and noise-aware acceptance criteria determine whether revisions are retained.
Experiments on LiTS and KiTS23 demonstrate substantial improvements over thirteen frozen vision–language models, supported by matched-protocol comparisons and component ablations. The revised reader achieves 79% liver-lesion recall and 83% kidney-tumor recall at fewer than 0.4 false positives per patient, with particularly strong improvements on small lesions. The analyses further show that recipe-level revisions primarily refine existing detections, whereas perception adaptation expands the detectable lesion set.
Critical strengths
-
The paper addresses an important limitation of existing self-improvement approaches: knowing that a system failed does not reveal which component should change. Using annotated volumes for counterfactual view rendering and spatial verification provides a concrete, externally grounded mechanism for diagnosing failures. Separating proposal generation from a fixed evaluation and acceptance process also makes the improvement procedure more auditable. This conceptual contribution is the main reason for my positive assessment.
-
The evaluation includes thirteen frozen VLM baselines, a matched dense-reading comparison, and ablations separating reading protocol, training recipe, and perception adaptation. These analyses support the conclusion that the gains are not simply explained by observing more slices. Particularly compelling is the finding that recipe-level improvements largely refine existing detections, whereas perception adaptation substantially expands the detectable lesion set. The large small-lesion gains and reporting of rejected revisions strengthen the empirical contribution.
Critical weaknesses
-
Validation is restricted to two abdominal CT datasets, with test cohorts of 31 liver patients and 70 kidney patients. Some important subgroups are particularly small—for example, the kidney test cohort contains only 14 lesions in the 1–5 mL range. Although the reported gains are encouraging, these sample sizes limit confidence in subgroup robustness and generalization to different acquisition protocols, patient populations, or organs. Broader evaluation is the main missing evidence preventing a stronger recommendation.
-
The paper explicitly states that the routing rule in Eq. (5) was formulated retrospectively and did not schedule the reported experiments. Consequently, the results provide stronger evidence for the effectiveness of the selected interventions than for the complete attribution-driven decision process. This does not negate the framework’s value, but it leaves uncertainty about whether the proposed routing procedure reliably selects useful interventions in a new setting.
Questions for authors
-
Table 2 reports liver test (J) values of 0.71 and 0.54 for the two seeds of the designer-selected configuration. Are the overall and small-lesion recall improvements consistent across these seeds? Per-seed results with patient-level confidence intervals would help assess robustness given the limited test cohorts.
-
Since Eq. (5) was formulated retrospectively, what procedure selected intervention levels during the reported experiments? Please distinguish decisions made by the designer from those made by researchers, and clarify which parts of the attribution-guided loop were executed automatically.
-
Do the authors have any existing results on an independent cohort or acquisition-protocol subgroups that could help assess generalization? Such evidence would materially strengthen my confidence. Evaluation on additional organs or modalities would be valuable future work rather than a prerequisite for acceptance.
-
The reproducibility statement mentions that code, designer prompts, playbooks, and trial registries are prepared for release. Will these materials, together with the trained reader adapters and evaluation scripts, be publicly available? Please clarify the release timeline and whether an anonymized repository is available during review. Open-source availability is important to my assessment because independently reproducing the recursive improvement process requires more than access to the final model.
4 comments
Response to Reviewer Y4oS (Part 2 of 3)
Response to W1 and Q3: Cohort Size and Generalization across Acquisition Protocols and Organs
The cohorts are the complete held-out test splits of both datasets; to quantify the subgroup uncertainty the reviewer points to, we add patient-bootstrap intervals to every per-size cell; the kidney 1–5 mL bin (n = 14) has an interval of [0.33, 0.82] for seed 0. On protocol: the test volumes span 0.7–1.5 mm slice spacing on liver and 0.5–5.0 mm on kidney. Recall of seed 0 at its ≤0.5 FP operating point (liver support ≥2; kidney support ≥1; hand adapters at theirs), by spacing group:
| Spacing group (patients) | Designer seed 0 | Hand adapters |
|---|---|---|
| Liver ≤1.0 mm (22) | 0.80 @ 0.32 FP | 0.25 @ 0.05 |
| Liver 1.0–2.5 mm (9) | 0.74 @ 0.44 | 0.26 @ 0.11 |
| Kidney ≤1.0 mm (14) | 0.75 @ 0.86 | 0.44 @ 1.07 |
| Kidney 1.0–2.5 mm (8) | 1.00 @ 0.12 | 1.00 @ 0.25 |
| Kidney >2.5 mm (48) | 0.82 @ 0.27 | 0.74 @ 0.31 |
Recall is higher or equal in every group (the eight kidney patients at 1.0–2.5 mm are a tie at 1.00, with fewer false positives); thin-slice kidney volumes draw more false positives for both readers (more slices are read), which we will note. An independent liver or kidney cohort is the natural next step, and the limitations section says so.
We do have one existing result on a further organ, which we add to the paper: the same readers were also trained and evaluated on pancreatic tumours (PanTS; 100 training, 20 development, 30 test patients), a much harder target on which the current candidate space is not sufficient. Test J is 0.05 with the frozen encoder and 0.08 / 0.13 for the two seeds of the designer's configuration, with 0.90 / 0.75 of lesions never detected on any slice. The attribution places the loss at perception, as on liver and kidney, but the perception revisions in the current candidate space reduce it only slightly (never-detected share 0.95 with the frozen encoder). This is the bounded case the third contribution describes: the loop locates the limitation and does not claim progress it has not verified. We add it as a third column of Table 1 and Table A6 and discuss it in Section 6.
Response to W2 and Q2: Selection of Intervention Levels
We did, at every decision point; the designer chose what to try within a level, and the fixed gate accepted or rejected. Recipes were searched first because that loop was implemented first; the vision-encoder adapters were our decision after the designer's playbook diagnosed a perceptual plateau ("hand, on designer's diagnosis" in Table A9); the perception-configuration search (Table A8) and the inference programs were the designer's under the gate. Eq. (5) was written after the liver-study decisions and scheduled nothing. Automated: proposal generation, training, evaluation, acceptance and playbooks. Not automated: the choice and order of levels. We will add a table of decision points with the decision-maker (given in full in our response to Reviewer i6HP).
We also ran a control during the discussion period: the perception search of Table A8 repeated with the attribution terms removed from the designer's prompt and feedback, everything else identical. The designer proposed the same six configurations in the same order and the same configuration was accepted. This matches the reviewer's reading: the results establish the effectiveness of the selected interventions and the value of attribution as a measurement of what they changed, and the accepted revision is robust to whether the designer sees the attribution terms. Whether the routing rule selects useful levels in a new setting is the question for prospective work, and the second contribution and the abstract are scoped accordingly.
Response to Q4: Code and Model Release
Yes. The supplementary zip already contains the core implementation (replay world, objective, attribution, acceptance rule, designer interface and inference programs, about 450 lines), a demo that runs on synthetic volumes without API keys, and the verbatim designer prompts, playbooks and 18 trial registries behind Tables A8–A11 and A16. We are preparing an anonymized repository with the full training and evaluation pipeline (data preparation, reader training, the three search levels, test scoring), the saved development and test reads of all four readers, from which every table can be recomputed without a GPU, and the scripts for the new analyses in this rebuttal; we will post the link in this thread during the discussion period. The trained adapters will be released with the camera-ready version.
Response to Reviewer Y4oS (Part 3 of 3)
Additional Update: Acceptance Rule at the Inference Level
While preparing the threshold details requested by Reviewer i6HP we tightened the acceptance test at the inference level, whose bootstrap had re-picked operating thresholds within each resample. The new rule chooses thresholds on other development patients; under it we re-scored all 60 logged proposals (none beats the reader alone at P ≥ 0.9) and re-ran the search; the one program per organ that the stricter rule accepts on development does not improve on the reader alone on test (liver exceeds every false-positive budget; kidney ΔJ +0.009, interval includes zero). We remove the inference-level test claims (Table 2 bottom rows and the corresponding sentences in the introduction and Section 5.3). The perception-level results, on which the reviewer's assessment rests, use a different test (retrained readers against the seed-noise floor, with integer thresholds that transfer from development to test) and are unaffected; the full table is in our response to Reviewer i6HP.
I thank the authors for the detailed response and the experiments. I appreciate the clarifications on the method's positioning and limitations. I therefore decide to maintain my score.
Response to Reviewer Y4oS (Part 1 of 3)
We thank the reviewer for the thoughtful review and the accurate summary of the paper. We appreciate the recognition of the problem the paper addresses (knowing that a system failed does not reveal which component should change), of annotated volumes as a concrete, externally grounded mechanism for diagnosing failures, and of the separation between proposal generation and a fixed evaluation and acceptance process. We are also glad the reviewer found the presentation clear and the evaluation convincing, in particular the matched dense-reading comparison, the finding that recipe revisions refine existing detections while perception adaptation expands the detectable set, and the reporting of rejected revisions. We answer the four questions in order. The per-seed and subgroup numbers come from the saved test predictions of the submitted readers; no reader was re-run on test.
Response to Q1: Per-Seed Results with Patient-Level Confidence Intervals
Yes: detection is consistent across seeds. The table gives both seeds of the designer's configuration at the operating threshold chosen on development (the same threshold for both seeds and for the baseline: support ≥2 on liver, ≥1 on kidney), with 95% patient-bootstrap intervals.
Overall recall is 0.79 and 0.82 on liver and 0.83 and 0.81 on kidney, and small-lesion (1–5 mL) recall is 0.69 and 0.75 on liver and 0.57 and 0.50 on kidney, against 0.38 and 0.29 for the hand-chosen adapters. Both seeds therefore expand the detectable lesion set by a similar amount. What differs between seeds is the false-positive rate at that threshold: seed 1 produces more false positives (0.87 vs 0.35 per patient on liver, 0.53 vs 0.37 on kidney). This explains the J values in Table 2 (liver 0.71 / 0.54, kidney 0.80 / 0.78, against 0.39 and 0.68 for the hand-chosen adapters): J enforces hard false-positive budgets, so under the ≤0.5 FP budget seed 1 moves one threshold step tighter, to 0.51 recall at 0.32 FP on liver and 0.75 at 0.17 on kidney. The seed effect is thus one of false-positive calibration, not of which lesions the reader can detect, and both seeds exceed the hand-chosen adapters on J on both organs. We will report both seeds in the abstract and Table 1 with these intervals. Three further seeds of the designer's configuration are training now, and we will add them to this table in this thread.