- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
3DMedEvo improves frozen models’ performance on CT question answering through an evolving multimodal skill repository, combining trajectory-to-skill distillation, clinical evidence checking, adoption-aware credit, and posterior-interval governance. The paper reports notable benchmark improvements, but questions remain regarding the visual input pipeline, novelty relative to closely related methods, and evidence for generalization through continual evolution.
Critical strengths
1.Substantial empirical improvements. Under frozen memory, the method improves overall accuracy across two primary backbones and two benchmarks, outperforming the strongest same-backbone memory baseline by approximately 4.3–6.9 percentage points.
2.Extensive supplementary analyses. Ablations, cross-backbone evaluation, repository transfer, construction-order experiments, and physician verification provide complementary evidence supporting the framework.
3.Transparent failure analysis. The case studies disclose concrete limitations, including tool incompatibility, infeasible recovery actions, and insufficient interaction budgets.
Critical weaknesses
W1. The visual input pipeline for GLM-5.2 is unclear
The official GLM-5.2 documentation lists text as its input modality, whereas Appendices A and G describe supplying current-case images and tool-returned images to the same frozen backbone. The paper does not specify the actual model endpoint, visual frontend, or image-to-text mechanism. A text model can process structured measurements, but this does not explain how chest images and historical visual exemplars inform its decisions. This gap affects the interpretation of the multimodal mechanism, visual ablations, and baseline comparisons, and requires clarification.
W2. Methodological advances over closely related frameworks are not sufficiently established
SkeMex already includes trajectory distillation, skill creation and revision, branch-aware retrieval, adoption attribution, relative-advantage estimation, and repository governance. The main additions in 3DMedEvo are visual exemplars, clinical evidence checking, and posterior-interval governance. The ablations support the usefulness of these components, but do not sufficiently separate the contributions of additional information, additional processing, and new algorithmic mechanisms. Comparisons against adapted baselines with the same visual information, evidence access, and comparable budgets would better establish the methodological advance.
W3. Evidence for continual improvement on unseen patients is limited
The evolving-memory protocol revisits the same evaluation questions, retaining skills and correctness feedback derived from reference answers across rounds. The R1→R3 gains in Table 2 are 0.92–1.72 percentage points, primarily supporting adaptation to repeated tasks rather than establishing continual improvement on unseen patients. The partial-freezing comparison in Table 3 provides some support, but an independent held-out evaluation and uncertainty estimates across multiple runs are still missing. This concern does not negate the frozen-memory results.
Questions for authors
1.What exact model ID and visual processing pipeline were used for GLM-5.2? If an additional vision model was involved, was it equally available to all baselines?
2.Can the authors compare against an adapted SkeMex under matched visual information, evidence access, and comparable budgets?
3.Starting from the same end-of-R1 snapshot, how do freezing and continued evolution compare on a patient-disjoint held-out set that never contributes to updates? Are the gains stable across runs?
4.How frequently are tool-incompatible skills retrieved? Can the authors release the evidence-retrieval configuration, evidence snapshots, and task-baseline estimation procedure?
7 comments
W2 / Q2: Methodological advances over closely related frameworks are not sufficiently established — SkeMex already includes trajectory distillation, skill creation and revision, branch-aware retrieval, adoption attribution, relative-advantage estimation, and repository governance. The main additions in 3DMedEvo are visual exemplars, clinical evidence checking, and posterior-interval governance. The ablations support the usefulness of these components, but do not sufficiently separate the contributions of additional information, additional processing, and new algorithmic mechanisms. Comparisons against adapted baselines with the same visual information, evidence access, and comparable budgets would better establish the methodological advance. Can the authors compare against an adapted SkeMex under matched visual information, evidence access, and comparable budgets?
Thank you for your valuable feedback regarding the methodological advance over SkeMex. Following your advice, we compared 3DMedEvo with an adapted SkeMex under matched visual information and evidence access, which separates the contribution of the new mechanisms from that of the additional information.
The adapted SkeMex keeps its own utility estimation, skill revision, and repository governance, and receives the two kinds of information that 3DMedEvo adds: historical CT exemplars are attached to its retrieved skills, and the same clinical references are supplied during its skill learning. All three methods use Qwen3.6-Plus with the same acquisition cases, current-case images, CT tools, and tool budget per question, and are evaluated with frozen memory on all 1,740 DeepTumorVQA questions, as in Table 1.
Regarding budgets, we match the budget available to each method rather than the cost it finally consumes. The consumed cost is determined by each framework's own learning pipeline: SkeMex and 3DMedEvo distill, review, and govern skills through different stages, so forcing identical consumption would require truncating one pipeline and would no longer compare the two methods as designed. The resulting costs are nevertheless of a similar scale. In Table 7, building the library with Qwen3.6-Plus takes 69.40M tokens for 3DMedEvo and 59.80M for SkeMex, and the adapted SkeMex additionally processes the added exemplars and references.
| Method | Visual exemplars | Clinical references | Accuracy (%) |
|---|---|---|---|
| SkeMex | ✗ | ✗ | 49.71 |
| Adapted SkeMex | ✓ | ✓ | 51.49 |
| 3DMedEvo | ✓ | ✓ | 54.71 |
The matched information improves SkeMex by 1.78 points, while 3DMedEvo remains 3.22 points higher with the same information. Most of the 5.00-point advantage of 3DMedEvo over SkeMex therefore comes from how the information is used, namely evidence-checked skill synthesis, adoption-aware credit, and posterior-interval governance, rather than from the additional inputs. The ablations in Figure 3 are consistent with this: with the information unchanged, crediting all injected skills instead of only the adopted ones lowers accuracy by 3.7 points, and replacing credit intervals with direct SCS comparisons also lowers accuracy.
W3 / Q3: Evidence for continual improvement on unseen patients is limited — The evolving-memory protocol revisits the same evaluation questions, retaining skills and correctness feedback derived from reference answers across rounds. The R1→R3 gains in Table 2 are 0.92–1.72 percentage points, primarily supporting adaptation to repeated tasks rather than establishing continual improvement on unseen patients. The partial-freezing comparison in Table 3 provides some support, but an independent held-out evaluation and uncertainty estimates across multiple runs are still missing. This concern does not negate the frozen-memory results. Starting from the same end-of-R1 snapshot, how do freezing and continued evolution compare on a patient-disjoint held-out set that never contributes to updates? Are the gains stable across runs?
Thank you for this constructive suggestion. We ran the requested comparison of freezing and continued evolution from the same end-of-R1 snapshot on a patient-disjoint held-out set that never contributes to updates.
Skill acquisition follows the DeepChestVQA evolving-memory setting in the paper. Freezing keeps the library saved at the end of R1, while continued evolution proceeds from this same snapshot for one and two more rounds, giving the R2 and R3 libraries. We repeat this comparison three times, where every run continues evolution from the same end-of-R1 snapshot, and report the mean and range. The held-out set contains 900 questions sampled from the Existence Detection task of the 3D-RAD dataset at https://huggingface.co/datasets/Tang-xiaoxiao/3D-RAD, each paired with the full 3D CT volume of a distinct CT-RATE patient who does not appear in the acquisition set. During held-out evaluation, each library and its credit statistics are frozen, so these cases never contribute to updates. All conditions use Qwen3.6-Plus with the same CT tools, prompts, and tool budget as in the paper, and the no-skill baseline uses the same tools with an empty library.
| Memory | Mean accuracy (%) | Range (%) |
|---|---|---|
| No skill, same tools | 54.78 | – |
| Frozen at the end of R1 | 57.11 | 56.78–57.67 |
| Continued evolution to R2 | 60.74 | 60.56–60.89 |
| Continued evolution to R3 | 62.41 | 61.89–62.78 |
Continued evolution exceeds freezing by 3.63 points after one more round and by 5.30 points after two, and the R3 library is 7.63 points above the no-skill baseline. Since the held-out patients never contribute feedback or skills, this gain reflects reusable skill improvement rather than adaptation to repeated questions. The gains are also stable across runs: the accuracy of each library varies by at most 0.89 points, well below the gain of continued evolution, and the R3 library exceeds the frozen R1 library in every run. Consistently, Appendix B.5 rebuilds the library under four different case orders, and every order improves over ReAct by 7.07–9.02 points.
We greatly appreciate your detailed and constructive review of our work. We are grateful for your recognition of the "substantial empirical improvements" of our method and the "extensive supplementary analyses" that support the framework. We are also encouraged by your positive assessment of our "transparent failure analysis". In addition, we are very thankful for your insightful questions and suggestions; we take every comment seriously and provide point-by-point responses below.
W1 / Q1: The visual input pipeline for GLM-5.2 is unclear — The official GLM-5.2 documentation lists text as its input modality, whereas Appendices A and G describe supplying current-case images and tool-returned images to the same frozen backbone. The paper does not specify the actual model endpoint, visual frontend, or image-to-text mechanism. A text model can process structured measurements, but this does not explain how chest images and historical visual exemplars inform its decisions. This gap affects the interpretation of the multimodal mechanism, visual ablations, and baseline comparisons, and requires clarification. What exact model ID and visual processing pipeline were used for GLM-5.2? If an additional vision model was involved, was it equally available to all baselines?
Thank you for raising this important point. We would like to clarify the visual input pipeline of GLM-5.2, which follows its official configuration.
The backbone is the official GLM-5.2 model. Since GLM-5.2 accepts text only, Z.ai provides an official Vision MCP Server at https://docs.z.ai/devpack/mcp/vision-mcp-server that gives it image understanding through a GLM vision model, and we follow this official configuration with its default vision model GLM-5.3-Flash. All images take this route, including the views of the current case, the images returned by CT tools, and the visual exemplars of retrieved skills. The vision model reads each image and returns a textual description to GLM-5.2, which can then compare the current case with historical exemplars. GLM-5.2 remains the only model that plans tool calls, uses skills, answers the question, and performs every LLM-based stage of skill evolution. Both models stay frozen.
This vision model belongs to the backbone configuration rather than to 3DMedEvo. All GLM-5.2 methods in the paper, including ReAct and every memory baseline, use the same vision model with the same settings, so the baseline comparisons and visual ablations are all made under an identical visual frontend. In addition, 3DMedEvo achieves consistent gains with Qwen3.6-Plus, which reads images natively, indicating that our conclusions do not depend on this visual frontend. We will add this clarification to the implementation details in the main text.