- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes a framework that improves frozen LLMs and VLMs by updating three forms of external expertise: a Skill document containing reusable procedures, a Knowledge Memory containing evidence-supported facts, and a Multimodal Knowledge Base (MMKB) containing visual references and their source answers. Candidate Skill and Memory updates are evaluated on both an incoming batch and a buffer of historical cases before deployment. MMKB additionally guides the model to analyze retrieved references and compare their evidence with the current case. Experiments across six benchmarks and four base models evaluate online improvement, generalization with frozen expertise, and cross-model transfer.
Critical strengths
- The framework provides a coherent approach to managing deployment experience. Separating procedural guidance, factual knowledge, and visual examples is well motivated. The online protocol clearly separates deployed predictions from subsequent feedback and candidate evaluation. Evidence checks, retrieval exclusions, and historical replay further strengthen the experimental design.
- The evaluation covers several complementary dimensions of utility. Beyond aggregate performance, the paper studies component removal, generalization to held-out cases, cross-model reuse, and visual reference utilization. The comparison using identical retrieved references is particularly useful: it provides evidence that explicitly analyzing and comparing references can outperform simply supplying them as context.
Critical weaknesses
-
The experiments do not sufficiently isolate the mechanisms underlying the central contribution. The validation ablation compares the proposed strategy primarily against accepting every update, which establishes the importance of filtering updates but does not establish the advantage of dynamic validation over simpler alternatives. Likewise, much of the visual improvement is already present without adding deployment cases: on QCalEval, Base, Initial MMKB, and growing MMKB score 72.3, 83.3, and 84.2, respectively; on MedChain Task 3, the corresponding scores are 0.514, 0.630, and 0.637. These results support reference-guided reasoning, but provide more limited evidence for the incremental value of accumulating visual deployment experience. This distinction matters because continual learning from deployment is the paper’s central framing.
-
The transfer protocol supports a narrower conclusion than transfer to unseen cases. Expertise is learned from the complete benchmark and then transferred to another model evaluated on that same benchmark. Excluding matching references prevents direct retrieval of a case’s answer, but does not remove its possible influence on the learned Skill or Knowledge Memory. Consequently, this experiment demonstrates cross-model reuse of benchmark-informed expertise rather than generalization to cases unseen during expertise construction. Furthermore, GPT-5.6-Sol optimizes expertise for all source models, which complicates attributing the resulting expertise solely to the source model.
Questions for authors
- Under matched optimization and evaluation budgets, how does the proposed validation strategy compare with a fixed validation set, incoming-batch-only validation, and historical-buffer-only validation? Which component accounts for its advantage?
- Can you more explicitly separate gains from initial reference resources, reference-analysis instructions, Skill/Memory adaptation, and newly accumulated visual cases? How does the incremental benefit of MMKB growth change with the initial library size?
- Can cross-model transfer be evaluated on an independent held-out split that was unavailable during source expertise construction? Does the benefit persist when the source model also serves as its own optimizer?
- What are the total inference-token, model-call, and latency costs, including optimization and validation? Are the improvements maintained against baselines given comparable computational budgets?
8 comments
Response to Reviewer rRSz (2/2)
W2 / Q3. Transfer on unseen cases and the role of the optimizer. We agree that Section 4.5 measures reuse on the same stream, so we repeated the transfer on unseen cases. We use the split of Section 4.4. The expertise learned with GPT-5.6-Sol on the learning split is frozen and given unchanged to each other model, which is evaluated only on the test split. No test case shares a patient, source group, or quantum experiment with the learning split. The table reports the relative change from each target's Base on the test split:
| Benchmark | GPT-4o-mini | Nemotron-3-Nano-Omni-30B | Qwen3.8-27B |
|---|---|---|---|
| AgentClinic | +16.0% | +7.7% | −2.3% |
| MedChain | +9.0% | +3.2% | +10.1% |
| MedThink-Bench | +9.5% | +4.7% | +3.1% |
| MedThinkVQA | +6.5% | +6.0% | +0.9% |
| VisuLogic | +3.7% | +76.9% | +42.9% |
| QCalEval | +58.2% | +26.6% | +47.3% |
On the optimizer: in this experiment and in Figure 4(a), the source is GPT-5.6-Sol, which both answers the cases and serves as the optimizer. The transferred expertise is therefore produced by the source model alone, and the benefit persists on unseen cases.
The learned expertise also generalizes to a different dataset. We froze the expertise learned with GPT-5.6-Sol on the complete stream and applied it, without further learning, to another dataset of the same task type, so none of these cases was available when the expertise was built:
| Learned on → evaluated on | Base | Ours (frozen) |
|---|---|---|
| VisuLogic → MM-IQ (955 questions after removing duplicates) | 59.7 | 64.2 (+7.5%) |
| AgentClinic-MedQA → MedR-Bench (957 cases) | 52.7 | 58.4 (+10.8%) |
Q4. Cost and comparable compute. Ours costs about 5× Base, between ACE (2.4×) and SkillOpt (7.4×), and the gain does not come from more compute. Averaged over the benchmarks with GPT-5.6-Sol, including optimization and validation:
| Per benchmark | Base | ACE | SkillOpt | Ours |
|---|---|---|---|---|
| Tokens | 7.6M | about 18M | about 56M | about 38M |
| Relative to Base | 1× | 2.4× | 7.4× | about 5× |
Ours makes 7,485 model calls per benchmark on average, compared with 3,261 for Base.
Relative to answering the stream once with the base model alone, our framework adds about 2.3× extra cost on text-only tasks and 4.8× on image tasks.
On comparable compute, SkillOpt uses more compute than Ours, yet its average gain on unseen cases is 8.3%, compared with 20.7% for Ours (Section 4.4). In addition, the fixed validation set in the Q1 table uses the same number of validation cases per batch as Ours and falls below Base on all five benchmarks.
Dear Reviewer rRSz,
Thank you again for your thoughtful comments and for taking the time to consider our responses.
As the discussion period approaches its conclusion, we wanted to kindly follow up and ask whether our additional analyses and clarifications have addressed your main concerns.
If there are any remaining questions or points you would like us to clarify further, please feel free to let us know. We would greatly appreciate any additional feedback while there is still time for discussion.
Thank you again for your time and consideration.
Best regards,
The Authors of Submission 21400
Response to Reviewer rRSz (1/2)
Thank you for the detailed review. We answer each point below.
W1 (validation) / Q1. Validation rules under matched budgets. We ran this comparison under matched budgets: Ours is the only rule that stays above Base, and neither part alone accounts for the advantage. Only the acceptance rule changes; the optimizer, the three components, the batch size, the case order, and the seeds are the same as in Ours. All runs use GPT-5.6-Sol and the online protocol on the complete stream. The fixed validation set has 64 cases, the same number of validation cases per batch as Ours (32 new and 32 buffer cases).
The new batch alone and the buffer alone are both below Base on all five benchmarks, and only requiring both stays above Base. The reason is that the optimizer receives the validation feedback when it proposes the next update (Figure 1). When the same cases are used for validation again and again, as with a fixed set or the buffer alone, the optimizer writes very detailed Skills for those cases, and these Skills do not generalize to new cases. The new batch alone avoids this reuse, but it does not check whether the update still works on earlier cases.
W1 (visual gains) / Q2. Sources of the visual gain. We ran two new experiments, and both show that deployment cases matter most when the initial library is small. First, we removed the initial references, so MMKB contains only cases completed earlier in the stream (GPT-5.6-Sol; improvement over Base):
MMKB built only from deployment cases improves over no MMKB on all four tasks. Second, we varied the size of the initial library (MMKB with deployment cases; improvement over Base):
Without initial references, deployment cases alone give +9.9% and +4.4%.
The existing results separate the other sources (relative to Base):
Continual learning in our framework is also not limited to MMKB. On the text benchmarks, the Skill and Knowledge Memory start empty and all gains come from deployment cases (e.g., 79.0 → 83.6 on AgentClinic).