acceptodds
Under review as a conference paper at ICLR 2027

Frozen Models, Evolving Expertise: Model-Agnostic Learning from Deployment Experience for Multimodal Medical AI

Abstract

Large language models (LLMs) and vision-language models (VLMs) are usually frozen after deployment, so they do not learn from the cases they solve. This is especially concerning in medicine, where new clinical evidence, updated guidelines, and new therapies can change established practice. Fine-tuning can update the model, but it requires access to model weights and additional training. Parameter-free methods avoid training, but they may overfit a fixed validation set, lack reliable domain knowledge, or lose visual details by saving experience only as text. To address these limitations, we present a model-agnostic framework that allows frozen LLMs and VLMs to learn from deployment experience through three forms of external expertise: a Skill that guides reasoning and tool use, a Knowledge Memory that stores reliable facts supported by earlier cases or trusted external evidence, and a Multimodal Knowledge Base that keeps visual examples and guides the model to relate each retrieved case to the current image. Instead of relying on a fixed validation set, a validation strategy keeps an update only if it helps on new cases without degrading performance on earlier ones. Across six benchmarks covering clinical diagnosis, clinical workflows, medical reasoning, and medical and non-medical visual reasoning, and with four open-weight and closed-source base models, our framework improves performance during online deployment by up to 34.2% over the base model on medical tasks, generalizes to unseen cases, transfers to other models without further optimization, and works in non-medical domains.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Healthcare, Self-evolving, Large language models, Vision language models

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_rRSzOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a framework that improves frozen LLMs and VLMs by updating three forms of external expertise: a Skill document containing reusable procedures, a Knowledge Memory containing evidence-supported facts, and a Multimodal Knowledge Base (MMKB) containing visual references and their source answers. Candidate Skill and Memory updates are evaluated on both an incoming batch and a buffer of historical cases before deployment. MMKB additionally guides the model to analyze retrieved references and compare their evidence with the current case. Experiments across six benchmarks and four base models evaluate online improvement, generalization with frozen expertise, and cross-model transfer.

Critical strengths
  1. The framework provides a coherent approach to managing deployment experience. Separating procedural guidance, factual knowledge, and visual examples is well motivated. The online protocol clearly separates deployed predictions from subsequent feedback and candidate evaluation. Evidence checks, retrieval exclusions, and historical replay further strengthen the experimental design.
  2. The evaluation covers several complementary dimensions of utility. Beyond aggregate performance, the paper studies component removal, generalization to held-out cases, cross-model reuse, and visual reference utilization. The comparison using identical retrieved references is particularly useful: it provides evidence that explicitly analyzing and comparing references can outperform simply supplying them as context.
Critical weaknesses
  1. The experiments do not sufficiently isolate the mechanisms underlying the central contribution. The validation ablation compares the proposed strategy primarily against accepting every update, which establishes the importance of filtering updates but does not establish the advantage of dynamic validation over simpler alternatives. Likewise, much of the visual improvement is already present without adding deployment cases: on QCalEval, Base, Initial MMKB, and growing MMKB score 72.3, 83.3, and 84.2, respectively; on MedChain Task 3, the corresponding scores are 0.514, 0.630, and 0.637. These results support reference-guided reasoning, but provide more limited evidence for the incremental value of accumulating visual deployment experience. This distinction matters because continual learning from deployment is the paper’s central framing.

  2. The transfer protocol supports a narrower conclusion than transfer to unseen cases. Expertise is learned from the complete benchmark and then transferred to another model evaluated on that same benchmark. Excluding matching references prevents direct retrieval of a case’s answer, but does not remove its possible influence on the learned Skill or Knowledge Memory. Consequently, this experiment demonstrates cross-model reuse of benchmark-informed expertise rather than generalization to cases unseen during expertise construction. Furthermore, GPT-5.6-Sol optimizes expertise for all source models, which complicates attributing the resulting expertise solely to the source model.

Questions for authors
  1. Under matched optimization and evaluation budgets, how does the proposed validation strategy compare with a fixed validation set, incoming-batch-only validation, and historical-buffer-only validation? Which component accounts for its advantage?
  2. Can you more explicitly separate gains from initial reference resources, reference-analysis instructions, Skill/Memory adaptation, and newly accumulated visual cases? How does the incremental benefit of MMKB growth change with the initial library size?
  3. Can cross-model transfer be evaluated on an independent held-out split that was unavailable during source expertise construction? Does the benefit persist when the source model also serves as its own optimizer?
  4. What are the total inference-token, model-call, and latency costs, including optimization and validation? Are the improvements maintained against baselines given comparable computational budgets?
8 comments
AuthorsOpenReview ↗

Response to Reviewer rRSz (1/2)

Thank you for the detailed review. We answer each point below.

W1 (validation) / Q1. Validation rules under matched budgets. We ran this comparison under matched budgets: Ours is the only rule that stays above Base, and neither part alone accounts for the advantage. Only the acceptance rule changes; the optimizer, the three components, the batch size, the case order, and the seeds are the same as in Ours. All runs use GPT-5.6-Sol and the online protocol on the complete stream. The fixed validation set has 64 cases, the same number of validation cases per batch as Ours (32 new and 32 buffer cases).

BenchmarkBaseNo validation (Table 3)New batch onlyBuffer onlyFixed validation setOurs
AgentClinic79.053.563.659.860.383.6
MedThink-Bench68.451.250.755.656.971.8
MedThinkVQA68.352.158.452.260.668.9
VisuLogic45.838.137.240.442.350.0
QCalEval72.352.452.070.866.184.2

The new batch alone and the buffer alone are both below Base on all five benchmarks, and only requiring both stays above Base. The reason is that the optimizer receives the validation feedback when it proposes the next update (Figure 1). When the same cases are used for validation again and again, as with a fixed set or the buffer alone, the optimizer writes very detailed Skills for those cases, and these Skills do not generalize to new cases. The new batch alone avoids this reuse, but it does not check whether the update still works on earlier cases.

W1 (visual gains) / Q2. Sources of the visual gain. We ran two new experiments, and both show that deployment cases matter most when the initial library is small. First, we removed the initial references, so MMKB contains only cases completed earlier in the stream (GPT-5.6-Sol; improvement over Base):

BenchmarkNo MMKBMMKB without initial referencesInitial MMKBMMKB
MedChain Task 3+1.8%+9.9%+22.6%+23.9%
MedThinkVQA0.0%+0.7%+0.3%+0.9%
VisuLogic+0.2%+4.4%+8.5%+9.2%
QCalEval+0.1%+9.4%+15.2%+16.5%

MMKB built only from deployment cases improves over no MMKB on all four tasks. Second, we varied the size of the initial library (MMKB with deployment cases; improvement over Base):

Initial library size0%25%50%100%
MedChain Task 3+9.9%+18.6%+19.8%+23.9%
VisuLogic+4.4%+5.0%+8.3%+9.2%

Without initial references, deployment cases alone give +9.9% and +4.4%.

The existing results separate the other sources (relative to Base):

  • Retrieved references given only as context (Standard retrieval): −3.5% to +5.0% (Figure 3).
  • Reference analysis on the initial references (Initial MMKB): +0.3% to +22.6% (Figure 3).
  • Skill and Memory: on MedChain Task 3, removing the Skill or the Memory lowers the score from 0.637 to 0.604 and 0.613 (Table 6).

Continual learning in our framework is also not limited to MMKB. On the text benchmarks, the Skill and Knowledge Memory start empty and all gains come from deployment cases (e.g., 79.0 → 83.6 on AgentClinic).

AuthorsOpenReview ↗

Response to Reviewer rRSz (2/2)

W2 / Q3. Transfer on unseen cases and the role of the optimizer. We agree that Section 4.5 measures reuse on the same stream, so we repeated the transfer on unseen cases. We use the split of Section 4.4. The expertise learned with GPT-5.6-Sol on the learning split is frozen and given unchanged to each other model, which is evaluated only on the test split. No test case shares a patient, source group, or quantum experiment with the learning split. The table reports the relative change from each target's Base on the test split:

BenchmarkGPT-4o-miniNemotron-3-Nano-Omni-30BQwen3.8-27B
AgentClinic+16.0%+7.7%−2.3%
MedChain+9.0%+3.2%+10.1%
MedThink-Bench+9.5%+4.7%+3.1%
MedThinkVQA+6.5%+6.0%+0.9%
VisuLogic+3.7%+76.9%+42.9%
QCalEval+58.2%+26.6%+47.3%

On the optimizer: in this experiment and in Figure 4(a), the source is GPT-5.6-Sol, which both answers the cases and serves as the optimizer. The transferred expertise is therefore produced by the source model alone, and the benefit persists on unseen cases.

The learned expertise also generalizes to a different dataset. We froze the expertise learned with GPT-5.6-Sol on the complete stream and applied it, without further learning, to another dataset of the same task type, so none of these cases was available when the expertise was built:

Learned on → evaluated onBaseOurs (frozen)
VisuLogic → MM-IQ (955 questions after removing duplicates)59.764.2 (+7.5%)
AgentClinic-MedQA → MedR-Bench (957 cases)52.758.4 (+10.8%)

Q4. Cost and comparable compute. Ours costs about 5× Base, between ACE (2.4×) and SkillOpt (7.4×), and the gain does not come from more compute. Averaged over the benchmarks with GPT-5.6-Sol, including optimization and validation:

Per benchmarkBaseACESkillOptOurs
Tokens7.6Mabout 18Mabout 56Mabout 38M
Relative to Base1×2.4×7.4×about 5×

Ours makes 7,485 model calls per benchmark on average, compared with 3,261 for Base.

Relative to answering the stream once with the base model alone, our framework adds about 2.3× extra cost on text-only tasks and 4.8× on image tasks.

On comparable compute, SkillOpt uses more compute than Ours, yet its average gain on unseen cases is 8.3%, compared with 20.7% for Ours (Section 4.4). In addition, the fixed validation set in the Q1 table uses the same number of validation cases per batch as Ours and falls below Base on all five benchmarks.

Dear Reviewer rRSz,

Thank you again for your thoughtful comments and for taking the time to consider our responses.

As the discussion period approaches its conclusion, we wanted to kindly follow up and ask whether our additional analyses and clarifications have addressed your main concerns.

If there are any remaining questions or points you would like us to clarify further, please feel free to let us know. We would greatly appreciate any additional feedback while there is still time for discussion.

Thank you again for your time and consideration.

Best regards,

The Authors of Submission 21400

Reviewer_7WhbOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a parameter-free framework that improves frozen language and vision-language models using deployment feedback. It maintains a natural-language Skill, an evidence-checked Knowledge Memory, and a Multimodal Knowledge Base (MMKB) of visual references. Proposed Skill and Memory updates are accepted only after comparison on a fresh batch and a replay buffer, while MMKB accumulates completed cases with released source answers. Evaluation covers six benchmarks and four base models, including online adaptation, component ablations, frozen evaluation on unseen cases, and cross-model reuse. The framework improves all 24 online benchmark-model pairs over Base and 21 of 24 pairs in the frozen evaluation.

Critical strengths
  1. Broad evaluation with a meaningful held-out test. The experiments cover clinical workflows, medical reasoning, and non-medical visual tasks across open-weight and closed-source models. The frozen evaluation separates learning from testing, preserves source groups, and disables further updates and external searches. Its improvements in 21 of 24 settings provide substantive evidence that the complete framework can help beyond the cases used for adaptation.

  2. A sensible division of external expertise. Skill captures reusable procedures, Knowledge Memory stores declarative facts with supporting evidence and applicability conditions, and MMKB preserves original visual cases. This separation matches distinct reasoning needs and helps avoid mixing instructions with facts or losing visual details through textual compression. It provides a practical structure for reusing experience with frozen models.

Critical weaknesses
  1. The distinctive validation mechanism is not sufficiently isolated. Removing all validation produces a large collapse, but this establishes the importance of screening updates, rather than the advantage of the proposed fresh-batch-plus-buffer protocol over fixed validation or fresh-batch validation alone. Comparisons with ACE and SkillOpt also change other aspects of the system, and total optimization and evaluation costs are not reported. Consequently, the central claim that this validation design addresses fixed-set overfitting remains insufficiently supported by a controlled comparison.

  2. The update acceptance rule does not adequately account for statistical uncertainty. Candidate updates are evaluated on a 32-case fresh batch and a 32-case buffer, with one execution per case and version. Acceptance relies on empirical mean gains and win-loss counts, without a confidence requirement or minimum improvement margin. Matching random seeds reduces controllable variation but does not eliminate sampling uncertainty. Consequently, it remains unclear how reliably the rule distinguishes genuine improvements from fluctuations, especially when acceptance depends on only a few cases.

Questions for authors
  1. Clarification of validation evidence: Do any existing experiments compare the proposed validation rule with fixed validation or fresh-batch-only validation while keeping the optimizer, components, and evaluation budget unchanged? Such a comparison would materially affect my assessment of the validation contribution.

  2. Reliability of accepted updates: How often do accepted updates retain their gains on subsequent cases or repeated evaluations? Are paired uncertainty estimates available, particularly for updates whose acceptance depends on only one or a few case-level improvements?

  3. Sensitivity of the acceptance rule: How were the 32-case batch and buffer sizes chosen? Do available results show whether acceptance decisions and downstream performance are sensitive to these sizes or to requiring a minimum improvement margin?

8 comments
AuthorsOpenReview ↗

Response to Reviewer 7Whb (1/2)

Thank you for the careful review. We answer each point below.

W1 / Q1. Controlled comparison of validation rules. We ran this controlled comparison: Ours is the best rule on all five benchmarks, and it is the only rule that stays above Base. The submission did not include it. Only the acceptance rule changes; the optimizer, the three components, the batch size, the case order, and the seeds are the same as in Ours. All runs use GPT-5.6-Sol and the online protocol on the complete stream. The evaluation budget is also matched: the fixed validation set has 64 cases, the same number of validation cases per batch as Ours (32 new and 32 buffer cases).

BenchmarkBaseNo validation (Table 3)New batch onlyBuffer onlyFixed validation setOurs
AgentClinic79.053.563.659.860.383.6
MedThink-Bench68.451.250.755.656.971.8
MedThinkVQA68.352.158.452.260.668.9
VisuLogic45.838.137.240.442.350.0
QCalEval72.352.452.070.866.184.2

This supports the claim about overfitting to a fixed set. With the same number of validation cases, the fixed validation set is better than no validation on all five benchmarks, but it remains below Base on all five. The reason is that the optimizer receives the validation feedback when it proposes the next update (Figure 1). When the same cases are used for validation again and again, as with a fixed set or the buffer alone, the optimizer writes very detailed Skills for those cases, and these Skills do not generalize to new cases. The new batch alone avoids this reuse, but it does not check whether the update still works on earlier cases. Requiring both keeps an update only if it helps on cases the optimizer has not seen and does not hurt earlier ones.

We agree that the comparisons with ACE and SkillOpt are comparisons between complete systems; the table above is the controlled one.

W1. Total optimization and evaluation cost. Ours costs about 5× Base in total. Averaged over the benchmarks with GPT-5.6-Sol, including optimization and validation:

Per benchmarkBaseACESkillOptOurs
Tokens7.6Mabout 18Mabout 56Mabout 38M
Relative to Base1×2.4×7.4×about 5×

Ours makes 7,485 model calls per benchmark on average, compared with 3,261 for Base. Relative to answering the stream once with the base model alone, our framework adds about 2.3× extra cost on text-only tasks and 4.8× on image tasks. For comparison, the total cost of ACE is 2.4× Base and that of SkillOpt is 7.4× Base.

The cost is set by the mechanism. Per batch of 32 new cases, the current version's answers on the new batch are the deployed answers and are reused. Validation runs the candidate on the new batch and both versions on the 32-case buffer, i.e., 3 batch-sized passes, and the periodic check every four batches adds at most 0.75 pass per batch. The optimizer makes one call per batch. Runs of an unchanged version are reused, and no validation is run when there is no candidate. This cost is fixed per batch, does not grow with the length of the stream, and does not delay deployed answers, because validation runs after the batch has been answered.

AuthorsOpenReview ↗

Response to Reviewer 7Whb (2/2)

W2 / Q2. Reliability of accepted updates. Accepted updates keep their gains on subsequent cases: on the next batch, which neither version had been validated on, 79 of the 99 accepted updates score at least as high as the version they replaced, and only 3 of the 99 were later restored. We answer the three parts of the question in turn.

Subsequent cases. With GPT-5.6-Sol, the optimizer proposed 394 updates and 99 (25%) were accepted:

BenchmarkProposedAcceptedAccepted (%)Restored
AgentClinic6233%0
MedChain (five tasks)3138226%3
MedThink-Bench15427%0
MedThinkVQA22523%0
VisuLogic31516%0
QCalEval7114%0
Total3949925%3

The next batch is a clean test of an accepted update, because it was not used to accept it. 79 of the 99 pass this test, and the periodic check restored an earlier version only 3 times.

Repeated evaluations. At the level of the whole system, the gains are stable under five random case orders (Table 2: 79.3 ± 0.7 → 83.8 ± 0.9 on AgentClinic and 71.3 ± 1.8 → 83.1 ± 1.5 on QCalEval), and they hold on unseen cases in 21 of 24 settings (Table 4).

Paired uncertainty and updates accepted on a few cases. The description of the rule is correct: each comparison uses one execution per case, with no confidence requirement, and we agree that matching seeds does not remove sampling uncertainty. We do not have paired confidence intervals for single decisions, and one comparison on 32 + 32 cases is too small for a significance test. Such updates are common: 54 of the 99 accepted updates had a net win of at most two cases. The rule does not need each single decision to be statistically significant, for two reasons.

  1. An accepted update is not final. Every later candidate is compared against it, and every four batches the current version is compared with a retained earlier version and restored if it is worse.
  2. The rule is strict. An update must not lose on two different sets of cases, with at least as many wins as losses on both, and three of four proposals are rejected.

Q3. Batch and buffer sizes, and a minimum margin. We tested sizes 16, 32, and 64 on AgentClinic with GPT-5.6-Sol (Base 79.0), and all three improve over Base:

Batch and buffer size1632 (paper)64
AgentClinic84.183.680.3

We chose 32 as a trade-off between the number of cases per decision and the number of updates. Sizes 16 and 32 give similar scores (the difference is within the standard deviation in Table 2). With 16, the optimizer is called twice as often and each decision rests on only 16 + 16 cases; with 64, the 214-case stream has only three full batches, so few updates can be proposed.

We also tested a minimum improvement margin on AgentClinic: when an update must have a net win of at least three cases to be accepted, the final score is 79.9, only slightly above Base (79.0) and below the 83.6 of the rule in the paper. A margin rejects too many useful updates: 54 of the 99 accepted updates had a net win of at most two cases, and, as shown above, most accepted updates keep their gains on the next batch.

Dear Reviewer 7Whb,

Thank you again for your thoughtful comments and for taking the time to consider our responses.

As the discussion period approaches its conclusion, we wanted to kindly follow up and ask whether our additional analyses and clarifications have addressed your main concerns.

If there are any remaining questions or points you would like us to clarify further, please feel free to let us know. We would greatly appreciate any additional feedback while there is still time for discussion.

Thank you again for your time and consideration.

Best regards,

The Authors of Submission 21400

Reviewer_Dn9yOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes a model-agnostic framework that lets a frozen LLM improve from deployment experience by maintaining 3 external expertis including Skill, Knowledge Memory, and a Multimodal Knowledge Base (MMKB). The core mechanism is a validation gate that deploys a candidate Skill or Memory update only if it improves the next batch and does not degrade past cases. Experiments span six benchmarks and four base models, demonstrating online improvement against Base, ACE and SkillOpt. Extensive analysis covering component ablations, held-out evaluation, and cross-model transfer also help prove the effectiveness.

Critical strengths
  1. The progressive validation to update through deployment experience is a sensible design, and is proved effective. Testing each candidate on the next batch and on a buffer of earlier cases shows both improvement and regression, and avoids reusing a fixed validation set.
  2. The experiments are thorough, covering six benchmarks and four base models, predictions fixed before feedback is released, patient- and source-level splits, and robustness check over five random orders.
Critical weaknesses
  1. Generalization is only evaluated within the same benchmark stream without any cross-dataset experiment. Thus, the claims of generalization to unseen cases is not adequately proved.
  2. On visual tasks, the improvement mostly comes from the predefined knowledge base instead of the progressive learning. As shown in Table 3, removing Skill or Memory barely changes image-task scores on multimodal datasets. Besides, in Figure 3, Initial MMKB gains largely compared with Standard retrieval while additional updates bring little increase. The improvement on VQA should be credited to retrieval and prompting rather than to the framework's core mechanism.
  3. The transfer experiment in Section 4.5 is not a fair comparison. The source model learns on the entire stream with every answer released, and the target model is then scored on those same cases, so the transferred Skill and Memory includes leaked knowledge in the evaluation set. A clean version would let the target answer batch tt using only what the source learned through batch t−1t−1.
  4. Validation cost is substantial and not quantified. Each batch requires running both the current and candidate versions on the new cases and on the 32-case buffer, and replays against a retained version and an optimizer call with high reasoning effort.
  5. Visual baselines are not given the reference knowledge base (Initial MMKB). Base explicitly excludes MMKB, and whether ACE or SkillOpt received the knowledge base is not mentioned.
Questions for authors
  1. Were the learned Skills and Knowledge Memory entries on the medical benchmarks reviewed by clinicians?
  2. How many updates were proposed, how many were accepted, and how many rollbacks occurred during the optimization?
8 comments
AuthorsOpenReview ↗

Response to Reviewer Dn9y (1/2)

Thank you for the careful review. We answer each point below.

W1. Generalization beyond one dataset. We ran the cross-dataset experiment, and the learned expertise generalizes to a different dataset. We froze the expertise learned with GPT-5.6-Sol on the complete stream and applied it, without further learning, to another dataset of the same task type:

Learned on → evaluated onBaseOurs (frozen)
VisuLogic → MM-IQ (955 questions after removing duplicates)59.764.2 (+7.5%)
AgentClinic-MedQA → MedR-Bench (957 cases)52.758.4 (+10.8%)

In addition, the test split in Section 4.4 is already separated from learning within each benchmark: no test case shares a patient, source group, or quantum experiment with the learning split, test feedback is never released, and learning and new external searches are disabled. This matches deployment, where learned expertise is reused on new patients of the same task. In this setting, Ours improves over Base in 21 of 24 pairs (Table 4).

W2. Source of the visual gains. We ran a new experiment without the initial references, and MMKB built only from deployment cases still improves over no MMKB on all four tasks. In this experiment, MMKB contains only cases completed earlier in the stream (GPT-5.6-Sol, online protocol, Skill and Memory learned as usual). The table reports the improvement over Base:

BenchmarkNo MMKBMMKB without initial referencesInitial MMKBMMKB
MedChain Task 3+1.8%+9.9%+22.6%+23.9%
MedThinkVQA0.0%+0.7%+0.3%+0.9%
VisuLogic+0.2%+4.4%+8.5%+9.2%
QCalEval+0.1%+9.4%+15.2%+16.5%

For example, deployment cases alone raise the gain from +0.1% to +9.4% on QCalEval and from +1.8% to +9.9% on MedChain Task 3. The extra gain from deployment cases is smaller when initial references are present, because the initial libraries are already large (e.g., 4,292 references for VisuLogic vs. 1,000 stream cases; Table 13).

MMKB, including its initial references, is part of our method (Section 3.4). In practice, deployment rarely starts from nothing; for example, hospitals keep past images with their reports. MMKB is designed to use such references from the first case and to keep adding completed cases. The gain also does not come from retrieval itself: with identical retrieved references, Standard retrieval changes the score by −3.5% to +5.0% relative to Base, whereas MMKB improves all four image tasks (Figure 3). The gain comes from how MMKB guides the model to analyze each reference and compare it with the current case. Skill and Memory also matter when an image task needs a procedure: on MedChain Task 3, removing them lowers the score from 0.637 to 0.604 and 0.613 (Table 6).

W3. Transfer protocol. We agree that Section 4.5 measures reuse on the same stream, so we ran the transfer on unseen cases. We use the split of Section 4.4. The expertise learned with GPT-5.6-Sol on the learning split (Table 4) is frozen and given unchanged to each other model, which is evaluated only on the test split. The table reports the relative change from each target's Base on the test split (Table 4):

BenchmarkGPT-4o-miniNemotron-3-Nano-Omni-30BQwen3.8-27B
AgentClinic+16.0%+7.7%−2.3%
MedChain+9.0%+3.2%+10.1%
MedThink-Bench+9.5%+4.7%+3.1%
MedThinkVQA+6.5%+6.0%+0.9%
VisuLogic+3.7%+76.9%+42.9%
QCalEval+58.2%+26.6%+47.3%

We did not use the suggested version, which answers batch t with what the source learned through batch t−1. It avoids using the current batch, but it does not test whether a fixed, learned expertise transfers: early batches would be answered with nearly empty expertise and later batches with expertise that keeps changing, so the score would mix transfer with the source's own learning.

AuthorsOpenReview ↗

Response to Reviewer Dn9y (2/2)

W4. Validation cost. Relative to answering the stream once with the base model alone, our framework adds about 2.3× extra cost on text-only tasks and 4.8× on image tasks on average.

The cost is set by the mechanism. Per batch of 32 new cases, the current version's answers on the new batch are the deployed answers and are reused. Validation runs the candidate on the new batch and both versions on the 32-case buffer, i.e., 3 batch-sized passes, and the periodic check every four batches adds at most 0.75 pass per batch. The optimizer makes one call per batch; its input is the batch's public trajectories plus the Skill and Memory. Runs of an unchanged version are reused, and no validation is run when there is no candidate. Image tasks cost more because each run includes reference images and MMKB changes after every batch, so fewer runs can be reused.

W5. Baselines and MMKB. ACE and SkillOpt use the same MMKB references as Ours, through standard retrieval; they do not receive MMKB's instructions to analyze each reference and compare it with the current case. Base is the native model without references. The effect of giving the same references as plain context is shown by Standard retrieval in Figure 3.

Q1. Review of learned expertise. We used GPT-6-Astra to review all learned Skills and Knowledge Memory entries on the medical benchmarks against medical guidelines, and it judged them reasonable. Each Memory entry also keeps its supporting evidence (quoted case evidence or a PubMed excerpt) and its conditions of use, and the Skill is a readable document, so both can be checked by people. An LLM review does not replace clinicians; we plan to have clinicians review the learned expertise in future work.

Q2. Proposed, accepted, and restored updates. With GPT-5.6-Sol, the optimizer proposed 394 updates; 99 (25%) were accepted, and the periodic check restored an earlier version 3 times, all on MedChain. Most proposals are rejected, which is consistent with Table 3: accepting every update lowers the score by 21.8% to 37.8%.

BenchmarkProposedAcceptedAccepted (%)Restored
AgentClinic6233%0
MedChain Task 1731926%1
MedChain Task 2731115%0
MedChain Task 321419%0
MedChain Task 4731926%1
MedChain Task 5732940%1
MedThink-Bench15427%0
MedThinkVQA22523%0
VisuLogic31516%0
QCalEval7114%0
Total3949925%3

Dear Reviewer Dn9y,

Thank you again for your thoughtful comments and for taking the time to consider our responses.

As the discussion period approaches its conclusion, we wanted to kindly follow up and ask whether our additional analyses and clarifications have addressed your main concerns.

If there are any remaining questions or points you would like us to clarify further, please feel free to let us know. We would greatly appreciate any additional feedback while there is still time for discussion.

Thank you again for your time and consideration.

Best regards,

The Authors of Submission 21400

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.