- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies how to estimate the current performance of a deployed LLM-based medical AI system whose underlying model is updated often, when only a few outcomes are available after each update. The proposed method, ReMix, combines several online estimators using the squared-loss Aggregating Algorithm:
- (i) six methods from clinical trials for borrowing historical information (Chu & Yi, rMAP, SAM, EMAP, CP, NPP);
- (ii) for each of M external benchmarks, a "replay-informed" Gaussian shrinkage estimator whose borrowing weight decreases as d_k grows, where d_k is the mean absolute paired score change between the old and new versions on J = 20 fixed benchmark cases;
- (iii) a "direct" estimator equal to the new version's mean benchmark score.
The paper gives a standard regret bound relative to the best fixed estimator. The experiments replay stored per-instance scores:
- MedHELM: 13 models from five vendors and 50 "patients" per version. The targets are HealthCareMagic, iCliniq, and MedicationQA (LLM-jury scores), and the replay sources are ACI-Bench, HeadQA, MedBullets, and MedCalc-Bench.
- Supplementary testbed: 6 models and 30 "patients" per version. HealthBench, MedAgentBench, and MedSP each serve as a target, with the other two as replay sources.
Critical strengths
- Well-designed method. Any online estimator can be added, and the aggregation is lightweight and well understood.
- Evaluating the old and new versions on the same cases, before new outcomes arrive, to judge how much to trust history is a sensible idea. Proposition 2 states clearly when this works.
- All methods share the same patient orderings and anchors, and the paper includes ablations, model subsets, and a sensitivity analysis on version length.
Critical weaknesses
W1. Most of the gain comes from aggregating existing methods rather than from replay, and the main table leaves out the key baseline.
- Table 1 compares ReMix, a regret-bounded ensemble, with its individual components, and an aggregate is expected to beat single estimators. Since the paper's novelty is replay, the key baseline is the same aggregation without replay ("Prior-work only"). This baseline appears only in Appendix F.
- On MedHELM, Prior-work only already reaches a Total MSE of 0.797 and an MSE@5 of 0.663 (ECSS +2.51), versus 0.706 and 0.610 (ECSS +3.14) for ReMix, so replay adds only a modest improvement.
- On the supplementary testbed, Replay only (0.619) beats ReMix (0.662), so adding the prior-work methods actually hurts there.
- The results therefore do not consistently show that combining replay with historical borrowing is the right design. The main table should include both the prior-work-only and the replay-only mixtures.
W2. The replay signal rests on assumptions that do not hold in the experiments, and its own reliability is not tested.
- Violated assumptions. Proposition 2 requires the benchmark and the target to share the case space and reward function, with a bounded density ratio. In the experiments, however, ACI-Bench (note generation), HeadQA and MedBullets (exam MCQs), and MedCalc-Bench (calculations) are replayed to inform jury-graded patient Q&A, so the guarantee does not apply.
- Noisy dissimilarity. d_k is a mean absolute paired difference (Table 2) computed from a single rollout per model and case (Sec. 6). With stochastic generation and grading, |Δ_j| > 0 even when nothing has changed, so d_k mixes noise with real change.
- Calibration never checked. The paper never checks empirically whether d_k ≥ |δ_k| holds (calibration coverage), even though Proposition 1 depends on this condition.
- A single anchor draw. Only J = 20 anchors per source are drawn, once, and reused in all 500 replays. So the results depend on a single anchor draw and ignore its variance. The sources were "the first four eligible external sources alphabetically", and sensitivity to the choice of sources and anchors (and to J) is not studied.
W3. The simulated deployment is unrealistic in ways that may favor borrowing.
- Cross-vendor switching. The "updates" are switches between models from different vendors, ordered by release date (GPT-4o → Gemini-1.5-Pro → GPT-4o-mini → Claude-3.5-Sonnet → Llama-3.3-70B → …). This is not the motivating scenario of a vendor silently updating a single product line, and it creates large, arbitrary jumps.
- Idealized timing and population. Update times are known, versions have fixed lengths (25–50), patients are drawn i.i.d. from a fixed pool, and neither the population nor the grader ever drifts. Real streams show case-mix shifts, seasonality, and grader changes. Hidden updates are listed as future work.
- Unexplained detail. The protocol "reserve[s] the first 20 positions" of each permutation without explanation.
W4. Important alternative methods are missing from the baselines.
- Tracking regret / Fixed Share. This is the textbook approach to aggregating experts when the best expert changes over time (here, at each update). It is simple, yet it is neither cited nor compared against, and strongly adaptive methods are mentioned only as future work.
- State-space tracking (Kalman filter / dynamic linear model) of v(k), with a jump at each update whose variance is informed by replay. This is the natural Bayesian version of the paper's idea.
- Prediction-powered inference, PPI++, and control variates, which are cited but not compared against. If the deployment's own interactions can be graded by an automated judge, combining many proxy scores with a few verified outcomes is the standard approach.
- Regression calibration from benchmark scores to target performance, fitted on earlier versions, which is a stronger form of the "direct benchmark-score" estimator.
- Drift detectors with adaptive windows rather than a fixed 40-patient window.
W5. Robustness and validity of conclusions. Some claims are stronger than the evidence, and some negative results are understated.
- Early CVaR. The text states that "after five patients, NPP has lower CVaR on MedHELM (0.309 versus 0.439)". In fact, four methods beat ReMix on early CVaR on MedHELM: NPP (0.309), Pooling (0.322), rMAP (0.376), and SAM (0.395) (Table 1), so ReMix ranks only fifth.
- "On both testbeds." The abstract states that ReMix "achieves lower average cumulative error than methods for borrowing historical information in clinical trials on both testbeds". This holds for the full sequences, but not for the open-model subset (NPP 0.530 vs. ReMix 0.539; Table 4), nor against the replay-only mixture on the supplementary testbed (0.619 vs. 0.662).
- Unexplained ingredients. The most useful component differs across testbeds (the prior-work methods help on MedHELM but hurt on the supplementary testbed), and the paper does not explain when each ingredient helps. So practitioners have no guidance on when to include each family of estimators.
W6. Some parts are hard to follow:
- Algorithm 1 mixes predicting the next reward with estimating the current mean: the estimator losses use ṽ_{t−1}, whereas the reported estimate v̂_t already includes r_t. It takes effort to see what is being optimized and what is being reported.
- "Replay" here means re-running fixed benchmark cases, not experience replay. This should be stated early to avoid confusion.
- Figure 1 is dense, and using the term "patients" for benchmark cases blurs the clinical meaning of the results.
- The abstract should state that all experiments are offline simulations based on stored benchmark scores with automated grading.
Questions for authors
Please refer to Weaknesses above.
4 comments
Response to Reviewer 8oMt [2/3]
Response to W3
"Cross-vendor switching."
Our experiments already include related-model updates. In the Supplementary Testbed (Appendix B.2), Llama-3-8B → Med42-v2-8B and Gemma-3-4B → MedGemma-4B are within-family fine-tunes, and GPT-5.4-mini → GPT-5.6-luna is a same-vendor update. On this sequence, ReMix achieves lower MSE@5, Total MSE, and Total CVaR than every standalone baseline (Table 1).
Our repeated stochastic rollouts regenerate this same-vendor update in four independent runs, where ReMix (tuned) lowers MSE@5 by 67% on MedSP and 73% on HealthBench relative to current-only. Please see Table R3 in our global response (G2) for details.
Cross-vendor updates are the hard case for borrowing, and ReMix performs well on both related-model and cross-vendor sequences. On the open-model subset, where two of the three updates are within-family, even pooling all past outcomes beats current-only (Total MSE 0.644, Table 4). On MedHELM, where eleven of the twelve updates are cross-vendor, pooling and NPP reach 2.656 and 1.619, while full ReMix stays at 0.706 (Table 1).
"Idealized timing and population."
Appendix G already varies the version length from 25 to 50, and ReMix has the lowest Total MSE at every length (Table 5). We have extended the fixed version lengths to random update times, with version lengths drawn uniformly from 21 to 50 cases, and the fixed case mix to topic proportions that change over the stream.
Table R5. ReMix under random update times and changing case mix. Each change is tested on five streams. Each cell is the range of ReMix's reduction relative to the named baseline.
| Deployment change | MSE@5 vs. prior-work-only | MSE@5 vs. current-only | Total MSE vs. current-only |
|---|---|---|---|
| Random update times | 7.0–22.9% | 40.1–46.9% | 19.8–25.9% |
| Changing case mix | 1.4–23.9% | 36.1–56.8% | 14.0–26.4% |
Results. ReMix's early gains persist in all ten streams. Details are in Appendix H.8, Table 15.
Grading. Grading follows each benchmark's own procedure, held fixed across versions (Section 3).
"The protocol “reserve[s] the first 20 positions” of each permutation without explanation."
Thank you for pointing this out. The first 20 positions are held-out target cases that no estimator uses. We have added this explanation to Appendix B.3.
Response to W4
"Important alternative methods are missing from the baselines."
Thank you for the suggestions. We have now added further baselines or extensions of ReMix (e.g., Fixed Share and CBCE) for comparison.
Setup. All methods use the original case orders of the two full sequences. The added baselines are:
- Fixed Share [1]: ReMix with a share of uniform weight mixed in after each weight update, for version length .
- CBCE [2]: copies of the ReMix aggregation algorithm started at different times. Fixed Share and CBCE use the same candidate estimators as full ReMix.
- Kalman filter: ReMix's replay-informed borrowing estimators use a zero-mean jump with variance equal to the squared replay dissimilarity (Section 4.1). The standalone filter uses the mean dissimilarity across sources.
- Linear calibration: least squares from benchmark scores to target means on retired versions.
- ADWIN [3]: the mean of an adaptively retained window, with .
Table R6. Baseline comparisons on the two full sequences. Each cell is Total MSE / MSE@5 relative to current-only.
| Method | MedHELM | Supplementary |
|---|---|---|
| Full ReMix | 0.706 / 0.610 | 0.662 / 0.652 |
| ReMix (tuned) | 0.682 / 0.545 | 0.603 / 0.561 |
| ReMix with Fixed Share | 0.691 / 0.602 | 0.641 / 0.626 |
| CBCE over ReMix | 0.876 / 0.637 | 0.632 / 0.591 |
| Standalone Kalman filter | 0.744 / 0.716 | 0.675 / 0.692 |
| Linear calibration | 3.705 / 1.876 | 11.281 / 8.621 |
| ADWIN | 2.656 / 1.554 | 2.339 / 2.112 |
| Sliding window (40) | 1.502 / 1.818 | 0.926 / 1.024 |
Results. ReMix (tuned) achieves the lowest Total MSE and MSE@5 in Table R6 on both full sequences. Full ReMix also outperforms the standalone Kalman filter, linear calibration, and ADWIN. Please see our global response (G1) for details on ReMix (tuned).
Linear calibration is unstable with few retired versions.
PPI and PPI++ need a different observation protocol: proxy and reference grades on the same target cases. In our setting each target case reveals one outcome (Section 3); where proxy grades exist, a PPI estimator can be added to ReMix as another candidate.
Appendix H.7 gives the baseline comparisons and setup, and Related Work now cites Fixed Share and strongly adaptive methods.
References
[1] Herbster, M., and Warmuth, M. K. Tracking the Best Expert. Machine Learning, 1998.
[2] Jun, K.-S., Orabona, F., Wright, S., and Willett, R. Improved Strongly Adaptive Online Learning using Coin Betting. AISTATS, 2017.
[3] Bifet, A., and Gavaldà, R. Learning from Time-Changing Data with Adaptive Windowing. SDM, 2007.
Response to Reviewer 8oMt [3/3]
Response to W5
"Robustness and validity of conclusions. Some claims are stronger than the evidence, and some negative results are understated."
Thank you for pointing these out. We address these shared concerns in our global response (G1). Please see the discussion following Table R1 for the open-model subset and replay-only, and Table R2 for early CVaR.
When to include each family of estimators. Which estimator or replay source is most reliable after an update is not known in advance. This uncertainty is precisely what motivates the mixture in ReMix, which adapts the weights as target feedback arrives.
To provide practical guidance:
- Start from the full collection. It is a tuning-free default. Theorem 1 bounds regret by , and full ReMix improves on prior-work-only on all four model sequences (Table 3).
- Tune on a past sequence when one is available. ReMix (tuned) was chosen among six options on the full MedHELM sequence only. Applied unchanged, it lowers every metric relative to full ReMix on the Supplementary Testbed and under repeated stochastic rollouts. Details are in Appendix H.1, Table 8.
- Use a replay source for its paired changes first. On MedHELM, adding a source's paired-change (replay-informed borrowing) estimator lowers Total MSE for all four sources (0.6–8.6%), while adding its direct benchmark-score estimator alone changes it by −2.6% to +2.8%. Details are in Appendix H.4, Figure 6.
Response to W6
Thank you for these suggestions. We have adopted them in the revised manuscript.
Algorithm 1. The loss is used only to update the weights (Theorem 1). The reported estimate uses the updated weights and includes (Appendix E.3). We have labeled these two steps in Algorithm 1.
Other edits. We have stated in the Introduction that paired replay differs from experience replay, in Section 5.2 that “patients” are target benchmark cases, and in the abstract that the experiments are “simulations with automated grading”. We will simplify Figure 1 during revision.
Thank you again for your careful review. We would be grateful if you could consider the new experiments and theoretical results in your assessment.
Follow-up before the discussion closes
Thank you again for your careful review. As the discussion period ends soon, we wanted to gently follow up. We would like to highlight two new results addressing your main concerns. Replay gives gains on its own: replay-only, which uses none of the existing methods, has 10.7–34.9% lower Total MSE than prior-work-only in all eight repeated stochastic runs. After minimal tuning, ReMix achieves the lowest Total MSE and MSE@5 in Table R6 on both full sequences, including against the baselines you suggested. We look forward to any further questions or comments you may have.
Response to Reviewer 8oMt [1/3]
Thank you for your careful reading and helpful comments. We appreciate your recognition of the value of paired replay, the flexible aggregation algorithm, and the shared evaluation setup.
We respond to W1–W6 below. The appendix provides the full analysis and experimental details, with additions marked in red.
Thank you for your helpful comment. We agree that prior-work-only and replay-only are the key ablations and have promoted both to the main table (Table 1), listed under ReMix. Both use the same aggregation algorithm as ReMix and differ only in the candidate estimators.
Effectiveness of replay. Replay-only uses replay to inform historical borrowing and achieves lower Total MSE than prior-work-only on all four model sequences in Table 3 (Appendix F). This is direct evidence that replay gives gains even without the existing methods. The same holds under repeated stochastic rollouts, where replay-only has 10.7–34.9% lower Total MSE than prior-work-only in all eight runs. Please see our global response (G2) for details.
Robustness to estimator choice. We did not tune the original estimator collection, since in the online setting, hyperparameters cannot be tuned in hindsight. Every tested collection that uses replay improves on prior-work-only in Total MSE (Table 3), so the gains do not hinge on the exact choice. After minimal tuning, ReMix (tuned) combines the replay estimators with current-only and NPP and has lower Total MSE than the replay-only configuration in Table 3 on both full sequences. Please see our global response (G1) for details.
The new Theorem 2 (Appendix I.2) extends our borrowing guarantee to heterogeneous benchmarks. It shows when the variance reduction from borrowing outweighs the squared bias, even if replay understates the target drift.
We address this shared concern in our global response (G2). Please see Table R3 for repeated stochastic rollouts on MedSP and HealthBench. Noise also makes borrowing more conservative: a larger dk puts less weight on history.
We have added a direct check of this condition. The replay dissimilarity dk(m) upper-bounds observed target drift ∣δk∣ in 70.8% of MedHELM and 83.3% of Supplementary comparisons. The single-source replay-informed estimator beats current-only in all 127 comparisons where the inequality holds and in 21 of 47 where it does not. Details are in Appendix H.3, Table 12 and Figure 5.
Sensitivity to anchor cases. We have extended the single anchor draw to 20 independent draws per target, and J=20 to J∈{5,10,20,40,80}. At J=20, Full ReMix has lower Total MSE than current-only for every draw on all six targets (highest ratio 0.883), and average Total MSE decreases as the number of anchor cases increases. Details are in Appendix H.5, Figure 7. The repeated stochastic rollouts in our global response (G2) also draw a new set of anchor cases for paired replay in each run.
Replay sources. The four MedHELM replay sources were chosen alphabetically, which is a reproducibility choice and not one selected for performance. We have also added each external source separately to prior-work-only.
Table R4. Adding replay sources to prior-work-only on MedHELM. Each cell is the reduction in Total MSE relative to prior-work-only.
Results. Using all four sources gives the largest overall reduction. No single replay source is best on every target, so hand-picking one requires knowledge of the target that we do not have before deployment. This is precisely why we use an adaptive method. Details are in Appendix H.4, Figure 6.