acceptodds
Under review as a conference paper at ICLR 2027

ReMix: Online Evaluation of Evolving Medical AI Systems

Abstract

Medical AI systems built on large language models can change faster than evidence about their performance accumulates. How can we evaluate a new version when only a few patient outcomes are available? We introduce ReMix, an adaptive mixture of historical-borrowing and direct benchmark-score estimators. At each update, paired benchmark evaluations calibrate borrowing in replay-informed estimators, while the new version's benchmark scores provide direct estimates. ReMix adjusts their weights as patient outcomes arrive. We bound its cumulative squared prediction loss relative to the best constituent estimator for any reward sequence. In offline experiments using real patient questions from MedHELM, ReMix reduces mean squared estimation error by 39% after just five post-update patients and cumulative squared estimation error by 29%, compared with evaluating each version from scratch. It also achieves lower average cumulative error than methods for borrowing historical information in clinical trials on both testbeds.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM Evaluation, Historical Data Borrowing, Sequential Estimation

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 2 reviews, above 68% of the others.
Reviewer_8oMtOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies how to estimate the current performance of a deployed LLM-based medical AI system whose underlying model is updated often, when only a few outcomes are available after each update. The proposed method, ReMix, combines several online estimators using the squared-loss Aggregating Algorithm:

  • (i) six methods from clinical trials for borrowing historical information (Chu & Yi, rMAP, SAM, EMAP, CP, NPP);
  • (ii) for each of M external benchmarks, a "replay-informed" Gaussian shrinkage estimator whose borrowing weight decreases as d_k grows, where d_k is the mean absolute paired score change between the old and new versions on J = 20 fixed benchmark cases;
  • (iii) a "direct" estimator equal to the new version's mean benchmark score.

The paper gives a standard regret bound relative to the best fixed estimator. The experiments replay stored per-instance scores:

  • MedHELM: 13 models from five vendors and 50 "patients" per version. The targets are HealthCareMagic, iCliniq, and MedicationQA (LLM-jury scores), and the replay sources are ACI-Bench, HeadQA, MedBullets, and MedCalc-Bench.
  • Supplementary testbed: 6 models and 30 "patients" per version. HealthBench, MedAgentBench, and MedSP each serve as a target, with the other two as replay sources.
Critical strengths
  1. Well-designed method. Any online estimator can be added, and the aggregation is lightweight and well understood.
  2. Evaluating the old and new versions on the same cases, before new outcomes arrive, to judge how much to trust history is a sensible idea. Proposition 2 states clearly when this works.
  3. All methods share the same patient orderings and anchors, and the paper includes ablations, model subsets, and a sensitivity analysis on version length.
Critical weaknesses

W1. Most of the gain comes from aggregating existing methods rather than from replay, and the main table leaves out the key baseline.

  • Table 1 compares ReMix, a regret-bounded ensemble, with its individual components, and an aggregate is expected to beat single estimators. Since the paper's novelty is replay, the key baseline is the same aggregation without replay ("Prior-work only"). This baseline appears only in Appendix F.
  • On MedHELM, Prior-work only already reaches a Total MSE of 0.797 and an MSE@5 of 0.663 (ECSS +2.51), versus 0.706 and 0.610 (ECSS +3.14) for ReMix, so replay adds only a modest improvement.
  • On the supplementary testbed, Replay only (0.619) beats ReMix (0.662), so adding the prior-work methods actually hurts there.
  • The results therefore do not consistently show that combining replay with historical borrowing is the right design. The main table should include both the prior-work-only and the replay-only mixtures.

W2. The replay signal rests on assumptions that do not hold in the experiments, and its own reliability is not tested.

  • Violated assumptions. Proposition 2 requires the benchmark and the target to share the case space and reward function, with a bounded density ratio. In the experiments, however, ACI-Bench (note generation), HeadQA and MedBullets (exam MCQs), and MedCalc-Bench (calculations) are replayed to inform jury-graded patient Q&A, so the guarantee does not apply.
  • Noisy dissimilarity. d_k is a mean absolute paired difference (Table 2) computed from a single rollout per model and case (Sec. 6). With stochastic generation and grading, |Δ_j| > 0 even when nothing has changed, so d_k mixes noise with real change.
  • Calibration never checked. The paper never checks empirically whether d_k ≥ |δ_k| holds (calibration coverage), even though Proposition 1 depends on this condition.
  • A single anchor draw. Only J = 20 anchors per source are drawn, once, and reused in all 500 replays. So the results depend on a single anchor draw and ignore its variance. The sources were "the first four eligible external sources alphabetically", and sensitivity to the choice of sources and anchors (and to J) is not studied.

W3. The simulated deployment is unrealistic in ways that may favor borrowing.

  • Cross-vendor switching. The "updates" are switches between models from different vendors, ordered by release date (GPT-4o → Gemini-1.5-Pro → GPT-4o-mini → Claude-3.5-Sonnet → Llama-3.3-70B → …). This is not the motivating scenario of a vendor silently updating a single product line, and it creates large, arbitrary jumps.
  • Idealized timing and population. Update times are known, versions have fixed lengths (25–50), patients are drawn i.i.d. from a fixed pool, and neither the population nor the grader ever drifts. Real streams show case-mix shifts, seasonality, and grader changes. Hidden updates are listed as future work.
  • Unexplained detail. The protocol "reserve[s] the first 20 positions" of each permutation without explanation.

W4. Important alternative methods are missing from the baselines.

  • Tracking regret / Fixed Share. This is the textbook approach to aggregating experts when the best expert changes over time (here, at each update). It is simple, yet it is neither cited nor compared against, and strongly adaptive methods are mentioned only as future work.
  • State-space tracking (Kalman filter / dynamic linear model) of v(k), with a jump at each update whose variance is informed by replay. This is the natural Bayesian version of the paper's idea.
  • Prediction-powered inference, PPI++, and control variates, which are cited but not compared against. If the deployment's own interactions can be graded by an automated judge, combining many proxy scores with a few verified outcomes is the standard approach.
  • Regression calibration from benchmark scores to target performance, fitted on earlier versions, which is a stronger form of the "direct benchmark-score" estimator.
  • Drift detectors with adaptive windows rather than a fixed 40-patient window.

W5. Robustness and validity of conclusions. Some claims are stronger than the evidence, and some negative results are understated.

  • Early CVaR. The text states that "after five patients, NPP has lower CVaR on MedHELM (0.309 versus 0.439)". In fact, four methods beat ReMix on early CVaR on MedHELM: NPP (0.309), Pooling (0.322), rMAP (0.376), and SAM (0.395) (Table 1), so ReMix ranks only fifth.
  • "On both testbeds." The abstract states that ReMix "achieves lower average cumulative error than methods for borrowing historical information in clinical trials on both testbeds". This holds for the full sequences, but not for the open-model subset (NPP 0.530 vs. ReMix 0.539; Table 4), nor against the replay-only mixture on the supplementary testbed (0.619 vs. 0.662).
  • Unexplained ingredients. The most useful component differs across testbeds (the prior-work methods help on MedHELM but hurt on the supplementary testbed), and the paper does not explain when each ingredient helps. So practitioners have no guidance on when to include each family of estimators.

W6. Some parts are hard to follow:

  • Algorithm 1 mixes predicting the next reward with estimating the current mean: the estimator losses use ṽ_{t−1}, whereas the reported estimate v̂_t already includes r_t. It takes effort to see what is being optimized and what is being reported.
  • "Replay" here means re-running fixed benchmark cases, not experience replay. This should be stated early to avoid confusion.
  • Figure 1 is dense, and using the term "patients" for benchmark cases blurs the clinical meaning of the results.
  • The abstract should state that all experiments are offline simulations based on stored benchmark scores with automated grading.
Questions for authors

Please refer to Weaknesses above.

4 comments
AuthorsOpenReview ↗

Response to Reviewer 8oMt [1/3]

Thank you for your careful reading and helpful comments. We appreciate your recognition of the value of paired replay, the flexible aggregation algorithm, and the shared evaluation setup.

We respond to W1–W6 below. The appendix provides the full analysis and experimental details, with additions marked in red.

Response to W1


"Most of the gain comes from aggregating existing methods rather than from replay, and the main table leaves out the key baseline."

Thank you for your helpful comment. We agree that prior-work-only and replay-only are the key ablations and have promoted both to the main table (Table 1), listed under ReMix. Both use the same aggregation algorithm as ReMix and differ only in the candidate estimators.

Effectiveness of replay. Replay-only uses replay to inform historical borrowing and achieves lower Total MSE than prior-work-only on all four model sequences in Table 3 (Appendix F). This is direct evidence that replay gives gains even without the existing methods. The same holds under repeated stochastic rollouts, where replay-only has 10.7–34.9% lower Total MSE than prior-work-only in all eight runs. Please see our global response (G2) for details.

Robustness to estimator choice. We did not tune the original estimator collection, since in the online setting, hyperparameters cannot be tuned in hindsight. Every tested collection that uses replay improves on prior-work-only in Total MSE (Table 3), so the gains do not hinge on the exact choice. After minimal tuning, ReMix (tuned) combines the replay estimators with current-only and NPP and has lower Total MSE than the replay-only configuration in Table 3 on both full sequences. Please see our global response (G1) for details.

Response to W2


"Violated assumptions."

The new Theorem 2 (Appendix I.2) extends our borrowing guarantee to heterogeneous benchmarks. It shows when the variance reduction from borrowing outweighs the squared bias, even if replay understates the target drift.

"dkd_k mixes noise with real change."

We address this shared concern in our global response (G2). Please see Table R3 for repeated stochastic rollouts on MedSP and HealthBench. Noise also makes borrowing more conservative: a larger dkd_k puts less weight on history.

"Calibration never checked."

We have added a direct check of this condition. The replay dissimilarity dk(m)d_k^{(m)} upper-bounds observed target drift ∣δk∣|\delta_k| in 70.8% of MedHELM and 83.3% of Supplementary comparisons. The single-source replay-informed estimator beats current-only in all 127 comparisons where the inequality holds and in 21 of 47 where it does not. Details are in Appendix H.3, Table 12 and Figure 5.

"sensitivity to the choice of sources and anchors (and to JJ) is not studied."

Sensitivity to anchor cases. We have extended the single anchor draw to 20 independent draws per target, and J=20J=20 to J∈{5,10,20,40,80}J\in\lbrace 5,10,20,40,80\rbrace. At J=20J=20, Full ReMix has lower Total MSE than current-only for every draw on all six targets (highest ratio 0.883), and average Total MSE decreases as the number of anchor cases increases. Details are in Appendix H.5, Figure 7. The repeated stochastic rollouts in our global response (G2) also draw a new set of anchor cases for paired replay in each run.

Replay sources. The four MedHELM replay sources were chosen alphabetically, which is a reproducibility choice and not one selected for performance. We have also added each external source separately to prior-work-only.

Table R4. Adding replay sources to prior-work-only on MedHELM. Each cell is the reduction in Total MSE relative to prior-work-only.

Replay source addedHealthCareMagiciCliniqMedicationQAMedHELM overall
ACI-Bench8.1%2.8%9.4%6.5%
HeadQA−5.6%−2.6%1.9%−2.2%
MedBullets10.5%9.8%7.2%9.3%
MedCalc-Bench−3.6%4.8%9.0%3.3%
All four (full ReMix)7.9%10.3%16.4%11.4%

Results. Using all four sources gives the largest overall reduction. No single replay source is best on every target, so hand-picking one requires knowledge of the target that we do not have before deployment. This is precisely why we use an adaptive method. Details are in Appendix H.4, Figure 6.

AuthorsOpenReview ↗

Response to Reviewer 8oMt [2/3]

Response to W3


"Cross-vendor switching."

Our experiments already include related-model updates. In the Supplementary Testbed (Appendix B.2), Llama-3-8B → Med42-v2-8B and Gemma-3-4B → MedGemma-4B are within-family fine-tunes, and GPT-5.4-mini → GPT-5.6-luna is a same-vendor update. On this sequence, ReMix achieves lower MSE@5, Total MSE, and Total CVaR than every standalone baseline (Table 1).

Our repeated stochastic rollouts regenerate this same-vendor update in four independent runs, where ReMix (tuned) lowers MSE@5 by 67% on MedSP and 73% on HealthBench relative to current-only. Please see Table R3 in our global response (G2) for details.

Cross-vendor updates are the hard case for borrowing, and ReMix performs well on both related-model and cross-vendor sequences. On the open-model subset, where two of the three updates are within-family, even pooling all past outcomes beats current-only (Total MSE 0.644, Table 4). On MedHELM, where eleven of the twelve updates are cross-vendor, pooling and NPP reach 2.656 and 1.619, while full ReMix stays at 0.706 (Table 1).

"Idealized timing and population."

Appendix G already varies the version length from 25 to 50, and ReMix has the lowest Total MSE at every length (Table 5). We have extended the fixed version lengths to random update times, with version lengths drawn uniformly from 21 to 50 cases, and the fixed case mix to topic proportions that change over the stream.

Table R5. ReMix under random update times and changing case mix. Each change is tested on five streams. Each cell is the range of ReMix's reduction relative to the named baseline.

Deployment changeMSE@5 vs. prior-work-onlyMSE@5 vs. current-onlyTotal MSE vs. current-only
Random update times7.0–22.9%40.1–46.9%19.8–25.9%
Changing case mix1.4–23.9%36.1–56.8%14.0–26.4%

Results. ReMix's early gains persist in all ten streams. Details are in Appendix H.8, Table 15.

Grading. Grading follows each benchmark's own procedure, held fixed across versions (Section 3).

"The protocol “reserve[s] the first 20 positions” of each permutation without explanation."

Thank you for pointing this out. The first 20 positions are held-out target cases that no estimator uses. We have added this explanation to Appendix B.3.

Response to W4


"Important alternative methods are missing from the baselines."

Thank you for the suggestions. We have now added further baselines or extensions of ReMix (e.g., Fixed Share and CBCE) for comparison.

Setup. All methods use the original case orders of the two full sequences. The added baselines are:

  • Fixed Share [1]: ReMix with a share α=1/(4L)\alpha=1/(4L) of uniform weight mixed in after each weight update, for version length LL.
  • CBCE [2]: copies of the ReMix aggregation algorithm started at different times. Fixed Share and CBCE use the same candidate estimators as full ReMix.
  • Kalman filter: ReMix's replay-informed borrowing estimators use a zero-mean jump with variance equal to the squared replay dissimilarity (Section 4.1). The standalone filter uses the mean dissimilarity across sources.
  • Linear calibration: least squares from benchmark scores to target means on retired versions.
  • ADWIN [3]: the mean of an adaptively retained window, with δ=0.002\delta=0.002.

Table R6. Baseline comparisons on the two full sequences. Each cell is Total MSE / MSE@5 relative to current-only.

MethodMedHELMSupplementary
Full ReMix0.706 / 0.6100.662 / 0.652
ReMix (tuned)0.682 / 0.5450.603 / 0.561
ReMix with Fixed Share0.691 / 0.6020.641 / 0.626
CBCE over ReMix0.876 / 0.6370.632 / 0.591
Standalone Kalman filter0.744 / 0.7160.675 / 0.692
Linear calibration3.705 / 1.87611.281 / 8.621
ADWIN2.656 / 1.5542.339 / 2.112
Sliding window (40)1.502 / 1.8180.926 / 1.024

Results. ReMix (tuned) achieves the lowest Total MSE and MSE@5 in Table R6 on both full sequences. Full ReMix also outperforms the standalone Kalman filter, linear calibration, and ADWIN. Please see our global response (G1) for details on ReMix (tuned).

Linear calibration is unstable with few retired versions.

PPI and PPI++ need a different observation protocol: proxy and reference grades on the same target cases. In our setting each target case reveals one outcome (Section 3); where proxy grades exist, a PPI estimator can be added to ReMix as another candidate.

Appendix H.7 gives the baseline comparisons and setup, and Related Work now cites Fixed Share and strongly adaptive methods.


References

[1] Herbster, M., and Warmuth, M. K. Tracking the Best Expert. Machine Learning, 1998.

[2] Jun, K.-S., Orabona, F., Wright, S., and Willett, R. Improved Strongly Adaptive Online Learning using Coin Betting. AISTATS, 2017.

[3] Bifet, A., and Gavaldà, R. Learning from Time-Changing Data with Adaptive Windowing. SDM, 2007.

AuthorsOpenReview ↗

Response to Reviewer 8oMt [3/3]

Response to W5


"Robustness and validity of conclusions. Some claims are stronger than the evidence, and some negative results are understated."

Thank you for pointing these out. We address these shared concerns in our global response (G1). Please see the discussion following Table R1 for the open-model subset and replay-only, and Table R2 for early CVaR.

When to include each family of estimators. Which estimator or replay source is most reliable after an update is not known in advance. This uncertainty is precisely what motivates the mixture in ReMix, which adapts the weights as target feedback arrives.

To provide practical guidance:

  • Start from the full collection. It is a tuning-free default. Theorem 1 bounds regret by (log⁡N)/η(\log N)/\eta, and full ReMix improves on prior-work-only on all four model sequences (Table 3).
  • Tune on a past sequence when one is available. ReMix (tuned) was chosen among six options on the full MedHELM sequence only. Applied unchanged, it lowers every metric relative to full ReMix on the Supplementary Testbed and under repeated stochastic rollouts. Details are in Appendix H.1, Table 8.
  • Use a replay source for its paired changes first. On MedHELM, adding a source's paired-change (replay-informed borrowing) estimator lowers Total MSE for all four sources (0.6–8.6%), while adding its direct benchmark-score estimator alone changes it by −2.6% to +2.8%. Details are in Appendix H.4, Figure 6.

Response to W6


Thank you for these suggestions. We have adopted them in the revised manuscript.

Algorithm 1. The loss (rt−v~t−1(i))2(r_t-\tilde v_{t-1}(i))^2 is used only to update the weights (Theorem 1). The reported estimate v^t\hat v_t uses the updated weights and includes rtr_t (Appendix E.3). We have labeled these two steps in Algorithm 1.

Other edits. We have stated in the Introduction that paired replay differs from experience replay, in Section 5.2 that “patients” are target benchmark cases, and in the abstract that the experiments are “simulations with automated grading”. We will simplify Figure 1 during revision.


Thank you again for your careful review. We would be grateful if you could consider the new experiments and theoretical results in your assessment.

AuthorsOpenReview ↗

Follow-up before the discussion closes

Thank you again for your careful review. As the discussion period ends soon, we wanted to gently follow up. We would like to highlight two new results addressing your main concerns. Replay gives gains on its own: replay-only, which uses none of the existing methods, has 10.7–34.9% lower Total MSE than prior-work-only in all eight repeated stochastic runs. After minimal tuning, ReMix achieves the lowest Total MSE and MSE@5 in Table R6 on both full sequences, including against the baselines you suggested. We look forward to any further questions or comments you may have.

Reviewer_4FphOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes ReMix, an online framework for estimating the performance of evolving medical AI systems when only a few outcomes are available after each model update. It combines historical-borrowing, replay-informed, and direct benchmark estimators, and uses Hedge to adapt their weights as new patient outcomes arrive. The paper provides theoretical guarantees and reports lower estimation error than evaluating each version from scratch on two medical evaluation suites.

Critical strengths
  • The paper studies an important and underexplored problem: estimating the current performance of a medical AI system soon after its underlying model changes.
  • ReMix provides a practical way to use paired benchmark results to control how much evidence is borrowed from previous versions, while adapting the mixture as new outcomes arrive.
  • The evaluation includes multiple medical tasks, model sequences, historical-borrowing baselines, component ablations, sensitivity analyses, and a theoretical regret guarantee.
Critical weaknesses
  • The methodological novelty is limited. ReMix mainly combines existing historical-borrowing and replay-based estimators using the classical Hedge algorithm.
  • Each model-case pair is represented by only one stored rollout. The experiments do not examine whether the results remain stable under repeated stochastic generations or grading variability.
  • ReMix is not consistently the best-performing method across evaluation settings. Replay-only performs better on the full supplementary sequence, NPP performs better on the open-model subset, and simpler baselines achieve lower early tail risk.
Questions for authors

See Weaknesses.

2 comments
AuthorsOpenReview ↗

Response to Reviewer 4Fph

Thank you for recognizing that we study an “important and underexplored problem,” and for highlighting the practicality of paired replay, the breadth of our evaluation, and the theoretical regret guarantee. We address each concern below.

Response to W1


"The methodological novelty is limited. ReMix mainly combines existing historical-borrowing and replay-based estimators using the classical Hedge algorithm."

Hedge is classical, but the replay-informed borrowing estimators are introduced in this paper. ReMix makes three contributions beyond combining existing estimators.

1. A clean mathematical formulation of a novel and important problem. Both you and Reviewer nyh4 recognize the importance of estimating a medical AI system's performance soon after its underlying model changes. To our knowledge, we are the first to formalize this problem, as a setting in which historical target observations are available, feedback on the updated medical LLM agent is scarce, and external benchmarks can be replayed. Estimation must balance noisy current data, potentially biased history, and uncertain benchmark-to-target transfer. The borrowing methods we compare with use target observations only and do not use the paired benchmark changes available at each update.

2. A finding on benchmark transfer. A benchmark can be a poor proxy for performance on a particular target distribution yet reveal important aspects of how an LLM's capabilities and behavior change across versions. For example, adding ACI-Bench's pairwise-change estimator to prior-work-only lowers MedHELM Total MSE by 8.6%, whereas adding its direct-score estimator increases it by 1.7%. Details are in Appendix H.4, Figure 6.

3. A simple adaptive solution. ReMix combines source-specific replay estimators with existing borrowing methods and adapts their weights using target feedback, requiring no knowledge of the best source in advance. Reducing this aggregation to the classical experts problem gives a simple algorithm with regret guarantees.

The new estimators account for most of ReMix's gain over combining existing methods. Adding them to prior-work-only lowers Total MSE by 10.8% on MedHELM and 6.3% on Supplementary, about 95% and 55% of full ReMix's improvement over prior-work-only (Table 3). Under our new repeated stochastic rollouts, replay-only, which contains none of the existing borrowing methods, achieves 10.7–34.9% lower Total MSE than prior-work-only. Please see our global response (G2) for details.

Response to W2


"Each model-case pair is represented by only one stored rollout. The experiments do not examine whether the results remain stable under repeated stochastic generations or grading variability."

All three reviewers raised this concern. Please see our global response (G2), where Table R3 reports the repeated stochastic rollout results.

Response to W3


"ReMix is not consistently the best-performing method across evaluation settings."

We address this shared concern in our global response (G1). Please see the discussion following Table R1 for replay-only and the open-model subset, and Table R2 for early tail risk.


Thank you again for your careful review. We hope our response has fully addressed your concerns, and we are happy to answer any further questions.

AuthorsOpenReview ↗

Follow-up before the discussion closes

Thank you again for your careful review. As the discussion period ends soon, we wanted to gently follow up. We hope our response above has addressed your concerns, and we are happy to answer any further questions.

Comments on the paper

6 comments
AuthorsOpenReview ↗

Global response

We thank all reviewers for their careful reading and constructive feedback. We address two concerns raised by all three reviewers, each with new experiments.

G1. Why does the current ReMix not achieve the lowest MSE or CVaR in every setting?


(Addressing 4Fph – W3, 8oMt – part of W5, and nyh4 – part of W3.)

The submitted version of ReMix is a tuning-free default. We intentionally did not tune it, since in the online setting, hyperparameters cannot be tuned in hindsight. It includes every estimator with η=2\eta=2, and including an estimator costs at most (log⁡N)/η(\log N)/\eta in regret (Theorem 1).

After minimal tuning, we now present ReMix (tuned), which lowers every metric on both testbeds and outperforms the baselines in Table R1 on every metric on the full Supplementary sequence. We chose among six options on the full MedHELM sequence only: two collections (all estimators, or the replay estimators with current-only and NPP) and η∈{2,1,0.5}\eta\in\lbrace 2,1,0.5\rbrace. We included NPP for its strong performance, as highlighted in the review. The selected option, the smaller collection with η=0.5\eta=0.5, is applied unchanged elsewhere.

Table R1. Full and minimally tuned ReMix. Ratios to current-only. Best baseline is the lowest standalone baseline per column; bold is lowest.

MethodMSE@5CVaR@5MSE@10CVaR@10Total MSETotal CVaR
(MedHELM)
Best baseline0.7470.3090.8400.5680.8160.703
Full ReMix0.6100.4390.8340.6190.7060.575
ReMix (tuned)0.5450.3880.7760.5820.6820.526
(Supplementary)
Best baseline0.7440.4640.8780.7500.7210.748
Full ReMix0.6520.5740.8210.7460.6620.697
ReMix (tuned)0.5610.4360.7400.5960.6030.644

Results. Minimal tuning removes three of the four exceptions raised. ReMix (tuned) now outperforms the submitted replay-only configuration on the full Supplementary sequence, NPP on the open-model subset in Total MSE, and the sliding window on Supplementary CVaR@5. Full results are in Appendix H.1, Tables 6–9.

The remaining exception, early CVaR on MedHELM, resembles the classical bias–variance tradeoff when borrowing historical data [1–2]. NPP, pooling, and rMAP borrow more heavily from earlier versions, lowering variance but adding bias when performance has changed.

Table R2. MSE@5 decomposition and CVaR by horizon on MedHELM. Ratios to current-only.

MethodBias$^2$VarianceCVaR@5CVaR@10CVaR@20
NPP1.1020.1080.3090.5991.073
Pooling1.5270.0260.3220.6661.289
rMAP0.5650.2020.3760.5680.887
ReMix (tuned)0.2060.3390.3880.5820.821

Results. The three baselines with lower CVaR@5 than ReMix (tuned) have 2.7–7.4 times its squared bias, and their advantage does not last: at 20 cases ReMix (tuned) has the lowest CVaR.

G2. Do ReMix’s gains persist under repeated stochastic rollouts?


(Addressing 4Fph – W2, 8oMt – part of W2, and nyh4 – part of W3/Q4.)

We have extended the single stored rollout per model–case pair to four independent runs, regenerating and regrading all randomly sampled target and replay cases. We consider a realistic version update (GPT-5.4-mini → GPT-5.6-luna) on MedSP and HealthBench.

Table R3. Repeated stochastic rollouts. Means over the four runs; format as in Table R1.

MethodMSE@5CVaR@5MSE@10CVaR@10Total MSETotal CVaR
(MedSP)
Best baseline0.6070.4450.7950.6720.7710.838
Full ReMix0.5120.5310.6350.6210.7170.791
ReMix (tuned)0.3260.3400.4070.4060.6290.728
(HealthBench)
Best baseline0.2730.1870.4500.3280.6510.731
Full ReMix0.4050.4820.4800.5390.6940.766
ReMix (tuned)0.2750.3050.3300.3480.6400.726

Results. Full ReMix lowers MSE relative to current-only and prior-work-only on both targets in all four runs. Replay gives the larger gain: replay-only has 10.7–34.9% lower Total MSE than prior-work-only across both targets. ReMix (tuned) lowers MSE further and outperforms every standalone baseline on MSE and CVaR on MedSP in every run; replay-only, itself a ReMix collection, is lower still. On HealthBench, where the target mean changes by only 0.04–0.09, the best baseline keeps lower early CVaR, consistent with the bias–variance tradeoff discussed in G1. Full results are in Appendix H.2, Tables 10–11.


References

[1] Kopp-Schneider, A., Calderazzo, S., and Wiesenfarth, M. Power gains by using external information in clinical trials are typically not possible when requiring strict type I error control. Biometrical Journal, 2020.

[2] Galwey, N. W. Supplementation of a clinical trial by historical control data: is the prospect of dynamic borrowing an illusion? Statistics in Medicine, 2017.

AuthorsOpenReview ↗

Response to Reviewer nyh4 [1/2]

Thank you very much for your careful reading and helpful comments. We really appreciate your recognition of the importance of early evaluation after updates and the breadth of our empirical study. Please see below for our point-by-point responses to your questions.

Response to Q1 and part of W1


"For the actual target/source pairs used in the experiments, how often does the replay statistic satisfy the condition required by Proposition 1, i.e., does dkd_k actually upper-bound or reliably track ∣v(k)−v(k−1)∣\lvert v^{(k)}-v^{(k-1)}\rvert?"

Thank you for this question. We checked all 174 target–source–update comparisons, with δk=v(k)−v(k−1)\delta_k=v^{(k)}-v^{(k-1)}.

Table R7. Replay dissimilarity and borrowing accuracy. Lower error compares the single-source replay-informed borrowing estimator in the submission manuscript with current-only, using cumulative squared estimation error over the new version.

CheckMedHELMSupplementary
dk≥∣δk∣d_k\ge\lvert\delta_k\rvert102/144 (70.8%)25/30 (83.3%)
Lower error when dk≥∣δk∣d_k\ge\lvert\delta_k\rvert102/10225/25
Lower error when dk<∣δk∣d_k<\lvert\delta_k\rvert19/422/5

Results. The drift inequality holds in most comparisons. Borrowing improves on current-only in all 127 comparisons where it holds, consistent with the benefit predicted by Proposition 1. The same estimator also improves in 21 of 47 comparisons where the inequality fails. Appendix H.3, Table 12 and Figure 5 give the results for every source and update, including the tracking diagnostics.

"what theoretical justification remains for using dk2d_k^2 as the update-specific borrowing penalty?"

The inequality in Proposition 1 is a sufficient condition. Our new Theorem 2 also permits dk<∣δk∣d_k<\lvert\delta_k\rvert, as explained in our response to Q2 below.

Response to Q2 and part of W1


"Can the authors provide a theoretical result applicable to the heterogeneous benchmark setting actually used in MedHELM and the supplementary testbed?"

Thank you for this insightful question. We have extended our borrowing analysis to benchmarks with different cases, tasks, and grading rules in Theorem 2 (Appendix I.2). The result quantifies when borrowing improves MSE, including cases where replay understates the target change, and bounds cumulative excess MSE when borrowing is harmful.

Borrowing guarantee. Let δk=v(k)−v(k−1)\delta_k=v^{(k)}-v^{(k-1)} be the target mean change, dd the observed replay dissimilarity, μ\mu and PP the carried mean and variance, and hh the number of current cases. Retaining Proposition 1's independent current rewards, known variance σ2\sigma^2, and historical error bound ∣μ−v(k−1)∣≤P\lvert\mu-v^{(k-1)}\rvert\le\sqrt P, borrowing matches or improves on current-only whenever

∣δk∣≤2(P+d2)+σ2/h−P.\lvert\delta_k\rvert\le\sqrt{2(P+d^2)+\sigma^2/h}-\sqrt P.

Intuition. Borrowing historical target data reduces variance but can introduce bias after an update. The carried mean's error relative to the new target mean is bounded by P+∣δk∣\sqrt P+\lvert\delta_k\rvert, and the displayed sufficient condition ensures that the variance reduction offsets this possible bias. Early after an update, current-only variance σ2/h\sigma^2/h is large, allowing a wider range of target changes. The threshold on the right-hand side is always at least dd, so the guarantee can hold even when d<∣δk∣d<\lvert\delta_k\rvert. The condition relates the magnitudes of changes across versions. The benchmark and target need not have similar mean scores or change in the same direction.

When borrowing is harmful. Theorem 2 also bounds the cumulative positive excess MSE of borrowing over current-only within a version, requiring neither d≥∣δk∣d\ge\lvert\delta_k\rvert nor the historical error bound. Appendix I.2 gives the full statement and proof.

AuthorsOpenReview ↗

Response to Reviewer nyh4 [2/2]

Response to Q3 and W2


"Can the authors provide an analogous bound for the finite-population, without-replacement protocol used in the experiments?"

Thank you for this question. We have added an estimation-MSE guarantee for this exact protocol in Appendix I.1, Proposition 3.

Bound for the experimental target. Let vtv_t be the full-pool mean of the active version, and let ET=∑t=1T(v^t−vt)2\mathcal E_T=\sum_{t=1}^T(\hat v_t-v_t)^2 denote ReMix's cumulative squared estimation error. Define ETi\mathcal E_T^i analogously for constituent estimator ii. For 0<η≤20<\eta\le2, Proposition 3 gives

\le \min_{i\in[N]}\mathbb E[\mathcal E_T^i] +\frac{\log N}{\eta}+K+B_{\mathrm{FP}}.$$ Here $N$ is the number of estimators, $\eta$ the learning rate, $K$ the number of versions, and $B_{\mathrm{FP}}$ an explicit finite-population correction. Expectations are over random case orders, conditional on the fixed score pools, replay anchors, and version schedule. **Why the correction is needed.** Under sampling without replacement, the next reward's conditional mean is the remaining-pool mean. Its difference from the full-pool mean depends on previously observed cases, as do the adaptive estimates. The correction bounds the resulting dependence term when converting prediction regret into estimation error. Appendix I.1 gives its explicit formula in terms of pool size, sampling horizon, and score variances. **For a fixed horizon, this correction bound vanishes as the pool grows.** **Remark.** Proposition 3 applies to the full ReMix mixture and does not require $d_k\ge\lvert\delta_k\rvert$. It bounds estimation error against the full-pool mean used in our experiments. Corollary 2 continues to cover the population mean under independent arrivals. > ## Response to Q4 and W3 --- > "evidence under a setting with stochastic repeated model rollouts, changing case mix, or a prospective outcome stream rather than resampling fixed precomputed benchmark scores" Please see our global response (G2) for the repeated stochastic rollout results in Table R3. We discuss the experiments with changing case mix in our response to reviewer 8oMt (W3). Although the benchmark scores are precomputed, each random permutation simulates a different order of case arrivals during deployment. Prospective clinical-outcome validation remains future work, as stated in the Limitations. > "how does ReMix behave when the external replay benchmarks fail to track a clinically important regression in the deployment population?" **Our main experiments already contain 24 regressions, and none was missed by every replay source.** Target performance drops at 19 of the 36 MedHELM updates and 5 of the 15 Supplementary updates, by up to 0.17. At nine of them, at least one replay source understated the drop ($d_k<\lvert\delta_k\rvert$), but never all. We have reanalyzed the main experiments over these regression updates only. **Table R8. MSE@5 at regression updates, relative to current-only.** Each regression's MSE@5 is divided by current-only's at the same update, then averaged over regressions. |Method|Avg over 24 regressions|9 regressions understated by ≥ 1 replay source| |---|--:|--:| |Pooling|0.894|1.749| |Sliding window (40)|0.886|1.960| |Chu-Yi|0.673|0.806| |rMAP|0.540|0.846| |SAM|0.544|0.835| |EMAP|0.887|1.370| |CP|0.857|0.881| |NPP|0.711|1.378| |Prior-work-only|0.503|0.736| |Replay-only|0.629|0.765| |**Full ReMix**|**0.490**|**0.687**| **Results.** **Full ReMix has the lowest average MSE@5 among the methods in Table R8, with about half the error of current-only.** It also has the lowest average MSE@5 over the nine regressions that at least one replay source understated, where pooling, the sliding window, EMAP, and NPP are all worse than current-only. Full ReMix also improves on prior-work-only in both columns, so the replay estimators help even when some replay sources understate the drop. Details are in Appendix H.6, Table 13. **Potential silent failures.** In a separate setting, one may expect an *unannounced change* that triggers a regression that is difficult to catch in real life. We view it as important future work that may need dedicated techniques, such as change-point or anomaly detection. > "ReMix is not uniformly best even within the offline testbeds." Please see our global response (G1) where we address this common concern in full. --- *Thank you again for your careful review. We would be grateful if you could consider the new theoretical results and experiments in your assessment.*

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.