- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper introduces Diagnose-Before-Adapt (DBA), a retrospective framework for comparing memory updates, calibration, learned aggregation, and model adaptation under delayed label availability. Its proposed Learned Neighbourhood Aggregation (LNA) uses a lightweight gradient-boosted scorer over retrieval statistics from frozen encoders. Experiments use 117,328 consultations spanning 51 months from one clinical service, with additional physician-specific analyses and a static public benchmark. Cumulative memory provides the largest improvement, while LNA adds 1.05–3.53 balanced-accuracy points over recalibrated cumulative memory with lower reported GPU occupancy than parameter adaptation.
Critical strengths
- The temporal protocol is carefully specified. Predictions use only prior-month state, evaluation cases remain excluded from subsequent training and memory, and interventions are compared on identical monthly cases. The size-matched freshness control also separates recency from the amount of stored evidence.
- The paper tests several plausible explanations for the gains. Dense-retrieval and scorer-refitting ablations, alternative supervised aggregators, prior controls, and label-blind overlap filtering provide more convincing evidence than a comparison against frozen memory alone. Reporting CPU costs separately from GPU occupancy is also appropriate.
- The limitations are unusually explicit. The authors acknowledge overlapping streams, the static nature of the public benchmark, uncertainty across physicians, and the failure of the exploratory router to generalize to one stream. They also correctly state that the support oracle does not bound LNA, which can predict classes absent from the retrieved candidates.
Critical weaknesses
- The methodological contribution remains limited. Conditional headroom is a matched performance difference, the support ceiling follows directly from candidate inclusion, and the interaction proposition is a standard difference-in-differences identity. LNA combines established retrieval features with supervised boosting. This could still support a valuable empirical paper, but DBA’s additional diagnostic value is not yet demonstrated convincingly: it requires executing the interventions, and the exploratory routing rules reduce to constant actions within each evaluated stream. A direct comparison with equally lightweight classifiers trained on frozen query embeddings would also help establish the benefit of neighbourhood-based learning.
- The primary inference does not adequately address the temporal structure of the experiment. Base-LNA superiority uses month resampling, although successive months share accumulated memory and fitted models. Bonferroni correction addresses multiplicity, not serial dependence. The appendix’s dependence-aware superiority analysis concerns LNA+, while the base-LNA block analysis tests non-inferiority against different references. This matters because LNA+’s SingleMD comparison changes from significant under IID resampling to nonsignificant with longer blocks. The confirmatory base-LNA versus BL2b comparison needs the same sensitivity analysis.
- The breadth of evidence is narrower than the number of reported streams suggests. Full contains both principal physician streams, and all longitudinal results come from one service. TCM-SD validates static aggregation rather than adaptation under drift. Overlap filtering preserves relative gains, but absolute LNA performance falls substantially—for example, from 78.85 to 57.72 on Full—and filtering is applied at retrieval time without changing model fitting. These are useful controls, but they do not establish patient-disjoint generalization. The physician-level Hartung–Knapp interval also crosses zero, leaving uncertainty about how consistently the benefit extends beyond the dominant physicians.
Questions for authors
- Can you report moving-block or HAC confidence intervals for the primary base-LNA versus BL2b contrasts, including sensitivity to block length and the stated multiplicity correction?
- How does LNA compare with a linear classifier or small MLP trained directly on frozen query embeddings using the same historical labels, monthly update schedule, and tuning budget? The existing alternative-aggregator comparison keeps the neighbourhood representation fixed.
- Which record fields are available to each predictor at inference time? Please explicitly distinguish chief complaint and examination findings from prescriptions, syndrome names, and other information recorded after the clinical decision. Also clarify whether repeat patients can occur across training and evaluation consultations.
- Figure 17 shows approximately chance-level performance with 1,000–2,000 fitting neighbourhoods, followed by a sharp jump at 3,000 across all streams. What explains this discontinuity? Please clarify whether any fitting condition, fallback, or implementation threshold changes at that point.
10 comments
W1 & Q2: Direct Frozen-Query Classifiers, Aggregators, & DBA Scope
Your point in W1 and Q2—asking how matched parametric classifiers trained directly on frozen query embeddings perform, alongside bounding DBA's audit versus adaptive routing—identifies a key experimental control that we want to address with complete candor.
1. Candid Status of Direct Classifiers on Raw Frozen Query Embeddings (W1, Q2)
You asked how monthly Logistic or MLP heads trained directly on frozen query features (sparse IDF + 512-D bge-small-zh query embeddings, without neighbour retrieval) perform under the exact same <t monthly budget (3,000 or 12,000 cases).
We want to be completely transparent: we agree this is the cleanest direct parametric control for isolating retrieval-mediated classification from direct embedding classification, and indirect comparisons cannot substitute for it. Because raw consultation text and per-visit query embeddings (all_records.jsonl) reside exclusively on the restricted hospital server and are not exported in our local artefact bundle, direct frozen-query Logistic/MLP runs on the hospital streams remain unexecuted (we state this gap explicitly in Section 4.3 and Appendix E, and include a <t prequential runner script for host execution).
To situate what our completed experiments do and do not show in the meantime:
- Holding
<tretrieved neighbourhoods fixed (Table 4b, Section 4.3): On identical dual-retriever relational matrices (),LNA–HGB(77.74% / 65.19% / 76.83%) consistently beats a linearLogisticranker (76.11% / 62.74% / 74.12%), aTwo-layer MLP(76.48% / 63.61% / 74.49%), and permutation-invariantDeepSets(69.88% / 59.27% / 69.31%), isolating the benefit of gradient-boosted threshold splits given the retrieved neighbourhood representation. - Direct end-to-end encoders & parameter-updated LLMs: On public 148-way
TCM-SD(Table 12),Base LNA(82.72%Acc,56.25%Macro-F1) andLNA+(83.19%,58.50%) outperform end-to-end fine-tunedRoBERTa(82.26%,47.55%) and domain-pretrainedZY-BERT(82.19%,51.01%). On the longitudinal hospital streams (Tables 1, 2, 15),Base LNAstays within-1.27to+1.49pts of parameter-updatedQwen3.5-4B(CDA-L3LoRA andBL5monthly full-history retraining, passing 2-pt Bonferroni non-inferiority in3/6pairs inTable 15) at0.0020–0.0111vs.2.28–2.64GPU-hours.
2. Bounding DBA as a Counterfactual Audit Rather Than a Dynamic Controller (W1)
We also agree with both of your points in W1 regarding DBA's diagnostic role: DBA requires executing the compared arms offline, and in Table 19 and Appendix K.3 (Table 27) our exploratory LOSO threshold router selects a constant action (12/12 quarterly probes) on each stream (LNA+ on Full and IndepMD, LoRA on SingleMD), incurring 1.83 pts of static regret on IndepMD and not executing a late-horizon switch on SingleMD (Q8–Q12). In revised Section 4.2 and Appendix G, we therefore bound DBA strictly to offline counterfactual prequential attribution—measuring size-matched freshness gaps (Figure 3), retention (Table 2), multi-candidate coverage (Table 26), and memory–aggregation substitution (Proposition 2, Table 17) on a shared timeline—rather than claiming a zero-execution predictor or adaptive online controller.
W3 & Q3: Field Contract, Text-Overlap Controls, & 4-MD Pooling
Turning to W3 and Q3, we found your questions on exact inference-time schema fields, patient-level repeat visits under lexical filtering, and cross-physician uncertainty across the four eligible clinicians especially helpful for tightening our external-validity claims.
1. Exact Pre-Decision Field Contract Across Code Modules (W3, Q3)
To remove any ambiguity about whether post-diagnosis information leaks into encoders or prompts, new Appendix K.1 (Table 25) (referenced in Section 3.1) maps every schema field to its exact pipeline stage and function:
Record Field (Appendix Table 25) | Clinical Stage | Online Query, Retrieval, & Prompt (S_{t-1}) | Post-Lock Role (After y_t Locks) |
|---|---|---|---|
chief_complaint | Pre-decision intake | Sparse IDF (record_features) + dense BGE (presentation_of) + LLM prompt | Stored in <t+1 keys & exemplars |
tongue_pulse | Pre-decision exam | Sparse IDF (record_features) + dense BGE (presentation_of) + LLM prompt | Stored in <t+1 keys & exemplars |
diagnosis_western | Pre-decision (optional) | Sparse IDF (record_features) + LLM prompt (not in dense BGE) | Stored in <t+1 lexical index |
symptoms / age / sex | Pre-decision (optional) | Formatted in LLM prompt (cda/fuse.py) if present (not in LNA) | Retained in consultation record |
prescription (herbs) | Post-decision order | Strictly withheld (never in encoders, features, or prompts) | Model-free herb-drift audit (Fig. 2b) |
syndrome / zangfu | Post-decision note | Strictly withheld (never in encoders, features, or prompts) | Novel-term drift audit (Fig. 2d) |
gold_meridian () | Post-decision target | Strictly withheld until month- predictions lock (score_month) | Scores month ; non-eval <t+1 updates |
2. Repeat Visits, Text-Overlap Drop (78.85% -> 57.72%), & Patient-ID Boundary (W3)
As noted in revised Sections 3.1 and 4.3, de-identified records are keyed by consultation ID without a cross-visit patient_id, so exact repeat-patient rates cannot be audited. In Table 3, withholding all strictly earlier normalized-exact or high-similarity character-trigram matches affects 46.08%–61.21% of queries and substantially lowers absolute accuracy as close clinical templates are removed (e.g., Full drops from 78.85% to 57.92% / 57.72%, SingleMD to 43.57% / 43.29%, IndepMD to 56.14% / 55.92%), while paired LNA gains over BL2b widen to +4.09 / +4.36, +2.16 / +2.11, and +4.84 / +4.78 pts. Crucially, we state explicitly in Section 4.3 that retrieval-time lexical filtering bounds template-copying sensitivity during inference, but does not remove repeat patients from <t scorer fitting and cannot establish patient-disjoint generalization.
3. Four-Clinician Heterogeneity & Zero-Crossing Hartung–Knapp Intervals (W3)
Across the mutually exclusive clinicians meeting outcome-blind volume thresholds (summarized directly in Section 4.3 and Appendix I, Tables 21–22), we report all uncertainty estimators side-by-side without selective emphasis:
Clinician (Table 21) | Cases | Eval / Active Mo | Policy (95% CI) | Active | Meta-Estimator (Table 22, ) | Pooled (95% Interval) |
|---|---|---|---|---|---|---|
8056e996 (IndepMD) | 79,535 | 38 / 38 | +3.53 ([+2.57, +4.49]) | +3.53 | Normal random-effects CI | +1.73 ([+0.15, +3.31]) |
f3ec4945 (SingleMD) | 26,099 | 39 / 20 | +1.05 ([+0.02, +2.07]) | +1.96 | Hartung–Knapp () CI | +1.73 ([-0.71, +4.17]) |
69d22f35 (pre-8k) | 6,065 | 35 / 0 | +1.77 ([+0.14, +3.39]) | — | Modified Hartung–Knapp CI | +1.73 ([-0.84, +4.30]) |
a90628ca (pre-8k) | 2,092 | 11 / 0 | -0.27 ([-2.93, +2.40]) | — | Between-MD prediction interval | +1.73 ([-5.16, +8.62]) |
Leave-one-physician-out point estimates stay positive (+1.11 to +2.14 pts), but only 2 of 4 clinicians reach the 8,000-case ranker activation threshold, and both Hartung–Knapp ([-0.71, +4.17]) and prediction ([-5.16, +8.62]) intervals cross zero. We highlight this explicitly in Section 4.3 (Page 9) so cross-clinician generalizability is not overstated.
W2 & Q1: Dependence-Aware Inference (Table 4a & 28) & SingleMD
We are indebted to your rigorous statistical and code-level audit, which directly strengthened our uncertainty quantification and uncovered a real implementation bug in our exploratory budget script. Keeping manuscript edits targeted (preserving all
Figures 1–20andTables 1–3, adding two-panelTable 4in Section 4.3, and placing full audit tables inAppendix K), we respond in four focused comments: Part 1 covers autocorrelation-robust inference (W2, Q1), Part 2 covers frozen-query classifiers and DBA's scope (W1, Q2), Part 3 covers the pre-decision field contract, overlap filtering, and four-clinician pooling (W3, Q3), and Part 4 covers theFigure 17code audit (Q4).1. Dependence-Aware Inference for Confirmatory
Base LNA vs. BL2b(W2, Q1)We fully agree that IID month resampling can understate variance when monthly evaluation series are autocorrelated, and that confirmatory
Base LNArequires the same dependence-aware scrutiny previously reported only forLNA+(Table 20). Using the exact monthly paired series, we addedTable 4adirectly in Section 4.3 (Page 10) alongside the complete 6-estimatorAppendix K.4(Table 28) reporting circular moving-block bootstrap (L = 3, 6, 12;Künsch, 1989), finite-sample Student-t Newey–West HAC (lag = 12;Newey & West, 1987), and non-overlapping quarterly block 95% CIs plus raw and 63-test Bonferroni-adjusted p-values:Table 4a&Table 28)Full(T=39)[+2.06,+3.61](0.0063)[+2.04,+3.63][+2.03,+3.58][+2.15,+3.45][+2.05,+3.61](<0.0001 / <0.0001)[+1.86,+3.80]IndepMD(T=38)[+2.61,+4.51](0.0063)[+2.48,+4.76][+2.47,+4.86][+2.55,+4.64][+2.34,+4.71](<0.0001 / <0.0001)[+2.13,+4.93]SingleMDpolicy (T=39)[+0.07,+2.09](1.0000)[-0.05,+2.18][-0.21,+2.35][-0.24,+2.48][-0.44,+2.53](0.1621/1.0000)[-0.22,+2.32]SingleMDpre-8k (T=19)[-1.00,+1.10](1.0000)[-1.08,+1.12][-1.16,+1.07][-0.82,+1.00][-0.73,+0.90](0.8339/1.0000)[-1.80,+2.22]SingleMDactive (T=20)[+0.34,+3.55](1.0000)[+0.31,+3.59][+0.17,+3.62][+0.56,+3.37][+0.23,+3.70](0.0285/1.0000)[-0.57,+4.34]2. What These Estimators Establish (
Full&IndepMDvs.SingleMDPhases)FullandIndependentMDare robust to serial dependence and 63-test multiplicity: Across every moving-block length (3, 6, 12), quarterly aggregation, andNW-12HAC, unadjusted 95% CIs remain strictly above+1.86pts onFulland+2.13pts onIndepMD. UnderNW-12, both streams survive 63-test Bonferroni correction at p63<10−4 (IIDbootstrap p63=0.0063 reflects the 63/10,000 resampling resolution floor).SingleMD39-month policy vs. pre-/post-8k phase decomposition: OnSingleMD, the 39-month deployable policy (+1.05pts) is marginally positive under unadjusted IID resampling ([+0.07,+2.09], praw=0.0432) but crosses zero under all block and HAC estimators (NW-12[-0.44,+2.53], praw=0.1621, p63=1.0000). Decomposingfit_logdescriptively—while noting explicitly that8kranker activation co-varies with calendar time and historical archive size rather than being randomized—the19pre-8k fallback months average+0.08pts (NW-12[-0.73,+0.90]), whereas the20post-8k active months average+1.96pts (unadjustedNW-12[+0.23,+3.70], praw=0.0285, but p63=1.0000 and quarterly[-0.57,+4.34]). Thus, active-phase single-physician gains are directionally positive on unadjusted monthly HAC, but remain suggestive rather than multiplicity-robust (Section 4.3&Appendix K.4).