acceptodds
Withdrawn from ICLR 2027

Multi-Fidelity Learning under Update Budgets: Cross-Fidelity Spectral Filtering

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Updating a low-fidelity predictor with scarce high-fidelity labels can alter behavior that should be retained. We study multi-fidelity residual learning under an explicit budget on changes to the stored predictor. For fixed kernel models, the weighted norm of the cross-fidelity residual-to-anchor map gives the exact worst-case residual-normalized update, termed the scale-separation index (SSI). We construct a spectral filter that attenuates high-gain residual directions, decreases design-anchor SSI monotonically, and minimizes it among permutations of a fixed shrinkage spectrum. Validation selects the filter and blend step subject to an independently audited update cap. For a finite candidate family, we derive a selection-uniform population bound under independent and identically distributed audit sampling and bounded updates. Across 20 seeds at a normalized cap of 0.20, filtering reduces root mean squared error relative to matched quadratic regularization by 0.590% on QM7b and 0.241% on MF-PCBA, and outperforms equal-spectrum scrambled controls. The filter uses more of the available update budget than matched controls. Stronger external predictors remain more accurate in several comparisons, often with larger changes to the stored predictor. A separate quadratic-control study satisfies all 160 anchor certificates but meets the same cap on only 133 test blocks. Together, these results establish a mechanism for directional update control and delineate the role of anchor coverage in transferring local guarantees.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
multi-fidelity learning, continual learning, spectral filtering, update budgets, kernel methods, residual learning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_ym7gOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies multi-fidelity learning under a budget on changes to an existing predictor. It introduces the Scale-Separation Index (SSI) to characterize worst-case normalized updates under fixed-kernel assumptions and proposes spectral filtering to suppress high-gain residual directions. The paper establishes theoretical properties of the filter and a population bound under independent audit sampling. Experiments on QM7b and MF-PCBA show modest improvements over matched controls, while revealing limitations in predictive competitiveness and the transfer of update-budget guarantees.

Critical strengths
  1. Careful and transparent experimental evaluation. The paper distinguishes predictive error, update magnitude, and low-fidelity agreement, and reports both positive and negative findings. Its detailed protocols and uncertainty estimates make the empirical claims relatively easy to assess.
  2. Controlled evaluation of directional filtering. The equal-spectrum scrambled control isolates directional alignment from shrinkage strength. The direct filter outperforms scrambled directions in 21 of 22 sensitivity configurations, providing evidence for a directional effect.
Critical weaknesses
  1. Limited practical gains and insufficiently isolated benefits. Improvements over matched controls are small, and the proposed method uses more of the update budget. Comparisons at matched realized update magnitudes are missing. Stronger feasible predictors perform substantially better in several settings, while isotropic regularization also outperforms SSI regularization in a reported QM7b experiment.
  2. Limited validation of preservation and generalization. The experiments do not explicitly separate retention and correction regions, leaving the motivating preservation objective insufficiently tested. Moreover, test-budget violations in the pooled experiment and the weak correlation between SSI and neural-network interference raise concerns about practical transfer beyond the fixed-kernel setting.
Questions for authors
  1. Contribution of directional control beyond regularization strength. Could the authors report direct-filter comparisons against trace-matched isotropic and training-covariance controls at \tau=0.20 and across additional budget levels? In particular, how should the substantially better isotropic-control result in the reported \tau=0.50 QM7b experiment be interpreted? Does directional filtering retain an advantage when regularization strength is appropriately matched?

  2. Accuracy gains at matched realized update magnitudes. Could the authors provide paired accuracy-update frontiers, with confidence intervals, comparing direct filtering with \gamma=0 scalar blending and strong baselines such as NARGP and AR1 at matched realized update magnitudes? This would help determine how much of the reported improvement is attributable specifically to directional filtering rather than greater budget utilization.

  3. Validity of budget transfer and the preservation use case. Could the authors clarify whether the pooled-gate protocol uses audit samples independent of candidate fitting, as required by Theorem 2, and explain the discrepancy between anchor feasibility and the 133/160 test-feasibility result? In addition, can they evaluate a setting with distinct retention and correction regions, reporting the update on the retained region and predictive error on the correction region?

Reviewer_YQpvOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies how to update an existing predictor, built from cheap low-fidelity data, using a small number of expensive high-fidelity labels, while limiting how much the update changes the predictor's outputs on a chosen set of reference inputs. For a fixed kernel residual model, the authors define the scale-separation index (SSI): the largest factor by which the learned correction can change the reference outputs. A spectral filter then shrinks the correction most along the directions with the largest amplification. Validation data choose the filter strength and a blending step so that the update stays within a fixed budget on a separate audit set. A Hoeffding-style bound extends this budget from the audit set to the full input distribution. At a tight budget on QM7b and MF-PCBA datasets with 20 seeds, the filter reduces RMSE by 0.59% and 0.24% relative to a quadratic penalty, and also beats a control that applies the same shrinkage to random directions. Under the same budget, however, standard multi-fidelity models (NARGP, AR1, MFNN) are more accurate on QM7b, though they change the existing predictor more.

Critical strengths

Strength: The paper measures both accuracy and how much the model changes, and ablation experiments exist.

  1. The paper uses three separate measures: error on high-fidelity data, how much the update changes the old model, and agreement with low-fidelity labels (Section 2). It reports error and update size together. For example, this shows that the filter gets lower error but also uses a larger update under the same cap (Table 2).
  2. The existing ablations are over multiple seeds and well designed. The scrambled control uses the same amount of shrinkage but in random directions, so it tests whether the choice of directions really matters. The proposed filter beats this control and quadratic regularization at τ\tau = 0.20, over 20 seeds with paired confidence intervals. It also beats quadratic regularization in all 22 settings of feature size, ridge and anchor count (Section 5).
  3. The paper is honest about its limits. The cap is not met on some test blocks (it holds in 133 of 160). The safety margin lowers accuracy without improving feasibility. In trained neural networks, SSI is only weakly related to the real change (Section 5).
Critical weaknesses

Weakness 1: The empirical validation is incomplete and does not establish competitiveness.

  1. The gains are measured only against the method's own variants, and they are very small in absolute terms. For example, on QM7b, RMSE falls from 0.953 (quadratic penalty) to 0.948 (Table 2). That is a difference of 0.005, about 5 meV if the units are the eV shown in Figure 3, and 99.5% of the error remains. The difference is about one-twentieth of the gap to capped NARGP (0.851; Table 16) and one-third of the width of the method's own 95% interval. The two methods' unpaired intervals overlap almost completely, so the difference shows up only in paired contrasts.
  2. It is not state of the art. Under the paper's own objective to minimize EhE_h s.t. Ū ≤ τ\tau, capped NARGP is feasible (Ū = 0.177 < 0.2) and more accurate (0.851 vs. 0.948) show in the results. NARGP, AR1 and MFNN all beat the filter on QM7b by 6~12% (Table 3). The paper never shows that the method achieves the lowest error in any setting.
  3. The methods are evaluated under different rules, so the accuracy–preservation comparison is inconclusive. The filter is set to pass an additional worst-case check that the external baselines apparently do not face. It consequently makes much smaller changes to the stored predictor: its average update uses 37% of the QM7b budget, compared with 89~91% for NARGP and AR1 (Tables 2 and 16). These results do not establish whether its lower update and higher error come from directional filtering or simply from the stricter check. The authors should compare methods under the same constraints, match their realized updates, and report how often each satisfies the cap on test inputs.
  4. The filter is never applied to a strong model. It is applied only to an RFF residual. Applying it to a stronger model is deferred to future work so the generalizability of this method is not tested.
  5. The motivation for limiting the update is never tested. The method is meant for cases where the existing model (built from cheap data before the update) has been validated somewhere and should be protected (Section 1). No experiment uses such a model. On QM7b, the existing model is less accurate than a model trained on the expensive labels alone (Table 6). On MF-PCBA they tie (Table 8). The paper also never measures a downstream benefit from changing the model less. The method therefore gives up accuracy to protect a model with no demonstrated value, and the benefit of that protection is never measured.
  6. Baselines, benchmarks and metrics are thin.
    a. Baselines: three classical families, with simplified AR1 and an MFNN that is 2.3~8.9× worse than not updating (Table 16). No recent deep GP/BayesianNN or in-context multi-fidelity method.
    b. The benchmark coverage does not support the method generalizability. (1) The evidence comes from three target pairs in one domain: HOMO and LUMO on QM7b (simulated labels) and one MF-PCBA assay. There are also small Cu and neural-Jacobian checks (Section 6). One of the two main benchmarks has almost no room for improvement. On MF-PCBA, every method in Table 8, including leaving the model unchanged (0.254), falls within 0.251–0.255, so it cannot separate methods. (2) There are no analytical synthetic multi-fidelity functions with known ground truth (such as the MF2 benchmark suite https://mf2.readthedocs.io/en/latest/), where preservation could be checked directly.
    c. Metrics: no uncertainty metrics, and no runtime against the baselines.
  7. Overall: the evidence supports one narrow claim: that this is the best regularizer among its own variants for an RFF residual. It is far from the breadth expected of a multi-fidelity regression method.

Weakness 2: The theoretical guarantees are standard and do not support the method at its primary evaluation setting.

  1. Theorem 1’s presentation as a contribution is overstated. The paper lists an “exact update coefficient” as its first contribution. However, SSI is defined as the operator norm of the residual-to-reference map (Eq. 7), whose largest possible amplification is precisely that norm. Theorem 1 therefore follows directly from the definition after a change of variables (Eq. 17). This is a useful diagnostic, but the paper does not establish what is theoretically new.
  2. “Exact” refers to the worst possible residual, not the actual update, and this gap makes the method costly. Each update uses one observed residual, and the paper concedes that "a large SSI need not produce a large observed change" (Sec. 4). Because the step size is limited by this worst case (Eq. 15), the method can reject updates that would have stayed within budget. It uses only 0.074 of the 0.20 budget on QM7b (Table 2), and the worst-case gate costs 0.060 RMSE relative to limiting the observed change (Sec. 5.5). In trained networks, SSI barely tracks the actual change (rank correlation 0.163; Sec. 5.2).
  3. Proposition 1 provides a narrow design-anchor guarantee. It shows that stronger filtering reduces design SSI and that assigning stronger shrinkage to larger singular values minimizes SSI among permutations of a fixed shrinkage spectrum. It does not establish better prediction, control on other inputs, or superiority over quadratic regularization or other shrinkage profiles. The scrambled control’s design SSI cannot improve on the proposed assignment by construction; the meaningful empirical evidence is its held-out accuracy and audit SSI, including the modest accuracy gains of 0.575% and 0.131% in Table 1.
  4. The filter is a standard regularization technique and closely resembles its main comparator. The paper derives the filter as the solution to "stay close to the original residual, plus a penalty on its effect on the design anchors" (Eq. 12). This is classical penalized least squares. The quadratic surrogate (Eq. 23) also penalizes the effect on the design anchors; it applies the penalty to the model weights rather than directly to the residual. These are two versions of the same idea, which is consistent with differences below 1% (Table 1). The paper does not relate the filter to classical spectral regularization or state what is new.
  5. Theorem 2 cannot certify the main MF-PCBA setting. The result applies Hoeffding’s inequality and a union bound to convert measured updates into a population-average guarantee. With 65 audit inputs, B=1B=1, and δ=0.05\delta=0.05, its sampling margin is at least 0.152, even for one candidate that is already above the squared budget of 0.04 at τ=0.20\tau=0.20. The paper evaluates the margin at τ=0.60\tau=0.60 instead, citing the small audit set (Appendix B.3), but this limitation is not made explicit alongside the headline results. The QM7b nested audit size is also unspecified.
  6. The population margin has no demonstrated empirical benefit at the tested settings. In the nested experiments, the nominal rule already satisfies the test cap in every block. Adding the finite-family margin leaves observed compliance unchanged while increasing QM7b RMSE by 0.0151 (Sec. 5.7; Table 11). This does not invalidate the guarantee, but the experiments do not identify a setting where its additional conservatism improves held-out compliance.
  7. Anchor certificates do not reliably transfer to held-out inputs. In the pooled quadratic study, all 160 blocks satisfy the anchor certificate, but only 133 satisfy the same cap on test molecules and compliance falls to 20/40 at the largest label budget (Sec. 5.7). This does not contradict the local theorem, but it limits the certificate’s practical reach when an independent population guarantee is unavailable.
  8. Overall: the "exact coefficient" is the definition of an operator norm. Proposition 1 holds by construction and says nothing about accuracy. The filter is a standard regularizer. The population bound is a textbook result that cannot be used at the main setting. The theory therefore does not support the paper's framing of a certified update method.
Questions for authors

Major questions:

  1. Does directional filtering beat simply shrinking a strong model? The central practical question is whether the filter improves prediction while limiting changes to the stored model. Yet capped NARGP and AR1 are about 12% more accurate on QM7b (Table 3), and filtering these models is left to future work (Sec. 6). Can the authors filter one of these stronger predictors and compare it with scalar blending at the same realized update? Without this comparison, the experiments establish small gains over matched controls on a weaker residual model, but leave the practical benefit unresolved.
  2. Is the reported trade-off partly caused by unequal constraints? The direct filter appears to face a worst-case certificate in addition to the measured-update cap, while external models face only the latter (Eq. 15; Appendix B). This could explain why the filter changes the stored predictor less but has higher error. Can the authors compare all methods under the measured-update cap alone, report accuracy–update curves, and show which constraint limits the selected step? Can you also report the fraction of external-model runs satisfying the cap on test inputs?
  3. What useful behavior is actually preserved? The paper motivates the method by a stored predictor that is trustworthy in some region, but a small change from that predictor is not itself evidence of retained accuracy. Can the authors show a task where the old model is accurate in a specified region, standard updates damage that region, and filtering prevents the damage? This would connect the proposed update measure to the application benefit used to motivate it.
  4. Does the population guarantee cover the headline results? For MF-PCBA’s 65 audit inputs, Theorem 2’s sampling margin is at least about 0.152 with the stated (B=1) and (δ\delta=0.05), even before accounting for multiple candidates. This exceeds the squared budget of 0.04 at the main cap of 0.20. Can the authors confirm that the theorem cannot certify this setting, report the corresponding calculation for QM7b, and state which experimental settings have a non-vacuous population guarantee? Empirical compliance on the test set should be distinguished from that guarantee.
  5. Is the theoretical contribution more than a standard spectral regularizer? The exact coefficient in Theorem 1 follows from the operator-norm definition, the ordering result in Proposition 1 uses an exchange argument, and Eq. 12 gives a quadratic-regularization interpretation. What new theoretical result remains beyond applying these constructions to residual updates? Please explain which part requires multi-fidelity structure rather than applying to any fixed-kernel correction of a stored predictor.
  6. Are the gains large and general enough to support the method’s value? The main improvements over quadratic regularization are below 0.6%, occur alongside larger realized updates, and do not extend consistently to the Cu and neural Jacobian tests (Tables 1, 2, and 15). Can the authors show where the direct filter improves on the unfiltered model at matched update size across the existing cap sweep? Also report paired results by target and label budget, and explain whether the gains in native units affect a practical decision.

Minor questions:

  1. Are the reference model and update units shared? Please confirm whether every external method is compared with the same stored predictor and uses the same residual scale. Otherwise, the same numerical cap may represent different amounts of change.
  2. How severe are the held-out cap violations? The pooled quadratic study passes every anchor certificate but satisfies the test cap in only 20/40 runs at the largest label budget (Sec. 5.7). How far above the cap are the failed runs, and does a disjoint audit set resolve the problem under otherwise matched settings?
  3. Could baseline tuning explain the unusual learning curves? High-only and random-feature NARGP errors increase as more labels are provided (Table 6), while native MFNN predictions are unstable (Table 16). Can the authors report validation tuning for feature count, regularization, and the neural model, and check whether these trends persist.
  4. Does clipping drive any of the results? How often does the pointwise clipping used for the bounded-update experiments alter a prediction? Please separate its effect from the effect of spectral filtering.
Reviewer_KxmfOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper reformulates the usual multi-fidelity prediction objective by introducing an update constraint, min⁡Eh,such thatUˉ≤τ,\min E_h, \text{such that} \bar U \le \tau, where EhE_h measures the high-fidelity prediction error and Uˉ\bar U measures the normalized change caused by the update.

The paper introduces SSI, which acts as an operator norm under weighted geometries, resulting in a bound of the form ∥AR∥≤SSIW∥R∥.\|AR\| \le \mathrm{SSI}_W \|R\|.

The method then uses spectral filtering based on the singular value decomposition of the relevant operator. Directions corresponding to large singular values are shrunk, with the goal of controlling predictor changes while improving predictive accuracy.

The experiments show that direct spectral filtering consistently but modestly improves over matched quadratic and scrambled controls under a tight update budget. However, the practical advantage is less clear when compared with established multi-fidelity predictors, and the experiments also reveal limitations in the Pareto trade-off, feature-learning setting, and transfer of the anchor certificate to held-out inputs.

Critical strengths

I have no major concerns regarding soundness. The main mathematical results appear correct, the experimental protocol appears appropriate, and I did not identify any significant technical errors.

  1. The central fixed-kernel mathematics appears correct and clean once the notation is reconstructed.

  2. The weighting scheme appears effective at removing sensitivity to anchor density. Across the anchor-density experiment, the coefficient of variation is 0.0057 for weighted SSI versus 0.478 for the unweighted quantity.

  3. The reported 22/22 and 21/22 results indicate consistency across the tested cases.

  4. The authors compare against established external multi-fidelity predictors rather than only against weak internal baselines.

  5. The paper is candid about unfavorable findings, including the weak neural interference correlation, the negative Pareto-frontier result, imperfect anchor-to-test transfer, and cases in which simpler alternatives perform better.

  6. The distinction between observed update and worst-case operator gain is conceptually useful.

Critical weaknesses

The paper is substantially harder to follow than necessary.

The main problem is not stated clearly. Some definitions appear to be missing or are very easy to miss, including WcW_c and WfW_f. The reader is therefore required to reconstruct, partly by inference, the main sequence of ideas:

\rightarrow A \rightarrow \mathrm{SSI}_W \rightarrow Q_\gamma \rightarrow \alpha \rightarrow \text{audit gate} \rightarrow \text{anchor/test transfer}.$$ Operators such as $\Theta_{cf}$ and $\Theta_{ff}$ are also introduced with insufficient explanation for readers outside the immediate kernel-literature context. The sequence above is my current understanding of the paper, but I am not fully confident that it matches the intended presentation of the authors. The experimental organization in Section 5 is also difficult to track. Tables 1, 2, and 3 appear to use closely related protocols and should be discussed more directly together. In particular, Tables 1 and 3 should be placed or discussed together to facilitate direct comparison. Sections 5.5--5.7 appear to address questions different from those in Sections 5.3--5.4. The authors should state more clearly at the beginning of each subsection what question the corresponding experiment is intended to answer. The formulation is sound, and the paper develops a coherent fixed-kernel framework for controlling predictor updates. However, I find the overall contribution limited. First, the theoretical ingredients are relatively elementary. The central SSI result is largely an induced operator-norm characterization, while the spectral result concerns monotonicity and optimal assignment of a fixed shrinkage spectrum. The theory also does not extend convincingly to general feature-learning dynamics. Second, the practical improvement demonstrated by the proposed method is limited. The gains over the matched internal controls are consistent but small, while stronger established multi-fidelity predictors remain more accurate in important comparisons. Third, the broader motivation of preserving a previously validated predictor is only partially supported by the current certificate. Section 5.7 shows that satisfying the anchor-level preservation constraint does not always imply satisfaction of the same numerical budget on held-out inputs. 1. **The main predictive gains are small.** At $\tau=0.20$, direct spectral filtering improves RMSE relative to quadratic regularization by 0.590% on QM7b and 0.241% on MF-PCBA. The 22/22 and 21/22 results show that these gains are consistent, but their magnitude remains small. 2. **External multi-fidelity predictors are substantially stronger in important comparisons.** Table 3 is unfavorable to the proposed method on QM7b. Direct spectral filtering has +11.6%, +11.8%, and +6.3% RMSE relative to exact NARGP, AR1, and composite MFNN, respectively. At the reported mean operating points, these external predictors also remain within the nominal update budget $\bar U\le 0.20$. 3. **SSI regularization does not improve the reported accuracy-update Pareto frontier.** Figure 2 shows that the proposed SSI regularization does not improve the reported frontier. Section 5.6 gives a related result: the quadratic SSI model wins 0/160 error comparisons and 64/160 update comparisons, while the normalized update differences are near zero. 4. **The anchor-level preservation certificate does not always transfer to held-out inputs.** Section 5.7 shows that satisfying the anchor-level update constraint does not necessarily imply satisfaction of the same numerical budget on test inputs. This weakens the practical interpretation of the preservation guarantee. 1. **Restriction to the fixed-kernel regime.** The theory is strongest in the fixed-kernel setting. The reported Spearman correlation with measured interference is only $0.163$ under feature-learning dynamics, suggesting that SSI may become a weak predictor once the model leaves this regime. 2. **Narrow empirical scope.** The main tight-budget results are demonstrated on only two datasets, QM7b and MF-PCBA. 3. **Limited transfer of the certificate.** The current theory directly controls the chosen anchor set, while the experiments show that the same numerical constraint does not always transfer to held-out inputs. This limits how broadly the preservation guarantee can be interpreted.
Questions for authors
  1. Table 3 appears to raise the most serious practical question. On QM7b, NARGP, AR1, and MFNN achieve substantially lower RMSE than the proposed direct spectral method, while their reported mean capped updates remain below τ=0.20\tau=0.20.

    Can the authors report the per-block test-feasibility rates of these external methods under the same τ=0.20\tau=0.20 constraint? If these methods also satisfy the update budget reliably, what practical advantage does the proposed method provide over them?

  2. Why does SSI regularization fail to improve the Pareto frontier?

    Section 5.5 states: The equal-sized frontier comparison gives a different result: test hypervolume changes by -2.61%

How should this negative result be reconciled with the motivation that controlling SSI should improve the accuracy-update trade-off? Under what conditions should one expect SSI regularization to improve the Pareto frontier rather than merely change the selected operating point?

  1. Section 5.7 raises an important transfer issue. The paper reports that all 160/160 pooled QM7b blocks satisfy the nominal anchor certificate, while only 133/160 satisfy the same numerical cap on held-out test inputs under the worst-case gate, with feasibility decreasing to 20/40 at the largest high-fidelity budget.

    Given these results, should SSI be interpreted only as a certificate on the chosen anchor set rather than as a guarantee over the retained-input distribution?

    A stronger result would be of the form Uˉanchor≤τ⟹Uˉtest≤τ\bar U_{\mathrm{anchor}}\le\tau\quad \Longrightarrow \bar U_{\mathrm{test}}\le\tau at least with high probability. The experimental results do not appear to establish such a transfer guarantee.

    What exactly is certified by the current theory? Is the guarantee only Uˉanchor≤τ\bar U_{\mathrm{anchor}}\le\tau, or are additional assumptions required for transfer to held-out inputs? How useful is the current certificate in practice?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.