acceptodds
Under review as a conference paper at ICLR 2027

Model Replacement under Limited Adaptation

Abstract

When a model is retired, unavailable, or too costly to call, one may still want to reproduce its behavior by querying other available models and retaining a small amount of target-specific information that specifies how their responses should be combined. Which available models provide the best replacement when this retained information is limited? We show that the answer can depend fundamentally on the budget. Even when the original model is known completely, two sets of available models can achieve exactly the same replacement error with unrestricted adaptation yet reverse their optimal ordering when the retained state increases from one to two bits, with arbitrary encoders and nonlinear predictors allowed. Replacement error separates into an irreducible component and the error of compressing the target behavior predictable from training observations and the available models’ responses. Under Gaussian predictable behavior, the covariance spectrum of this behavior determines the entire information–error tradeoff and gives a necessary and sufficient condition for one model set to dominate another at every budget. Ex- periments with language models on three multiple-choice tasks show that matching retained-state budgets changes both adapter comparisons and which models are selected, while compact states can preserve much of the attainable performance: on Gemma/HellaSwag, 848 target-specific bits achieve test total variation 0.1951, compared with 0.1933 for full-precision kernel ridge regression.

Primary area:
learning theory
Keywords:
model replacement, model evaluation, limited adaptation, model ecosystems, information-constrained learning, kernel methods

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_X4HDOpenReview ↗
Rating
2(+1)
Confidence
1
Informativeness
2(+1)
Presentation
1
Soundness
2(+1)
Read review
Paper summary

The paper investigates scenarios when a model becomes unavailable but an user would like to obtain similar outputs from the other models. The replacement uses a merge of the outputs of alternative models, and the merge requires us to save some information (e.g., terms of the regression). The paper discusses the behavior of optimal solutions of such merges, especially, how largely it changes depending on the size of conserved information.

Critical strengths
  • The background of the problem, i.g., how to imitate outputs of a model from those of other models, is of interest
  • The paper consists of a detailed theory and related experiments
Critical weaknesses
  • The target problem discussed in the paper seems very unrealistic or less important. In my understanding, the discussion is related to scenarios where we want to, literally, minimize the size of the conserved information (e.g., less than 1-1000 bits?).
  • The presentation of the manuscript is still not very good and difficult to follow despite careful reading. I'm not really a specialist in this theory field. Even so, with that in mind, I still believe the explanations, even in the introduction, are not clear, a part of which might be derived from the gap between the discussions and practical advantages (related to weakness 1).
Questions for authors

Is there a practical use case, where the finding of this paper is useful? At least, the experiments in the paper does not apply to it, as a system for the router or orchestrator of LLMs may have standard capacity of memory, storage, and compute (much larger than 1-1000 bits). Users would maximize the performance for model replacement with more representation power from more conserved information.

Although I may be missing an important point, it seems intuitively natural for me that the optimal model choice can change with the allowed storage budget. Apart from its practical relevance, could you clarify what makes the finding or result surprising or nontrivial beyond this intuition?

1 comment

I do not agree with this review or that the "presentation is poor and very difficult to follow," it seems that this reviewer did not have the expertise to actually evaluate this paper (stated in confidence score). In my opinion this is a decently written paper (even if my review argues it is too incremental in current form).

Reviewer_xyZxOpenReview ↗
Rating
2(−1)
Confidence
1(−1)
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies model replacement using peer model's responses under limited target-specific storage. It proves that optimal peer rankings can reverse as the storage budget changes and characterizes the error–budget tradeoff with gaussian assumptions. Experiments across three llm targets and three tasks demonstrate budget-dependent adapter and peer selection, with compact states preserving much of the full-precision performance.

Critical strengths
  • The theory provides profound insights. Thms show that, even with full access to target information and no restrictions on the classes of encoders and predictors, the optimal ranking of peers can reverse as the retained-state budget changes.
Critical weaknesses
  • The spectral characterization relies on assumptions 'gaussian predictable behavior" and SE, but the experiments evaluate answer distributions using "total variation." While the results show that rankings shift with the budget, they do not directly verify the spectral explanatory mechanism.

  • Peer-derived bases and fitting responses are excluded from the target-specific budget. This does not establish the total storage cost of replacing a single target.

Questions for authors
  • Can the existing results clarify whether the observed ranking reversals reflect the proposed behavioral-spectrum mechanism or differences in adapter optimization and regularization?

  • What is the size of the shared infrastructure excluded from the budget, and how many replaced targets are needed to amortize this cost?

3 comments

Hi xyZx,

The end of the rebuttal period is approaching. I was wondering if you could go into more detail on the profound insights the theory provides and how you position this paper's theory within the existing literature.

sorry for late reply. I am not very familiar with this research area, so my confidence score is 2. I gave the paper a rating of 3 because, with the assistance of an LLM, I was able to understand most of its content. Compared with other submissions I reviewed, I find this paper relatively well written, appreciate its problem setting, and consider its theoretical analysis relatively complete. I will further consider the other reviewers' scores and comments when reassessing my evaluation.

Can you post me some of what the LLM provided you if possible? Specifically, the arguments the LLM came up with that "its theoretical analysis relatively complete." I would like to consider if I should increase my score. This also isn't my area of expertise to be honest.

I would also appreciate what the AC has to say on this paper.

Reviewer_9HySOpenReview ↗
Rating
2(+1)
Confidence
1(−1)
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies approximating a target model from callable peer responses while limiting the target-specific information retained after training. Its main result constructs two peer interfaces with equal unrestricted replacement error whose exact optimal ordering reverses between one and two retained bits, despite complete target knowledge and arbitrary encoders and nonlinear predictors. It reduces squared-error replacement to quantization of predictable target behavior and derives a Gaussian error curve and an all-budget comparison criterion under a mutual-information constraint. Experiments compare encoded convex, linear, random-feature, kernel-ridge, and MLP adapters on multiple-choice response distributions, reporting budget-dependent comparisons, changes in validation-selected peers, and compact states that approximately preserve full-precision adapter performance.

Critical strengths

Theorem 1 demonstrates that complete target knowledge removes training uncertainty, equal unrestricted error controls for eventual predictability, and the optimization over arbitrary encoders and nonlinear predictors removes adapter-specific optimization as an explanation. The proof also establishes minimax optimality, and independent enumeration of the finite quantization problem reproduces the stated risks. This is a useful conceptual separation: equal eventual replacement accuracy does not determine which interface is preferable under a finite storage constraint.

Critical weaknesses

1. The theoretical synthesis makes sense, but it is an incremental contribution and the demonstrated consequences are limited.

A synthesis can make an important contribution by resolving a consequential ambiguity, deriving a broadly useful characterization, or explaining an observed phenomenon. This paper provides some such value, especially through Theorem 1, but I do not find that value sufficiently substantial beyond the specific separation it establishes.

There are close antecedents to the underlying formulation. Gao et al., Rate Distortion For Model Compression: From Theory To Practice (ICML 2019, Section 3), already define compression distortion through differences in model predictions rather than parameter reconstruction. Hafez-Kolahi et al., Rate-Distortion Analysis of Minimum Excess Risk in Bayesian Learning (ICML 2021, Equation 12 and Section 6), constrain information retained from training in a learned predictor and express squared prediction distortion in an L2L^2 function space. Both are cited in the manuscript, but the relationship needs to be explained at the level of the mathematical formulations, not only their general subject matter.

Within this setting, Lemma 2 applies the established conditional-mean estimation-and-quantization structure to functions of peer responses; a closely related finite-quantizer decomposition is stated in Proposition 1 of On Noisy Source Vector Quantization via a Subspace Constrained Mean Shift Algorithm. Once Gaussian predictable behavior is expanded in orthonormal coordinates, Theorem 3 is Gaussian vector rate-distortion. Theorem 4 then derives the explicit all-budget comparison through concave duality. The latter is not empty: it supplies a necessary-and-sufficient criterion within the Gaussian model and covers cases beyond coordinatewise eigenvalue dominance. Nevertheless, I view it as a semi-interesting direct corollary of the known error curve, rather than a separate foundational advance. I am not asserting that the exact finite-state reversal or this particular spectral transform has already been published.

The clearest additional inference is therefore that unrestricted accuracy cannot serve as a budget-independent ordering, even after eliminating training uncertainty and restrictions on predictor expressiveness. This strengthens the rationale for budget-matched evaluation. However, the broader characterization remains specific to a Gaussian mutual-information model. It does not supply the corresponding general comparison for literal finite-state storage; the paper correctly states that ordering the relaxed risks does not order the fixed-state risks. Proposition 5 provides a more general finite-state lower bound, but not an all-budget ordering criterion for that problem.

Nor do the experiments establish an additional explanatory consequence of the spectral analysis. They neither estimate the covariance of predictable behavior across targets and training observations nor test a quantitative prediction derived from it. The empirical KRR eigenbasis comes from the peer kernel matrix, not that behavioral covariance, and the comparisons use TV rather than squared prediction loss. These distinctions are acknowledged and do not invalidate the theory. They do mean that the empirical crossovers cannot currently be credited as evidence that the spectral characterization explains the observed model preferences.

I therefore regard the work as a sound conceptual separation accompanied by useful specializations of established compression theory, but a very incremental theoretical advance. There needs to be some further inference from the synthesis, or strong independent empirical evidence that its selection perspective matters in the proposed replacement setting.

2. The experimental protocol has a significant sampling defect, and the comparisons do not establish the additional peer-selection benefit needed to strengthen the contribution.

The most serious empirical issue is the distribution used for fitting, selection, and evaluation. In kernel_isqed/data.py:417–423, clusters are sorted by domain|gold, with a hash only as a secondary key. After minimum coverage is allocated, lines 481–497 fill residual capacity from the same sorted records. Thus, alphabetically early domains consume the residual sample budget. In the released MMLU-Pro test set, 996 of 1,024 items come from six included domains, while each of the remaining seven contributes exactly four. The manuscript does not specify a target query distribution that justifies this allocation.

The HellaSwag records establish a validation/test weighting discrepancy directly. Sixty-six domains have exactly four test examples each, accounting for 264/1,024 = 25.8% of test. Because the allocator requires at least four examples from every included domain in each partition, those same domains occupy at least 264/512 = 51.6% of validation. Validation therefore evaluates substantially different domain weights from the nominally IID test, by construction.

This matters specifically to the paper's selection claims: validation jointly selects peers, regularization, rank, precision, and encoding, so the selected systems can be optimized for a different mixture from the one on which their performance is reported. The published Qwen/MMLU-Pro crossover reproduces from the released losses, and the defect does not prove that it would disappear under corrected sampling. It does, however, prevent the current protocol from establishing reliable same-distribution selection. Bootstrapping fixed test predictions cannot repair a mismatched selection distribution; fitting and selection need to be repeated under a consistently specified distribution.

There is also a problem in the direct evidence for peer selection. The prominent convex-versus-KRR crossover compares adapter families using the same peers. Table 2 is closer to the theoretical interface comparison, and the released records support its literal claim that the validation-selected pair changes with budget. However, both competing pairs' held-out errors are not reported at both budgets. Consequently, the study does not quantify whether changing peers improves held-out performance over retaining a fixed pair and changing only the adapter. This missing comparison is consequential because choosing the peer interface, rather than merely choosing a regressor, motivates the paper.

The inexpensive calibration comparator is likewise important. Appendix C.6 reports mean TV 0.2892 from three nonnegative coefficients, compared with 0.3028 for convex mixing and 0.2784 for full-precision KRR, recovering approximately 55.6% of that improvement. Yet it is absent from the matched-budget comparison. Three FP64 coefficients plus the stated 176-bit header would require 368 bits, assuming a public decoder for this family. It fits below 1,024 bits without low-bit quantization. A per-cell, budget-matched evaluation is therefore needed to distinguish the additional benefit of richer adaptation from inexpensive logit rescaling. The aggregate result does not establish that calibration eliminates the reported crossover, but it leaves the practical comparison incomplete.

The 848-bit result remains useful evidence that a fitted replacement can be compressed with little change in its measured fidelity. Its reference, however, is full-precision KRR—not an established irreducible error floor or an application-defined replacement requirement. Together with the adapter-family crossover, it supports compressibility and budget sensitivity of particular fitted methods, but does not independently establish a reliable benefit from budget-dependent peer choice.

Overall, I recommend Reject. I find the incremental theoretical contribution modest, and the empirical study does not yet supply the reliable selection insight that would make the combined contribution compelling. Correcting the split protocol is necessary; merely reproducing the convex-versus-KRR crossover and fixing the split protocol would not fully resolve the contribution concern. A stable held-out benefit from budget-aware peer selection beyond inexpensive alternatives (note that I will be rigorous in analyzing your reply), or a clearer substantial consequence of the theoretical synthesis, would materially change my assessment.

I aim to reply every 24-48 hours during review period. Please submit a https://anonymous.4open.science for all claims made in rebuttal to me.

Questions for authors
  1. On the theoretical contribution — How do the function-space reduction and information-constrained formulation differ precisely from prediction-oriented rate–distortion in Gao et al. (2019) and Hafez-Kolahi et al. (2021, Equation 12 and Section 6)? I credit Theorem 1’s removal of training uncertainty and adapter restrictions, and Theorem 4’s explicit Gaussian comparison. Which consequence of the combined results provides the principal advance over applying existing rate–distortion theory separately to fixed interfaces? Please distinguish consequences established within the Gaussian relaxation from claims about literal finite-state storage and the real-model experiments.
  2. Sampling and selection — validation of the current empirical claims. What query distribution is the split builder intended to represent? Do the main comparisons persist after complete refitting and validation selection on cluster-disjoint partitions with consistent, explicitly justified domain weights, repeated across split seeds? Please include the Qwen/MMLU-Pro budget comparisons and the Gemma/HellaSwag compact-state result, and provide domain/label counts for all partitions. Reweighting the existing test losses alone would not resolve the selection issue.
  3. Peer-selection benefit — missing comparisons within the current setup. Under the corrected protocol, what are both Table 2 peer pairs’ held-out errors at each displayed budget, with paired differences and uncertainty? How much is gained by changing peers rather than retaining a fixed pair while adapting its encoding? Please include the encoded three-coefficient calibration at matched budgets and call counts, with per-target/task results and appropriate predetermined zero-state references. The matched response arrays and scripts for the C.6 calibration would allow these comparisons to be independently checked. These requests concern validation of the existing claims rather than a new benchmark campaign.
  4. Is there anyway you can show something directly consequential resulting from the paper's theory, besides those I have mentioned in questions 1-3?

Comments on the paper

7 comments

I agree that this paper isn't easy to understand why it matters (which is why I gave a reject too). I think we are disagreeing about types of writing quality.

"transition of discussions in the paper was a bit abrupt, especially in terms of connections to practical scenarios and benefits" - I fully agree with this, I also feel like the paper does not do a very good job on the practical scenarios and benefits.

In my opinion, the individual axioms are clear; the flow of where the axioms/theory connect and who cares about how all of these axioms/theory combine are clear but to me they look like most of this paper's work is just minor specializations of existing work.

I hope the authors can address better where this entire flow all synthesizes to in the end that makes this a solid ICLR contribution. I defer to the authors of this paper to provide a better answer for that.

Thank you—writing the objective as ΦB⋆∈arg⁡min⁡Φ∈FqDΦ(B)\Phi_B^\star \in \arg\min_{\Phi\in\mathcal F_q}D_\Phi(B) makes the intended joint-selection question clearer. I agree that Theorem 1 establishes a genuine distinction between unrestricted replacement error and finite-budget adaptation efficiency.

What I would like to understand more precisely is the role of the admissible family Fq\mathcal F_q. A reversal between two interfaces does not necessarily imply that the best available interface changes with the budget: another feasible interface may remain optimal throughout.

For example, in Theorem 1, if all six coordinate peers can be freely combined into three-peer sets, the mixed interface (X1,X4,X5)(X_1,X_4,X_5) has energies (6,5,4)(6,5,4) and risks (20,14,9,5)(20,14,9,5) at budgets 0,1,2,≥30,1,2,\geq3. It remains optimal across budgets even though the two displayed interfaces cross. This does not contradict the theorem—restricting the family to the two displayed interfaces is valid—but it makes the feasible-family assumption important to the proposed joint-selection interpretation.

Appendix A.6 seems a useful place to clarify the broader consequence. What does the structure of the available interfaces, through HΦH_\Phi and Σlearn\Sigma_{\mathrm{learn}}, tell us about when one fixed selection remains optimal and when budget-dependent selection is genuinely necessary? A result establishing either outcome could be informative. I am asking for the strongest consequence supported by the submitted framework, not requiring a new algorithm or a general finite-bit theory during rebuttal.

The empirical items in Questions 2 and 3 remain important for the same reason: corrected sampling and refitted selection, both Table 2 peer pairs evaluated at each budget on held-out data, and the inexpensive calibration comparator. These would help distinguish a change in the validation winner from a reliable benefit of changing the deployed peer set.

Please do keep pushing the rebuttal—I do not regard my current score as fixed, and information-theory papers can sometimes move surprisingly quickly from a reject to a strong accept when a sharper formulation brings out the real contribution. A clearer account of what the joint formulation establishes, together with evidence addressing the empirical concerns, could substantially change my view. Again, I will reply every day or so as you iterate.

Thank you for clarifying the motivation and the relationship to Gao et al. and Hafez-Kolahi et al.

I agree that changing Φ\Phi changes the predictable behavior being compressed. The (6,2,2)(6,2,2) versus (5,4,2)(5,4,2) example removes the unrestricted-error tie, while your earlier four-peer example addresses the distinction between pairwise crossings and changes in the global optimum. I credit these as stronger existence results.

My concern remains their significance. You have shown that choosing peers under unrestricted adaptation can produce a worse choice under a finite-state constraint. But if the deployment budget is known, evaluating candidate peer sets at that budget is already the natural selection procedure. What additional conclusion about model replacement follows from your theory?

Theorem 4 provides an exact all-budget comparison for Gaussian predictable behavior, which I recognize as a useful characterization. But it follows from established Gaussian rate-distortion and duality, assumes the relevant behavioral spectra are specified, and does not give the corresponding comparison for literal finite-bit storage. I do not yet see a sufficiently substantial advance beyond the counterexamples and this specialization of existing theory.

Your service example makes the setting plausible, but it does not establish the importance of the proposed constraint. The full-precision adapters in your experiments require tens to hundreds of kilobytes per target, while the peer models are shared and remain necessary at inference. What scale or deployment requirement makes reducing this state consequential enough to change which peers should be called?

I appreciate your clarification that the experiments do not verify the spectral explanation. They could still establish a separate empirical contribution. The decisive question is whether budget-aware peer selection gives a meaningful held-out advantage over keeping a fixed peer set and retuning its adapter at each budget, under corrected sampling and against the inexpensive C.6 calibration baseline.

I will assess your response to Weakness 2 on that basis. If your response seems strong, I will proceed to a literature checks to see how it differs from existing papers. The stronger constructions improve the theoretical case, but I still find the incremental contribution limited. Correcting the experiments is necessary; demonstrating another crossover alone would not resolve my concern about what the paper adds.

(some of what is above is polished by an LLM; what is below is written totally by me)

The good thing for your paper is you have many paths that could exist to a full ICLR contribution here. You can either target the theoretical contribution for a theoretical breakthrough in information theory and/or an empirical contribution that will require you to show something real. I urge you to pay attention to the entire chain of evidence that creates a conceptional breakthrough in the theory or getting something working in deployment that matters.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.