acceptodds
Under review as a conference paper at ICLR 2027

Measuring Heterogeneous Adaptation: From Nominal Configuration to Intervention Dependence

Abstract

Modern frozen-backbone adaptation systems combine heterogeneous controls, including available transformation banks, conditional allocation, and learned prompt-derived states. For such systems, nominal capacity scalars do not identify mechanism usage, intervention dependence, or downstream utility. We study how nominal configuration, structural availability, conditional allocation, realized actuation, same-checkpoint dependence, and downstream utility should be measured separately. We instantiate the study in VEFT+, which exposes these controls to observation and intervention. By varying residual-transformation availability (), the number of subnetworks, and soft-prompt conditioning capacity (), the number of prompt states, we examine whether configurations with equal nominal products yield equivalent mechanism use or utility. Within the evaluated configurations, increasing either or does not produce a metric-independent ordering, learned nonuniformity does not consistently outperform uniform composition, and evaluation-time prompt removal is not equivalent to training without the prompt branch. Ultimately, nominal allocation, realized composition, intervention dependence, and downstream utility operate as distinct properties in dual-axis adaptation.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Heterogeneous adaptation, dynamic capacity control, frozen language models, conditional allocation, soft prompts, subnetwork weighting, same-checkpoint intervention, mechanism characterization

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_UZrDOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper investigates how to characterize heterogeneous adaptation mechanisms in frozen language models. The authors identify six core attributes: nominal configuration, the set of available transformations, conditional allocation, actual representational changes, intervention dependency within a given checkpoint, and utility for downstream tasks. Using VEFT+ as an experimental framework, the authors vary the number of sub-networks (KK) and the soft-prompt length (\texttt{sp_len}), comparing configurations with identical nominal products while conducting routing and soft-prompt intervention experiments on fixed model checkpoints. The primary experiments are based on the AlpaGAsus dataset, supplemented by analyses involving instruction generation, fact-based question answering, and cross-backbone model evaluations. The findings indicate that even when nominal products are identical, model performance varies across different metrics; furthermore, the non-uniform weights learned by the model do not consistently outperform uniform combinations. The paper also notes that zeroing out soft prompts during inference only captures dependencies at a specific checkpoint and does not reflect the impact of omitting such soft-prompt branches during the training phase. The primary contribution of this work is the proposal of a methodology for measuring and characterizing heterogeneous adaptation mechanisms.

Critical strengths
  • The intervention-based experimental design is a notable strength (presented in Tables 2 and 3). By adjusting routing combinations or soft-prompt values ​​while keeping the trained model and evaluation conditions constant, the authors effectively mitigate the confounding effects stemming from different training trajectories. Furthermore, the authors clearly distinguish between setting soft prompts to zero during inference and disabling specific branches during training, explicitly noting that these two approaches address distinct scientific questions.

  • The research problem itself is highly significant. The paper argues for the independent examination of nominal capacity, actual component utilization, intervention dependencies, and task utility. The experimental results in Table 2 demonstrate that the non-uniform combinations learned by the model do not outperform uniform combinations across all metrics; this finding serves as a compelling illustration that the effectiveness of underlying mechanisms cannot be judged solely by the number of components or the activation frequency of routing paths.

Critical weaknesses
  • There are contradictions in the evidence for the core mechanisms, and the validity of the measurement framework has not been sufficiently verified. In Figure 3, a set of configurations is labeled with a fixed K=1K=1, yet it yields Neff>1N_{\mathrm{eff}} > 1 and p1max⁡<1p1_{\max} < 1. If these metrics describe normalized allocations within a single sub-network, both should theoretically equal 1. Furthermore, the main experiments only compare results across multiple random seeds under a limited set of conditions, and no empirical data is provided for the actual representation shift (UactU_{\mathrm{act}}) within the six-attribute framework. Relying solely on the existing task-based metrics is insufficient to fully explain the underlying mechanisms.

  • A lack of necessary technical details hinders the reproducibility of the experiments. Equation (2) fails to fully specify the computational relationships between the sub-networks, soft prompts, and the routing mechanism. The specific procedures for the sequence-mean and zero-prompt interventions are not clearly described. The main experiments omit critical configuration details, such as learning rates, batch sizes, total training steps, data splitting strategies, and parameters related to decoding. Appendix C contains only an introductory paragraph and fails to provide the operational definitions promised in the main text, making it difficult for others to independently replicate the experiments and verify the results.

  • The layout of figures and the Related Work section require improvement. In Figure 1, text and graphical elements overlap, and the caption at the bottom extends beyond the page margins. The "Related Work" section merely lists existing methods and repeatedly discusses the connections between KADA, DHKR, and VEFT, without focusing on the measurement issues central to this paper or clearly distinguishing existing research from the authors' own contributions. The main text frequently reiterates the research motivation while providing insufficient discussion on the core implementation and experimental analysis, which obscures the paper's actual contributions.

Questions for authors
  • Mechanism Metrics and Experimental Setup. What are the precise definitions of NeffN_{\mathrm{eff}} and p1max⁡p1_{\max} in Figure 3? When KK is fixed at 1, why do we observe an effective sub-network count greater than 1 and a maximum routing weight less than 1? Do the curves in the figure indeed correspond to the experimental setting mentioned in the main text where expansion (scaling) is disabled?
  • Technical Details and Reproducibility Protocol. Is there missing content in Appendix C? Please provide the complete forward-pass procedure, all intervention operations, and the specific configurations used for training and evaluation. To clarify: when soft prompts are zeroed out, does the model recompute the routing assignments, or does it continue to use the original routing weights?
  • Specific Contributions of the Measurement Framework. What are the operational definitions and experimental results for each of the six categories of attributes? In particular, where are the empirical results for UactU_{\mathrm{act}} presented in the paper? Compared to standard configuration comparisons or component ablation studies, what specific characteristics can this framework uncover that are otherwise difficult to identify?
  • Statistical Stability of Experimental Results. How robust are the rankings of different configurations across the metrics shown in Table 1? Given the three sets of random seeds used, could you provide paired differences or an appropriate uncertainty analysis to verify that these ranking variations are not merely due to fluctuations in the mean values ​​across different runs?
Reviewer_6h14OpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper investigates how subnetwork count and soft-prompt length affect LLM adaptation, using different configurations and controlled interventions to study routing behavior, model performance, and efficiency.

Critical strengths
  • The paper raises a reasonable question about how different adaptation components contribute to model behavior and whether nominal capacity reflects their actual utilization.

  • The use of same-checkpoint interventions is a useful approach to understanding the contribution of individual components without retraining the entire model.

  • The experimental evaluation considers multiple aspects, including generation quality, routing behavior, calibration, and computational efficiency.

  • The authors are careful in acknowledging several limitations and reporting findings where learned routing or soft prompts do not necessarily improve performance.

Critical weaknesses
  • The paper is quite difficult to read and follow. It introduces many terms and concepts without sufficient background or intuitive explanations. The presentation seems to assume that readers are already familiar with VEFT, DHKR, and their underlying mechanisms. As a result, it is difficult to understand the motivation, the proposed framework, and even some of the main claims.

  • I find the novelty rather limited. The central observation that different adaptation components affect model behavior differently seems fairly intuitive. It is not clear what fundamentally new understanding this work provides beyond experimentally confirming something that one might reasonably expect.

  • I am particularly unconvinced by the motivation for defining the nominal capacity as the product of two different parameters, K and sp_len. Unless there is a mathematical or architectural justification for combining them this way, why should we expect different configurations with the same product to behave similarly? The observation that different combinations produce different results seems unsurprising.

  • The significance of the findings is unclear. Even if we accept the experimental observations, what should researchers or practitioners learn from them? The paper does not provide a clear design principle, actionable insight, or guidance on how to allocate adaptation capacity more effectively. This makes the overall contribution appear rather incremental.

  • The experimental evidence could be stronger. Several comparisons rely on a small number of seeds or individual runs, and some reported performance differences are quite small. Certain numerical results and parameter accounting, particularly the total parameter counts underlying Table 6, also need clearer explanations to make the comparisons convincing.

  • I find the novelty and significance of this paper below average for ICLR. The measurement perspective is potentially useful, but the paper needs a much clearer motivation, better presentation, and more compelling evidence that its findings offer something beyond expected differences between architectural configurations.

Questions for authors
  1. What is the mathematical or architectural justification for defining nominal capacity as K × sp_len? Since these two parameters control fundamentally different components, why should we expect configurations with the same product to behave similarly in the first place?

  2. Beyond showing that different configurations produce different results, what is the main new insight researchers should take away from this study? How can the findings help someone make better decisions when designing or configuring an LLM adaptation system?

Reviewer_F6C4OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

VEFT+ investigates whether nominal configuration adequately characterizes heterogeneous adaptation in frozen-backbone models. By varying subnetwork count and soft-prompt length, this work compares configurations with equal nominal products and performs same-checkpoint routing and prompt interventions. Experiments primarily use AlpaGAsus, supplemented by instruction-following and factual-QA evaluations. Results motivate separating configured capacity, realized allocation, intervention dependence, and downstream utility. However, insufficient methodological specification, questionable quantitative reporting, and substantial presentation problems weaken the contribution.

Critical strengths

S1. Distinguishing cross-configuration comparisons from same-checkpoint interventions is useful. In particular, evaluating a checkpoint with its prompt removed measures dependence on that prompt, rather than the effect of training without the prompt branch.

S2. Three-seed comparisons and multiple quality metrics provide some empirical support. Explicit acknowledgment that broader sweeps remain descriptive and that VEFT+ does not consistently outperform external baselines appropriately limits the claims.

Critical weaknesses

W1. Using K \times \text{sp_len} as the central comparison variable lacks sufficient justification. Section 4.1 states that this product represents neither parameter count nor compute, memory, or information capacity. Demonstrating different behavior among configurations sharing this product offers limited insight without explaining why it should be sufficient or comparing it against more meaningful capacity measures.

W2. Figure 3 contains results inconsistent with the stated configuration. Its K=1, sp=4K=1,\ \text{sp}=4 curve shows an effective subnetwork count greater than one and a maximum routing weight below one. Both quantities should equal one for a normalized distribution over a single available subnetwork. Since capacity growth is reportedly disabled, these curves require correction or a precise explanation.

W3. Insufficient implementation details prevent reproduction. Equation 2 leaves Δ\Delta abstract without adequately defining the subnetworks, routing computation, or prompt integration. Learning rates, batch sizes, data splits, and evaluation sample counts are also missing. Appendix C promises operational definitions but provides only an introductory paragraph.

W4. Evidence for metric-dependent allocation preferences remains weak. In Table 1, BLEU scores for (2,2)(2,2) and (4,1)(4,1) are 0.0771±0.00070.0771\pm0.0007 and 0.0774±0.00200.0774\pm0.0020, respectively. Different rankings of sample means do not establish robust trade-offs. Paired statistical analysis and additional seeds are needed, especially because larger-product sweeps use only one run.

W5. Additional reporting inconsistencies raise concerns about quantitative reliability. Table 6 lists approximately 638 million trainable parameters as 10.97%, implying a denominator of roughly 5.82 billion, whereas the BitFit and AdaLoRA rows imply approximately 8.04 billion. A consistent denominator or explanation is needed. Section 5.2 also describes early- and late-stage results, although Table 4 reports only Epoch 1.

Questions for authors

See weaknesses

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.