- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper investigates how to characterize heterogeneous adaptation mechanisms in frozen language models. The authors identify six core attributes: nominal configuration, the set of available transformations, conditional allocation, actual representational changes, intervention dependency within a given checkpoint, and utility for downstream tasks. Using VEFT+ as an experimental framework, the authors vary the number of sub-networks () and the soft-prompt length (\texttt{sp_len}), comparing configurations with identical nominal products while conducting routing and soft-prompt intervention experiments on fixed model checkpoints. The primary experiments are based on the AlpaGAsus dataset, supplemented by analyses involving instruction generation, fact-based question answering, and cross-backbone model evaluations. The findings indicate that even when nominal products are identical, model performance varies across different metrics; furthermore, the non-uniform weights learned by the model do not consistently outperform uniform combinations. The paper also notes that zeroing out soft prompts during inference only captures dependencies at a specific checkpoint and does not reflect the impact of omitting such soft-prompt branches during the training phase. The primary contribution of this work is the proposal of a methodology for measuring and characterizing heterogeneous adaptation mechanisms.
Critical strengths
-
The intervention-based experimental design is a notable strength (presented in Tables 2 and 3). By adjusting routing combinations or soft-prompt values while keeping the trained model and evaluation conditions constant, the authors effectively mitigate the confounding effects stemming from different training trajectories. Furthermore, the authors clearly distinguish between setting soft prompts to zero during inference and disabling specific branches during training, explicitly noting that these two approaches address distinct scientific questions.
-
The research problem itself is highly significant. The paper argues for the independent examination of nominal capacity, actual component utilization, intervention dependencies, and task utility. The experimental results in Table 2 demonstrate that the non-uniform combinations learned by the model do not outperform uniform combinations across all metrics; this finding serves as a compelling illustration that the effectiveness of underlying mechanisms cannot be judged solely by the number of components or the activation frequency of routing paths.
Critical weaknesses
-
There are contradictions in the evidence for the core mechanisms, and the validity of the measurement framework has not been sufficiently verified. In Figure 3, a set of configurations is labeled with a fixed , yet it yields and . If these metrics describe normalized allocations within a single sub-network, both should theoretically equal 1. Furthermore, the main experiments only compare results across multiple random seeds under a limited set of conditions, and no empirical data is provided for the actual representation shift () within the six-attribute framework. Relying solely on the existing task-based metrics is insufficient to fully explain the underlying mechanisms.
-
A lack of necessary technical details hinders the reproducibility of the experiments. Equation (2) fails to fully specify the computational relationships between the sub-networks, soft prompts, and the routing mechanism. The specific procedures for the sequence-mean and zero-prompt interventions are not clearly described. The main experiments omit critical configuration details, such as learning rates, batch sizes, total training steps, data splitting strategies, and parameters related to decoding. Appendix C contains only an introductory paragraph and fails to provide the operational definitions promised in the main text, making it difficult for others to independently replicate the experiments and verify the results.
-
The layout of figures and the Related Work section require improvement. In Figure 1, text and graphical elements overlap, and the caption at the bottom extends beyond the page margins. The "Related Work" section merely lists existing methods and repeatedly discusses the connections between KADA, DHKR, and VEFT, without focusing on the measurement issues central to this paper or clearly distinguishing existing research from the authors' own contributions. The main text frequently reiterates the research motivation while providing insufficient discussion on the core implementation and experimental analysis, which obscures the paper's actual contributions.
Questions for authors
- Mechanism Metrics and Experimental Setup. What are the precise definitions of and in Figure 3? When is fixed at 1, why do we observe an effective sub-network count greater than 1 and a maximum routing weight less than 1? Do the curves in the figure indeed correspond to the experimental setting mentioned in the main text where expansion (scaling) is disabled?
- Technical Details and Reproducibility Protocol. Is there missing content in Appendix C? Please provide the complete forward-pass procedure, all intervention operations, and the specific configurations used for training and evaluation. To clarify: when soft prompts are zeroed out, does the model recompute the routing assignments, or does it continue to use the original routing weights?
- Specific Contributions of the Measurement Framework. What are the operational definitions and experimental results for each of the six categories of attributes? In particular, where are the empirical results for presented in the paper? Compared to standard configuration comparisons or component ablation studies, what specific characteristics can this framework uncover that are otherwise difficult to identify?
- Statistical Stability of Experimental Results. How robust are the rankings of different configurations across the metrics shown in Table 1? Given the three sets of random seeds used, could you provide paired differences or an appropriate uncertainty analysis to verify that these ranking variations are not merely due to fluctuations in the mean values across different runs?