acceptodds
Under review as a conference paper at ICLR 2027

Effective Does Not Mean Useful: Conditional Functional Substitutability for Redundancy and Scaling in Transformers

10 views

Abstract

Modern neural networks scale predictably, yet the mechanisms behind these regularities remain unclear. Neural redundancy is typically characterized by component importance or representational similarity, both indirect proxies. We view redundancy as an input-conditioned, dynamic relation: intermediate computational states are functionally redundant when they induce similar downstream responses. We introduce Conditional Functional Substitutability (CFS) to directly characterize such functional substitution. CFS exposes functional relations and reduction potential missed by conventional importance- and similarity-based measures. Across modalities and Transformer families, CFS reveals systematic functional reorganization with scale. Controlled scaling further shows that performance gains need not track growth in substitutability, while fixed-capacity models with more independent functional structure perform better, providing a functional account of diminishing returns. Predicted CFS further enables dynamic computation with a better performance–computation trade-off than importance-based component selection, suggesting new directions for redundancy-aware computation and more efficient model scaling.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Conditional Functional Substitutability, Neural Redundancy, Transformer Scaling, Functional Organization, Adaptive Computation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_UWwPOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

There are two branches of study: neural scaling laws and neural redundancy. This paper proposed a method to measure the functional redundancy of attention heads by silencing certain heads and measuring the performance change. They can also measure how much other heads can compensate for the silenced head. Across model families, larger models tend to have more diverse heads but also increasing redundancy, potentially explaining the diminishing returns of scaling laws. Based on the new metric, the paper can also prune the model better by removing the redundant heads, reaching better performance than pruning based on importance-based metrics.

Critical strengths
  1. The metric is creative and intuitively reasonable. Empirical results support the direction of the claims. I think such a metric can be potentially useful.
Critical weaknesses
  1. In the paper, the actual analysis focused on the redundancy of attention heads. A similar idea may be easily generalized to MoE. But it is hard for me to imagine how to generalize it to other modules like MLP. The method is therefore too specific and has a limited scope.
  2. Scientifically, although it is appealing to connect increasing redundancy to diminishing returns of scaling laws, the paper does not provide quantitative evidence for this connection, like predicting scaling laws from redundancy or vise versa. The connection between redundancy and scaling laws is not convincing. And defining redundancy only helps to describe what we have, but not to predict what changes can reduce the redundancy.
  3. Beyond scientific contribution, the practical usage of the metric also feels limited.

In general, the metric is interesting, but the current paper's limitations make it hard for me to recommend it for acceptance now.

Questions for authors
  1. Line 146, when first introducing the output discrepancy, D(⋅,⋅)D(\cdot, \cdot), please provide more details. No idea what this function is.
  2. By reducing the redundancy, can we improve the coefficients or exponents of the scaling laws? What is the theoretical connection between redundancy and scaling laws?
  3. Based on your results, can you provide an explanation of why redundancy arises? Can you predict what kind of changes (e.g., architectures, optimizers) can reduce the redundancy?
4 comments

We thank the reviewer for recognizing CFS's novelty and potential. We address the concerns about generality, scaling laws, and practical utility with new experiments and clarifications.

1. Beyond attention heads

CFS applies to intervenable components with measurable downstream effects; attention heads offer a convenient additive decomposition, but are not essential.

We additionally applied the same protocol to grouped MLP neurons in ViT-B (layers 6/11; 8/16 groups; 64 inputs; Appendix C.4, Table 10). Raw Oracle is 0.6120/0.6856 and 0.5221/0.5813, respectively. These groups also exhibit input-dependent substitution (cross-input SD 0.232–0.283) and directionality (mean directional gap 0.146–0.186). These properties persist under alternative groupings, although recovery magnitude varies with grouping. This empirically extends CFS beyond attention heads.

2. Functional redundancy and scaling laws

As outlined in Discussion, predicting scaling-law coefficients or exponents and ultimately improving scaling curves by shaping functional organization are long-term research goals. This work takes an important step toward them by directly measuring functional substitutability and quantitatively examining its relationship with model capacity and parameter efficiency.

In controlled GPT-2 scaling (Table 4), starting from the same W768/D12 baseline, widening to W1152 increases Raw Oracle by 0.0397, while deepening to D18 increases it by only 0.0029. Consistent with this difference, depth scaling achieves approximately 23% greater validation-loss reduction per added parameter.

At identical parameter count (124.44M; Table 5), changing head decomposition from 6×128 to 24×32 raises Raw Oracle from 0.2012 to 0.2583, reduces normalized effective functional rank from 0.9507 to 0.8725, and increases validation loss from 3.5707 to 3.5780.

New Matched-N and fixed-K Cover/Rank experiments (Figure 2; Appendix Figure 7) further show that the natural-family functional-organization trends persist after controlling for candidate count and graph size.

These controlled results provide quantitative empirical evidence relating functional substitutability to parameter efficiency, though they do not yet establish a causal prediction of scaling-law exponents. Deriving such predictions and testing whether directly modifying functional redundancy can improve scaling curves remain central goals of our future work. The proposed measure, observed structural regularities, and controlled experiments provide an empirical foundation and concrete, testable directions toward these goals.

3. Why redundancy arises and what could reduce it

Training dynamics (§5.4; Appendix A) show that substitution relations form and reorganize during optimization, following non-monotonic trajectories with different stabilization rates across layers. Changes in substitutability are not always synchronized with performance improvements. Thus, functional redundancy is part of an evolving computational organization rather than merely a static post-training property.

Our controlled experiments further reveal two design-relevant patterns: (i) from a shared baseline, width scaling produces greater growth in substitutability and lower improvement per added parameter than depth scaling; (ii) at fixed parameter count, fewer, wider heads exhibit lower Raw Oracle, higher normalized functional rank, and better validation loss.

These observations suggest that architecture and capacity allocation affect the emergence of independent and substitutable functions. While they do not establish a unique causal mechanism or optimizer-level explanation, they motivate concrete tests, including changing capacity allocation and introducing CFS-aware regularization under fixed architecture and parameter budgets, to examine whether functional organization can be deliberately improved.

4. Practical utility and clarification of D

A lightweight router predicts CFS from forward features, requiring no oracle interventions, labels, or gradients at inference. On ViT-B, CFS routing reduces dense-output KL by 43.5–49.2% relative to Taylor under equal logical head budgets (Table 7). On GPT-2 layers 8/11, CFS R2 achieves lower mean KL than a similarly sized learned gating baseline in five of six settings, with reductions up to 58.4% (Table 8; Appendix F.5). These demonstrate improved functional preservation under logical budgets; wall-clock acceleration requires a sparsity-aware backend.

For Q1, §3.2 now explicitly defines D(y,y~)=DKL(py∥py~)D(y,\tilde y)=D_{\mathrm{KL}}(p_y\|p_{\tilde y}): the KL divergence between dense and intervened predictive distributions, over classes or next-token probabilities as appropriate.

We thank the reviewer again for the thoughtful questions. We hope the additional evidence clarifies CFS's scientific contribution and practical value, and welcome further discussion.

Follow-up: Revised manuscript and additional evidence

We thank the reviewer again for the thoughtful comments. We have completed a revised manuscript with substantive main-text additions marked in blue. To facilitate reassessment, we summarize the new evidence and its implications for the three main concerns.

1. Generality beyond attention heads

We applied the unchanged CFS protocol to grouped MLP neurons in ViT-B (Appendix C.4, Table 10):

LayerMLP groupsRaw Oracle95% CI
680.6120[0.5769, 0.6496]
6160.6856[0.6551, 0.7153]
1180.5221[0.4907, 0.5548]
11160.5813[0.5484, 0.6119]

MLP substitution also exhibits input dependence (cross-input SD 0.232–0.283) and directionality (mean gap 0.146–0.186), including under alternative groupings. This directly demonstrates applicability beyond attention heads, addressing the specific concern about MLP modules.

2. Quantitative links to scaling efficiency

Our controlled GPT-2 experiments compare two scaling paths from the identical W768/D12 baseline (Table 4):

Scaling pathAdded paramsΔ Raw OracleLoss reduction
Width → W1152125.92M+0.03970.1752
Depth → D1842.53M+0.00290.0728

Depth scaling produces much less growth in substitutability and approximately 23% greater validation-loss reduction per added parameter.

At a fixed parameter count of 124.44M, the head-decomposition experiment provides a complementary comparison (Table 5):

HeadsRaw OracleRank/HValidation loss ↓
6×1280.20120.95073.5707
12×640.21250.93353.5752
24×320.25830.87253.5780

These controlled results quantitatively relate functional organization to parameter efficiency and performance at fixed capacity. Additional Matched-N and fixed-K Cover/Rank controls (Figure 2; Appendix Figure 7) further show that the natural-family trends persist after controlling for candidate count and graph size.

Predicting scaling-law exponents and improving scaling curves through direct interventions remain long-term goals. The current measurements and controlled findings provide an empirical foundation and testable directions toward them.

3. Redundancy formation and design implications

Training dynamics (§5.4; Appendix A) reveal evolving, non-monotonic substitution structure with different stabilization rates across layers. The width/depth and head-granularity results above also identify concrete architectural variables that change functional organization. These observations motivate causal tests of capacity allocation and CFS-aware regularization, rather than merely describing redundancy in a fixed model.

4. Practical utility

CFS-guided routing reduces ViT-B dense-output KL by 43.5–49.2% relative to Taylor under equal logical head budgets (Table 7). We further extended routing to GPT-2 layers 8 and 11, comparing CFS R2 against Taylor and parameter-matched learned gating (825,091 vs. 822,460 trainable parameters).

The mean dense-output KL results (lower is better) are:

LayerKeep KTaylorLearned GatingCFS R2
1130.062680.031160.02910
1160.026910.017330.01624
1190.012000.009100.00732
830.040730.028830.04412
860.024080.019840.01580
890.011400.011150.00463

CFS R2 achieves lower mean KL than learned gating in five of six settings, with a maximum reduction of 58.4% (layer 8, K=9). The layer-8, K=3 exception is reported and discussed in §6.2. These results demonstrate functional preservation across vision and autoregressive language models under logical head budgets; realized wall-clock acceleration requires a sparsity-aware backend.

We also explicitly defined D(y,y~)=DKL(py∥py~)D(y,\tilde y)=D_{\mathrm{KL}}(p_y\|p_{\tilde y}) in §3.2.

Reviewer_UWwPOpenReview ↗

Reply to author rebuttal

I thank the authors for their detailed responses with new experiments.

I wrote 3 weaknesses: the first is about the generality of the proposed metric, the second is about scientific understanding, and the third is about the practical usage of the metric. The response addressed the first weakness very well and the third point acceptably. But the second weakness remains, and I do not think it can be addressed easily.

The current paper remains largely empirical. Although the proposed metric has some theoretical heuristics, the rest of the paper tries to show empirical relations to a range of other performance metrics, supporting the heuristics but without theoretical analysis or further understanding. This is also why I chose "Informativeness: 1: I learned little or nothing new from this paper."

Based on my standard, if a paper is for better understanding, certain theoretical analysis or direct connections between things is needed beyond heuristic connections and empirical correlation or trends.

Therefore, I tend to keep my original score now. But I am still open to increasing my score after more careful reading and thinking.

We thank the reviewer for the further clarification. We understand that the central remaining concern is whether CFS can provide more direct theoretical analysis and further understanding of functional organization beyond empirical trends. To address this point, we conducted additional theoretical derivations and real-model analyses, leading to the following results.

  1. CFS admits a clear local geometric structure. Under the local Fisher geometry of the predictive distribution, we show that the quadratic form of CFS can be expressed as constrained projection recovery between component functional directions. This yields a rigorous functional-packing lower bound in a finite-dimensional functional space: when the task-relevant local effects of (H) components lie in a (d)-dimensional space with (H>d), they cannot all be mutually orthogonal, implying unavoidable functional overlap. We further derive the effect of the finite (\alpha)-grid on this limit and obtain an attainable exact minimum Raw Local Oracle value of 0.24 for (H=3,d=2) under the original 16-point (\alpha)-grid.

  2. Full CFS captures more than local similarity. We further prove that two models can have exactly the same dense prediction, deletion loss, and Fisher Gram, while their Raw Oracle CFS under full intervention can take different values spanning from 0 to 1. Thus, Full CFS cannot be reduced to representation similarity, effective rank, or local Fisher geometry; it additionally captures conditional functional interactions and compensatory structure under finite interventions.

  3. This theoretical distinction explains the source of CFS growth in real scaling experiments. In controlled BERT width scaling, Full CFS increases by 0.07657 from B1 to B3, while the Local component increases by 0.08063. For ViT Small→Large, Full CFS increases by 0.18963, of which 0.14972 is contributed by the Local component. Therefore, in these two independent model families, the observed growth of CFS with scale is primarily associated with increasing overlap among task-relevant local functional directions, rather than merely being an empirical statistic that co-varies with model size.

Together, these results provide a theoretical account of CFS spanning local functional geometry, functional-packing limits, and full conditional intervention structure, and further clarify the source of the functional-organization changes observed during scaling.

If the paper is accepted, we will incorporate the full derivations, proofs, and corresponding experimental results into the camera-ready version. We sincerely thank the reviewer again for raising this important concern, which motivated us to further clarify and deepen the theoretical interpretation of CFS. We hope that these additional analyses address the main concern regarding theoretical analysis and further understanding; if so, we would be very grateful if the reviewer could reconsider the current score. Thank you again for your time and careful evaluation.

Reviewer_WCQnOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

The paper studies transformer scaling and redundancy from a functional substitutability or replaceability perspective. The core premise being that the current metrics in literature using importance scoring, or representation similarity are indirect proxies to quantify redundancy. With the argument being that those measure either importance to downstream outputs or whether the component outputs look alike, not whether those can be replaced. The authors focus on downstream (output) behavioral effect, and study the effect when an architectural component is replaced by another.

The authors introduce CFS, conditional functional substitutability, a measure that for an input, quantifies whether an attention head's output can reproduce the same downstream behavioral effect as another. CFS is derived out of an intervention where one attention head is removed, and another is utilized with appropriate scaling, and measures what proportion of the output deterioration is recovered. In short, the measure is functional equivalence with respect to the input specific output behavior. Additionally, the measure is also directional, where head i substituting for head j may not be equivalent to head j substituting for head i. Further, the authors propose using "raw oracle", average CFS across best available substitute for each head as a measure for total substitutability, and functional coverage (how small a set can cover / substitute for all components) as a measure of how compact is the functional organization.

The paper establishes CFS as uncorrelated with importance based scores and with representational similarity, showing that CFS captures distinct information about redundancy. The authors then utilize CFS and the defined metrics for studying scaling, showing that the raw oracle rises as the number of attention heads rise, however proportional growth in functional coverage is not observed. This shows that scaling creates redundancies without a proportional growth in independent new functional capabilities. Additionally, isolating the effect of width, depth, and number of heads reveals that the scaling path with less growth in the CFS through depth results in more parameter efficient loss improvement. Finally, the paper demonstrates that a lightweight model based on forward computations to predict CFS can lead to better KL divergence under the same number of attention heads as compared to importance based methods.

Critical strengths

CFS is a novel perspective and well motivated measure for capturing functional redundancy, and it captures information missed by the existing classes of methods in the literature. The experiment comparing the correlations with Taylor importance and other similarity measure is effective, and provides insights into how small variations can have a large effect on the downstream behavior.

The experiments are design to test specific questions, and the empirical results provided support the arguments made.

Critical weaknesses

Narrow scope on two experiments: 1. controlled capacity scaling only utilizes GPT-2 style models, and 2. the forward computation based routing experiment with the dynamic routing evaluated on a single layer and a single model class. The narrow scope make the generality claims made weaker.

Minor (does not materially move the score) - the writing of the main paper can be substantially improved by making appropriate definitions and defining variables clearer.

Questions for authors

Clarification:

  1. In the Taylor based baseline, are scores utilized per input or averaged over the entire set?
  2. How is fixed alpha and oracle alpha selected in section 4.1?

Suggestion: Comparing CFS based dynamic routing against learned dynamic routing (gating) baseline with the same number of parameters can substantially strengthen the final sections.

2 comments

We thank the reviewer for the positive assessment of CFS's novelty, motivation, and experimental design. We have added controlled scaling experiments across architectures, cross-model and cross-layer routing evaluations, and a parameter-matched learned gating baseline to address the concerns about generality.

1. Controlled scaling beyond GPT-2

We trained three additional BERT-style bidirectional encoders from scratch under a shared masked language modeling protocol, fixing depth at 8 and head dimension at 64 while varying width (384/512/640).

As width increases, validation MLM CE decreases from 2.6030 to 2.4574/2.3468, while Raw Oracle rises from 0.1288 to 0.1729/0.2058 and Matched-2 from 0.0696 to 0.0862/0.0960 (Appendix E.4, Table 11).

These results provide additional controlled evidence across different architectures and training objectives, extending our original scaling observations beyond GPT-2.

Given the limited rebuttal window and substantial computational cost of retraining language models, we could not complete the full BERT depth and fixed-capacity sweeps. If the reviewer finds these additional controls useful, we would be happy to extend them in the camera-ready version.

2. Cross-model/layer routing and learned gating

We extended routing from the final ViT-B layer to GPT-2 autoregressive language modeling, evaluating layers 8 and 11 at K=3/6/9 retained heads (§6.2, Table 8).

We also implemented the suggested parameter-matched learned dynamic gating baseline. CFS R2 and learned gating have 825,091 and 822,460 trainable parameters, respectively (0.32% difference). They share input features, data splits, joint-subset KL targets, hard-gating evaluation, and three seeds (17/23/41), using 256 held-out documents. R2 additionally uses pairwise CFS supervision as its method-specific signal (Appendix F.5).

CFS R2 achieves lower mean KL in five of six layer-budget settings, with a maximum relative reduction of 58.4% (layer 8, K=9).

The exception is layer 8, K=3 (0.04412 vs. 0.02883 for learned gating). As discussed in §6.2, this may reflect insufficient functional coverage under aggressive pruning: when nine heads are removed jointly, pairwise substitutions need not compose, potentially leaving some independent functions uncovered.

We do not claim uniform superiority over learned gating at every budget. Nevertheless, the new results extend CFS-guided routing to autoregressive language modeling and multiple layers, with advantages in most evaluated settings.

3. Technical clarifications

Q1. Is Taylor computed per input or averaged?

Taylor scores are computed separately for each input using ground-truth classification cross-entropy (true next-token NLL for GPT-2), without dataset averaging before head selection. Taylor thus requires target information and backpropagation, whereas the CFS router uses only forward features at inference.

Q2. How are fixed and oracle α selected?

Fixed α is calibrated separately for each layer and directed head pair on training data by minimizing mean normalized intervention KL, then frozen at evaluation. Oracle α minimizes each input-pair's dense-to-intervention KL over the shared 31-point grid {0, 0.1, …, 3}. All oracle-α policies, including Taylor, receive identical search permission; α=1 corresponds to source deletion without modifying the substitute. The two fixed-representative policies are fitted separately rather than constituting a strict α-only ablation.

These definitions and protocols are clarified in §3.2 and Appendices B.1/C.1.

Summary

The new BERT scaling experiments and GPT-2 routing evaluations directly address the two scope limitations, while the matched-parameter gating comparison and protocol clarifications strengthen the empirical evidence.

We thank the reviewer again for these constructive suggestions. We hope the additions address your concerns and welcome further discussion.

Follow-up: Revised manuscript and additional evaluations

We thank the reviewer again for the constructive suggestions. We have prepared a revised manuscript with the substantive main-text additions marked in blue. We would like to summarize the new evidence directly, particularly the additional controlled scaling experiments and the parameter-matched learned gating comparison you suggested.

1. Controlled scaling beyond GPT-2

We trained three BERT-style bidirectional encoders from scratch under a shared MLM protocol, varying width while holding depth and head dimension fixed (Appendix E.4, Table 11).

WidthValidation MLM CE ↓Raw OracleMatched-2
3842.60300.12880.0696
5122.45740.17290.0862
6402.34680.20580.0960

Both Raw Oracle and Matched-2 increase consistently as validation loss decreases. These controlled results extend the original GPT-2 observations to a bidirectional encoder with a different training objective.

2. Cross-model/layer routing and learned gating

Following your suggestion, we extended routing to GPT-2 autoregressive language modeling at layers 8 and 11 and implemented a learned dynamic gating baseline with nearly identical trainable parameters: 825,091 (CFS R2) vs. 822,460 (Learned Gating), a 0.32% difference.

The methods share input features, data splits, joint-subset KL targets, and hard-gating evaluation. R2 additionally uses pairwise CFS supervision. Results use three seeds and 256 held-out documents (Table 8; Appendix F.5).

Mean dense-output KL (lower is better):

LayerKeep KTaylorLearned GatingCFS R2
1130.062680.031160.02910
1160.026910.017330.01624
1190.012000.009100.00732
830.040730.028830.04412
860.024080.019840.01580
890.011400.011150.00463

CFS R2 achieves lower mean KL than learned gating in five of six settings, with a maximum reduction of 58.4%. The layer-8, K=3 exception may reflect insufficient functional coverage under aggressive joint pruning, as discussed in §6.2.

3. Protocol clarifications

We also clarified both questions in the revised manuscript:

  • Taylor: Scores are computed per input using ground-truth cross-entropy (next-token NLL for GPT-2), without dataset averaging before head selection.
  • Fixed/oracle α: Fixed coefficients are calibrated per directed head pair on training data and frozen during evaluation. Oracle coefficients minimize per-input KL over the shared 31-point grid [0, 3], with identical search permission across oracle policies (Appendices B.1/C.1).

We sincerely appreciate your suggestions, which directly motivated these new evaluations. We hope the additional evidence addresses your concerns about generality and strengthens the assessment of the paper. If you find these concerns sufficiently addressed, we would be grateful if you would consider increasing your rating. We welcome further discussion.

Reviewer_nLkzOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper defines Conditional Functional Substitutability (CFS) on transformer models, to characterise the extent to which deleting an attention head ii can be compensated for (on a particular input) by rescaling another attention head jj in the same layer. The authors argue this is a more direct measure of redundancy in transformer models than existing similarity and importance measures. Across ViT, BERT, Qwen2.5 and controlled GPT-2 sweeps, the paper reports that larger models tend to have more substitutability and that fixed-capacity models with more independent functional structure perform better; this is presented as a functional account of diminishing returns. Finally, the authors train a small router that predicts CFS and is used to select which final-layer heads of a ViT model to keep, cutting KL divergence to the dense model by over 40% relative to Taylor-importance selection

Critical strengths

The idea behind CFS is clear and well-motivated - it is made obvious to the reader why CFS is a natural and direct measure to use for quantifying substitutability.

The authors demonstrate a practical application of CFS - the routing experiments show that selecting heads based on predicted CFS performs substantially better than doing so based on Taylor importance.

Critical weaknesses

The paper introduces a "matched oracle" that controls for the number of candidate heads that a given attention head ii could be substituted by. This seems like an important control - more candidate heads could artificially increase the substitutability. But it seems that the majority of results do not account for this concern, e.g. it seems the Cover/H\text{Cover/H} and Rank/H\text{Rank/H} measures do not account for the number of candidate heads (these are computed over all H−1H-1 candidates and dividing by HH is not equivalent to controlling for this) and even all of the matched oracle results are done with Match-2\text{Match-2}, i.e. choosing between 2 candidate heads, which seems oddly small. Where the number of heads is held constant (depth scaling experiment in table 4) there isn't a clear trend in substitutability. This slightly undermines the paper's "diminishing returns" argument.

While CFS is generally a well-motivated idea, it is unclear to me that the exact definition used by this paper is the right one for measuring redundancy - in particular, it seems odd that a given head ii must be substituted for by only one other head jj, rather than some linear combination of all the retained heads.

Questions for authors
  1. Can you report results using Rank/H\text{Rank/H} and Cover/H\text{Cover/H} statistics that are adjusted to account for candidate count, i.e. perhaps in eqn 8 you could use some random subset VsampledV_{\text{sampled}} consisting of HminH_\text{min} heads, where HminH_{\text{min}} is the minimum number of heads across the models being compared; and then average the statistics over many random samples of VsampledV_{\text{sampled}}. Or some other well-motivated approach to controlling for candidate count.
  2. Is there a reason for using Match-2\text{Match-2} in your reporting rather than Match-N\text{Match-}N for some higher NN? Intuitively it seems to me that choosing the better of 2 random candidate heads is not much better than just choosing a random head.
  3. Have you considered a definition of CFS that allows a linear combination of retained heads to substitute for the removed head? Your current definition seems to understate the redundancy in the network as it only allows one head to be scaled up to substitute for another - I am curious how the conclusions in this paper would change if the definition was broadened in this way.
2 comments

We thank the reviewer for recognizing CFS's motivation and routing application. We address both concerns with new experiments and a clearer account of the pairwise definition.

1. Candidate-count and graph-size controls (Q1–Q2)

We agree that more substitute candidates can inflate the observed maximum, and dividing Cover/Rank by head count alone does not control graph size. Raw Oracle measures the best available path, including alternatives introduced by scaling; Matched-N fixes candidate count to assess substitutability independently of that availability.

Matched-2 was not arbitrary: ViT-Tiny has only three heads, leaving two candidates per source, so N=2 is the largest common choice for the full ViT family. We now extend to N=2/4/8/11 for Qwen2.5 and N=2/4/5 for ViT-Small/Base/Large. Figure 2 shows an overall increase with scale at fixed N; Qwen2.5 Matched-2, for example, rises from 0.1620 (1.5B) to 0.2745 (14B).

For Cover and Rank, we sample fixed-K induced subgraphs and recompute Cover/K and Rank/K, rather than merely dividing full-graph scores by H. The decreasing trends persist in both Qwen2.5 and ViT (Figure 2; Appendix Figure 7). Thus, head-count and graph-size effects cannot fully explain the observed functional reorganization.

2. Why depth scaling differs

Our claim is not that CFS rises monotonically along every scaling axis, but that different scaling paths organize added capacity differently. Relative to the shared W768/D12 baseline, widening to W1152 increases Raw Oracle by 0.0397, compared with only 0.0029 when deepening to D18. Consistent with its smaller growth in functional substitutability, depth scaling achieves approximately 23% greater validation-loss reduction per added parameter. These controlled results quantify the corresponding differences in functional organization and marginal parameter efficiency across the two scaling paths.

3. Why pairwise rather than multi-head substitution (Q3)

Pairwise CFS asks whether one existing head can directly replace another's downstream role; multi-head substitution measures reconstruction after jointly adjusting several heads. Greater flexibility may raise oracle recovery without implying greater direct redundancy. Pairwise CFS also yields an interpretable, input-conditioned directed graph.

We tested two-head substitution (Joint-2): normalized selection regret changes from 0.5522 to 0.5437 (1.54% relative reduction), but batched oracle-label measurement costs over 20× more per label. With 12 heads, the directed relation space grows from 132 to 660. Since CFS is input-dependent, the expanded relation space also increases measurement and prediction demands. These results motivate pairwise CFS as a tractable basic relation, not a claim that coalitional reconstruction is unimportant.

Nor do we assume pairwise effects compose exactly: Section 4.2 measures joint removal using the Exact Joint-Subset Oracle, and R2 uses joint-subset supervision to address interactions beyond pairwise relations.

In sum, the new controls support functional-organization trends beyond candidate-count effects, while the Joint-2 comparison clarifies the distinct scope and cost–benefit trade-off of our pairwise definition.

We thank the reviewer again for the constructive feedback. We hope these additions address your concerns and welcome further discussion.

Follow-up on the revised manuscript

We thank the reviewer again for the constructive suggestions. We have completed a revised manuscript with the substantive main-text additions marked in blue. We would like to briefly consolidate how the new evidence addresses your concerns.

1. Candidate-count and graph-size controls (Q1–Q2)

As clarified in our earlier response, Matched-2 was originally chosen because ViT-Tiny has only three heads. We have now extended it to multiple fixed candidate counts (Qwen2.5: N=2/4/8/11; ViT Small/Base/Large: N=2/4/5).

The new Figure 2 shows that matched substitutability retains an overall increasing trend with scale across these candidate counts. More importantly, we independently control graph size by recomputing Cover/K and Rank/K on fixed-K induced subgraphs (Qwen2.5: K=4/8/12; ViT: K=4/6). Both metrics retain decreasing trends (Figure 2; Appendix Figure 7). These results directly address the concern that the observed scaling patterns might arise simply from larger candidate pools or graphs.

2. Depth scaling and parameter efficiency

The controlled results in Table 4 quantify the differences between two scaling paths from the same W768/D12 baseline:

Scaling pathAdded paramsΔ Raw OracleLoss reduction
Width → W1152125.92M+0.03970.1752
Depth → D1842.53M+0.00290.0728

Depth scaling shows substantially less growth in substitutability and approximately 23% greater loss reduction per added parameter, providing quantitative evidence of the relationship between functional organization and marginal parameter efficiency.

3. Pairwise versus joint substitution (Q3)

Pairwise CFS measures direct substitution between two existing functional units, whereas joint substitution measures reconstruction using multiple adjustable units. We additionally evaluated Joint-2 (Appendix B.3):

MetricPairwiseJoint-2
Normalized selection regret ↓0.55220.5437
Directed relations (H=12)132660
Batched oracle-label time per label1×>20×

Joint-2 reduces selection regret by 1.54% relative to Pairwise, with substantially higher measurement cost. This quantifies the trade-off and supports Pairwise CFS as a tractable measure of direct functional substitutability. As discussed previously, our Exact Joint-Subset Oracle and R2 supervision also account for interactions beyond pairwise relations.

We hope the additional controls and direct comparisons address your main concerns about scaling and the pairwise definition. If you find these concerns sufficiently addressed, we would sincerely appreciate your consideration of increasing your rating. We welcome any further questions or discussion.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Final Rebuttal Summary and Manuscript Revisions

Final Rebuttal Summary and Manuscript Revisions

We thank the Area Chair and reviewers for their constructive feedback. We added controlled experiments, broader evaluations, and protocol clarifications; substantive main-text revisions are marked in blue. Below, we summarize our responses, including the latest theoretical follow-up.

1. Reviewer nLkz: Candidate-count controls and pairwise substitution

We appreciate the reviewer's recognition of CFS's motivation and routing gains.

To control candidate count, we extended Matched-2 to N=2/4/8/11 (Qwen2.5) and N=2/4/5 (ViT), and recomputed Cover/K and Rank/K on fixed-size induced subgraphs. Scaling trends persist (new Figure 2; Appendix Figure 7). Matched-2 originally accommodated ViT-Tiny, which has only three heads.

For multi-head substitution, Joint-2 reduces normalized selection regret from 0.5522 to 0.5437 at over 20× batched oracle-label measurement time (Discussion; Appendix B.3). Controlled width/depth scaling further relates smaller substitutability growth to greater marginal parameter efficiency (Table 4).

2. Reviewer WCQn: Generality, learned gating, and clarity

We appreciate the reviewer's recognition of CFS's novelty and the insights from our experimental design.

Three controlled BERT-style MLM models show Raw Oracle increasing from 0.1288 to 0.2058 and Matched-2 from 0.0696 to 0.0960 as width increases and validation loss decreases (Appendix E.4, Table 11).

We extended routing to GPT-2 layers 8/11 and added the suggested parameter-matched learned gating baseline (825,091 vs. 822,460 parameters). CFS R2 achieves lower mean KL in five of six settings, with up to 58.4% reduction (§6.2; Table 8; Appendix F.5). We also clarified the output discrepancy definition (§3.2), per-input Taylor scoring (Appendix C.1), and fixed/oracle alpha calibration (Appendix B.1).

3. Reviewer UWwP: Generality, scientific understanding, and utility

We appreciate the reviewer's recognition of CFS's creativity and potential practical value.

We extended CFS to grouped ViT-B MLP neurons, demonstrating input-dependent and directional substitution beyond attention heads (Discussion; Appendix C.4, Table 10). Controlled scaling and fixed-capacity experiments quantitatively relate functional organization to parameter efficiency (Tables 4–5). Routing evaluations across vision and language models further demonstrate improved functional preservation under equal logical budgets (Tables 7–8).

4. Latest theoretical follow-up to UWwP (November 18)

On November 17, UWwP acknowledged that we addressed generality "very well" and practical utility "acceptably", identifying theoretical understanding as the remaining concern. We subsequently reported new derivations and real-model analyses on November 18:

Local functional geometry and packing. Under local Fisher geometry, quadratic CFS admits a constrained-projection interpretation between functional directions. We derive unavoidable overlap when H>d, including an attainable minimum Raw Local Oracle of 0.24 for H=3, d=2 under the original 16-point alpha grid.

Beyond local geometry. Models can share identical dense predictions, deletion losses, and Fisher Gram matrices while Full CFS ranges from 0 to 1, demonstrating additional information captured by finite conditional interventions.

Mechanistic scaling evidence. Full CFS increases by 0.07657 in controlled BERT width scaling (Local component: +0.08063), and by 0.18963 for ViT Small→Large (Local contribution: +0.14972). These results identify local functional overlap as a major contributor to CFS growth across both families.

Together, these findings establish a geometric interpretation and packing limits for local CFS, distinguish Full CFS from local geometric measures, and explain a substantial portion of the observed scaling trends.

These results were shared after the revised PDF was prepared. We plan to include complete proofs and supporting analyses in the camera-ready version if accepted.

We hope this summary helps the Area Chair assess the complete rebuttal record, particularly our latest theoretical response to UWwP's remaining concern. We recognize that opportunities for further discussion during rebuttal may be limited. If any questions or concerns remain, we would welcome further feedback and, if accepted, carefully address them in the camera-ready revision. We sincerely thank the Area Chair and all reviewers for their time, thoughtful feedback, and careful consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.