acceptodds
Under review as a conference paper at ICLR 2027

Is Depth All You Need? Rethinking ViT Capacity via Structural Re-parameterization

Abstract

Vision Transformers (ViTs) are computationally expensive largely due to their deeply stacked layer designs, yet prevailing efficiency research has concentrated on algorithmic-level strategies such as token pruning and attention mechanism acceleration. This raises a fundamental but underexplored architectural question: is it possible to decrease ViT depth without sacrificing model capacity? We address this question by introducing a branch-based structural re-parameterization framework that augments transformer blocks with parallel branches during training, thereby enhancing optimization flexibility and model capacity. After training, these branches are systematically consolidated into a single linear pathway through exact re-parameterization applied at the inputs of nonlinear activations, ensuring strict functional equivalence between the expanded and collapsed models. Experiments across multiple ViT backbones demonstrate that this approach achieves a 2–4 reduction in network depth while maintaining competitive classification performance on ImageNet-1K. Beyond depth reduction, we further demonstrate that re-parameterization functions as a general capacity enhancer, boosting the accuracy of iso-depth ViTs and improving vision encoders within vision-language models (VLMs). These findings indicate that heavily stacked transformer architectures may be redundant for error-tolerant and resource-constrained deployment scenarios, opening promising avenues for the design of efficient vision transformers.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Vision Transformers, Structural Re-parameterization, Depth Reduction, Efficient Transformers, Vision-Language Models

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_Vy8gOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper presents an approach that targets real problems in the deployment of vision transformers under resource constraints. It argues that instead of deploying ViTs with significant depth, it could establish an effective framework that enables shallower models to have similar performance compared to their deeper peers. Specifically, such a framework can be achieved by so-called multi-branch training that could later be collapsed into a single-branch inference network without performance loss when the relevant parameter λ\lambda converges to 1. The paper devised several re-parameterization aspects to aid the training and the final branch collapse, and it later puts its method to the test against several baselines with further ablation studies. However, there appears to be a major mathematical flaw that would significantly weaken one of the central claims made in the paper.

Critical strengths
  1. The research question is real and well motivated, and this paper serves as a good exploration of methods that would turn multi-branch training into single-branch inference that could bring real benefits under resource constraints.
  2. The method is tested on several different datasets and compared against competing methods, providing evaluation beyond a single benchmark, with substantial ablation studies that help interpret the method.
Critical weaknesses
  1. The key claim: "Our approach mitigates this by training parallel transformer branches that are later collapsed into a single branch at inference, drastically reducing depth while preserving representational capacity. The key idea is that, with proper constraints, parallel branches can be algebraically re-parameterized into one equivalent branch without approximation error" is not supported by the method in its current form. Specifically, for the two branches AA and BB, even at the merge point λ=1\lambda = 1, the softmax part should be identical, while the sum of the outputs from different branches should have WA,iVWA,iO+WB,iVWB,iOW_{A,i}^V W_{A,i}^O + W_{B,i}^V W_{B,i}^O as the last multiplier, but in the proposed formulation after Equation 1, the output would have (WA,iV+WB,iV)(WA,iO+WB,iO)(W_{A,i}^V+W_{B,i}^V)(W_{A,i}^O+W_{B,i}^O) instead, which introduces additional cross terms. As a result, in its current form, the method would not in general produce results identical to those of the multi-branch model at single-branch inference, as claimed in the paper.
  2. Several numerical claims in the text are inconsistent with the paper's own tables, and there are inconsistencies across different tables. Section 4.1 reports that MAE-Ti-6-R has an improvement of 6.7% against its baseline, while Tables 1, 2, and 4 all have 3.8%. The claimed accuracy of Swin-S-12-R is reported at 81.5% in Table 2 but 82.0% in Table 15, with no explanation for the difference. These inconsistencies could generally be reporting mistakes and could sometimes be traced in the supplement, but unfortunately they still raise concerns about the reliability of the reported results.
Questions for authors
  1. Can you reexamine whether the exact algebraic equivalence is established in the current framework and is supported by numerical results? Furthermore, is the current form of branch collapsing implemented in the supplemented code and used to evaluate the performance of the model?
  2. Can you check the discrepancies between the text and tables and between different tables against the experiment records? Please clarify the correct values and settings, and whether the corrections affect the conclusions.
2 comments

We thank the reviewer for the careful reading and for the two concrete findings: the merge rule as written in §3.3 is not exact, and the text and tables disagreed in places. Both are correct, and we address them in turn.

W1 / Q1. Exact algebraic equivalence, and what the code does

The reviewer's analysis is right. At λ=1\lambda=1 the softmax of every branch receives the same input, so the attention map is shared; the sum of the branch outputs then has ∑bWbVOb\sum_b W^V_bO_b as its last factor, whereas the merge written after Eq. (1), (∑bWbV)(∑bOb)(\sum_b W^V_b)(\sum_b O_b), contains the additional cross terms ∑b≠b′WbVOb′\sum_{b\ne b'}W^V_bO_{b'}. Written that way the merge is not exact, and the paper's claim would not hold.

What is implemented. The collapse code in the supplementary material (inference_fusion.py) does not sum the two projections separately. For every head ii it forms

Ci=∑bWb,iQ(Wb,iK)⊤,Mi=∑bWb,iVOb,i=WA,iVOA,i+WB,iVOB,i,C_i=\sum_b W^Q_{b,i}(W^K_{b,i})^{\top},\qquad M_i=\sum_b W^V_{b,i}O_{b,i}=W^V_{A,i}O_{A,i}+W^V_{B,i}O_{B,i},

where Ob,iO_{b,i} is the row block of WbOW^O_b for head ii, and the FFN merges as W1=∑bW1,bW_1=\sum_bW_{1,b}, W2=∑bW2,bW_2=\sum_bW_{2,b}. With Si=softmax⁡(XCiX⊤/(2dk))S_i=\operatorname{softmax}\big(XC_iX^{\top}/(2\sqrt{d_k})\big) common to all branches, the sum of the branch outputs is exactly ∑iSiXMi\sum_i S_iXM_i: the per-branch products are formed first and summed afterwards, so no cross term arises and no approximation is made. The FFN merge is exact because all branches apply GELU to the same pre-activation at λ=1\lambda=1.

Was this used for the reported results? Yes. Every accuracy, latency, throughput and memory number in the paper was obtained from models collapsed by this rule; the error was confined to the exposition in §3.3 and Algorithm 1. We have rewritten Eq. (1), the Inference Representations and Implementation Details paragraphs, the collapse step of Algorithm 1 and Figure 1 to state the rule above, and we have attached the deployed collapsed checkpoints so that the merge can be checked directly against the multi-branch checkpoints.

Numerical support. Appendix J (new) compares the fully joined training graph, which evaluates each branch's own QQ, KK, VV, OO and sums the branch outputs, against the collapsed model, end to end on the trained six-layer two-branch model (N=197N{=}197, d=192d{=}192, h=3h{=}3), including biases, LayerNorm, residual connections and the classification head, on eight independent random input batches. The maximum absolute logit difference is 7.30×10−57.30\times10^{-5} (dense Ci,MiC_i,M_i) and 3.28×10−53.28\times10^{-5} (factorized form) in FP32, and 1.22×10−91.22\times10^{-9} / 7.77×10−87.77\times10^{-8} in FP64.

Because the exact merge stores CiC_i and MiM_i per head, a collapsed layer is not the same size as a standard layer. The revised §3.3 gives the parameter and MAC counts in closed form, and every table was recounted for the collapsed model actually deployed (see below).

W2 / Q2. Discrepancies between text and tables

We checked every number cited in the text against the tables and every table against the experiment logs. The two the reviewer found, and what we did:

  • MAE-Ti-6-R gain. §4.1 stated +6.7%. The tables are correct: MAE-Ti-6-R reaches 72.4% against 68.6% for the iso-depth MAE-Ti-6†, a gain of +3.8%. The +6.7% figure belongs to MAE-Ti-3-R (60.5% vs 53.8%) and was carried into the wrong sentence. §4.1 now reads +6.8% (DeiT-Ti-6-R) and +3.8% (MAE-Ti-6-R†).
  • Swin-S-12-R. The experiment log for the final Swin-S-12-R checkpoint gives 82.0%, the value in Table 15 and in the cross-depth comparison. The 81.5% in Table 2 was from a previous run; Table 2 now reads 82.0%, and the parameter and FLOP entries of the Swin rows were reconciled across Tables 2 and 15 at the same time.

The audit also found stale cost numbers in the prose that pre-dated the recount of parameters and FLOPs (for example, the FLOP figures quoted for MAE-L-6-R in the token-reduction section and for MAE-L-16-R in Appendix B); all now match the tables. The full list of corrections is in the revision, marked in blue.

Do the corrections affect the conclusions? The accuracy results do not change except for the two reporting corrections above, and the pattern they support is unchanged: the recovered gain over the iso-depth baseline is +6.7, +6.0 and +3.8 points at 3, 4 and 6 layers for MAE-Ti, and +6.8 points for DeiT-Ti-6-R. The recounted parameters and FLOPs do change one statement: a re-parameterized model is not free at iso-depth, costing 6–16% more parameters and 8–22% more FLOPs than its iso-depth baseline (DeiT-Ti-6-R: 3.27M / 774M vs 3.05M / 641M), and the text now says so where it previously said "without additional inference cost". Relative to the full-depth models the six-layer models remain 38–43% smaller in both parameters and FLOPs, and the measured latency and throughput are unchanged because they were always measured on the collapsed models.

We are grateful for the scrutiny; the corrected formulation, the attached code and collapsed checkpoints, and the end-to-end equivalence test allow the central claim to be checked directly, and the reconciled tables now agree with the text and with each other.

Reviewer_YZdcOpenReview ↗
Rating
4
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes progressive structural re-parameterization to reduce ViT depth. Parallel attention and feed-forward branches are trained, gradually joined before softmax and GELU, and collapsed for inference. The approach is also evaluated for improving fixed-depth models and vision-language encoders. Experiments cover ImageNet-1K, transfer classification, COCO detection and segmentation, and LLaVA benchmarks across multiple backbones. GPU, CPU, and ARM measurements assess practical efficiency. The authors report improved accuracy over shallow baselines, faster inference than deeper models, and compatibility with ToMe and VisionZip.

Critical strengths
  1. The idea of adopting structural reparameterisation is interesting and worth exploring.

  2. The evaluation and ablation studies are comprehensive. The experiments extend beyond ImageNet classification to transfer learning, COCO detection and segmentation, and LLaVA benchmarks. Compatibility with token reduction is also examined. The ablations distinguish direct depth scaling, unmerged parallel branches, progressive joining, and diversity regularization, while the appendix studies alternative shallow-model baselines and joining schedules.

Critical weaknesses
  1. The emphasis on ViTs with six or fewer layers is insufficiently motivated. Although the authors highlight potential benefits on specialized hardware, particularly photonic processors, no experiments are conducted on these platforms. The measurements on the RTX 4080, Intel i9-9900X, and Cortex-A72 demonstrate speedups, but do not clearly establish why such aggressive depth reduction is preferable to deeper alternatives with comparable inference budgets. Moreover, the improvements on deeper, fixed-depth models suggest that the main contribution is a broadly applicable structural re-parameterization strategy. The paper should better distinguish the benefits of re-parameterization from those of depth reduction and provide stronger justification for prioritizing models with six or fewer layers.
Questions for authors

None

1 comment

We thank the reviewer for the positive assessment and for the breadth of the reading, which covers the transfer, detection, VLM and token-reduction experiments as well as the ablations. The one weakness raised is a fair one, and we address its three parts: the hardware motivation, the choice of six layers, and the separation of re-parameterization from depth reduction.

Photonic and analog hardware

The reviewer is right that we have no measurements on photonic or analog in-memory processors; the motivation there rests on published characterizations of those platforms (per-layer optical–electrical conversion, ADC energy and area shares, noise accumulation with depth). Evaluating the collapsed models on such an accelerator is part of our future work.

Why six layers

The choice is empirical, and we have made the justification explicit in the revision rather than leaving it implicit in the appendix. The depth-scaling analysis (Appendix B.1) shows a boundary: at six layers re-parameterization recovers most of the gap to the full-depth model (DeiT-Ti-6-R 70.2% vs 72.2%; DeiT-B-6-R 80.3% vs 81.8%; MAE-B-6-R 81.4% vs 83.6%, at 38–43% fewer parameters and FLOPs and about 1.6–1.8× the GPU throughput), whereas at four layers or fewer the recovered models, although still 6–7 points above their iso-depth baselines, fall below deeper and narrower models at comparable or lower compute (e.g., MAE-B-4-R at 77.4% against MAE-Ti-12 at 76.4–78.4%). Six layers is therefore the most aggressive depth at which the method yields a model that is competitive at its own compute budget; the intermediate-depth results (MAE-L at 18, 16 and 12 layers) show that the same procedure gives a smooth frontier for less aggressive targets.

At comparable inference budgets, the six-layer model matches deeper alternatives in accuracy: DeiT-Ti-6-R (0.774G FLOPs, 70.2%) against 12-layer compressed DeiT-Tiny variants SCOP (0.80G, 68.9%), HVT (0.80G, 69.7%), S²ViTE (1.0G, 70.1%) and UPDP-P6 (0.90G, 70.3%) (Table 19). At equal accuracy and FLOPs, the shallower model has half the sequential layers, and therefore half the kernel launches and synchronization points, which is what sets batch-1 latency on CPUs and edge devices. Thus, we consider depth reduction preferable to within-layer compression at the same budget.

Separating re-parameterization from depth reduction

We agree that the general mechanism is the broader contribution, and our experiments already separate the two effects: Table 4 isolates depth scaling without re-parameterization, un-joined parallel branches, and re-parameterized models at each depth; Table 6 applies re-parameterization at fixed depth (gains of +0.2 to +2.3 points on DeiT, Swin and MAE from Ti to L); Table 19 places each re-parameterized model between its iso-depth and full-depth counterparts; and Appendix B.2 compares against alternative ways of building a shallow model under the same training budget. In the revision we restructured the framing to match this evidence: the contributions paragraph now presents structural re-parameterization as the general method, with (i) iso-depth capacity boosting and (ii) depth reduction to six layers as its two applications, and states that the gains are largest in the second regime (+3.8 to +8.1 points at six layers vs +0.2 to +2.3 at full depth) because that is where the iso-depth model is most capacity-limited.

We thank the reviewer again; the reframing above makes the paper's contribution clearer, and we are glad to make any further adjustment the reviewer thinks would help.

Reviewer_EWRaOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The method trains shallow transformer blocks with parallel branches, progressively mixes their inputs to softmax and GELU, and then merges the branches for deployment.

Critical strengths

The paper measures actual deployment performance on multiple processor types rather than relying only on FLOPs.

The appendices provide alternative depth-reduction baselines, scheduling ablations, and training-cost measurements.

Critical weaknesses

its strongest claimed advantage—exact attention collapse—is currently unsupported by the presented algebra.

The paper needs to state whether every baseline receives equivalent total training exposure, including joining, fine-tuning, and distillation. Per-step timing alone cannot establish total training-cost fairness.

the most serious concern is in §3.3, Eq. 1 and Algorithm 1. separately summing the value and output projection weights introduces cross terms absent from the original branch outputs, invalidating the claimed exact equivalence as written. The authors must provide a corrected merging rule and numerical equivalence tests, then reassess the resulting parameter and computational costs.

Questions for authors

How does the actual attention merge avoid the value/output cross terms?

What are the deployed query/key or bilinear-matrix dimensions, and how are FLOPs and parameters counted?

What are the maximum output differences before and after merging, including LayerNorm, biases, and residuals?

2 comments

We thank the reviewer for the careful reading, and for noting that we report measured latency on three processor types and include the depth-targeting baselines, schedule ablations and training-cost measurements in the appendix. The reviewer's main concern is correct: Eq. (1) and the last step of Algorithm 1, as written, summed the value and output projections separately, which introduces cross terms and is not an exact merge. Below we give the corrected rule, the deployed dimensions and counting conventions, the end-to-end numerical equivalence test, and the total training exposure of every model.

Q1 / W1 / W3. How the actual merge avoids the value/output cross terms

Separately summing the projections gives (∑bWbV)(∑bOb)=∑bWbVOb+∑b≠b′WbVOb′(\sum_b W^V_b)(\sum_b O_b)=\sum_b W^V_bO_b+\sum_{b\ne b'}W^V_bO_{b'}, and the second sum is not part of the branch outputs. Our implementation (inference_fusion.py in the supplementary material) does not form it. At λ=1\lambda=1 every branch's softmax receives the same mixed logits, so for head ii the attention map Si=softmax⁡(XCiX⊤/(2dk))S_i=\operatorname{softmax}\big(XC_iX^{\top}/(2\sqrt{d_k})\big) with Ci=∑bWb,iQ(Wb,iK)⊤C_i=\sum_b W^Q_{b,i}(W^K_{b,i})^{\top} is common to all branches, and the sum of the branch outputs is

∑b∑iSiXWb,iVOb,i=∑iSiX(∑bWb,iVOb,i)=∑iSiXMi,Mi=∑bWb,iVOb,i=WA,iVOA,i+WB,iVOB,i.\sum_b\sum_i S_iXW^V_{b,i}O_{b,i}=\sum_i S_iX\Big(\sum_b W^V_{b,i}O_{b,i}\Big)=\sum_i S_iXM_i,\qquad M_i=\sum_b W^V_{b,i}O_{b,i}=W^V_{A,i}O_{A,i}+W^V_{B,i}O_{B,i}.

The per-branch products Wb,iVOb,iW^V_{b,i}O_{b,i} are formed first and summed afterwards; nothing is summed before it is multiplied, so no cross term can arise. The FFN merge, W1=∑bW1,bW_1=\sum_b W_{1,b} and W2=∑bW2,bW_2=\sum_b W_{2,b}, is exact because at λ=1\lambda=1 all branches apply GELU to the same pre-activation, and the LayerNorm before the FFN is absorbed into W1W_1 exactly (W1 diag(γ)W_1\,\mathrm{diag}(\gamma), W1β+b1W_1\beta+b_1). Eq. (1), the Inference Representations paragraph, Implementation Details, Algorithm 1 and Figure 1 have all been rewritten to this rule. Every reported accuracy, latency, throughput and memory number was produced by models collapsed by this rule, and the collapsed checkpoints are attached to the supplementary material.

Q2. Deployed dimensions and how parameters and FLOPs are counted

Deployed per-head quantities. For each head ii the collapsed block stores the bilinear matrix Ci∈Rd×dC_i\in\mathbb{R}^{d\times d} and the fused value–output matrix Mi∈Rd×dM_i\in\mathbb{R}^{d\times d}, and computes XCi∈RN×dXC_i\in\mathbb{R}^{N\times d}, (XCi)X⊤∈RN×N(XC_i)X^{\top}\in\mathbb{R}^{N\times N}, Si=softmax⁡(⋅/2dk)∈RN×NS_i=\operatorname{softmax}(\cdot/2\sqrt{d_k})\in\mathbb{R}^{N\times N}, XMi∈RN×dXM_i\in\mathbb{R}^{N\times d} and Si(XMi)∈RN×dS_i(XM_i)\in\mathbb{R}^{N\times d}, summed over the hh heads. The attention map has the same N×NN\times N shape as in a standard block; what changes is only the rank of the per-head value–output map (a standard block is the special case Mi=WiVOiM_i=W^V_iO_i of rank dhd_h). For DeiT-Tiny, d=192d=192, h=3h=3, N=197N=197. A factorized representation, which keeps the branch-specific query/key factors and sums the logits before one softmax, computes the same function and is also described; the two differ only in cost.

Counting convention. Parameters are all stored coefficients of the collapsed model: the CiC_i, the value–output path, the merged W1W_1 and W2W_2, biases and LayerNorm affine parameters, plus the patch embedding, position embedding, class token and classifier. FLOPs are counted as multiply–accumulates on a 224×224224\times224 input (N=197N=197), the convention of DeiT and fvcore under which DeiT-Ti-12 is 1.25G, over all matrix products of the collapsed block including the N2dN^2d attention terms; softmax, GELU, LayerNorm and bias additions are excluded for all models alike. §3.3 of the revision gives the resulting per-layer parameter and MAC counts in closed form as functions of hh, BB, dd and NN for the standard, dense-merged and factorized layers, together with the condition under which each representation is cheaper.

Reassessed costs. All parameter and FLOP entries in every table were recounted under this convention for the collapsed model actually deployed, and all prose that cites them was updated. Relative to the iso-depth model, DeiT-Ti-6-R costs 7% more parameters (3.27M vs 3.05M) and 21% more FLOPs (774M vs 641M); across the backbones the overhead is 6–16% in parameters and 8–22% in FLOPs. Relative to the full-depth model the six-layer models remain 38–43% smaller in both (e.g., DeiT-Ti-6-R 3.27M / 774M vs DeiT-Ti-12 5.72M / 1.25G; DeiT-B-6-R 51.1M / 10.41G vs 86.6M / 17.56G). The measured latency and throughput did not change, because they were always measured on the collapsed models.

Q3. Maximum output differences before and after merging, including LayerNorm, biases and residuals

Appendix J (new) compares the fully joined training graph, which evaluates each branch's own QQ, KK, VV, OO and sums the branch outputs, against the collapsed model, end to end on the trained six-layer two-branch model (N=197N{=}197, d=192d{=}192, h=3h{=}3), including biases, LayerNorm, residual connections and the classification head, on eight independent random input batches. The maximum absolute logit difference is 7.30×10−57.30\times10^{-5} (dense Ci,MiC_i,M_i) and 3.28×10−53.28\times10^{-5} (factorized form) in FP32, and 1.22×10−91.22\times10^{-9} / 7.77×10−87.77\times10^{-8} in FP64.

W2. Total training exposure of every model

We agree that per-step timing alone does not establish fairness, and the revised Appendix G now states the total exposure explicitly. Every model in a comparison receives the same base recipe: DeiT and Swin models, 300 epochs of supervised training; MAE models, 400 (Ti) or 800 (B/L) pre-training epochs followed by the fine-tuning of Table 13. The attention-map distillation from an MAE-B teacher is applied to every MAE-Ti model marked †, baselines and ours alike, with the loss computed on the branch-averaged attention maps for multi-branch models. The depth-targeting baselines of Appendix B.2 (LayerDrop, progressive pruning, weight sharing, depth-adaptive, KD, NAS) use the same 300-epoch DeiT-Tiny recipe. On top of the base recipe, the re-parameterized models alone receive the joining phase: a 10k-step λ\lambda warm-up followed by 50k adjustment steps at λ=1\lambda=1 (60k steps at batch 1024, about 48 ImageNet epochs), after which MAE models are fine-tuned with the same recipe as their baselines and DeiT/Swin models receive no further training.

In total compute, a six-layer -R model therefore costs about 1.8×348300≈2.1×1.8\times\tfrac{348}{300}\approx2.1\times its iso-depth baseline and about 0.83×348300≈0.96×0.83\times\tfrac{348}{300}\approx0.96\times the full-depth model it replaces. The comparison to the full-depth model, which motivates the method, is compute-neutral. For the iso-depth comparison, two existing results indicate that the gain is not explained by training exposure alone: instant joining, which uses the same compute as our schedule, loses 4–7 points (Table 5), and the KD and NAS baselines trained under the same 300-epoch budget reach at most 65.2% at six layers against 70.2% (Table 16). We have also completed an additional compute-matched experiment, reported in Appendix G.2, in which the iso-depth DeiT-Ti-6 baseline is trained for 630 epochs to match the total training compute of DeiT-Ti-6-R (8 GPUs, same base training recipe). Under approximately equal GPU-hour budgets (63.00 vs. 61.96 GPU-hours), DeiT-Ti-6-R achieves 70.2% Top-1 accuracy, still substantially outperforming the iso-depth baseline (63.4%) by 6.8 percentage points. Notably, extending the baseline to 630 epochs yields no further accuracy improvement, demonstrating that the gains cannot be attributed solely to additional training compute.

We hope the corrected rule, the explicit counting convention, the end-to-end equivalence test and the stated training exposure address the reviewer's three concerns, and we would be glad to provide any further detail.

Reviewer_422SOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes a structural re-parameterization framework that trains parallel Transformer branches and progressively aligns their inputs before merging them into a shallower inference network. Experiments span several vision backbones, downstream tasks, and token-reduction methods, with latency and throughput measurements on GPU, CPU, and ARM. The results suggest that branch-based training can substantially improve accuracy at a fixed inference depth.

Critical strengths
  1. The problem is practically relevant. Reducing sequential depth targets a source of latency that token reduction alone does not address. The proposed approach could therefore complement existing efficiency methods.
  2. The empirical coverage is broad. For example, DeiT-B-6-R reaches 80.3% accuracy at 9.1G FLOPs, compared with 81.8% at 16.8G for the 12-layer baseline. Results across architectures and hardware platforms make the deployment benefit worth investigating.
Critical weaknesses
  1. The stated merge rule appears incompatible with exact function preservation. Under the described branch computation, independently parameterized value and output projections contribute terms of the form ΣᵢVᵢOᵢ. Independently summing the projections before multiplying instead gives (ΣᵢVᵢ)(ΣᵢOᵢ), introducing cross-branch terms. Aligning branch inputs does not, by itself, remove these terms. Unless the implementation imposes additional constraints or merges the composite maps differently, this challenges the paper's central claim of an exact, approximation-free merge.

  2. The deployed attention computation is unclear. The described merged per-head matrices appear to change the head dimension, whereas Figure 1 retains a conventional concatenation path. The paper should specify the inference tensor shapes and operations explicitly. This is necessary both to assess algebraic equivalence and to substantiate compatibility with standard ViT deployment.

  3. The training comparison does not isolate the source of the accuracy gains. Multi-branch training uses roughly 1.8× the per-step time and memory of the iso-depth baseline. A control with comparable total training compute would distinguish benefits of the proposed parameterization from benefits of additional optimization or capacity during training. Inconsistent temperature expressions and merging schedules further make the actual training procedure difficult to reconstruct.

Questions for authors

Please give the exact branch and merged computations, identify how cross terms are avoided, and report numerical output agreement before and after merging. Can you also reconcile the inference shapes and training specifications, and provide a compute-matched training comparison?

2 comments

We thank the reviewer for a careful and precise review. The three weaknesses identify a real error in our formulation and a missing control; we address each below and have revised the paper accordingly.

W1 / Q1. Exact branch and merged computations, and where the cross terms go?

The reviewer's algebra is correct, and it is exactly what was wrong with our text. Summing value and output projections separately gives

(WAV+WBV)(OA+OB)=WAVOA+WBVOB+WAVOB+WBVOA(W_A^V+W_B^V)(O_A+O_B)=W_A^VO_A+W_B^VO_B+W_A^VO_B+W_B^VO_A

so a merge written that way is not exact. Our implementation never forms the left-hand side. The computation that is implemented (inference_fusion.py in the supplementary material) is the following.

Branch computation at λ=1\lambda=1. For head ii and branch bb, with Ob,i∈Rdh×dO_{b,i}\in\mathbb{R}^{d_h\times d} the ii-th row block of WbOW^O_b:

Si=softmax⁡ ⁣(12dk X(∑bWb,iQ(Wb,iK)⊤)X⊤),Yb=∑i=1hSi X Wb,iV Ob,i.S_i=\operatorname{softmax}\!\Big(\tfrac{1}{2\sqrt{d_k}}\,X\Big(\textstyle\sum_{b}W^Q_{b,i}(W^K_{b,i})^{\top}\Big)X^{\top}\Big),\qquad Y_b=\sum_{i=1}^{h} S_i\,X\,W^V_{b,i}\,O_{b,i}.

SiS_i carries no branch index because, at λ=1\lambda=1, every branch's softmax receives the identical mixed logits (that is what progressive joining enforces). The block output is ∑bYb\sum_b Y_b.

Merged computation. Since SiS_i and XX are common to all branches,

∑bYb=∑i=1hSi X(∑bWb,iV Ob,i)=∑i=1hSi X Mi,Mi:=∑bWb,iVOb,i∈Rd×d,\sum_b Y_b=\sum_{i=1}^{h} S_i\,X\Big(\sum_b W^V_{b,i}\,O_{b,i}\Big)=\sum_{i=1}^{h} S_i\,X\,M_i,\qquad M_i:=\sum_b W^V_{b,i}O_{b,i}\in\mathbb{R}^{d\times d},

and Ci:=∑bWb,iQ(Wb,iK)⊤C_i:=\sum_b W^Q_{b,i}(W^K_{b,i})^{\top}. The cross terms never arise because the per-branch products Wb,iVOb,iW^V_{b,i}O_{b,i} are formed first and then summed; nothing is summed before being multiplied. The FFN merge, W1=∑bW1,bW_1=\sum_b W_{1,b} and W2=∑bW2,bW_2=\sum_b W_{2,b}, is exact for the same reason: at λ=1\lambda=1 both branches apply GELU to the same pre-activation. No additional constraint on the branches is required.

Revision. Eq. (1), the "Inference Representations" paragraph, the Implementation Details paragraph, and the collapse step of Algorithm 1 state this rule; the old text that summed WVW^V and WOW^O separately has been removed everywhere.

Numerical agreement before and after merging. Appendix J (new) compares the fully joined training graph, which evaluates each branch's own QQ, KK, VV, OO and sums the branch outputs, against the collapsed model, end to end on the trained six-layer two-branch model (N=197N{=}197, d=192d{=}192, h=3h{=}3), including biases, LayerNorm, residual connections and the classification head, on eight independent random input batches. The maximum absolute logit difference is 7.30×10−57.30\times10^{-5} (dense Ci,MiC_i,M_i) and 3.28×10−53.28\times10^{-5} (factorized form) in FP32, and 1.22×10−91.22\times10^{-9} / 7.77×10−87.77\times10^{-8} in FP64.

W2 / Q2. Inference tensor shapes and compatibility with standard ViT deployment

Per head ii, per layer, the deployed block computes

stepoperationshape
1XCiXC_iN×dN\times d
2(XCi)X⊤(XC_i)X^{\top}, scaled by 1/(2dk)1/(2\sqrt{d_k})N×NN\times N
3Si=softmax⁡(⋅)S_i=\operatorname{softmax}(\cdot)N×NN\times N
4XMiXM_iN×dN\times d
5Si (XMi)S_i\,(XM_i)N×dN\times d
6∑i=1h(⋅)\sum_{i=1}^{h}(\cdot) over heads, then residualN×dN\times d

followed by the merged FFN. The head dimension does not change: each head still computes one N×NN\times N attention map over all tokens. What changes is only the rank of the per-head value–output map. A standard block computes concat⁡i(SiXWiV) WO=∑iSiX (WiVOi)\operatorname{concat}_i(S_iXW^V_i)\,W^O=\sum_i S_iX\,(W^V_iO_i), i.e., it is already of the form ∑iSiXMi\sum_i S_iXM_i with rank⁡(Mi)≤dh\operatorname{rank}(M_i)\le d_h; ours has the same form with rank⁡(Mi)≤B dh\operatorname{rank}(M_i)\le B\,d_h. Steps 1–6 are two GEMMs, a softmax, and two GEMMs per head, with no custom operator, which is how all latency and throughput numbers in Tables 1 and 15 were measured (plain PyTorch on the RTX 4080, the i9-9900X, and the Cortex-A72). Equivalently, the block can be run through any standard attention kernel by passing Q′=XCiQ'=XC_i, K′=XK'=X, V′=XMiV'=XM_i with an identity output projection.

The cost consequences are now stated in §3.3 as closed-form parameter and MAC counts in hh, BB, dd, NN for the standard, dense-merged, and factorized layers, and all tables were recounted accordingly (e.g., DeiT-Ti-6-R: 3.27M / 774M against 3.05M / 641M for the iso-depth model and 5.72M / 1.25G for 12 layers).

W3 / Q3. Isolating the source of the gains; training specification

What the training-cost numbers are. All three models in Appendix G are trained for the same 300 epochs. Per step, DeiT-Ti-6-R costs 157 ms against 86 ms for the iso-depth DeiT-Ti-6 (1.8×) and 189 ms for the full-depth DeiT-Ti-12 (0.83×). So relative to the full-depth model, which is the model we are proposing to replace, our training is cheaper, not more expensive; the reviewer's question concerns the iso-depth comparison, and we agree it needs a compute-matched control.

Evidence already in the paper that the gain is not generic "more training." (i) Instant joining (λ\lambda set to 1 from the start) uses the same compute as our schedule and loses 4–7 points (Table 5), so the gain depends on the parameterization, not on the step count. (ii) At the same 300-epoch budget, KD from a DeiT-Ti-12 teacher, AutoFormer-style NAS at ±15% FLOPs, weight sharing, LayerDrop, and progressive pruning reach at most 65.2% at six layers against our 70.2% (Table 16). (iii) The un-joined two-branch model (MAE-Ti-6-2†), which has the full training-time capacity but none of the merging, reaches 75.4% at 2× the inference cost (Table 4); our collapsed model recovers 3.8 of the 6.8 points separating the iso-depth model from it, at iso-inference cost. The gain therefore comes from training-time capacity that the merge preserves, which is the paper's thesis; what remains to show is that the iso-depth model cannot buy the same gain with the same compute.

Compute-matched control. To isolate the benefits of re-parameterization from additional training compute, we extend the iso-depth DeiT-Ti-6 baseline to 630 epochs, matching the total GPU-hour budget of DeiT-Ti-6-R (8 GPUs, same base training recipe).

ModelEpochsGPU-hoursTop-1 (%)
DeiT-Ti-630030.0063.4
DeiT-Ti-6 (compute-matched)63063.0063.6
DeiT-Ti-6-R (full training)300 + 8 λ warmup + 40 adjustment61.9670.2

Despite approximately matched training compute, DeiT-Ti-6-R still outperforms the iso-depth baseline by 6.6 percentage points. Extending the baseline to 630 epochs yields no further accuracy improvement, demonstrating that the gains cannot be attributed solely to additional training compute. These results are included in Appendix G.2.

Temperature and schedule, reconciled. There is one temperature and one schedule; the revision states them identically in §3.3, Appendix E, and Algorithm 1:

  • Pre-softmax scale during joining: (1+λ(t)2)dk(1+\lambda(t)^2)\sqrt{d_k}, derived from the variance of WA+λWBW_A+\lambda W_B under the assumption that the two branches' logit matrices are independent with equal variance. At λ=1\lambda=1 this is 2dk2\sqrt{d_k}, which is the constant that appears in Eq. (1).
  • Schedule: λ(t)=t\lambda(t)=t with t∈[0,1]t\in[0,1] the normalized progress within a 10k-step warm-up (the warm-up length and the linear form are ablated in Appendix E), after which λ\lambda is held at 1 for 50k adjustment steps. The same λ\lambda mixes the FFN pre-activations. Joining starts immediately after pre-training (for MAE) or after the 300-epoch supervised schedule (DeiT, Swin); MAE models are then fine-tuned with the recipe of Table 9, DeiT and Swin receive no further fine-tuning.

The sentence in the original §3.3 saying that tt varied "from 0 to 1 during training" was the source of the apparent inconsistency; it has been corrected.

We hope these clarifications, together with the corrected formulation, the attached collapsed models and code, and the verification in Appendix J, resolve the reviewer's concerns about the exactness of the merge and the reconstructibility of the procedure.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.