- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper presents an approach that targets real problems in the deployment of vision transformers under resource constraints. It argues that instead of deploying ViTs with significant depth, it could establish an effective framework that enables shallower models to have similar performance compared to their deeper peers. Specifically, such a framework can be achieved by so-called multi-branch training that could later be collapsed into a single-branch inference network without performance loss when the relevant parameter converges to 1. The paper devised several re-parameterization aspects to aid the training and the final branch collapse, and it later puts its method to the test against several baselines with further ablation studies. However, there appears to be a major mathematical flaw that would significantly weaken one of the central claims made in the paper.
Critical strengths
- The research question is real and well motivated, and this paper serves as a good exploration of methods that would turn multi-branch training into single-branch inference that could bring real benefits under resource constraints.
- The method is tested on several different datasets and compared against competing methods, providing evaluation beyond a single benchmark, with substantial ablation studies that help interpret the method.
Critical weaknesses
- The key claim: "Our approach mitigates this by training parallel transformer branches that are later collapsed into a single branch at inference, drastically reducing depth while preserving representational capacity. The key idea is that, with proper constraints, parallel branches can be algebraically re-parameterized into one equivalent branch without approximation error" is not supported by the method in its current form. Specifically, for the two branches and , even at the merge point , the softmax part should be identical, while the sum of the outputs from different branches should have as the last multiplier, but in the proposed formulation after Equation 1, the output would have instead, which introduces additional cross terms. As a result, in its current form, the method would not in general produce results identical to those of the multi-branch model at single-branch inference, as claimed in the paper.
- Several numerical claims in the text are inconsistent with the paper's own tables, and there are inconsistencies across different tables. Section 4.1 reports that MAE-Ti-6-R has an improvement of 6.7% against its baseline, while Tables 1, 2, and 4 all have 3.8%. The claimed accuracy of Swin-S-12-R is reported at 81.5% in Table 2 but 82.0% in Table 15, with no explanation for the difference. These inconsistencies could generally be reporting mistakes and could sometimes be traced in the supplement, but unfortunately they still raise concerns about the reliability of the reported results.
Questions for authors
- Can you reexamine whether the exact algebraic equivalence is established in the current framework and is supported by numerical results? Furthermore, is the current form of branch collapsing implemented in the supplemented code and used to evaluate the performance of the model?
- Can you check the discrepancies between the text and tables and between different tables against the experiment records? Please clarify the correct values and settings, and whether the corrections affect the conclusions.
2 comments
W2 / Q2. Discrepancies between text and tables
We checked every number cited in the text against the tables and every table against the experiment logs. The two the reviewer found, and what we did:
- MAE-Ti-6-R gain. §4.1 stated +6.7%. The tables are correct: MAE-Ti-6-R reaches 72.4% against 68.6% for the iso-depth MAE-Ti-6†, a gain of +3.8%. The +6.7% figure belongs to MAE-Ti-3-R (60.5% vs 53.8%) and was carried into the wrong sentence. §4.1 now reads +6.8% (DeiT-Ti-6-R) and +3.8% (MAE-Ti-6-R†).
- Swin-S-12-R. The experiment log for the final Swin-S-12-R checkpoint gives 82.0%, the value in Table 15 and in the cross-depth comparison. The 81.5% in Table 2 was from a previous run; Table 2 now reads 82.0%, and the parameter and FLOP entries of the Swin rows were reconciled across Tables 2 and 15 at the same time.
The audit also found stale cost numbers in the prose that pre-dated the recount of parameters and FLOPs (for example, the FLOP figures quoted for MAE-L-6-R in the token-reduction section and for MAE-L-16-R in Appendix B); all now match the tables. The full list of corrections is in the revision, marked in blue.
Do the corrections affect the conclusions? The accuracy results do not change except for the two reporting corrections above, and the pattern they support is unchanged: the recovered gain over the iso-depth baseline is +6.7, +6.0 and +3.8 points at 3, 4 and 6 layers for MAE-Ti, and +6.8 points for DeiT-Ti-6-R. The recounted parameters and FLOPs do change one statement: a re-parameterized model is not free at iso-depth, costing 6–16% more parameters and 8–22% more FLOPs than its iso-depth baseline (DeiT-Ti-6-R: 3.27M / 774M vs 3.05M / 641M), and the text now says so where it previously said "without additional inference cost". Relative to the full-depth models the six-layer models remain 38–43% smaller in both parameters and FLOPs, and the measured latency and throughput are unchanged because they were always measured on the collapsed models.
We are grateful for the scrutiny; the corrected formulation, the attached code and collapsed checkpoints, and the end-to-end equivalence test allow the central claim to be checked directly, and the reconciled tables now agree with the text and with each other.
We thank the reviewer for the careful reading and for the two concrete findings: the merge rule as written in §3.3 is not exact, and the text and tables disagreed in places. Both are correct, and we address them in turn.
W1 / Q1. Exact algebraic equivalence, and what the code does
The reviewer's analysis is right. At λ=1 the softmax of every branch receives the same input, so the attention map is shared; the sum of the branch outputs then has ∑bWbVOb as its last factor, whereas the merge written after Eq. (1), (∑bWbV)(∑bOb), contains the additional cross terms ∑b=b′WbVOb′. Written that way the merge is not exact, and the paper's claim would not hold.
What is implemented. The collapse code in the supplementary material (
inference_fusion.py) does not sum the two projections separately. For every head i it formsCi=∑bWb,iQ(Wb,iK)⊤,Mi=∑bWb,iVOb,i=WA,iVOA,i+WB,iVOB,i,
where Ob,i is the row block of WbO for head i, and the FFN merges as W1=∑bW1,b, W2=∑bW2,b. With Si=softmax(XCiX⊤/(2dk)) common to all branches, the sum of the branch outputs is exactly ∑iSiXMi: the per-branch products are formed first and summed afterwards, so no cross term arises and no approximation is made. The FFN merge is exact because all branches apply GELU to the same pre-activation at λ=1.
Was this used for the reported results? Yes. Every accuracy, latency, throughput and memory number in the paper was obtained from models collapsed by this rule; the error was confined to the exposition in §3.3 and Algorithm 1. We have rewritten Eq. (1), the Inference Representations and Implementation Details paragraphs, the collapse step of Algorithm 1 and Figure 1 to state the rule above, and we have attached the deployed collapsed checkpoints so that the merge can be checked directly against the multi-branch checkpoints.
Numerical support. Appendix J (new) compares the fully joined training graph, which evaluates each branch's own Q, K, V, O and sums the branch outputs, against the collapsed model, end to end on the trained six-layer two-branch model (N=197, d=192, h=3), including biases, LayerNorm, residual connections and the classification head, on eight independent random input batches. The maximum absolute logit difference is 7.30×10−5 (dense Ci,Mi) and 3.28×10−5 (factorized form) in FP32, and 1.22×10−9 / 7.77×10−8 in FP64.
Because the exact merge stores Ci and Mi per head, a collapsed layer is not the same size as a standard layer. The revised §3.3 gives the parameter and MAC counts in closed form, and every table was recounted for the collapsed model actually deployed (see below).