- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper suggests a method to approximate the sequential operation of deep nets using parallelized newton iterations. It shows empirical results on the tradeoffs between speedup and perplexity.
Critical strengths
- The idea is interesting and potentially impactful.
- The paper contain many experiments.
Critical weaknesses
- The empirical results are simply not good enough. We trade-off perplexity for speed, and the tradeoff doesn't seem reasonable at the 3B model (3.9% PPL increase for 1.2x speedup). Even at the 0.5B model, the strongest result in terms of quality, which achieves a speed-up of 1.4, has K=1 so I think this means there is no non-trivial Newton correction. Also, the downstream performance (accuracy gap) and of-the-shelf model performance (no speedup) aren't great.
- This method cannot compute a full path going from input to output through all the layers, if it uses less iterations that the number of layers. Therefore, it will only work if the original net depth was not strictly necessary and could be replaced with width. Thus, this method is essentially replacing a deep architecture with shallower and wider architecture. Consequently, a necessary baseline (currently missing from the paer) would be to compare against a computationally comparable and more standard architecture (e.g., just shallower and wider to allow a fair comparison in terms of memory/compute/parallelizability/latency) distilled from the original architecture.
Questions for authors
I don't have any questions, but I have some comments on the presentation, to help the authors revise their papers:
• The writing keeps discussing concepts that were not yet explained (e.g., “but for IDN/HCN the surrogate is trivial to compute or available from the architecture, making correction cheap relative to a Transformer block”), and there are vague sentences (such as “Off-the-shelf sequential models need not have layer dynamics that match a cheap surrogate”, or “stride-selected supervised layers”, “one-shot batched forward”).
• Eq. 3 is hard to understand without explaining what is the initialization.
• in eq. 8 it will be good to more precisely define what the mixing matrices do (or at least reference the appendix).
• I didn't understand what does the sum over N in eq. 10 means or why do we have it.
• Some notation is not defined when they are first introduced, such as \mathcal{S} in line 251, or what does “surrogate family A” means in line 253. It is later becomes a bit clearer, but it takes too long to get there and which makes this unnecessarily hard to read.
• Not all acronyms are defined (e.g. SFT, mHC, FHE). Even if some acronyms are well-known, they should be defined for better accessibility.
• The paragraph “Inference-side effects” was not clear to me, and includes vague references to the appendix. For example, “This can reduce layerwise error compounding under the approximation in Section B.1”. What is this approximation and why is this important? the main paper should be self contained, and the appendix include only additional details which the readers can decide if they want to read; here it is unclear what information is in the appendix and if we should read it.
• “The results suggest that SNLP benefits from structured depth coupling rather than removing depth interactions”. Why? what implies this?
• What is IDN. Reg. or HCN Reg. ? I don't see where this was defined as a regularization, I thought it means a specific Newton correction.
• The columns in Table 2 and Table 3 are not defined in the table, and it is hard to find where they are explained in the text (they are not immediately near the table, and they are not referenced in the table).
• In equation 12 we have \epsilon, \epsilon_l and \epsilon_{l,b}, and the relation between all these is not clear.
2 comments
I thank the authors for their response. However, following the same reasoning as reviewer CGhC, I still have the same concerns and vote for rejection.
We sincerely thank the reviewer for the very careful reading and the detailed presentation comments; they will substantially improve the manuscript.
W1 (trade-off; 1.40× has K=1): We agree that the 3B trade-off is modest (see our response to sK25 on the regime). Our claim is a speed-quality frontier rather than a single operating point. 1×F12 is its single-chunk end, where the correction is zero by construction; the correction is what enables the multi-chunk points. Note that K=1 does not by itself imply a trivial correction: the correction vanishes for 1×F12 only because it has a single chunk. With more than one chunk, the IDN correction is active and essential even at K=1 (e.g., 12×F2 at 2.58× in Table 1). In Table 3, removing it at K=1 raises PPL from 17.2/17.9/18.0 to 37.4/67.2/div. for N=8/12/16. Please see C1 for a matched comparison with Calvo-style layer pairing and with the uncorrected iteration.
On downstream accuracy and off-the-shelf models, we agree that gaps remain and state them in the limitations. Both point to the role of SNLP-aware training. After standard SFT, switching inference to IDN costs 8.34 accuracy points on the 0.5B base model but only 0.37 points on the IDN-regularized model (3.25 vs. 0.38 points at 3B; App. D.1). Off-the-shelf models were not trained for a cheap surrogate, so matching their sequential PPL post hoc requires multiple iterations and yields no speedup (App. E.4).
W2 (shallower-wider baseline): We agree this is a natural and important baseline. A distilled shallower-wider model requires a new training run, which is beyond our rebuttal budget; we now discuss this baseline and its trade-offs in the limitations. SNLP is not equivalent to removing depth. The suffix layers are not droppable: in our early-exit table, skipping 8–16 layers raises PPL from 16.8 to 50–1785. SNLP also keeps a single set of weights whose depth/latency trade-off can be chosen per request, by choosing K or falling back to exact sequential inference. That is also what makes it usable as a self-speculative drafter verified by the same model, which a separately distilled shallow model cannot provide.
Presentation. Thank you; we have revised all points in the uploaded PDF:
Concepts used before definition / vague phrases. Where Sec. 3.2 refers to the IDN/HCN surrogates, it now states what they are and points to Sec. 3.3. We also define precisely:
fwd);We also rephrased the vague sentence on off-the-shelf models.
Eq. 3 initialization. Stated right after Eq. 3: hl(0)=hS or hl(0)=fl(hS), referred to as
h0andfwdin Sec. 3.5.Eq. 8. We now define Hres in the main text: mHC keeps M residual streams, and each attention/MLP sub-block updates them as x′=Hresx+HpostF(Hprex), where Hres is the learned M×M stream-mixing matrix (details in App. B.2).
Sum over N in Eq. 10. S is the set of suffix lengths used in training, so one model is trained for several parallel-suffix sizes at once. This is now stated explicitly.
S and "surrogate family A". S is now defined at first use, and "surrogate family A" is replaced by "surrogate A∈{I,Hres,diag(J)} for IDN, HCN, and DiagN".
Acronyms. SFT, HC/mHC, and FHE are now spelled out at first use, and DEER and ELK are briefly described.
"Inference-side effects". Rewritten to be self-contained. One-step IDN evaluates every suffix branch at hS, so it differs from sequential execution by the summed substitution errors ∑lϵl, ϵl=gl(hl−1seq)−gl(hS). This is a bias that SNLP training keeps small, while, to first order, errors do not compound through later layers. The appendix only contains the derivation.
"Structured depth coupling". The sentence is softened and now cites its evidence: removing the correction degrades PPL sharply (Table 3, "No correction"; App. E correction propagation), and forward correction order is consistently best.
"IDN Reg." / "HCN Reg." Defined in Sec. 3.4 and the Table 1 caption: models pretrained with the matching loss of Eq. 10 using the IDN or HCN surrogate, versus "No Reg." (standard pretraining).
Table 2/3 columns. All columns are now defined in the captions, including "div." (PPL on the order of 104 or more).
Eq. 12 notation. ϵl is the substitution error of layer l and ϵl,b its value at token b; the stability constant in Eqs. 10 and 12 is renamed δ.