- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper adds loop-specific token embedding tables to Parallel Loop Transformers (PLT), with projections shared across loops. It compares this input channel, S-PLE, against loop specific zFLoRA adapters and their combination. Experiments at two model scales show lower perplexity for the tested input-only configurations than for the tested adapters. Separately measured mobile-NPU graphs retain a decoding-latency advantage over depth-doubled dense references.
Critical strengths
The approach is simple and addresses a relevant problem: improving shared-weight models while limiting active inference computation. The likelihood improvements are encouraging, and the 906M configuration offers a potentially useful storage-latency trade-off.
The authors also disclose capacity budgets, mixed downstream results, prefill regressions and limitations of their deployment measurements.
Critical weaknesses
- The contribution of loop-specific tables is not isolated: Given the established use of per-layer embeddings in Gema, the central question is whether making these embeddings private to each loop provides additional value. The paper does not compare against PLT with per-layer tables shared across loops. Its dense + S-PLE control changes recurrence, executed depth, and table capacity; its projection-sharing control leaves the tables private. I would like to see a shared-table controls at fixed width and across comparable storage budgets, together with a simple shared-table-plus-loop-embedding baseline. The current results establish that added embedding capacity helps, but do not isolate the benefit of the proposed loop-specific allocation.
- The comparisons do not establish how best to spend the parameter and compute budgets. At 135M, S-PLE32/64 add 95.5M/191M parameters, versus 2.95M for rank-8 adapters and 11.8M for rank 32. These results cannot separate the benefit of using embeddings from the enefit of adding more parameters. Please compare configurations with similar parameter bugets, include conventional LoRA, and report full forward/training FLOPs and training cost. The practical trade-off is also imxed: S-PLE64 has 325.5M total praameters versus dense's 241M, worse perplexity and avg-7, and only 0.5ms/token lower graph latency. The combined 135M model is larger, slower, and worse in quality than dense. The 906M results are more promising, but comparisons with dense models at similar quality and deployment budgets are still needed.
- The paper does not show that a deployed model achieves the reported quality and speed together. Quality is measured using floating-point models, while latency is measured using NPU graphs that are quantized. Please measure accuracy, end-to-end TPOT/TTFT, throughput and memory on the same deployed models.
- Important experimental controls are missing. At 906M, the paper does not report quality for PLT without S-PLE or adapters, so we cannot measure how much either addition improves the baseline. The main 135M ablation tests use only eight sequences to compute the perplexity differences. These tests show that the trained model uses the added components, but do not establish whether loop-specific tables are necessary or explain their advantage over adapters.
- Downstream evaluation is inconsistent across scales. the 906M study does not report Avg-7 or the full benchmark suite used at 135M. Appendix H includes CoQA and CNN/DailyMail, but the input-only model's better perplexity does not translate into higher point estimates than the weight-only model on these generative tasks. Reporting the same benchmark suite at both scales, with uncerntainty estimates, would clarify how broadly the improvements apply.
The presentation, particularly the appendix, needs substantial editing. Several passages retain internal experiment shorthand rather than clearly describing the methods: examples include “Hardened pair-2 grid” (Appendix G), “r16 died at 18B and joins at 15B” (Appendix E), and “pre-loss checkpoint” (Table 6). Appendix E also compresses multiple experimental protocols into one long sentence. More importantly, Table 11 does not clearly explain why its perplexities are incompatible with those in Table 12. Please use descriptive model names, separate experimental procedures from qualifications, and state evaluation differences explicitly. Figure 3 also needs its internal logging metadata removed and overlapping annotations corrected.
Questions for authors
-
How does performance change if each layer uses the same embedding table across all loops? Can adding a small embedding that identifies the loop match the performance of separate tables?
-
How well does the 906M PLT model perform without S-PLE or adapters? How do S-PLE and adapters compare when given similar numbers of additional parameters?
-
What quality and end-to-end latency do the quantized models achieve when timing includes table lookups, data transfers, and other application work? Does the speed advantage over depth-doubled dense models remain?
3 comments
Reply #2 to Reviewer exHu
Full 906M evaluation suite and metric corrections
As promised in our W4/W5 response, we report all nine tasks for the 906M comparison. These are single-run point estimates for each arm; no accuracy confidence intervals are supplied.
Metric definitions and corrections
We also correct the earlier presentation: per-task raw accuracy and length-normalized accuracy are now separated, and raw avg-7 is computed from full-precision JSON values. The 906M channel-free arm (E3) has raw avg-7 of 56.6719%, rather than 56.6714% from pre-rounded inputs. The earlier tabulation also multiplied WikiText perplexity by 100 through a percentage-formatting function. The summary below reports word-perplexity in its natural units; accuracy and CoQA F1 are percentages.
Model and control definitions
Arm key: T1 = same-backbone dense-906M; T2 = depth-doubled dense-1.71B; L6 = plain-loop control; E3 = 906M channel-free PLT (machinery only); A1 = input only (sp64); A2 = weight only (zf8); T3 = both channels (zf8+sp64); A3 = both channels (zf8+sp128).
Raw accuracy, acc (%):
| Task | T2 | A3 | T3 | A1 | A2 | L6 | T1 | E3 |
|---|---|---|---|---|---|---|---|---|
| HellaSwag | 43.79 | 40.88 | 40.83 | 40.58 | 40.32 | 40.73 | 39.41 | 40.20 |
| ARC-Easy | 66.58 | 64.06 | 63.22 | 62.25 | 62.08 | 61.20 | 62.71 | 62.88 |
| ARC-Challenge | 31.91 | 28.92 | 28.84 | 27.30 | 29.35 | 28.16 | 27.73 | 28.41 |
| PIQA | 73.07 | 71.76 | 72.03 | 72.31 | 71.55 | 72.03 | 71.16 | 71.65 |
| WinoGrande | 58.09 | 54.78 | 56.27 | 55.17 | 53.51 | 53.12 | 51.62 | 55.33 |
| OpenBookQA | 27.40 | 24.00 | 23.00 | 25.60 | 24.80 | 24.20 | 24.60 | 23.40 |
| BoolQ | 61.99 | 58.87 | 59.11 | 58.72 | 59.39 | 59.11 | 61.25 | 60.49 |
| SciQ | 90.60 | 88.20 | 87.50 | 87.00 | 87.10 | 87.20 | 85.80 | 87.10 |
| LAMBADA | 55.44 | 51.95 | 52.65 | 51.41 | 52.18 | 51.35 | 48.96 | 50.98 |
Length-normalized accuracy, acc_norm (%), where defined:
| Task | T2 | A3 | T3 | A1 | A2 | L6 | T1 | E3 |
|---|---|---|---|---|---|---|---|---|
| HellaSwag | 58.28 | 53.87 | 53.33 | 52.98 | 52.47 | 53.36 | 51.00 | 51.89 |
| ARC-Easy | 60.73 | 57.74 | 58.63 | 55.01 | 57.03 | 55.98 | 55.01 | 57.53 |
| ARC-Challenge | 34.64 | 31.66 | 31.14 | 29.95 | 31.31 | 30.55 | 30.12 | 31.40 |
| PIQA | 74.10 | 72.14 | 71.71 | 71.93 | 72.14 | 72.03 | 71.76 | 71.55 |
| OpenBookQA | 38.20 | 36.40 | 32.60 | 37.00 | 36.20 | 33.00 | 34.00 | 32.80 |
| SciQ | 82.90 | 79.40 | 78.20 | 76.30 | 79.50 | 76.30 | 76.40 | 78.80 |
The harness does not define acc_norm for WinoGrande, BoolQ or LAMBADA.
Summary and additional metrics:
| Arm | Raw avg-7 (%) | WikiText word-PPL | CoQA F1 (%) |
|---|---|---|---|
| T2 | 59.8383 | 18.61 | 52.46 |
| A3 | 57.1025 | 20.26 | 49.27 |
| T3 | 56.9058 | 20.43 | 48.57 |
| A1 | 56.8366 | 20.50 | 47.13 |
| A2 | 56.7738 | 20.78 | 47.69 |
| L6 | 56.5458 | 20.81 | 48.61 |
| T1 | 56.2721 | 21.64 | 45.29 |
| E3 | 56.6719 | 20.83 | 49.32 |
Raw avg-7 is the unweighted mean over the nine-task suite excluding ARC-Challenge and WinoGrande, following the original digest. It is computed before display rounding. For example, E3 and L6 have WikiText word-PPL 20.83 and 20.81; their earlier 2082.66/2080.60 entries were formatting errors, not different evaluations.
Interpretation of the results
All PLT-family raw avg-7 point estimates exceed T1's 56.27%, by 0.27–0.83 percentage points. The observed order is L6 < E3 < A2 < A1 < T3 < A3, with a 0.56-point range within this group. T2 has the highest raw avg-7 (59.84%); it is a larger dense reference, not a matched-parameter comparison. These differences do not establish statistically reliable gains or isolate individual design effects. The likelihood intervals discussed under W4 concern a separate measurement and do not supply accuracy intervals here.
Scope of the comparison
These results describe the evaluated 906M configurations. Because the 135M and 906M studies use different training schedules and primary likelihood metrics, this table does not establish a cross-scale trend.
Thank you for the additional experiments and clarifications. The new 906M baseline and full benchmark table address those reporting gaps. The shared-table and matched-parameter controls are also valuable additions.
My main concerns remain:
-
The benefit of loop-specific tables is still unclear. Shared width64 performs close to private width32 at similar storage. At fixed width64, private tables give only a small NLL improvement while using nearly twice the storage. These results do not establish equivalence, but they provide limited support for making the tables private to each loop. Please revise the main claims accordingly.
-
The matched-parameter result changes the comparison with adapters. Rank256 adapters achieve lower NLL than S-PLE32 at similar parameter counts. Input tables may still offer a compute or latency advantage, but this needs supporting measurements. Please also confirm that the new controls and original references use matching training recipes and evaluation procedures. In particular, the original S-PLE values come from 50B checkpoints within 100B-token runs; matching token counts alone would not control for differences in learning-rate decay.
-
The deployment concern remains unresolved. The simulator results do not establish the quality of the timed NPU models. Quality and end-to-end performance still need to be measured on the same deployed models.
The full 906M results also show small Avg-7 differences and mixed generative performance, so broader quality claims should remain qualified.
These additions improve the completeness of the evaluation, but they do not yet establish a clear benefit from loop-private allocation or a convincing deployment advantage. I therefore maintain my current recommendation.
Reply to Reviewer exHu
Thank you for separating allocation, controlled baselines, and deployment evidence. The new measurements narrow several claims while supplying the missing 906M quality baseline.
W1/Q1: shared tables and loop identity. Shared width64 gives 3.0073 NLL at 96.6M auxiliary parameters versus private S-PLE32's 3.0089 at 95.5M; shared width128 gives 3.0503 at 193.2M versus private S-PLE64's 3.0022 at 191.0M. Matching storage changes width and projection size. At fixed width64, shared/private are 3.0073/3.0022; at width128, 3.0503/3.0506, with private tables using almost twice the storage. Each new arm has one training seed; numerical closeness is not statistical equivalence.
The width64 loop-ID variant adds 1,152 parameters and changes NLL by +0.0000249. This does not answer the untested width128 loop-ID alternative or explain that budget's privacy gap. We limit the interpretation to the tested allocation and width.
W2/Q2: parameters, compute, and dense references. Matched-budget weight-only zFLoRA256 reaches 2.9835 NLL with 94.4M auxiliary parameters, versus S-PLE32's 3.0089 with 95.5M: the adapter wins by 0.0254. We have launched standalone conventional LoRA at approximately 95.55M auxiliary parameters, rank313 on all seven projections, shared across loops and without PLE. Its matched 50B result is not available; the old LoRA-plus-S-PLE experiment is not a substitute.
We agree about the 135M trade-off: S-PLE64 has 325.5M total parameters versus dense's 241M, worse perplexity/avg-7, and only 0.50 ms/token better graph TPOT. The combined 135M arm also fails to dominate dense. At 906M, input-only graph TPOT is 11.37 versus 19.12 ms/token for depth-doubled dense, but dense has better absolute quality. The 600/1057 MB values are nominal byte budgets, with int4 tables in the input estimate, not measured application peak memory or a verified joint quality/storage operating point. Matched-quality dense deployment comparisons remain open.
The new cost table measures allocated GPU-hours. It does not provide full forward/training FLOPs or energy. The inconsistent E9 absolute estimates are withdrawn; the existing backbone-only plotting proxy is a separate limited approximation that excludes channels, attention products, and head work. It cannot establish full compute matching. Corrected full accounting remains incomplete.
W3/Q3: quality and end-to-end deployment. Joint device quality/speed remains unmeasured. CPU fp32 timing uses random initialization, not NPU calls, and cannot supply handset TPOT/TTFT. Within-checkpoint W4A8 simulator degradation is +3.06 ppl [2.89,3.24] for plain-recurrent input-only, +3.64 [3.45,3.84] for plain-recurrent rank8, +2.01 [1.90,2.13] for PLT rank256, and +2.46 [2.33,2.60] for PLT shared width64. These are conditional paired CIs. The group128 RTN W4/per-token A8 policy with fp tables differs from handset W4A16-KV8; the model implementations differ too. Original speed claims remain graph-level. Lookup/data-transfer-inclusive latency, throughput, peak memory, and quantized quality on identical deployed artifacts remain unresolved; slower prefill remains a limitation.
W4/Q2: channel-free baseline and subset probes. New 906M channel-free PLT gives 14.097 validation perplexity at the decayed 50B endpoint, versus plain recurrence 14.20 and input-only PLT 13.86. This supplies the within-PLT baseline, not a complete factorial or a 135M recipe replicate. Shared-width64 probes use 130 sequences, 2000 sequence-bootstrap resamples and five shuffle seeds: zero +6.58 [6.54,6.63], shuffle +7.68 [7.62,7.75]. They measure channel dependence, not necessity of privacy. Legacy purged scratch checkpoints cannot be re-probed; their eight-sequence estimates remain limited. Permutation/bootstrap seeds are not training seeds.
W5: downstream consistency. Raw-accuracy avg-7 at 906M is input-only 56.84, weight-only 56.77, plain-loop 56.55, same-backbone dense 56.27, and depth-doubled dense 59.84. New channel-free PLT gives 56.67 and suite WikiText word-ppl 20.83. These small accuracy differences are descriptive; web-ppl CIs are not accuracy CIs. CoQA F1 is 0.471 input-only, 0.477 weight-only, and 0.493 for the new channel-free arm, on 500 samples using the train-consistent strategy-A decoder in App. I. Input-only also does not lead weight-only on CNNDM. We will add the complete accuracy table, separating point estimates from available uncertainty.
W6: presentation. The current revision replaces App. E/G shorthand, explains the interrupted checkpoint, and distinguishes subset-harness scoring from trainer-parity full-split values. Figure 3 now has descriptive labels, a separate legend, and an unclipped title. New results reported only in prose will not be described as completed table or figure additions. These responses state the qualified interpretation; remaining stronger PDF wording must be aligned with it.