acceptodds
Under review as a conference paper at ICLR 2027

LooPiT: Input-Space Specialization for Parallel Loop Transformers

11 views

Abstract

Weight-shared recurrence improves language modeling without duplicating the backbone, but sequential loops still execute the full recurrent depth. Cross-loop parallelism (CLP) addresses the decode schedule, but leaves the shared-backbone capacity constraint intact. We study loop-specific capacity under CLP, using Parallel Loop Transformers (PLT) as the substrate, without restoring a full set of unique layers. Alongside the established weight-space route of per-loop adapters, we investigate an input-space route: token-indexed tables private to each layer and loop, with shared projections (S-PLE). On a 135M backbone trained for 100B tokens, S-PLE improves geometric-mean perplexity from 27.50 to 22.91, compared with 26.09 for the tested adapters. A 906M study and two-seed checks support the input-over-weight ordering. Rank and width sweeps test how performance changes with added capacity, while channel ablations examine how each loop uses the learned channels. LooPiT uses approximately 45% fewer backbone parameters than depth-doubled dense models across both scales, with approximately 42% fewer total parameters at the larger scale. On a mobile NPU, the input-only configurations retain a graph decode-latency advantage at both scales. The evidence supports input-space specialization as an effective, compatible alternative to weight adaptation within the tested quality–storage–latency trade-offs; prefill efficiency remains an optimization target.

Primary area:
optimization
Keywords:
recurrent transformers, looped models, cross-loop parallelism, per-layer embedding, zflora, compander

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_exHuOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper adds loop-specific token embedding tables to Parallel Loop Transformers (PLT), with projections shared across loops. It compares this input channel, S-PLE, against loop specific zFLoRA adapters and their combination. Experiments at two model scales show lower perplexity for the tested input-only configurations than for the tested adapters. Separately measured mobile-NPU graphs retain a decoding-latency advantage over depth-doubled dense references.

Critical strengths

The approach is simple and addresses a relevant problem: improving shared-weight models while limiting active inference computation. The likelihood improvements are encouraging, and the 906M configuration offers a potentially useful storage-latency trade-off.

The authors also disclose capacity budgets, mixed downstream results, prefill regressions and limitations of their deployment measurements.

Critical weaknesses
  1. The contribution of loop-specific tables is not isolated: Given the established use of per-layer embeddings in Gema, the central question is whether making these embeddings private to each loop provides additional value. The paper does not compare against PLT with per-layer tables shared across loops. Its dense + S-PLE control changes recurrence, executed depth, and table capacity; its projection-sharing control leaves the tables private. I would like to see a shared-table controls at fixed width and across comparable storage budgets, together with a simple shared-table-plus-loop-embedding baseline. The current results establish that added embedding capacity helps, but do not isolate the benefit of the proposed loop-specific allocation.
  2. The comparisons do not establish how best to spend the parameter and compute budgets. At 135M, S-PLE32/64 add 95.5M/191M parameters, versus 2.95M for rank-8 adapters and 11.8M for rank 32. These results cannot separate the benefit of using embeddings from the enefit of adding more parameters. Please compare configurations with similar parameter bugets, include conventional LoRA, and report full forward/training FLOPs and training cost. The practical trade-off is also imxed: S-PLE64 has 325.5M total praameters versus dense's 241M, worse perplexity and avg-7, and only 0.5ms/token lower graph latency. The combined 135M model is larger, slower, and worse in quality than dense. The 906M results are more promising, but comparisons with dense models at similar quality and deployment budgets are still needed.
  3. The paper does not show that a deployed model achieves the reported quality and speed together. Quality is measured using floating-point models, while latency is measured using NPU graphs that are quantized. Please measure accuracy, end-to-end TPOT/TTFT, throughput and memory on the same deployed models.
  4. Important experimental controls are missing. At 906M, the paper does not report quality for PLT without S-PLE or adapters, so we cannot measure how much either addition improves the baseline. The main 135M ablation tests use only eight sequences to compute the perplexity differences. These tests show that the trained model uses the added components, but do not establish whether loop-specific tables are necessary or explain their advantage over adapters.
  5. Downstream evaluation is inconsistent across scales. the 906M study does not report Avg-7 or the full benchmark suite used at 135M. Appendix H includes CoQA and CNN/DailyMail, but the input-only model's better perplexity does not translate into higher point estimates than the weight-only model on these generative tasks. Reporting the same benchmark suite at both scales, with uncerntainty estimates, would clarify how broadly the improvements apply.

The presentation, particularly the appendix, needs substantial editing. Several passages retain internal experiment shorthand rather than clearly describing the methods: examples include “Hardened pair-2 grid” (Appendix G), “r16 died at 18B and joins at 15B” (Appendix E), and “pre-loss checkpoint” (Table 6). Appendix E also compresses multiple experimental protocols into one long sentence. More importantly, Table 11 does not clearly explain why its perplexities are incompatible with those in Table 12. Please use descriptive model names, separate experimental procedures from qualifications, and state evaluation differences explicitly. Figure 3 also needs its internal logging metadata removed and overlapping annotations corrected.

Questions for authors
  1. How does performance change if each layer uses the same embedding table across all loops? Can adding a small embedding that identifies the loop match the performance of separate tables?

  2. How well does the 906M PLT model perform without S-PLE or adapters? How do S-PLE and adapters compare when given similar numbers of additional parameters?

  3. What quality and end-to-end latency do the quantized models achieve when timing includes table lookups, data transfers, and other application work? Does the speed advantage over depth-doubled dense models remain?

3 comments
AuthorsOpenReview ↗

Reply to Reviewer exHu

Thank you for separating allocation, controlled baselines, and deployment evidence. The new measurements narrow several claims while supplying the missing 906M quality baseline.

W1/Q1: shared tables and loop identity. Shared width64 gives 3.0073 NLL at 96.6M auxiliary parameters versus private S-PLE32's 3.0089 at 95.5M; shared width128 gives 3.0503 at 193.2M versus private S-PLE64's 3.0022 at 191.0M. Matching storage changes width and projection size. At fixed width64, shared/private are 3.0073/3.0022; at width128, 3.0503/3.0506, with private tables using almost twice the storage. Each new arm has one training seed; numerical closeness is not statistical equivalence.

The width64 loop-ID variant adds 1,152 parameters and changes NLL by +0.0000249. This does not answer the untested width128 loop-ID alternative or explain that budget's privacy gap. We limit the interpretation to the tested allocation and width.

W2/Q2: parameters, compute, and dense references. Matched-budget weight-only zFLoRA256 reaches 2.9835 NLL with 94.4M auxiliary parameters, versus S-PLE32's 3.0089 with 95.5M: the adapter wins by 0.0254. We have launched standalone conventional LoRA at approximately 95.55M auxiliary parameters, rank313 on all seven projections, shared across loops and without PLE. Its matched 50B result is not available; the old LoRA-plus-S-PLE experiment is not a substitute.

We agree about the 135M trade-off: S-PLE64 has 325.5M total parameters versus dense's 241M, worse perplexity/avg-7, and only 0.50 ms/token better graph TPOT. The combined 135M arm also fails to dominate dense. At 906M, input-only graph TPOT is 11.37 versus 19.12 ms/token for depth-doubled dense, but dense has better absolute quality. The 600/1057 MB values are nominal byte budgets, with int4 tables in the input estimate, not measured application peak memory or a verified joint quality/storage operating point. Matched-quality dense deployment comparisons remain open.

The new cost table measures allocated GPU-hours. It does not provide full forward/training FLOPs or energy. The inconsistent E9 absolute estimates are withdrawn; the existing backbone-only plotting proxy is a separate limited approximation that excludes channels, attention products, and head work. It cannot establish full compute matching. Corrected full accounting remains incomplete.

W3/Q3: quality and end-to-end deployment. Joint device quality/speed remains unmeasured. CPU fp32 timing uses random initialization, not NPU calls, and cannot supply handset TPOT/TTFT. Within-checkpoint W4A8 simulator degradation is +3.06 ppl [2.89,3.24] for plain-recurrent input-only, +3.64 [3.45,3.84] for plain-recurrent rank8, +2.01 [1.90,2.13] for PLT rank256, and +2.46 [2.33,2.60] for PLT shared width64. These are conditional paired CIs. The group128 RTN W4/per-token A8 policy with fp tables differs from handset W4A16-KV8; the model implementations differ too. Original speed claims remain graph-level. Lookup/data-transfer-inclusive latency, throughput, peak memory, and quantized quality on identical deployed artifacts remain unresolved; slower prefill remains a limitation.

W4/Q2: channel-free baseline and subset probes. New 906M channel-free PLT gives 14.097 validation perplexity at the decayed 50B endpoint, versus plain recurrence 14.20 and input-only PLT 13.86. This supplies the within-PLT baseline, not a complete factorial or a 135M recipe replicate. Shared-width64 probes use 130 sequences, 2000 sequence-bootstrap resamples and five shuffle seeds: zero +6.58 [6.54,6.63], shuffle +7.68 [7.62,7.75]. They measure channel dependence, not necessity of privacy. Legacy purged scratch checkpoints cannot be re-probed; their eight-sequence estimates remain limited. Permutation/bootstrap seeds are not training seeds.

W5: downstream consistency. Raw-accuracy avg-7 at 906M is input-only 56.84, weight-only 56.77, plain-loop 56.55, same-backbone dense 56.27, and depth-doubled dense 59.84. New channel-free PLT gives 56.67 and suite WikiText word-ppl 20.83. These small accuracy differences are descriptive; web-ppl CIs are not accuracy CIs. CoQA F1 is 0.471 input-only, 0.477 weight-only, and 0.493 for the new channel-free arm, on 500 samples using the train-consistent strategy-A decoder in App. I. Input-only also does not lead weight-only on CNNDM. We will add the complete accuracy table, separating point estimates from available uncertainty.

W6: presentation. The current revision replaces App. E/G shorthand, explains the interrupted checkpoint, and distinguishes subset-harness scoring from trainer-parity full-split values. Figure 3 now has descriptive labels, a separate legend, and an unclipped title. New results reported only in prose will not be described as completed table or figure additions. These responses state the qualified interpretation; remaining stronger PDF wording must be aligned with it.

AuthorsOpenReview ↗

Reply #2 to Reviewer exHu

Full 906M evaluation suite and metric corrections

As promised in our W4/W5 response, we report all nine tasks for the 906M comparison. These are single-run point estimates for each arm; no accuracy confidence intervals are supplied.

Metric definitions and corrections

We also correct the earlier presentation: per-task raw accuracy and length-normalized accuracy are now separated, and raw avg-7 is computed from full-precision JSON values. The 906M channel-free arm (E3) has raw avg-7 of 56.6719%, rather than 56.6714% from pre-rounded inputs. The earlier tabulation also multiplied WikiText perplexity by 100 through a percentage-formatting function. The summary below reports word-perplexity in its natural units; accuracy and CoQA F1 are percentages.

Model and control definitions

Arm key: T1 = same-backbone dense-906M; T2 = depth-doubled dense-1.71B; L6 = plain-loop control; E3 = 906M channel-free PLT (machinery only); A1 = input only (sp64); A2 = weight only (zf8); T3 = both channels (zf8+sp64); A3 = both channels (zf8+sp128).

Raw accuracy, acc (%):

TaskT2A3T3A1A2L6T1E3
HellaSwag43.7940.8840.8340.5840.3240.7339.4140.20
ARC-Easy66.5864.0663.2262.2562.0861.2062.7162.88
ARC-Challenge31.9128.9228.8427.3029.3528.1627.7328.41
PIQA73.0771.7672.0372.3171.5572.0371.1671.65
WinoGrande58.0954.7856.2755.1753.5153.1251.6255.33
OpenBookQA27.4024.0023.0025.6024.8024.2024.6023.40
BoolQ61.9958.8759.1158.7259.3959.1161.2560.49
SciQ90.6088.2087.5087.0087.1087.2085.8087.10
LAMBADA55.4451.9552.6551.4152.1851.3548.9650.98

Length-normalized accuracy, acc_norm (%), where defined:

TaskT2A3T3A1A2L6T1E3
HellaSwag58.2853.8753.3352.9852.4753.3651.0051.89
ARC-Easy60.7357.7458.6355.0157.0355.9855.0157.53
ARC-Challenge34.6431.6631.1429.9531.3130.5530.1231.40
PIQA74.1072.1471.7171.9372.1472.0371.7671.55
OpenBookQA38.2036.4032.6037.0036.2033.0034.0032.80
SciQ82.9079.4078.2076.3079.5076.3076.4078.80

The harness does not define acc_norm for WinoGrande, BoolQ or LAMBADA.

Summary and additional metrics:

ArmRaw avg-7 (%)WikiText word-PPLCoQA F1 (%)
T259.838318.6152.46
A357.102520.2649.27
T356.905820.4348.57
A156.836620.5047.13
A256.773820.7847.69
L656.545820.8148.61
T156.272121.6445.29
E356.671920.8349.32

Raw avg-7 is the unweighted mean over the nine-task suite excluding ARC-Challenge and WinoGrande, following the original digest. It is computed before display rounding. For example, E3 and L6 have WikiText word-PPL 20.83 and 20.81; their earlier 2082.66/2080.60 entries were formatting errors, not different evaluations.

Interpretation of the results

All PLT-family raw avg-7 point estimates exceed T1's 56.27%, by 0.27–0.83 percentage points. The observed order is L6 < E3 < A2 < A1 < T3 < A3, with a 0.56-point range within this group. T2 has the highest raw avg-7 (59.84%); it is a larger dense reference, not a matched-parameter comparison. These differences do not establish statistically reliable gains or isolate individual design effects. The likelihood intervals discussed under W4 concern a separate measurement and do not supply accuracy intervals here.

Scope of the comparison

These results describe the evaluated 906M configurations. Because the 135M and 906M studies use different training schedules and primary likelihood metrics, this table does not establish a cross-scale trend.

Thank you for the additional experiments and clarifications. The new 906M baseline and full benchmark table address those reporting gaps. The shared-table and matched-parameter controls are also valuable additions.

My main concerns remain:

  1. The benefit of loop-specific tables is still unclear. Shared width64 performs close to private width32 at similar storage. At fixed width64, private tables give only a small NLL improvement while using nearly twice the storage. These results do not establish equivalence, but they provide limited support for making the tables private to each loop. Please revise the main claims accordingly.

  2. The matched-parameter result changes the comparison with adapters. Rank256 adapters achieve lower NLL than S-PLE32 at similar parameter counts. Input tables may still offer a compute or latency advantage, but this needs supporting measurements. Please also confirm that the new controls and original references use matching training recipes and evaluation procedures. In particular, the original S-PLE values come from 50B checkpoints within 100B-token runs; matching token counts alone would not control for differences in learning-rate decay.

  3. The deployment concern remains unresolved. The simulator results do not establish the quality of the timed NPU models. Quality and end-to-end performance still need to be measured on the same deployed models.

The full 906M results also show small Avg-7 differences and mixed generative performance, so broader quality claims should remain qualified.

These additions improve the completeness of the evaluation, but they do not yet establish a clear benefit from loop-private allocation or a convincing deployment advantage. I therefore maintain my current recommendation.

Reviewer_PkmSOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper adds separate token embeddings for each layer and loop of a Parallel Loop Transformer. These embeddings give each loop extra parameters while keeping the main Transformer weights shared. The authors compare this design with loop-specific adapters and with using both together. Experiments at two model sizes show improved language modeling results, and mobile measurements examine decoding time and model storage.

Critical strengths
  1. The embedding design improves language modeling results at both model sizes. This makes it a useful option for improving shared models without copying the full backbone.

  2. The study includes higher adapter ranks, different embedding sizes, and additional training seeds. These checks make the observed improvement more convincing than a single model comparison.

Critical weaknesses
  1. [Medium] The new idea is a modest extension of existing methods. Prior work already uses loop-specific adapters and per-layer embeddings. The main change is to give each layer and loop its own embedding table. The paper does not clearly show why separate tables for different loops are needed, rather than a simpler shared table with a comparable parameter budget. Removing a channel after training shows that the model uses it, but does not answer this design question. This limits what we learn beyond the performance of the chosen configuration (Sections 2, 3.2 and 5; Table 3).

  2. [Major] The better results may simply come from adding more parameters. The embedding models add many more parameters than the adapter models. Increasing adapter rank helps, but the tested adapters remain much smaller and the rank sweep stops before the full training budget. Appendix E also tests standard LoRA, but together with embeddings, rather than as a standalone alternative to them. A comparison of different adapter methods with the same total parameter count is needed to test whether embeddings are actually the better choice (Section 4.4; Appendix E, Table 14; Appendix J.1).

  3. [Major] Larger and alternative adapters are missing from the speed comparison. Figure 7 compares embeddings with a low-rank adapter at similar graph decoding times, but does not show the quality and speed of higher-rank or alternative adapter methods. The argument that larger adapters would be too slow therefore remains untested. A quality-versus-speed curve for these methods is needed to compare prediction quality at the same decoding speed. Timing should include embedding lookup, and quality should be measured on the same model used for timing (Section 6; Appendix J.2, Figure 7; Appendix K).

Questions for authors
  1. Clarification: Is there a training comparison between separate tables for each loop and a table shared across loops, with a similar total parameter budget? The removal tests in Section 5 do not answer whether separate tables are needed. If this comparison is unavailable, it is a targeted experiment that would help assess the main design choice.

  2. Requested comparison: Do embeddings still perform better than higher-rank and other adapter methods when the total number of parameters is the same? The standard LoRA experiment in Appendix E also uses embeddings, so it does not answer this question for an adapter-only model.

  3. Requested comparison: Can you extend Figure 7 to higher-rank and other adapter methods, so that prediction quality can be compared at the same decoding speed? Please include embedding lookup in the timing and measure quality on the model used for timing.

3 comments
AuthorsOpenReview ↗

Reply to Reviewer PkmS

Thank you. We agree that shared-table training, parameter matching, and joint quality/speed measurements answer different questions; the new evidence should not be used interchangeably across them.

W1/Q1: are private tables needed? At approximately matched auxiliary storage, shared width64 gives 3.0073 NLL with 96.6M parameters, versus private S-PLE32's 3.0089 with 95.5M. Shared width128 gives 3.0503 with 193.2M, versus private S-PLE64's 3.0022 with 191.0M. The first pair is numerically close; private allocation improves the second by 0.0481 NLL. Each new arm has one training seed. Matching storage changes width and projection size, so the experiment compares allocations rather than holding every non-privacy factor fixed.

Adding a small loop-ID embedding to shared width64 gives 3.0073468 versus 3.0073219 without it: +0.0000249 NLL in the tested run. Width128 plus loop-ID remains untested. The corresponding full-split probes are qualitatively similar: with/without ID, zeroing costs +6.41 [6.36,6.45]/+6.58 [6.54,6.63], and shuffling costs +7.62 [7.57,7.68]/+7.68 [7.62,7.75]. Both depend strongly on their channels. The 130-sequence bootstrap intervals are conditional on each trained model, not training-seed intervals. These probes establish dependence, not necessity of privacy. The contribution is an allocation study within PLT, not a claim that private embeddings are a new primitive or always required.

W2/Q2: is the advantage explained by parameter budget? The new matched-budget result is weight-only zFLoRA256: 94.4M auxiliary parameters and 2.9835 NLL, versus S-PLE32's 95.5M and 3.0089. The tested adapter wins by 0.0254 NLL. We therefore do not infer intrinsic input-space superiority from the original comparisons against smaller adapters. Width/rank sweeps describe the observed configurations; they do not replace this control or bound other adapter families.

The existing conventional-LoRA experiment is conditional on S-PLE and does not answer your adapter-only request. We have now launched standalone conventional LoRA, rank313 on all seven projections, no PLE, with one adapter pair shared across loops and approximately 95.55M auxiliary parameters. Its matched 50B result is not available yet. This is one explicit allocation choice, not a test of every loop-private LoRA variant. The new 135M comparisons use the fixed-50B stable-phase grid of 100B schedules; final-100B rankings remain unmeasured.

W3/Q3: speed curves, lookup, and quality on the timed model. This remains partially answered. The rank256 result extends the quality comparison, but no rank256 NPU timing or alternative-adapter device frontier is available. Our additional CPU fp32 microbenchmark uses randomly initialized models and does not invoke the NPU. It cannot establish handset latency, lookup/data-transfer-inclusive performance, or a speed advantage for the evaluated checkpoint.

We measured within-checkpoint fp→W4A8 simulation effects: plain-recurrent input-only 21.0705→24.1285 perplexity; plain-recurrent rank8 22.0145→25.6547; PLT rank256 19.7583→21.7712; PLT shared width64 20.2472→22.7075. These use group128 symmetric RTN W4, per-token A8, and floating-point tables. They differ from the calibrated handset W4A16-KV8 path, and absolute comparisons across the two model implementations retain their recipe qualifications.

Figure 7 still pairs separately measured floating-point quality with architecture-matched graph-only handset latency. It does not yet contain a verified rank256 deployment point. Joint quantized quality, lookup/transfer-inclusive latency, and storage on the same deployed checkpoints remain outstanding. We retain the original graph measurements within their stated boundary, including their storage and prefill limitations, rather than inferring the requested end-to-end frontier from CPU timing or policy simulation.

AuthorsOpenReview ↗

Reply #2 to Reviewer PkmS

Matched-budget LoRA comparison and training stability

In our initial comment, we committed to a conventional low-rank adapter comparison at an auxiliary parameter budget comparable to S-PLE32. We now report the fixed 50B-token reading, together with the training-instability record.

Setup and auxiliary parameter budget

Both arms use the SmolLM2-135M backbone, the same data and a single matched training seed, and a 100B-token WSD schedule with peak LR 3e-3 held through 80B tokens. The comparison is at the 50B schedule position, before LR decay. S-PLE32 uses private width-32 tables for each loop (95.5M auxiliary parameters). E11 uses rank-313 conventional LoRA on all seven projections, with each adapter pair shared across loops and no PLE tables (95.55M measured auxiliary parameters). This matches auxiliary parameter storage approximately, not full training compute.

Validation result at 50B tokens

ArmAuxiliary parameters50B validation NLLDifference from S-PLE32
S-PLE32, private tables95.5M3.0089—
E11, shared LoRA-r31395.55M3.4576+0.4487

Training stability and restart history

The original E11 attempt experienced five loss spikes, at 12B, 13.5B, 24.5B, 26B and 29B tokens. Four recovered within 0.5–4.5B tokens; the fifth remained at NLL 7.3–8.7 for 6.5B tokens, and that attempt was terminated at 35.5B. We resumed from its 25B checkpoint using the same recipe and the original schedule position. The resumed branch had three further spikes before the reading, at 28B, 30.5B and 36B, recovering after each. Following the 36B spike, its NLL remained above its earlier valley; the 50B reading was 3.4576. Another spike occurred at 51B, after the reported reading, and the job was then terminated. The 50B result is a fixed-checkpoint comparison; the discarded work before the restart adds training cost.

Best observed checkpoint

The best observed NLL on the resumed trajectory was 3.1985 at 35.5B, about 0.19 above S-PLE32's 50B value. This describes the observed checkpoints: it is neither a same-token comparison nor a bound on what a stable or retuned E11 run could achieve. Instability prevents us from attributing the full 50B gap to the parameterization alone.

Interpretation and limitations

The supported conclusion is that this shared conventional-LoRA configuration did not match S-PLE32 under the tested recipe. It does not establish that allocation and table capacity are jointly necessary, isolate table capacity from loop sharing and optimization, or rule out other adapter configurations. In particular, the previously reported matched-budget zFLoRA-r256 result (2.9835 NLL) remains better than S-PLE32 (3.0089) at the 50B grid.

Thank you for the new controls and the candid reporting. They confirm my concerns: a parameter-matched adapter outperforms the embeddings, shared tables match private tables at the smaller budget, and the speed comparison is still missing. I will keep my score.

Reviewer_csn2OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies loop-specific capacity in Parallel Loop Transformers. It introduces S-PLE, which gives each layer and loop a private token embedding table while sharing the projections across loops. Experiments with 135M and 906M backbones show improved perplexity over the tested adapter configurations, with favorable graph-level decoding latency in some settings.

Critical strengths
  • The design is simple and motivated: token-indexed storage adds capacity without increasing per-token computation as much as dense transformations.
  • S-PLE shows consistent perplexity improvements over the tested weight-adaptation configurations across both backbone sizes.
  • The paper explores several table widths and adapter ranks, and explicitly discusses limitations such as storage overhead, slower prefill, and the separation between quality and latency measurements.
Critical weaknesses
  1. The central design choice is not fully isolated. A parameter-matched comparison between loop-shared and loop-specific token tables is missing. The current results therefore do not clearly separate the benefit of additional input capacity from that of loop-specific specialization.

  2. The 906M study lacks a channel-free PLT quality baseline. The plain-loop baseline differs from PLT in several architectural components, so it cannot establish the incremental benefit of S-PLE within PLT at this scale.

  3. The seed replication changes the training recipe. Dropout is removed without a clear explanation, and the input-over-adapter gap becomes smaller. This supports the qualitative ordering, but does not directly replicate the original effect size. The channel-free PLT baseline is also absent from these repeats.

  4. Cross-scale conclusions remain limited. The two studies change backbone shape, attention configuration, training budget, and the main perplexity metric together. Larger experiments are not necessarily required, but a more consistent comparison would make the small-scale evidence more informative.

  5. Some baseline choices introduce avoidable differences. For example, the plain-loop control receives 2E2E on its first pass, while PLT receives EE. Its practical impact is unclear, but it makes the comparison less clean.

Overall, I understand that compute constraints may limit experiments at larger scales. Still, architectural choices can have substantial implications for large-scale training costs, so their scaling behavior deserves careful evaluation. Without larger-scale evidence, I place particular weight on how cleanly controlled and reliable the small-scale experiments are. This is the perspective behind my assessment, rather than a request for larger experiments.

Questions for authors

see weaknesses

1 comment
AuthorsOpenReview ↗

Reply to Reviewer csn2

Thank you for emphasizing clean small-scale comparisons. The new controls address the missing baselines and clarify which reliability questions remain open.

W1: shared versus private tables. We trained per-layer tables shared across loops. Approximately matched-storage pairs give: shared width64, 96.6M auxiliary parameters, 3.0073 NLL versus private S-PLE32, 95.5M, 3.0089; shared width128, 193.2M, 3.0503 versus private S-PLE64, 191.0M, 3.0022. The first pair is numerically close; the second favors private allocation by 0.0481 NLL. At fixed width64, shared/private are 3.0073/3.0022; at width128, 3.0503/3.0506, but private tables use almost twice the storage. These are single-seed comparisons, not equivalence tests. Matching storage changes width and projection size, so the evidence concerns the tested allocations rather than a pure privacy effect.

Full-split probes on shared width64 give zeroing +6.58 NLL [6.54,6.63] and shuffling +7.68 [7.62,7.75], over 130 sequences with 2000 sequence-bootstrap resamples and five shuffle seeds. Shared channels are strongly used too; dependence does not establish that private allocation is necessary. The matched-budget zFLoRA256 result, 2.9835 versus S-PLE32's 3.0089, additionally rules out a general equal-budget input-quality advantage.

W2: channel-free PLT at 906M. The new arm preserves the PLT machinery without tables or adapters and reaches 14.097 validation perplexity at the 50B endpoint. The comparison is plain recurrence 14.20, channel-free PLT 14.10, input-only PLT 13.86. It supplies the missing baseline for the input channel's incremental gain within PLT. Avg-7 is 56.67 versus 56.55 for plain recurrence; suite WikiText word-perplexity is 20.83 versus 20.81. Thus the machinery's gain is small and metric-dependent. This arm is neither a complete factorial nor the missing 135M recipe replication.

W3: recipe changes and missing repeats. We agree that the unified-codebase repeats do not reproduce the original effect size under an unchanged implementation. Dropout, model wrapper, and projection allocation differ. The additional intended dropout-restoration pair was invalidated by a flag-construction defect and is withdrawn; we do not attribute the old-to-new loss shift to dropout. A faithful PLT replication must preserve the wrapper and allocation and verify active attention/loop dropout. Channel-free 135M replication under the two recipes remains missing. The new 906M baseline does not fill those cells.

W4: cross-scale consistency. The 906M outputs cover the same nine accuracy tasks. Raw-accuracy avg-7 is 56.84 input-only and 56.77 weight-only, a small point-estimate difference, not an established accuracy gain. Existing web-perplexity CIs separate the channel arms from same-backbone dense but overlap between input-only and weight-only; those are likelihood intervals, not accuracy intervals. Shape, attention, training budget, endpoint scheduling, and the primary likelihood metric still differ across scales. New 135M controls are 50B stable-phase snapshots of 100B schedules, whereas 906M uses decayed 50B endpoints; we infer no scaling law from that comparison.

W5: first-pass 2E convention. Disabling additive embedding reinjection gives 3.1069 NLL versus 3.0912 with reinjection: +0.0157 NLL, or +0.35 perplexity. Both controls execute two backbone loops. The toggle removes E+H reinjection on later loops too, so it measures the whole convention, not first-pass scaling alone; it does not add or remove a backbone pass. The new control also trails the published non-looped baseline, 3.0980, by 0.0089 NLL. These single-seed results delimit the control convention rather than showing an improved baseline. An isolated first-pass E-versus-2E comparison remains untested.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response to all reviewers

We thank the reviewers for identifying the controls needed to distinguish input capacity, allocation across loops, and deployment efficiency. We have completed shared-table controls, a matched-budget zFLoRA comparison, and the missing channel-free PLT baseline at 906M. The results strengthen the controlled comparisons while narrowing the claims.

1. Shared and private allocation. At approximately matched auxiliary storage, shared per-layer tables of width 64 (96.6M parameters) give 3.0073 validation NLL, versus 3.0089 for private S-PLE32 (95.5M). At the larger budget, shared width 128 (193.2M) gives 3.0503, versus 3.0022 for private S-PLE64 (191.0M). Private allocation improves the latter comparison by 0.0481 NLL; the first pair is numerically close. Matching storage changes width and projection size, and the new arms have one training seed. These observations do not establish statistical equivalence or isolate privacy while holding all other dimensions fixed.

2. Parameter-matched adapters. Weight-only zFLoRA rank 256 (94.4M auxiliary parameters) reaches 2.9835 NLL, outperforming S-PLE32 by 0.0254 at the 50B snapshot. Thus the original input-over-small-adapter result does not establish an equal-budget advantage. The original 135M/100B operating points remain 27.50 geometric-mean perplexity without channels, 22.91 with S-PLE64, and 26.09 with rank-8 zFLoRA, at different storage budgets.

3. The missing 906M baseline. Channel-free PLT reaches 14.097 validation perplexity at the decayed 50B endpoint: plain recurrence 14.20 → channel-free PLT 14.10 → input-only PLT 13.86. This supplies the requested within-PLT baseline and a further 0.24 perplexity improvement from the input channel using rounded values. It is a three-arm comparison, not a complete factorial. Its avg-7 accuracy is 56.67, versus plain recurrence 56.55, weight-only PLT 56.77, and input-only PLT 56.84; the small differences are descriptive.

4. Scope and outstanding work. New 135M controls use the paper's fixed-50B stable-phase grid within 100B WSD schedules (App. J); the 906M result is a decayed 50B endpoint. Each new training arm has one seed. Final-100B results for the new controls are not yet delivered. We withdrew the planned dropout-restoration result after finding that the flag did not activate dropout in the intended modules; the architecture/projection differences also prevent treating it as faithful PLT replication.

We have launched conventional standalone LoRA at approximately 95.55M auxiliary parameters, rank 313 on all seven projections, with adapters shared across loops and no PLE. No matched 50B result is available yet; we will report it when available without assuming its outcome.

Joint deployed quality and end-to-end speed remain unmeasured. The additional CPU fp32 timing uses random initialization and does not invoke the NPU. W4A8 simulation is also different from the handset's calibrated W4A16-KV8 pipeline. Neither supplies the promised joint deployment measurement. We retain only the original graph-level speed claims, with table storage, prefill, calibration, and host-work limitations explicit.

The supported contribution is a controlled study of input-capacity allocation within PLT: input tables improve the channel-free model, the benefit of private allocation depends on the tested configuration, and a sufficiently large adapter can outperform the matched input configuration. The new controls delimit that contribution rather than establishing input-space superiority at every budget.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.