acceptodds
Under review as a conference paper at ICLR 2027

Beyond Routing: Matrix Zonotopic Attention for Set Transformers

Abstract

Standard attention adapts token routing to the input while sharing its value projections across contexts. We study a complementary design axis for set learning: making feature transformations explicitly context-dependent and carrying transformation-derived features across layers. We introduce Matrix Zonotopic Attention (MZAttn), which parameterizes a transformation family using a learned center matrix and generator matrices modulated by bounded, permutation-invariant context gates. Rather than using this family only to produce a single-vector update, MZAttn propagates separate center and generator feature streams and connects them to prediction through a generator-aware readout. To characterize operator variation, we introduce Transformation Degrees of Freedom (TDOF), which counts the independent matrix directions along which a specified operator varies with context. At the attention-core level, we establish exact representations for specified gated operator families and quantify approximation limits within fixed matrix-bank classes. We further show that identical signed and magnitude summaries can become distinguishable after the same rowwise normalization. For the normalized magnitude readout, we characterize local information capacity and construct a complementary two-row code supporting stable local recovery and scalar linear prediction through a finite head, with an explicit approximation bound. The resulting architecture preserves the permutation structure required for set prediction. We investigate its inductive bias through set-learning experiments. Together, these contributions frame set attention around three complementary choices: where to route information, how to transform it, and what transformation information to retain.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
attention mechanisms, set transformers, permutation-invariant learning, context-adaptive transformations, matrix zonotopes, neural network expressivity

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_SwKbOpenReview ↗
Rating
3(+1)
Confidence
2(−1)
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces Matrix Zonotopic Attention (MZAttn), a Set Transformer variant combining context-dependent matrix transformations with persistent center and generator feature streams. An invariant context modulates a learned matrix bank through bounded gates; generator rows undergo further processing before a magnitude-based readout.

The theoretical contributions address three distinct interfaces: expressivity and approximation of restricted gated cores, information lost when intermediate rows are replaced by signed/magnitude summaries, and local recovery after repeated rowwise LayerNorm. Particularly substantial results are the exact quadratic-prediction error curves under Gaussian and spherical inputs, and a complementary two-row construction providing stable local recovery and finite-head prediction. Validation includes direct training of the analyzed gated class and extensive complete-model comparisons. MZAttn improves several synthetic tasks at similar parameter counts, but benefits are task-dependent, transfer results are mixed, and computational overhead is substantial.

Critical strengths
  1. Sharp approximation results for a meaningful nonlinear function class. Theorems 4.5-4.6 characterize the exact error infimum at each gate count, including biased gates, rather than analyzing only a polynomial surrogate. The proofs combine constraints on quadratic projections with matching limiting constructions. The Gaussian/spherical distinction shows concretely how the input domain changes prediction complexity. Table 1 and Appendix C.4 also test the actual analyzed class. These results constitute a substantive theoretical contribution independently of whether the complete architecture proves advantageous.

  2. A constructive treatment of information retention and stable access. Theorem 4.7 provides an explicit example where equal signed and magnitude summaries become distinguishable under the same continuation. Theorems 4.8-4.10 then distinguish local dimension, conditioning, and access through a finite prediction head. The two-row construction is especially consequential: it supplies a recovery neighborhood independent of width and avoids relying on the radial direction that becomes ill-conditioned as the normalization stabilizer vanishes. This gives the proposed retention mechanism a precise mathematical motivation.

Critical weaknesses
  1. The complete-model evaluation contains unresolved implementation confounds. Section 5.1 states that variable-size sets are padded without attention masks and that the cosine scheduler is stepped per batch using an epoch-count period. Unmasked padding can affect attention, context gates, and pooling; if padding depends on the batch maximum, predictions can also depend on unrelated examples in the batch. The scheduler convention changes the intended learning-rate trajectory substantially. Applying these choices to every model does not establish that their effects are comparable across architectures. Without corrected controls, the reported gaps cannot confidently establish an architectural advantage, particularly since several baselines fit poorly and target normalization materially changes some comparisons.

  2. The evidence does not establish that persistent transformation-derived rows explain the empirical gains. The central theoretical constructions concern restricted cores or deliberately chosen normalization subpaths. The attention separation excludes FFNs and normalization; the depth–direction result restricts all directions to a square-zero algebra; and the recovery construction assumes prescribed encodings and normalization parameters. These restrictions are acknowledged, but the experiments do not bridge the resulting gap to trained complete models. Appendix I.2 varies bank size or changes both bank size and row count, without a matched comparison isolating retention from gating, additional feature capacity, and readout. Indeed, a single matrix direction already performs well on the interaction diagnostic, while the largest bank performs worst. Thus, those results do not demonstrate the proposed transformation-budget explanation. Given the substantial runtime overhead and inconsistent real-data benefits, this missing attribution materially weakens the case for the architectural contribution.

Overall, I lean toward rejection in the current form: the theoretical results are valuable, but the complete-model evidence needs stronger experimental controls and a clearer connection to the proposed mechanism.

Questions for authors
  1. What exact padding policy and scheduler configuration were used, including the scheduler period and number of optimizer steps per epoch? Does a fixed set receive the same prediction when evaluated alone and alongside sets of different cardinalities? Can you provide corrected masked-attention and properly scaled scheduler results for the main gains, particularly Tasks A, F, and G?

  2. Do you have comparisons at matched parameter counts that retain the same gates and matrix bank but replace persistent generator rows with their signed and/or magnitude summaries between blocks, retraining each variant? A gated model without persistent rows and a comparable multistream model without context-dependent gates would further distinguish the proposed mechanism from added capacity. Whether retention remains beneficial under these controls would materially affect my assessment.

  3. Is there evidence that trained generator features exhibit the complementary directions or stable readout geometry analyzed in Section 4.4? The parameterization probes in Appendix I.2.3 leave generator parameters untrained, so they do not answer this question. If no such evidence is available, should the recovery results be understood as existence results motivating the architecture, with its empirical mechanism still unresolved?

4 comments
AuthorsOpenReview ↗

Response to Reviewer SwKb (1/2)

Thank you for the detailed review and for your assessment of the theory. Your two concerns, implementation confounds and attribution to retention, were exactly right to raise. We addressed both with new experiments: the pipeline is now corrected for every model, and under the controls of your Q2, retention remains beneficial.

Q1 / W1: padding and scheduler.

Submitted pipeline.

  • Sets were zero-padded to the batch maximum without masks.
  • CosineAnnealingLR had an epoch-count horizon (Tmax⁡T_{\max} = 195) but was stepped every batch (79 steps per epoch on F/G, 157 on A), so the learning rate fell to zero and back every 390 steps (about 4.9 epochs; 2.5 on A) instead of decaying once.
  • Early stopping used patience 20.

Does a fixed set receive the same prediction alone and in mixed batches? In the submitted pipeline it did not, for any model; in the corrected pipeline it does.

Corrected pipeline.

  • Masks are applied in attention, pooling and the context mean.
  • Tmax⁡T_{\max} counts optimizer steps (15,405 on F/G, 30,615 on A).
  • Every model trains the full 200 epochs and is evaluated at its best-validation checkpoint.
  • Predictions are now identical alone and in batches (max deviation 1.7e-4 of the label s.d. over 4,642 corrected runs).

Corrected main results (raw targets as in the submitted Table 2; per-model learning-rate selection, which on F also repairs the poorly fitting baselines, e.g. HyperNet improves from 1373 to 357; see the general response). MZAttn still has the lowest error on A, F and G:

  • F: 97.2 ± 17 vs 198.8 (ST), 197.8 (ST-Large), 166.8 (FiLM).
  • G: 202.7 ± 20 vs 221.8 / 256.5 / 228.4, significant over 20 seeds (p = 0.007).
  • A: 0.38 ± 0.06 vs 0.63 ± 0.16 for ST-Large, 10 seeds each, p = 0.001.

Margin over the closest baseline, submission → now: 13% → 39% on A, 43% → 42% on F, 41% → 9% on G.

Q2 / W2: does retention, rather than gating, capacity or readout, explain the gains?

Under the controls you describe, retention remains beneficial: significantly in every cell on F, and significantly on A and G when the four cells are pooled (permutation test stratified by cell, p ≤ 0.003 and p = 0.03). On G, the main configuration (context gates, IHW) is also significantly better than each of the four controls that summarize the rows, keep the gated actions out of them or remove them (20 seeds each, p ≤ 0.01).

We ran this factorial at identical parameter count (h = 8, L = 4): the same gates and matrix bank, with generator rows either propagated or replaced by their signed summary after every block, each retrained with its own learning rate.

On Task F, propagated rows win in every cell:

Gates, readoutPropagatedCollapsedp
Context gates, IHW116.3 ± 18177.7 ± 3.81e-6
Context gates, signed97.2 ± 17158.7 ± 9.58e-7
Frozen gates, IHW112.1 ± 10171.8 ± 242e-5
Frozen gates, signed105.6 ± 3.3158.4 ± 272e-4

On A and G, propagated rows have the lower mean in all four cells (A, signed readout with context gates: 0.70 vs 1.16, p = 0.009, and with frozen gates: 0.50 vs 0.82, p = 0.02; G, context gates with IHW: 202.7 vs 245.4, p = 2e-4). Collapsing to the magnitude summary instead, which keeps IHW exactly, also hurts (context gates, IHW): 142.2 vs 116.3 on F (p = 0.015) and 226.8 vs 202.7 on G (p = 0.002).

For the two further controls you suggested:

  • a gated model without persistent rows: we removed the generator rows, so the same context-gated bank updates the centre, the head reads only the centre, and a wider centre FFN restores the parameter count. With its own learning rate it reaches 243.6 ± 21 on F, 2.27 ± 0.20 on A (10 seeds each) and 251.7 ± 14 on G (20 seeds), against 116.3 and 97.2 for the two propagated context-gate models on F, 1.19 and 0.70 on A, and 202.7 for the main configuration on G (p ≤ 1e-6). Keeping the rows as well, without gated actions written into them, lies in between (139.4 on F, 220.6 on G);
  • a multistream model without context-dependent gates (frozen tanh⁡(bl)\tanh(b_l)): rows matter equally there, see the frozen rows above.

To locate the effect, we collapsed rows only in the encoder or only before the readout:

Task FPropagatedCollapsed in encoder onlyCollapsed before readout only
Frozen gates112.1193.5102.7
Context gates116.3199.3122.8

Encoder-only vs readout-only collapse differs with p = 2e-6 (frozen gates) and p = 0.01 (context gates); readout-only collapse is within noise of full propagation. On F, retaining rows inside the encoder is therefore the operative mechanism, not added parameters. With the signed readout, collapsing before the readout leaves ∑gGg\sum_g G_g unchanged, and encoder-only collapse raises the error from 97.2 to 150.4 on F (p = 3e-6) and from 0.70 to 1.05 on A (p = 0.02, 10 seeds), within noise of collapsing after every block (158.7 and 1.16).

AuthorsOpenReview ↗

Response to Reviewer SwKb (2/2)

Localization on G (continued from part 1/2). Collapsing in either place hurts the main configuration (context gates, IHW): 249.3 with encoder-only and 234.3 with readout-only collapse, vs 202.7 with full propagation (p = 9e-5 and 0.02).

Q3: trained generator geometry.

We probed trained Task F models with zero-initialized context gates (five seeds per readout).

  • Within the encoder, generator rows remain distinct: mean pairwise cosine 0.54 and 0.47 after the two encoder blocks, versus 0.17 and 0.21 at initialization.
  • Toward the readout they align: cosine 0.81 entering the IHW readout, and κ=1−∥∑gGg∥1/∑g∥Gg∥1=0.05\kappa = 1 - \lVert\sum_g G_g\rVert_1 / \sum_g \lVert G_g\rVert_1 = 0.05, where κ=0\kappa = 0 would mean IHW equals ∣∑gGg∣|\sum_g G_g|.
  • Readout heads refit on frozen features reach the same test MSE whether they read IHW, ∣ΣG∣|\Sigma G| or ΣG\Sigma G (110.2 / 111.9 / 109.9). With centre features only, they reach 2726.
  • With the signed readout the rows stay less aligned (cosine 0.69, κ = 0.14), and the three refit heads are again equivalent (99.8 / 97.8 / 101.5).
  • Most of the predictive information therefore flows through the generator stream, and at the readout a refit head does as well with the signed sum as with IHW.

Together with the localization experiment above, the trained Task F models therefore show the empirical mechanism: multistream computation in the encoder, not readout geometry. This is the trained-model counterpart of Theorem 4.7, which you highlight: on F the signed summary suffices at the readout, where collapsing costs nothing, but not for the blocks that follow. As you suggest, Theorems 4.8–4.10 are existence results that motivate the multirow state, and the results for the restricted cores you list (Thm 4.1, no FFN or normalization; Thm 4.3, square-zero algebra) are statements about those interfaces (§4.1). The revised abstract and introduction state them this way, Thms 4.2–4.3 move to the appendix, and the probe is added to the appendix. In line with your first strength, the revised abstract and introduction present Theorems 4.5–4.6 as a contribution in their own right.

Bank size. We agree that the bank-size sweep of Appendix I.2 does not isolate the mechanism. The factorial does: it ties the gain to retained generator rows, and the revision states the mechanism this way.

Runtime. See the cost comparison in the general response: under the same protocol and a similar wall-clock time, the Set Transformer and ST-Large remain far behind MZAttn on F (194.3 and 177.9 vs 97.2) and significantly behind on G (p ≤ 0.006).

Real data. We agree that the real-data benefits are not consistent (§5.4). MZAttn's advantage is on the synthetic tasks A, F and G, whose targets depend on relations among the elements; on these, the controls above tie the gain to retained rows.

Your review asks for stronger experimental controls and a clearer connection to the proposed mechanism. The first is the corrected pipeline together with the factorial of your Q2. The second is the encoder result on F: the summary that is enough at the readout is not enough inside the encoder. Thank you for catching the padding and the scheduler. The paper is more reliable for it, and we are happy to go further into any of the controls.

Thank the author for the detailed responses to my questions. Most of my concerns are addressed. And I will raise my score to 3.

Besides this, I noticed that in J.2. Table S16, the author claim that MZAttn achieves lower test MSE than other methods on Task F in consideration of only successful runs. However, if taking failed runs into consideration, the average MSE may be much higher. How to explain this claim? It would not change my score, but it's better to clarify it.

AuthorsOpenReview ↗

Response to Reviewer SwKb

Thank you for the close reading of Appendix J. You are right: the Task F entry for MZAttn in Table S16 (97.2 ± 17) is the mean over the 16 successful runs out of 20. The other four runs fail (they revert to the constant predictor), and with them included the mean is much higher. They are identified by the failure rule of Appendix J.1, which uses the training loss, not the test error.

The claim of Table S16, that MZAttn has the lowest error on Task F, still holds in two checks that exclude no run. First, over all twenty runs, failed ones included, the median is 103.2 for MZAttn, against 167.3 for FiLM, 195.1 for the Set Transformer, 199.9 for ST-Large and 242.1 for the width-matched ST (Mann–Whitney p = 0.0012 against each). Second, the four failures occur at the selected learning rate, 3e-3; at the next lower rate, 1e-3, no run fails, and the mean over all runs is 112.6 ± 6.7 (5 runs), 32% below FiLM, the best baseline in Table S16 (166.8 ± 16).

Following your comment, in the camera-ready version we will state in the caption of Table S16 that the four failed runs are excluded, as the caption of Table 2 already does, and with that version we will release the code together with the per-seed results of Appendix J.

Reviewer_bPpKOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces Matrix Zonotopic Attention, combining context-dependent matrix transformations with persistent generator features and a magnitude-based readout. It analyzes transformation budgets, information preservation, and local recoverability, and evaluates the architecture on synthetic and real-world set-learning tasks.

Critical strengths
  1. The paper addresses a meaningful question: when does compressing an intermediate state preserve current predictions but lose information needed for subsequent computation? This provides a useful perspective on designing attention representations.
  2. The paper provides explicit separation proofs, sharp approximation bounds, and constructive recovery results. Targeted experiments support the approximation analysis, while parameter-matched comparisons demonstrate benefits on selected tasks.
Critical weaknesses
  1. Although the paper acknowledges this limitation, the full-model experiments do not isolate persistent feature retention from gating, capacity, and readout design. This leaves the empirical contribution of its central architectural choice unclear.
  2. Table S14 reports substantial runtime overhead, while the real-world evaluations show no consistent advantage. The practical benefits of the additional computation therefore remain insufficiently established.
Questions for authors
  1. On one representative task, can the authors compare persistent generator states with immediate compression, controlling for parameter count, gating, and readout access?
  2. On a task showing clear gains, can the authors compare against a baseline given a similar training-time budget to assess the performance–cost tradeoff?
3 comments
AuthorsOpenReview ↗

Response to Reviewer bPpK

Thank you for the clear review and its two precise questions. Your first strength puts the paper's question in one sentence: "when does compressing an intermediate state preserve current predictions but lose information needed for subsequent computation?" Your Q1 asks for the experiment that tests exactly this on trained models, and Q2 for what it costs. Both are now run. The Q1 comparison, which the other two reviews also request, has become the central experiment of the revision (§5.4, Appendix J.3).

W1 / Q1: persistent generator states vs immediate compression, controlling parameters, gating and readout.

On Task F we compared propagated generator rows with rows compressed to their signed summary after every block. Parameter count, gates and readout access are identical in each pair, and each variant has its own tuned learning rate. Propagation wins in every cell (5–20 seeds per cell):

Gates, readoutPropagatedCompressedp
Frozen gates, IHW112.1171.82e-5
Frozen gates, signed105.6158.42e-4
Context gates, IHW116.3177.71e-6
Context gates, signed97.2158.78e-7

Compressing only inside the encoder removes the gain (193.5 with frozen gates, 199.3 with context gates), while compressing only before the readout does not (102.7 and 122.8). On A, with the signed readout, compressing only inside the encoder also removes most of the gain (1.05, against 0.70 with propagated rows and 1.16 when compressing after every block). The benefit thus comes from persistent states during computation. This answers, on trained models, the question you highlight as the paper's contribution: compressing to the signed summary keeps what the signed readout sees, yet loses information that later blocks need.

W2 / Q2: equal training-time budget on a task showing clear gains.

We gave each baseline a similar wall-clock budget under the same protocol: MZAttn trained 200 epochs, and the baselines trained longer at their own selected learning rates, with all other hyperparameters shared. Minutes are medians over runs.

TaskMZAttn (200 epochs)Set TransformerST-LargeST + FiLM
F97.2 ± 17 (13.2 min)194.3 ± 24 (900 epochs, 11.5 min)177.9 ± 15 (650 epochs, 10.6 min)104.9 ± 17 (700 epochs, 14.5 min)
G202.7 ± 20 (8.1 min)222.3 ± 21 (650 epochs, 8.6 min)238.0 ± 12 (700 epochs, 9.6 min)226.3 ± 11 (650 epochs, 9.0 min)
  • On F, MZAttn's advantage over the plain and large Set Transformer is large at similar time (p < 1e-14), and their extra epochs close only a small part of the gap (198.8 → 194.3 and 197.8 → 177.9).
  • On G, MZAttn is significantly ahead of every time-matched baseline (p = 0.006 vs the Set Transformer, 9e-5 vs FiLM, 2e-7 vs ST-Large).

These comparisons are in the revision (§5.4; Appendix J.6, Table S20); the per-step costs remain in Table S14.

W2: real data. We agree: as §5.4 reports, MZAttn has no consistent advantage on the real-data tasks. Its advantage is on the synthetic tasks A, F and G, whose targets depend on relations among the elements; on these, the controls above tie the gain to retained rows.

You asked for two controls: persistent rows against immediate compression (Q1), and a baseline with a similar training-time budget on a task with clear gains (Q2). Retention accounts for the gain on F, and at a similar training time MZAttn stays ahead of the Set Transformer and ST-Large on F and of every baseline on G. Both experiments made the paper clearer, and we thank you for asking for them. If you see a clearer way to present either result, we would value your view.

Thank you for the thoughtful response and the additional experiments. The controlled ablation on Task F addresses my main concern by showing that retaining generator rows across layers improves performance over immediate compression. The comparisons under similar training-time budgets also answer my second question, showing that gains persist on Tasks F and G. The limitations on real-world performance and inference cost remain, so I will keep my current rating of 3.

AuthorsOpenReview ↗

Response to Reviewer bPpK

Thank you for checking the additional experiments point by point against your two questions. The Task F ablation controls for the three factors you listed in Q1 (parameter count, gating and readout access), and the comparison at similar training time comes from your Q2. In the final version we will keep both in the main text.

Reviewer_s9peOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Matrix Zonotopic Attention (MZAttn), a Set Transformer variant in which each token carries a center vector and hh generator rows. Routing is standard softmax attention on centers. The value transformation is a matrix bank Mc+∑l=1Ltanh⁡(wl⊤μ+bl)MlM_c + \sum_{l=1}^{L} \tanh(w_l^\top \mu + b_l) M_l gated by the permutation-invariant key/value-center mean μ\mu; the gated actions are mixed into the hh generator rows, which are propagated through their own LayerNorm/FFN path across layers (Eq. 3, §3). The head reads pooled centers concatenated with the interval-hull half-width (IHW), the coordinatewise sum of absolute generator rows.

Theoretical contributions: (i) Transformation Degrees of Freedom (TDOF), the affine dimension of an operator's image (Def. B.1); (ii) exact representation of gated operator families by one core and separations from restricted comparators: affine attention on constant-token sets (Thm 4.1), ReLU-FFN stacks (Thm 4.2), residual stacks in a square-zero algebra (Thm 4.3), plus a parameter-count bound (Thm B.4); (iii) exact MSE infima for approximating a quadratic target with LL gated-linear terms under Gaussian inputs, 2∑j>L(αj2+βj2)2\sum_{j>L}(\alpha_j^2+\beta_j^2) (Thm 4.5), versus uniform-on-sphere inputs (Thm 4.6), showing the zero-error gate budget depends on the input law; (iv) a continuation obstruction: states with equal signed sums and IHW can diverge after rowwise LayerNorm (Thm 4.7); (v) local information capacity of the normalized IHW readout, mmax⁡=min⁡{D,h(D−2)}m_{\max}=\min\{D, h(D-2)\} (Thm 4.8), ill-conditioning of one-row recovery as ϵ→0\epsilon\to0 (Thm 4.9), and a two-row affine code with a width-uniform recovery ball and a two-GELU-unit affine head (Thm 4.10).

Validation: training the gated class GL\mathcal{G}_L reproduces the Thm 4.5/4.6 curves (Table 1, App. C.4); an interaction diagnostic and bank-size sweep (§5.3, Fig. 2); seven set tasks versus Set Transformer, ST-Large, ST+FiLM, HyperNet, Perceiver, Deep Sets (Table 2/S1); and an extensive appendix (I.2–I.5) with depth, budget, scaling, equivariant, QM9, Neural-Process, tabular meta-regression and transfer comparisons, plus a cost table (S14).

Critical strengths

S1 — Correct, verifiable theory on the gated-linear class and the IHW readout (§4.2–4.4, App. C–E). Theorems 4.5/4.6 give exact, closed-form MSE infima and a clean mechanism (one gate contributes at most one eigenvalue of each sign; on the sphere quadratics are defined modulo tItI), confirmed by training (Table 1, C.4) and by my recomputation (60/10/2/0 Gaussian; 34.7/0.667/0/0 sphere). Thm 4.7's counterexample (Δ=8/(38/9)≈2.83\Delta = 8/(3\sqrt{8/9}) \approx 2.83) and Thm 4.10's spectrum (λmin⁡(TT⊤)=1/5\lambda_{\min}(TT^\top)=1/5, ∥B∥2=5\|B\|_2=\sqrt5) also check out. This is consequential because it rules out the usual failure mode of architecture-plus-theory papers (incorrect or vacuous theorems) and because Cor. C.8 makes the input-law result relevant to GLU/GELU networks generally.

S2 — Complete and candid reporting of scope and cost (§5.4, App. I.5, Table S14). The paper reports that MZAttn leads on no real-data task (ModelNet40, ScanObjectNN, all 12 QM9 targets, real-shape rotation matching R2=−0.066R^2=-0.066 vs 0.690 Perceiver, NP image completion, OpenML meta-regression, ResNet feature geometry) and costs 1.7–1.9× parameters and 2.4–5.5× time. This is consequential because it lets a reader calibrate the synthetic gains honestly; it also, however, is what makes W1–W2 decisive.

Critical weaknesses

W1 — The theory does not bear on the architecture claim.

  • Issue and location: Every separation is against a comparator that excludes what the baseline actually has. Thm 4.1/B.2(a): attention with affine maps, no FFN, no normalization, on constant-token sets. Thm 4.2: ReLU-only FFN stacks, "Nonidentical inducing seeds, LayerNorm or GELU would invalidate Step 3 and are explicitly excluded" (B.4). Thm 4.3: residual stacks confined to the square-zero algebra with "no intervening maps". Thm B.4: parameter counting over the bank only. The Set Transformer baseline in §5 has GELU FFNs and LayerNorm, and GELU$(t)=t,\Phi(t)$ is itself a gated-linear unit; Cor. C.8 shows the GL\mathcal{G}_L results hold verbatim for GELU gates. Thms 4.7–4.10 analyze MZAttn's own readout and never compare it with a plain vector state (a single stored vector is what Thm D.1 already shows preserves the signed sum exactly). The paper states this itself: "not lower bounds for general Transformers" (p. 5), "the isolated-core theorems do not by themselves explain the gap" (§5.3), "do not isolate retention" (§5.4).
  • Gap: No result in the paper distinguishes MZAttn from a GELU-FFN Set Transformer, from CondConv-style gated banks (App. G concedes the tanh bank is an affine reparameterization of a sigmoid-weighted bank), or from a hypernetwork-attention value network (Schug et al., ICLR 2025). TDOF is the affine dimension of an image set and, as C.1 shows, is not identified by scalar prediction, so it cannot be used to argue that a task needs LL directions.
  • Impact: The abstract frames three "complementary choices" (route, transform, retain) as a design axis; the submission establishes neither a theoretical nor (see W2) a controlled empirical reason to adopt the second and third. With the theory and the experiments decoupled, what remains is a collection of correct but self-referential results about the paper's own interfaces.
  • Suggested remedy: Either (a) prove one separation whose comparator class contains a Set Transformer block with a GELU FFN and LayerNorm, or (b) reframe the paper around the input-law gate-budget result (Thms 4.5/4.6 + C.8) as a contribution about GLU/GELU networks and drop the architecture claims to an application section; in both cases shorten substantially.

W2 — Complete-model evidence does not isolate the mechanism and shows signs of baseline under-tuning.

  • Issue and location: (a) H.1: one hyperparameter set (AdamW 10−410^{-4}, 200 epochs, patience 20) is shared by all architectures with no per-model sweep. HyperNet reaches MSE 1373 ± 142 on Task F versus 487 for a plain Set Transformer (Table 2), Perceiver sits at MSE ≈ 30.8 on every Table S9 row ("the scores alone do not establish convergence"), and a 1–4 layer Set Transformer reaches only R2≈0.40R^2\approx0.40 on a quadratic form of the set mean in d=32d=32 (§5.3), which a two-layer GELU head on PMA output can represent. These are optimization failures, not capacity limits, and they inflate every relative gain. (b) Prop. A.3 correctly notes that removing IHW disables the generator path, so the natural ablation is uninformative; the paper names the right control ("retains a trainable path and changes the summary or mixing") but reports it only as one sentence in I.2 (LN-on-readout variant, 0.966 vs 0.970), not at matched h,Lh, L across Tasks A–G. (c) Tables S2, S4, S9 report no seed uncertainty; S2 is used to claim compact configurations are "within 0.022". (d) The three tasks where MZAttn leads (A, F, G) are synthetic; on every real-data task it trails at 1.7–1.9× parameters.
  • Gap: Missing per-baseline tuning, missing retention/readout ablation at fixed budget, missing a same-width ST with matched parameter count (ST-Large changes width and dffd_{\text{ff}} rather than depth or FFN width alone).
  • Impact: Without (a)–(c) the synthetic advantages cannot be attributed to context-dependent transformation or to retained rows rather than to extra parameters, extra FFN capacity in the generator path, or baseline optimization; with (d) there is no practical case for the overhead.
  • Suggested remedy: Per-model learning-rate sweep (at least {3×10−5,10−4,3×10−4,10−3}\{3\times10^{-5},10^{-4},3\times10^{-4},10^{-3}\}) for all baselines on Tasks A, F, G and §5.3; a 2×2 ablation at fixed h=8,L=4h=8, L=4: {signed readout, IHW readout} × {generator stream propagated, generator stream reduced to one vector after each block}; seeds and std for S2/S4/S9; a width-matched ST (dmodeld_{\text{model}} chosen to hit ~370K).
Questions for authors
  1. (Clarification; drives W1) Is there any comparator class that contains a Set Transformer block with a GELU FFN and LayerNorm for which a separation from the MZ core holds? If not, what is the intended theoretical argument for preferring MZAttn over that baseline, given Cor. C.8?
  2. (Clarification; drives W2a) Were learning rates or schedules tuned per architecture? Please provide training/validation curves for HyperNet and Perceiver on Task F and for Perceiver in Table S9, and for the 1–4 layer Set Transformers in §5.3, so that non-convergence can be ruled out.
  3. (Clarification; drives W2b) At fixed h=8,L=4h=8, L=4 and matched parameters, what is the result of keeping the generator path trainable but (i) replacing IHW with a signed readout and (ii) collapsing the generator rows to a single vector after each block, on Tasks A, F, G? This is the control Prop. A.3 says is needed.
  4. (Clarification; drives W1/novelty) Functionally, how does MZAttn differ from the nonlinear value network of Schug et al. (ICLR 2025) combined with Hyper-Connections-style multi-stream residuals? A matched-parameter comparison against the former would settle whether the zonotope organization adds anything.
5 comments
AuthorsOpenReview ↗

Response to Reviewer s9pe (1/3)

Thank you for the exceptionally careful review, including recomputing the theorems. We have tried to keep the candour you valued in S2: every cell of the requested controls is reported. In brief: every baseline now has its own learning-rate sweep, as you suggested; the 2×2 at fixed h = 8, L = 4 is run on A, F and G; S2, S4 and S9 have seeds; width- and depth-matched Set Transformers, a CondConv-style control and a matched HYLA-style value network are added; and, in the direction of your remedy (b), the abstract and introduction are reframed and Thms 4.2–4.3 moved to the appendix. Details follow, W2 and the questions first.

W2a / Q2: per-architecture tuning and convergence.

You were right that several baselines were under-tuned: the submission shared one learning rate and schedule. We ran the per-model sweep you suggested. Every model now has its own learning rate, selected on seed-42 validation over {3e-5, 1e-4, 3e-4, 1e-3} and extended to 3e-3 / 1e-2 at a grid-edge optimum, then confirmed with further seeds. This covers all baselines on A, F and G and the §5.3 models. The schedule is corrected and all models are trained for the full budget. The revised appendix adds training and validation curves for HyperNet and Perceiver on Task F, Perceiver in Table S9, and the 1–4-layer Set Transformers in §5.3.

  • HyperNet on F improves from 1373 to 357. All baselines improve on F, and so does MZAttn (see the general response). With per-model learning rates, the gains on A and F remain: MZAttn still halves the Set Transformer's error on F (241 vs 488 in the submission, 97 vs 199 now), and its margin on A grows. On G the margin is smaller, 9% below the closest baseline, and remains significant (p = 0.007).
  • Convergence. At the selected learning rates the validation loss has plateaued within the 200 epochs: HyperNet and Perceiver on F reach their best validation loss at epochs 117 and 137 (medians over seeds) and improve by less than 1% after epoch 150; the §5.3 Set Transformers improve by at most 1.4% over the last 50 epochs (numbers in part 3/3).
  • §5.3 diagnostic. With tuning (including 3e-3), 1/2/4-layer Set Transformers reach R² 0.74–0.76 (K = 16) and 0.54–0.60 (K = 32), versus 0.95–0.997 for one-layer MZAttn.
    • Your point that a GELU head can represent the target is correct: an oracle that feeds the exact set mean into a one-hidden-layer GELU MLP reaches R² 0.999 (≥ 0.996 with only 64 hidden units).
    • The tuned Set Transformer's shortfall is therefore not one of head capacity, and it persists at every learning rate and after convergence: the Set Transformers stay far above the oracle (best validation MSE ≥ 0.378 vs 0.0007 for the oracle at K = 32).
  • Table S9. The 30.8 entries for HyperNet and LinearAttn were caused by early stopping (patience 10) on an initial plateau. Trained the full 50 epochs with the tuned learning rate, they reach 0.70 and 1.66 (two layers). At 3e-3 every model fails to train, so 1e-3 is an interior optimum. The NP Perceiver stays at the mean predictor for every learning rate (flat validation MSE). The revision reports this in Appendix J.8.

W2b / Q3: 2×2 at fixed h = 8, L = 4.

Readout {IHW, signed} × generator stream {propagated, reduced to one vector after every block} on A, F and G (5–20 seeds per cell). Both arms have the same parameters, including the generator-path FFN, which processes the collapsed rows as well. Test MSE, propagated vs collapsed (Welch p):

ReadoutAFG
IHW1.19 vs 1.30 (0.49)116.3 vs 177.7 (1e-6)202.7 vs 245.4 (2e-4)
Signed0.70 vs 1.16 (0.009)97.2 vs 158.7 (8e-7)236.4 vs 240.1 (0.8)

Propagation has the lower mean in all six cells, significantly in both on F and in one each on A and G. With frozen gates, the control Reviewer SwKb requested, it again has the lower mean in every cell (significantly in both on F and in one on A). Pooled over the four cells of a task (permutation test stratified by cell), its effect is significant on A (p ≤ 0.003) and G (p = 0.03). (On A, the main table uses the compact configuration selected on validation loss; the factorial keeps h = 8, L = 4 throughout.) For the full model, the signed readout is better than IHW on A and F and worse on G, so we now treat the readout as a configuration choice selected on validation loss.

At matched parameters and generator-path FFN capacity, the gain is therefore tied to retained rows. On F it arises inside the encoder: collapsing the rows only there gives 193.5 and 199.3 (frozen and context gates, IHW; five seeds each), collapsing them only before the readout 102.7 and 122.8, and full propagation 112.1 and 116.3. With the signed readout, encoder-only collapse raises 97.2 to 150.4 on F (p = 3e-6) and 0.70 to 1.05 on A (p = 0.02, 10 seeds), within noise of collapsing after every block (158.7 and 1.16). A summary sufficient for the readout thus does not suffice for later blocks (Thm 4.7).

AuthorsOpenReview ↗

Response to Reviewer s9pe (2/3)

W2c: seeds for S2, S4, S9. All three tables are rerun with five seeds and standard deviations under the corrected protocol (S9 with three; full tables in part 3/3), and the revised text states them with their seed uncertainty instead of "within 0.022".

  • S2: the compact configurations remain strong. MZ-Slim (signed readout) on Task A gives 0.38 ± 0.06 vs 0.63 ± 0.16 for ST-Large, 10 seeds each, p = 0.001; on F, every compact configuration (118–148) stays below every baseline (lowest: FiLM, 166.8).
  • S4: at each model's Task B learning rate, MZAttn is significantly below the Set Transformer at four of five noise levels (σ ≤ 0.5, p ≤ 0.04); at σ = 1.0 one of its five seeds diverges.

Width-matched Set Transformer. We added Set Transformers matched to MZAttn's parameter count by FFN width (ST-FFN) and by depth (ST-Deep, run on A, F and G). Both trail MZAttn on all three tasks, e.g. F: 236.2 and 262.6 vs 97.2.

W2d: real data. We agree. As §5.4 reports, MZAttn has no consistent advantage on real data; its advantage is on the synthetic tasks A, F and G, whose targets depend on relations among the elements. Within that scope the parameter overhead does not explain the gain: under the corrected protocol, ST-Large and the width-matched ST, with comparable parameter counts, trail MZAttn on A, F and G.

Q4: relation to HYLA / hyper-connections. We implemented the HYLA nonlinear value network (RMS-normalized head scores, ReLU value network; Schug et al., 2025) at MZAttn's parameter count, averaging instead of summing over the elements, as set sizes vary.

  • In every attention block: A 3.38 ± 0.06, F 276.6 ± 18, G 449.2 ± 9 (5 seeds; 10 on F).
  • In the encoder only (softmax pooling and decoder): A 3.55 ± 0.14, F 361.8 ± 26, G 257.8 ± 20.
  • MZAttn is far better on all three (0.38, 97.2, 202.7).

Functionally, HYLA uses the head scores to configure both layers of a nonlinear value network for each query–key pair, whereas MZAttn applies a context-gated bank of matrices to a centre and carries the resulting directions as persistent generator rows. Hyper-Connections instead keep several copies of the full hidden state; each layer reads and writes weighted combinations of these streams. MZAttn's rows are not copies of the hidden state: they are generated by gated matrix actions on the centre and inherited across blocks, only the centres route and set the gate context, and the head reads the rows only through IHW or their signed sum. Within MZAttn, the factorial shows that this persistence matters. The revised related-work section states both distinctions.

W1 / Q1: theory and architecture. To Q1: no. None of our separations covers a Set Transformer block with a GELU FFN and LayerNorm; as §4.1 says, they are not lower bounds for general Transformers. The argument we rest on for MZAttn is retention, not block expressivity: Thm 4.7 shows that a summary sufficient for the readout need not suffice for later computation, and the new controls test this on trained models. In the direction of your remedy (b), the revised abstract and introduction present the input-law gate-budget result (Thms 4.5–4.6, Cor. C.8) as a contribution about GLU/GELU networks and state the separations as results about the stated interfaces. The controlled experiments supply the link that W1 found missing:

  • under matched parameters, gating and readout, retention accounts for the gain on F and is significant on A and G when the cells are pooled (p ≤ 0.003 and 0.03);
  • it acts inside the encoder: on F, and on A with the signed readout, the signed sum suffices at the readout but not for the blocks that follow, the trained-model counterpart of Thm 4.7;
  • trained Set Transformers fall well short of an oracle with the exact mean on the §5.3 interaction diagnostic, at every learning rate.

On the comparators you list:

  • Plain vector state. The collapsed cells are this comparison: after every block the rows are replaced by their signed sum, which one stored vector preserves exactly (Thm D.1). Propagation is better on F in every cell and has the lower mean in every cell on A and G.
  • CondConv-style banks. Without generator rows, the same context-gated bank acts on a single vector state, the centre, at matched parameter count. The error more than doubles on F (243.6 vs 97.2–116.3) and rises on A (2.27 vs 0.70–1.19) and G (251.7 vs 202.7), p ≤ 1e-6. Carrying the rows as well, without gated actions, lies in between on F and G (139.4, 220.6).
  • TDOF. We agree that it is not identified by scalar predictions (App. C.1). We use it only for a specified operator (§4.1, Thm 4.3); the scaling studies in the appendix already decline to infer a task's direction count from it, and the revision says so in §4.1.

Every item of your W2 remedy is now run, and the gains remain. We hope the controlled evidence and the revised framing address W1 and W2, which you identified as decisive. We would be glad to discuss any point that remains open.

AuthorsOpenReview ↗

Response to Reviewer s9pe (3/3)

Numbers for Q2 and W2c. The curves and tables you asked for, in numeric form. The revised appendix contains the same tables and the plotted curves.

Q2: training and validation curves. Medians over seeds at each model's selected learning rate. "Val." is the lowest validation loss reached up to that epoch; "Train" is the training loss at that epoch.

Model (learning rate, seeds)Val. ep. 1050100150200Train ep. 1050100150200
F, HyperNet (3e-4, 5)507650841239639654392551148480
F, Perceiver (1e-3, 5)11503213002922908931971147966
§5.3 K=16, ST 1 layer (3e-3, 3)0.4530.2800.2520.2460.2440.3130.0590.0240.0140.010
§5.3 K=16, ST 2 layers (3e-3, 3)0.4470.3300.2950.2810.2790.3020.0580.0250.0120.008
§5.3 K=16, ST 4 layers (3e-3, 3)0.4610.2800.2540.2480.2480.3440.0790.0310.0140.010
§5.3 K=32, ST 1 layer (3e-4, 3)0.5930.4730.4600.4520.4500.5050.1290.0650.0500.046
§5.3 K=32, ST 2 layers (1e-3, 3)0.5500.4610.4260.4130.4090.3750.0570.0280.0180.015
§5.3 K=32, ST 4 layers (3e-4, 3)0.5540.4490.4160.4010.3970.4670.0790.0380.0250.023

In every row the validation loss has stopped improving (by at most 1.4% after epoch 150 in any seed) while the training loss keeps falling, so no run is cut off before convergence.

Table S9, Perceiver (50 epochs; lowest validation MSE up to epochs 5 / 10 / 25 / 50): 30.92 / 30.92 / 30.92 / 30.92 at depth 2 and 30.92 / 30.91 / 30.91 / 30.90 at depth 4; at the end of training it is between 29.9 and 31.0 at every learning rate. Its training loss, a Gaussian NLL, falls only slowly (3.04 at epoch 5 and 2.97 at epoch 50 at depth 2; 3.03 and 2.80 at depth 4). For comparison, HyperNet (depth 2) leaves the same plateau after epoch 10: 30.92 / 30.90 / 2.67 / 0.72.

W2c: Tables S2, S4 and S9 with seeds. Test MSE, mean ± s.d. (seeds).

S2, MZAttn configurations (∗ = selected on seed-42 validation loss and reported in the main table):

Configuration, readoutAFG
Slim (h=2, L=1), IHW0.37 ± 0.07 (5)139.8 ± 24 (5)204.7 ± 10 (5)
Slim, signed0.38 ± 0.06 (10)∗118.3 ± 8.3 (5)258.3 ± 52 (5)
Med (h=4, L=2), IHW0.42 ± 0.11 (5)147.8 ± 36 (5)207.7 ± 38 (10)
Med, signed0.44 ± 0.20 (5)131.0 ± 14 (5)204.1 ± 14 (10)
Full (h=8, L=4), IHW1.19 ± 0.34 (10)116.3 ± 18 (10)202.7 ± 20 (20)∗
Full, signed0.70 ± 0.30 (10)97.2 ± 17 (16)∗236.4 ± 18 (5)
Lowest baseline0.63 ± 0.16 (ST-Large)166.8 ± 16 (FiLM)221.8 ± 21 (ST)

On F every configuration is below every baseline; on A the four compact configurations are below ST-Large; on G four of the six are below the Set Transformer.

S4, noise level on set matching (the full configuration of the submitted table, h=8, L=4, here with the signed readout; each model at the learning rate selected for it on Task B):

σMZAttnSet Transformerp
0.059.00 ± 1.74 (5)13.76 ± 1.53 (5)0.004
0.18.18 ± 0.48 (5)11.48 ± 0.96 (5)0.0009
0.3 (Task B)8.44 ± 1.63 (9)15.27 ± 8.28 (10)0.04
0.59.09 ± 0.63 (5)10.46 ± 0.29 (5)0.009
1.086.2 ± 156 (5)12.04 ± 2.53 (5)0.40

At σ = 0.3 one of ten MZAttn seeds is excluded by the failure rule. At σ = 1.0 one of five MZAttn seeds reverts to the constant predictor (test MSE 399) and is included; the other four average 8.1. On larger test sets ([30, 50], [50, 80], [80, 120] points) the differences are not significant (p ≥ 0.3).

S9, meta-regression (three seeds; 50 epochs without early stopping; per-model learning rate):

ModelDepth 2Depth 4
Set Transformer0.46 ± 0.050.27 ± 0.09
ST-Large0.62 ± 0.220.092 ± 0.004
HyperNet0.70 ± 0.160.17 ± 0.03
Perceiver30.76 ± 0.0130.76 ± 0.01
LinearAttn1.66 ± 1.280.89 ± 0.21
MZ-Full0.61 ± 0.240.16 ± 0.01
MZ-Large0.47 ± 0.020.20 ± 0.02

Thank you again for the care you took in checking the theorems. We have tried to report the numbers above with the same care.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General response to all reviewers

We thank the reviewers for careful and constructive reviews. All three list the theory as a strength; their concerns are the complete-model evidence and its link to the theory. They converge on three requests: (i) fix two implementation issues and re-establish the results under a tuned protocol; (ii) isolate generator-row retention from gating, capacity and readout; (iii) assess the performance–cost trade-off. We have completed all three and summarize the results here. Numbers are test MSE, mean ± s.d. over seeds. Full tables, training curves and every variant are in Appendix J of the revision; the remaining studies will be rerun under the corrected protocol for the final version.

Corrected protocol (identical for every model).

  • Padding masks in all attention, pooling and context-mean operations.
  • Cosine decay over all post-warm-up optimizer steps.
  • 200 epochs with best-validation checkpointing.
  • Per-model learning-rate selection: a seed-42 sweep over {3e-5, 1e-4, 3e-4, 1e-3}, extended to 3e-3 / 1e-2 at a grid-edge optimum, then four further seeds (10–20 for the key comparisons).
  • All other training hyperparameters are shared by every model and unchanged from the submission.

After these corrections, every set receives the same prediction alone or in any padded batch (maximum deviation 1.7e-4 of the label s.d. over 4,642 corrected runs). For MZAttn, the context-gate weights are now zero-initialized, so the model starts at its context-independent special case tanh⁡(bl)\tanh(b_l), and its configuration (full or a compact configuration of Table S2; IHW or signed readout) is selected per task on validation loss, like the learning rate.

Main results under the corrected protocol (Tasks A, F and G, as requested).

ModelA: RegressionF: RotationG: MST
Set Transformer1.26 ± 0.24198.8 ± 17221.8 ± 21
ST-Large0.63 ± 0.16197.8 ± 13256.5 ± 56
ST + FiLM1.83 ± 0.22166.8 ± 16228.4 ± 12
ST, width-matched to MZAttn1.84 ± 0.06236.2 ± 19232.0 ± 12
Perceiver2.76 ± 0.46257.2 ± 31293.6 ± 16
HyperNet5.24 ± 0.14357.4 ± 32429.0 ± 30
MZAttn0.38 ± 0.0697.2 ± 17202.7 ± 20

On F and G, 20 seeds for MZAttn and the four Set Transformer variants above; on A, 10 for MZAttn, the Set Transformer and ST-Large; 5 otherwise. MZAttn configuration selected on seed-42 validation loss among {full h=8,L=4h{=}8, L{=}4; Slim h=2,L=1h{=}2, L{=}1; Med h=4,L=2h{=}4, L{=}2} × {IHW, signed}: A Slim-signed, F full-signed, G full-IHW. On F, four of the twenty MZAttn seeds diverged to the constant predictor, visible in the training loss by epoch 4; they are excluded by our pre-specified rule (lowest training loss > 5× that of seed 42) and listed in the appendix. Without exclusion, MZAttn's median over all twenty seeds is 103, vs 167 for FiLM and 195–242 for the three Set Transformers (Mann–Whitney, p = 0.001 each). At learning rate 1e-3 (selected: 3e-3) no seed diverges (112.6 ± 6.7, five seeds).

As in the submitted Table 2, MZAttn has the lowest error on A, F and G, significantly below every baseline (p ≤ 0.007):

  • on F it halves the error of the Set Transformer and ST-Large and is 42% below FiLM (p = 3e-13);
  • on A it is 39% below ST-Large (p = 0.001);
  • on G it is 9% below the closest baseline, the Set Transformer (p = 0.007).

Mechanism. A full factorial at identical parameter count (h = 8, L = 4) on A, F and G shows that retaining generator rows is what helps:

  • gates {context-dependent, frozen} × rows {propagated, collapsed after every block} × readout {IHW, signed}, 5–20 seeds per cell;
  • on F, propagated rows beat collapsed rows in every cell (p ≤ 2e-4), including the two frozen-gate cells, where no seed is excluded (112.1 vs 171.8 and 105.6 vs 158.4, 10 seeds each); on A and G they have the lower mean in every cell; pooled over the four cells (stratified permutation test), the effect is significant on A (p ≤ 0.003) and G (p = 0.03);
  • a model without generator rows, in which the same gated bank acts on the centre at matched parameter count, more than doubles the error on F (243.6 vs 116.3 and 97.2) and raises it on A (2.27 vs 1.19 and 0.70) and G (251.7 vs 202.7; p ≤ 1e-6);
  • on F, collapsing the rows inside the encoder removes the gain, while collapsing them only before the readout does not; on A, with the signed readout, collapsing them inside the encoder removes most of the gain (1.05 vs 0.70 propagated and 1.16 collapsed after every block; p = 0.02). So the benefit comes from multistream computation in the encoder. This is the trained-model counterpart of Thm 4.7: a summary sufficient for the readout need not suffice for later computation.

Cost. Given a similar wall-clock budget (650–900 epochs vs 200 for MZAttn, same protocol; table in our reply to Reviewer bPpK), the Set Transformer and ST-Large stay far behind on F (194.3 and 177.9 vs 97.2) and all three baselines stay behind on G (p ≤ 0.006).

AuthorsOpenReview ↗

Summary of changes in the revised PDF

We have uploaded a revised PDF. New and changed text is in blue. The title, the section structure and the theorem numbering are unchanged, and the main text stays within 10 pages. Reviewer points are given in parentheses.

Main text

  • Abstract and Introduction: the input-law gate-budget results (Thms 4.5–4.6, Cor. C.8) are presented as a contribution about gated-linear (GLU/GELU) networks in their own right; the separation results are stated as results about the stated interfaces; the case for MZAttn rests on the controlled experiments (s9pe W1, SwKb W2).
  • §4.1: TDOF is used only for a specified operator; the statements of Thms 4.2–4.3 move to Appendix B, with their numbers unchanged (s9pe W1/Q1).
  • §4.4: Thms 4.8–4.10 are stated as existence results that motivate the multirow state; the probe of trained models is in Appendix J.4 (SwKb Q3).
  • §5.1: the corrected protocol (padding masks, cosine schedule over optimizer steps, full 200 epochs with best-validation checkpointing, per-model learning-rate selection), and MZAttn's zero-initialized context gates and per-task configuration selection (SwKb W1/Q1, s9pe W2a).
  • §5.3: corrected numbers for the interaction diagnostic; the bank-size sweep is no longer used for the mechanism (SwKb W2).
  • §5.4: Table 2 rows for A, B, F and G under the corrected protocol, with a width-matched Set Transformer; the mechanism (factorial, localization, link to Thm 4.7); equal training time; the scope of the real-data results (all reviewers).
  • §6 and Related Work: updated accordingly; Related Work now states the differences from HYLA and Hyper-Connections (s9pe Q4).
  • Sentences on the submitted protocol and its caveats in §5.1, §5.4 and §6 are replaced by the blue text.

New Appendix J (pp. 53–59)

  • J.1–J.2: corrected protocol and main results for every model (Table S16), with the excluded seeds listed and the all-seed medians (SwKb Q1, s9pe W2a).
  • J.3: factorial at h = 8, L = 4 with pooled tests (Table S17), localization (Table S18) and the gated model without generator rows (SwKb Q2, bPpK Q1, s9pe Q3).
  • J.4: geometry of trained generator rows (SwKb Q3).
  • J.5: interaction diagnostic under the corrected protocol, with the oracle (Table S19; s9pe W2a).
  • J.6: equal training time and per-epoch costs (Table S20; bPpK Q2).
  • J.7: the HYLA-style value network (s9pe Q4).
  • J.8: Tables S2, S4 and S9 with seeds and standard deviations (Tables S21–S23; s9pe W2c); the "within 0.022" sentence of Appendix I.2 now points to Table S21.
  • J.9: training and validation curves (Figure 6, Table S24; s9pe Q2).
  • The captions of the submitted Tables S1–S4, S9 and S14 point to the tables that replace them.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.