- Rating
- 3(+1)
- Confidence
- 2(−1)
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduces Matrix Zonotopic Attention (MZAttn), a Set Transformer variant combining context-dependent matrix transformations with persistent center and generator feature streams. An invariant context modulates a learned matrix bank through bounded gates; generator rows undergo further processing before a magnitude-based readout.
The theoretical contributions address three distinct interfaces: expressivity and approximation of restricted gated cores, information lost when intermediate rows are replaced by signed/magnitude summaries, and local recovery after repeated rowwise LayerNorm. Particularly substantial results are the exact quadratic-prediction error curves under Gaussian and spherical inputs, and a complementary two-row construction providing stable local recovery and finite-head prediction. Validation includes direct training of the analyzed gated class and extensive complete-model comparisons. MZAttn improves several synthetic tasks at similar parameter counts, but benefits are task-dependent, transfer results are mixed, and computational overhead is substantial.
Critical strengths
-
Sharp approximation results for a meaningful nonlinear function class. Theorems 4.5-4.6 characterize the exact error infimum at each gate count, including biased gates, rather than analyzing only a polynomial surrogate. The proofs combine constraints on quadratic projections with matching limiting constructions. The Gaussian/spherical distinction shows concretely how the input domain changes prediction complexity. Table 1 and Appendix C.4 also test the actual analyzed class. These results constitute a substantive theoretical contribution independently of whether the complete architecture proves advantageous.
-
A constructive treatment of information retention and stable access. Theorem 4.7 provides an explicit example where equal signed and magnitude summaries become distinguishable under the same continuation. Theorems 4.8-4.10 then distinguish local dimension, conditioning, and access through a finite prediction head. The two-row construction is especially consequential: it supplies a recovery neighborhood independent of width and avoids relying on the radial direction that becomes ill-conditioned as the normalization stabilizer vanishes. This gives the proposed retention mechanism a precise mathematical motivation.
Critical weaknesses
-
The complete-model evaluation contains unresolved implementation confounds. Section 5.1 states that variable-size sets are padded without attention masks and that the cosine scheduler is stepped per batch using an epoch-count period. Unmasked padding can affect attention, context gates, and pooling; if padding depends on the batch maximum, predictions can also depend on unrelated examples in the batch. The scheduler convention changes the intended learning-rate trajectory substantially. Applying these choices to every model does not establish that their effects are comparable across architectures. Without corrected controls, the reported gaps cannot confidently establish an architectural advantage, particularly since several baselines fit poorly and target normalization materially changes some comparisons.
-
The evidence does not establish that persistent transformation-derived rows explain the empirical gains. The central theoretical constructions concern restricted cores or deliberately chosen normalization subpaths. The attention separation excludes FFNs and normalization; the depth–direction result restricts all directions to a square-zero algebra; and the recovery construction assumes prescribed encodings and normalization parameters. These restrictions are acknowledged, but the experiments do not bridge the resulting gap to trained complete models. Appendix I.2 varies bank size or changes both bank size and row count, without a matched comparison isolating retention from gating, additional feature capacity, and readout. Indeed, a single matrix direction already performs well on the interaction diagnostic, while the largest bank performs worst. Thus, those results do not demonstrate the proposed transformation-budget explanation. Given the substantial runtime overhead and inconsistent real-data benefits, this missing attribution materially weakens the case for the architectural contribution.
Overall, I lean toward rejection in the current form: the theoretical results are valuable, but the complete-model evidence needs stronger experimental controls and a clearer connection to the proposed mechanism.
Questions for authors
-
What exact padding policy and scheduler configuration were used, including the scheduler period and number of optimizer steps per epoch? Does a fixed set receive the same prediction when evaluated alone and alongside sets of different cardinalities? Can you provide corrected masked-attention and properly scaled scheduler results for the main gains, particularly Tasks A, F, and G?
-
Do you have comparisons at matched parameter counts that retain the same gates and matrix bank but replace persistent generator rows with their signed and/or magnitude summaries between blocks, retraining each variant? A gated model without persistent rows and a comparable multistream model without context-dependent gates would further distinguish the proposed mechanism from added capacity. Whether retention remains beneficial under these controls would materially affect my assessment.
-
Is there evidence that trained generator features exhibit the complementary directions or stable readout geometry analyzed in Section 4.4? The parameterization probes in Appendix I.2.3 leave generator parameters untrained, so they do not answer this question. If no such evidence is available, should the recovery results be understood as existence results motivating the architecture, with its empirical mechanism still unresolved?
4 comments
Response to Reviewer SwKb (2/2)
Localization on G (continued from part 1/2). Collapsing in either place hurts the main configuration (context gates, IHW): 249.3 with encoder-only and 234.3 with readout-only collapse, vs 202.7 with full propagation (p = 9e-5 and 0.02).
Q3: trained generator geometry.
We probed trained Task F models with zero-initialized context gates (five seeds per readout).
- Within the encoder, generator rows remain distinct: mean pairwise cosine 0.54 and 0.47 after the two encoder blocks, versus 0.17 and 0.21 at initialization.
- Toward the readout they align: cosine 0.81 entering the IHW readout, and , where would mean IHW equals .
- Readout heads refit on frozen features reach the same test MSE whether they read IHW, or (110.2 / 111.9 / 109.9). With centre features only, they reach 2726.
- With the signed readout the rows stay less aligned (cosine 0.69, κ = 0.14), and the three refit heads are again equivalent (99.8 / 97.8 / 101.5).
- Most of the predictive information therefore flows through the generator stream, and at the readout a refit head does as well with the signed sum as with IHW.
Together with the localization experiment above, the trained Task F models therefore show the empirical mechanism: multistream computation in the encoder, not readout geometry. This is the trained-model counterpart of Theorem 4.7, which you highlight: on F the signed summary suffices at the readout, where collapsing costs nothing, but not for the blocks that follow. As you suggest, Theorems 4.8–4.10 are existence results that motivate the multirow state, and the results for the restricted cores you list (Thm 4.1, no FFN or normalization; Thm 4.3, square-zero algebra) are statements about those interfaces (§4.1). The revised abstract and introduction state them this way, Thms 4.2–4.3 move to the appendix, and the probe is added to the appendix. In line with your first strength, the revised abstract and introduction present Theorems 4.5–4.6 as a contribution in their own right.
Bank size. We agree that the bank-size sweep of Appendix I.2 does not isolate the mechanism. The factorial does: it ties the gain to retained generator rows, and the revision states the mechanism this way.
Runtime. See the cost comparison in the general response: under the same protocol and a similar wall-clock time, the Set Transformer and ST-Large remain far behind MZAttn on F (194.3 and 177.9 vs 97.2) and significantly behind on G (p ≤ 0.006).
Real data. We agree that the real-data benefits are not consistent (§5.4). MZAttn's advantage is on the synthetic tasks A, F and G, whose targets depend on relations among the elements; on these, the controls above tie the gain to retained rows.
Your review asks for stronger experimental controls and a clearer connection to the proposed mechanism. The first is the corrected pipeline together with the factorial of your Q2. The second is the encoder result on F: the summary that is enough at the readout is not enough inside the encoder. Thank you for catching the padding and the scheduler. The paper is more reliable for it, and we are happy to go further into any of the controls.
Thank the author for the detailed responses to my questions. Most of my concerns are addressed. And I will raise my score to 3.
Besides this, I noticed that in J.2. Table S16, the author claim that MZAttn achieves lower test MSE than other methods on Task F in consideration of only successful runs. However, if taking failed runs into consideration, the average MSE may be much higher. How to explain this claim? It would not change my score, but it's better to clarify it.
Response to Reviewer SwKb
Thank you for the close reading of Appendix J. You are right: the Task F entry for MZAttn in Table S16 (97.2 ± 17) is the mean over the 16 successful runs out of 20. The other four runs fail (they revert to the constant predictor), and with them included the mean is much higher. They are identified by the failure rule of Appendix J.1, which uses the training loss, not the test error.
The claim of Table S16, that MZAttn has the lowest error on Task F, still holds in two checks that exclude no run. First, over all twenty runs, failed ones included, the median is 103.2 for MZAttn, against 167.3 for FiLM, 195.1 for the Set Transformer, 199.9 for ST-Large and 242.1 for the width-matched ST (Mann–Whitney p = 0.0012 against each). Second, the four failures occur at the selected learning rate, 3e-3; at the next lower rate, 1e-3, no run fails, and the mean over all runs is 112.6 ± 6.7 (5 runs), 32% below FiLM, the best baseline in Table S16 (166.8 ± 16).
Following your comment, in the camera-ready version we will state in the caption of Table S16 that the four failed runs are excluded, as the caption of Table 2 already does, and with that version we will release the code together with the per-seed results of Appendix J.
Response to Reviewer SwKb (1/2)
Thank you for the detailed review and for your assessment of the theory. Your two concerns, implementation confounds and attribution to retention, were exactly right to raise. We addressed both with new experiments: the pipeline is now corrected for every model, and under the controls of your Q2, retention remains beneficial.
Q1 / W1: padding and scheduler.
Submitted pipeline.
CosineAnnealingLRhad an epoch-count horizon (Tmax = 195) but was stepped every batch (79 steps per epoch on F/G, 157 on A), so the learning rate fell to zero and back every 390 steps (about 4.9 epochs; 2.5 on A) instead of decaying once.Does a fixed set receive the same prediction alone and in mixed batches? In the submitted pipeline it did not, for any model; in the corrected pipeline it does.
Corrected pipeline.
Corrected main results (raw targets as in the submitted Table 2; per-model learning-rate selection, which on F also repairs the poorly fitting baselines, e.g. HyperNet improves from 1373 to 357; see the general response). MZAttn still has the lowest error on A, F and G:
Margin over the closest baseline, submission → now: 13% → 39% on A, 43% → 42% on F, 41% → 9% on G.
Q2 / W2: does retention, rather than gating, capacity or readout, explain the gains?
Under the controls you describe, retention remains beneficial: significantly in every cell on F, and significantly on A and G when the four cells are pooled (permutation test stratified by cell, p ≤ 0.003 and p = 0.03). On G, the main configuration (context gates, IHW) is also significantly better than each of the four controls that summarize the rows, keep the gated actions out of them or remove them (20 seeds each, p ≤ 0.01).
We ran this factorial at identical parameter count (h = 8, L = 4): the same gates and matrix bank, with generator rows either propagated or replaced by their signed summary after every block, each retrained with its own learning rate.
On Task F, propagated rows win in every cell:
On A and G, propagated rows have the lower mean in all four cells (A, signed readout with context gates: 0.70 vs 1.16, p = 0.009, and with frozen gates: 0.50 vs 0.82, p = 0.02; G, context gates with IHW: 202.7 vs 245.4, p = 2e-4). Collapsing to the magnitude summary instead, which keeps IHW exactly, also hurts (context gates, IHW): 142.2 vs 116.3 on F (p = 0.015) and 226.8 vs 202.7 on G (p = 0.002).
For the two further controls you suggested:
To locate the effect, we collapsed rows only in the encoder or only before the readout:
Encoder-only vs readout-only collapse differs with p = 2e-6 (frozen gates) and p = 0.01 (context gates); readout-only collapse is within noise of full propagation. On F, retaining rows inside the encoder is therefore the operative mechanism, not added parameters. With the signed readout, collapsing before the readout leaves ∑gGg unchanged, and encoder-only collapse raises the error from 97.2 to 150.4 on F (p = 3e-6) and from 0.70 to 1.05 on A (p = 0.02, 10 seeds), within noise of collapsing after every block (158.7 and 1.16).