- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduces LOTFORMER, an efficient attention mechanism designed for bidirectional Transformers in which tokens serve as both information sources and receivers. LOTFORMER routes interactions through a learned low-rank pivot measure and composes two entropic transport couplings.
Critical strengths
LOTFORMER routes interactions through a learned low rank pivot measure and applies the resulting operator in O(nr) time. Composing two entropic transport couplings avoids the dense n × n attention matrix, finite Sinkhorn iterations approximate those marginals.
Critical weaknesses
- This paper investigates doubly stochastic attention, emphasizing its property of acting as both an emitter and a receiver; however, the distinctive characteristics are not demonstrated in the experiments.
- In Table 8: LOTFormer ablations, when the variable r of the low-rank pivot measure is increased from 32 to 64, Top 1 accuracy actually degrades, which is not reasonable. An explanation is necessary.
- The experimental results are not satisfied. In the retrieval experiments, comparing against only Hedgehog is insufficient. In the image recognition experiments, the results based on the Swin Tiny model are even worse than those of FLattenAttn, a 2023 algorithm.
- The paper is not written smoothly or clearly enough, especially the algorithm section, which is not concise and easy to understand.
Questions for authors
- What and where is the necessary and the unique of doubly stochastic attention?
- Why the performance is not better when the rank increases?
4 comments
(part 2)
W2 / Q2. Why does accuracy not improve when the rank grows from 32 to 64?
This is the classical bias–variance trade-off, and the pivot size r is exactly the knob that controls it.
-
Small r: high bias. The operator has rank at most r, so with few pivots every patch must route its attention through a coarse bottleneck. The operator cannot represent the interactions between 196 patches, and both training and test accuracy are low. This is why accuracy rises steeply up to r = 32 (64.5 at r = 4, 68.8 at r = 16, 74.8 at r = 32).
-
Large r: higher variance. Once r suffices to represent these interactions, additional pivots mainly add capacity: more pivot locations and masses per head, and a more flexible operator. Under a fixed data and training budget (ImageNet-1K, 400 epochs), this extra capacity fits the training set better without improving generalization. Test accuracy therefore drops, to 73.9 at r = 64 and 71.6 at r = 128.
-
Direct evidence from training and test accuracy (new Table 10). Training accuracy is one pass over the ImageNet-1K training set with evaluation transforms (no augmentation).
r 4 8 16 32 64 128 Train Top-1 70.2 71.7 75.6 80.5 84.6 89.2 Test Top-1 64.5 66.4 68.8 74.8 73.9 71.6 Gap 5.7 5.3 6.8 5.7 10.7 17.6 Up to r = 32, training and test accuracy rise together and the gap stays between 5.3 and 6.8 points (bias-limited). Beyond r = 32, training accuracy keeps rising (84.6, 89.2) while test accuracy falls (73.9, 71.6), and the gap grows to 10.7 and 17.6 points (variance-limited): the added capacity is spent on fitting the training set.
-
Consistent settings: every r uses the identical full DeiT-Tiny configuration and recipe (Table 9(iii) caption), so the trend reflects r alone.
-
r = 32 also costs less: the operator cost grows linearly in r (Eq. 8). r = 32 is therefore both the most accurate and cheaper than r = 64, and we use it throughout.
W3. Baselines and Swin.
The evidence supports two uses of LOTFormer: as a plug-and-play replacement in pretrained models, and as an attention layer trained from scratch, especially in vision.
Conversion: more baselines, and LOTFormer remains the strongest. Beyond Hedgehog, we added the closest transport-based alternatives under the identical conversion protocol on gte-Qwen2-1.5B (Table 16), with mean LongEmbed retention over three seeds:
DS: doubly stochastic. Positive-feature Sinkhorn uses r = 32 random features; the uniform-mass variant learns pivot locations only.
| Method | DS | Retention (%) |
|---|---|---|
| Hedgehog | ✗ | 93.2 |
| Positive-feature Sinkhorn | ✓ | 76.6 |
| LOTFormer, uniform masses | ✓ | 90.1 |
| LOTFormer | ✓ | 98.3 |
- Across model scales: LOTFormer retains 98.3–98.9% from 1.5B to 7B, against 93.2–94.1% for Hedgehog (Table 3).
- On GLUE: it recovers 99.6% of fine-tuned BERT, ahead of Hedgehog, T2R and T2R-HH under Hedgehog's own protocol (Table 4).
- Why few other conversion baselines apply: most recent conversion methods (Liger, SUPRA, DiJiang, RADLADS) linearize causal decoders through recurrent or causal structure and do not apply to bidirectional encoders. Hedgehog is the standard bidirectional plug-and-play baseline.
Training from scratch in vision: LOTFormer is the strongest under matched conditions.
- DeiT-Tiny, like for like. With identical auxiliary components and recipe (Table 12), LOTFormer is ahead both with and without DWC.
DeiT-Tiny, ImageNet-1K, Top-1 (%). Attention only: [CLS] softmax row, no DWC.
| Configuration | FLatten | PolaFormer | LOTFormer |
|---|---|---|---|
| Attention only | 71.8 | 61.9 | 73.2 |
| With DWC (and polarization) | 74.1 | 74.6 | 74.8 |
- DeiT-Small: LOTFormer reaches 80.1, against 79.8 for PolaFormer under the same 400-epoch recipe, with fewer parameters (22.3M vs. 23.0M) and the same FLOPs (4.8G).
- PVT-Tiny: 79.6, against 78.8 for PolaFormer.
- Limited supervision: LOTFormer leads softmax and PolaFormer at every data fraction (Table 6), by +4.5 points over softmax with 1% of the data.
Swin-Tiny.
- Result: LOTFormer reaches 81.8, which is +0.6 over softmax, within 0.3 of FLatten (82.1) and 0.8 of PolaFormer (82.6), with fewer parameters (28.0M vs. 29.0M).
- Why Swin differs: it attends within local 7×7 windows of only 49 tokens. This is the regime where a global low-rank operator has the least to offer, yet LOTFormer remains competitive there.
- Revised claim: Contribution 3 now states this precisely. LOTFormer outperforms the baselines on DeiT and PVT and is competitive on Swin.
Together, these results show that LOTFormer is effective in conversion and plug-and-play replacement, and that it is a strong choice when attention is trained from scratch, particularly for vision models.
(part 3)
W4. Clarity of the method section. We rewrote Section 3 with the same notation. It proceeds in order:
- entropic OT;
- queries, keys and the pivot measure;
- the two thin transport plans;
- gluing them through the pivot;
- the resulting attention operator, with the finite-T caveat;
- complexity and learning.
We also moved secondary tables and discussions to the appendix, with pointers from the main text.
End of Discussion Period
Dear Reviewer AyQD,
With the discussion period ending, we would like to briefly summarize how the revision addresses your questions, including results added after our first response.
Q1 (what is necessary and unique about double stochasticity). A matched control isolates the column (receiver) constraint, with the same pivots, rank, projections and recipe. It adds +1.7 to +2.0 points on DeiT-Tiny and +8.7 points with 1% of the data (Table 7). Accuracy also rises monotonically as the constraint is enforced more tightly (T = 1 → 5 → 10). In pretrained conversion on gte-Qwen2-7B, the same control trails LOTFormer (98.2% vs. 99.2% retention, Table R2).
Q2 (rank 32 vs. 64). New Table 10 shows a bias–variance trade-off. Up to r = 32, train and test accuracy rise together. Beyond r = 32, train accuracy keeps rising (84.6, 89.2) while test accuracy falls, and the gap grows from 5.7 to 17.6 points.
W3 (baselines and Swin).
- Conversion baselines. We added positive-feature Sinkhorn with both fixed and learned features, and a uniform-mass reference. They reach 76.6%, 94.3% and 90.1% retention respectively, against 98.3% for LOTFormer (Tables 16, R3).
- Swin-Tiny. We revised the claim to "competitive on Swin" (81.8 vs. 82.1 FLatten).
- Correction to our earlier reply. PolaFormer's 61.9 uses deformable convolution. Its convolution-free result is 68.1. Without convolution, LOTFormer reaches 73.2, against 71.8 for FLatten and 68.1 for PolaFormer. With DWC, we now describe LOTFormer as matching PolaFormer (74.8 vs. 74.6).
W4 (clarity). Section 3 has been rewritten in a linear order with consistent notation, and secondary material has moved to the appendix.
Reviewer arSF raised their score after these additions. We would sincerely appreciate it if you would consider them in your final assessment.
Response to Reviewer AyQD
We thank the reviewer for the review. The revision adds experiments that answer each question directly.
W1 / Q1. What is necessary and unique about doubly stochastic attention, and where is it demonstrated?
What is unique. Softmax attention constrains only the rows: each query distributes one unit of attention, but nothing controls how much attention each key receives. Attention can therefore collapse onto a few tokens while others are ignored. Double stochasticity also fixes the column sums, so every token both sends and receives exactly one unit of mass. This sender-and-receiver balance is the property the reviewer highlights, and it is natural for bidirectional encoders, where every token is both a query and a key.
Where it is demonstrated. We added a controlled experiment that isolates exactly this property (Table 7, Appendix E.3).
DeiT-Tiny, ImageNet-1K, Top-1 (%). Control: one softmax per plan instead of Sinkhorn.
What is necessary for this to be practical. Doubly stochastic attention existed before, but only at quadratic or superlinear cost: Sinkformer and ESPFormer. LOTFormer is the first to provide it in linear time, O(nr), which makes the property usable at long context and at the 7B scale.