acceptodds
Under review as a conference paper at ICLR 2027

ASAP: Amortized Doubly-Stochastic Attention via Sliced Dual Projection

Abstract

Doubly-stochastic attention has emerged as a transport-based alternative to row-softmax attention, with recent Transformer variants using it to reduce attention sinks and rank collapse while improving performance. In this family, the standard approach is Sinkhorn scaling, which trains more efficiently but still repeats matrix scaling in every inference forward pass. Sliced-transport attention removes the online iteration, but its soft sorting approximation materializes dense tensors for each slice, requiring substantially more training resources than Sinkhorn attention. We introduce ASAP (Amortized Doubly-Stochastic Attention via Sliced Dual Projection), a train-then-compile method that trains the doubly-stochastic layer with Sinkhorn, then replaces the iterative scaling loop at inference with a fixed sliced-dual operator. It learns a lightweight parametric map from exact one-dimensional Kantorovich potentials to the Sinkhorn query-side dual, then reconstructs the attention plan with a two-sided entropic c-transform. Across language and vision benchmarks, ASAP keeps the cheaper training setup and remains highly competitive with recent baselines. In the main frozen-layer benchmark, ASAP is 5.3× faster than the trained Sinkhorn teacher while matching its accuracy; in downstream replacements, ASAP recovers most of the teacher performance without any retraining.

Primary area:
optimization
Keywords:
optimal transport, doubly-stochastic attention, efficient attention, Sinkhorn algorithm, sliced optimal transport, amortized optimal transport, transformers

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_GMxyOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

ASAP replaces most iterations of a frozen Sinkhorn attention layer with a calibrated predictor of its source potential. The predictor uses one-dimensional sliced transport potentials and is fitted by ridge regression or a one-sided KL objective. The other potential is recovered analytically, followed by a fixed number of normalization passes. Experiments measure agreement with the teacher, classification accuracy, calibration cost and inference time.

Critical strengths

Predicting one potential and recovering the other analytically gives a compact replacement for a trained layer. The paper is careful about finite-step conventions: it distinguishes a change of potential coordinates from restarting Sinkhorn with a different kernel, and notes that the extra correction can move the reconstruction away from the finite teacher even with exact prediction.

The evaluation includes more than task accuracy. The reconstruction and predictor ablations help explain where the error comes from, and the shared-kernel comparison in Table E.3 addresses an important implementation concern. The reported full-model result on Cats and Dogs—1.53× faster with a 0.65-point accuracy loss—is also useful evidence of a deployment benefit.

Critical weaknesses

The short-Sinkhorn comparison changes the final constraint as well as the iteration budget. Appendix F.1 retains NN query rows and m=∣J∣m=|J| active key columns. A row-ending attention matrix has total mass NN, whereas a column-ending matrix has mass mm. Table H.1 states that S=3/5S=3/5 end in rows, while the teacher and ASAP end in columns. For m<Nm<N, these constraints are incompatible; a column-ending matrix necessarily has mean absolute row error at least 1−m/N1-m/N. This follows from the marginal mass identity underlying the transport constraints in Cuturi (2013), §2. The authors disclose this convention, but Table 1 does not control for it. Column-ending S=2/4/6S=2/4/6 on the existing frozen cases, preserving the teacher's score kernel and initialization, would determine how much of the fidelity gap remains after matching the terminal constraint.

The learned predictor's increment over inexpensive reconstruction is unresolved. Table H.3 includes useful fitted-feature controls, but lacks an unlearned source potential under the same column–row–column reconstruction. Besides zero potential in cost coordinates, use the known shift fi=∥qi∥2/(2dh)f_i=\|q_i\|^2/(2\sqrt{d_h}). With the paper's quadratic cost, this makes the first reconstructed attention exactly column-softmax of the scaled scores, without slicing, sorting or fitting. Appendix I.5 combines a learned query-norm coefficient with row log-sum-exp features; that ablation does not isolate the fixed analytic shift. These controls would distinguish the learned initialization from the normalization work, a relevant distinction given the established role of Sinkhorn initialization(https://proceedings.mlr.press/v206/thornton23a.html) and the credited precedent for sliced-potential regression(https://arxiv.org/html/2604.15114v2).

The shared-kernel timings in Table E.3 and the full-model result in Table 4 are strengths. However, the strongest long-sequence timings and the trained-model fidelity evidence use different settings. Testing the matched controls on one frozen trained model with the optimized backend would establish whether the learned predictor provides a useful practical improvement.

Questions for authors

How do column-ending S=2/4/6S=2/4/6, zero-cost-potential reconstruction and the analytic query-norm-source control compare on the Table 1 cases? Preserve each method's stated coordinates and report attention error, output RMSE and latency, including every normalization pass and grouping results by active-token fraction.

Eq. (F.2) includes every query, but feature construction and regression exclude padded positions. How are padded-query source potentials assigned and centered? Please specify which query positions enter the reported errors.

On one trained model, does ASAP retain an accuracy–latency advantage over these controls under the same optimized implementation? Report results by teacher seed; the 144 batch–layer cases are repeated measurements from three teachers.

2 comments

We have added controls for finite normalization and source initialization on the original cases, made the padded-position treatment explicit, and reported the trained teachers individually.

Q1. Marginal Mass and the Learned Source

The original S3/S5 comparison did not control for the effect of unequal total mass. We now use column-ending S2/S4/S6 from the teacher's score kernel and initialization, so the final target and total mass agree with the teacher and ASAP. We also evaluate the analytic norm source alongside zero cost potential, applying the same KQK reconstruction to both. Zero cost source corresponds to score source −ρ-\rho, while cost source ρ\rho corresponds to zero score source. The latter therefore tests the inexpensive column-softmax initialization, without slicing or fitting.

On the original cases, we measure output RMSE 0.002656 for ASAP, against 0.007806 for zero cost source, 0.005793 for the analytic source and 0.003135 for S6. The fixed-source controls show a fitted-source benefit beyond the shared transforms, while S6 remains slightly better on aggregate attention error. We grouped inputs by active-token fraction and evaluated every group on the same sequence IDs across teachers. The output-fidelity improvement over the analytic source persists throughout, while the comparison with S6's attention error depends on the group.

Table R8. IMDb by active fraction. Entries are output RMSE followed by attention error, using all retained source rows. Each teacher sees the same 256 sequences. Within a group, squared output errors are pooled over its examples and six layers before taking the root, then averaged over teachers. Attention error averages example-head relative ratios. These are group estimators, not the case-averaged RMSE of Table R1.

Active fractionSequences per teacherS2S4S6Zero costAnalytic normASAP
0 to 0.25220.026856 / 0.4603760.007298 / 0.1045000.003032 / 0.0414190.009357 / 0.1306240.005411 / 0.0821490.003019 / 0.048638
Above 0.25 to 0.501150.038606 / 0.4091870.010750 / 0.0958060.004607 / 0.0347580.012556 / 0.1119700.007866 / 0.0703050.003860 / 0.036951
Above 0.50 to 0.75500.045786 / 0.3669080.014286 / 0.0915420.006111 / 0.0330860.015697 / 0.1039420.008686 / 0.0590350.004495 / 0.032061
Above 0.75 to below 1280.057081 / 0.3259220.020335 / 0.0889730.008641 / 0.0327940.021047 / 0.0952520.011458 / 0.0527720.005934 / 0.030756
1410.063484 / 0.3138940.023019 / 0.0888770.009952 / 0.0327860.023090 / 0.0916340.013082 / 0.0526980.006382 / 0.029902

In the fully occupied group, we find lower output and attention error for ASAP than for every listed control, so this result does not depend on unequal support sizes. We also repeated calibration and full-model evaluation with compatible row targets of one and active-column targets of N/mN/m in Table R5, where the learned source again improves output fidelity over both fixed initializers. Table R3 reports the full cost of the original controls, including their initialization and all normalization passes.

Q2. Padded Positions and Centering

The active-only sentence did not describe the IMDb implementation, and we have corrected Appendix F.1 to state the support used by each experiment. For IMDb, we construct sliced features, fit the regression and center the source over all N retained positions, with the key mask applied during attention reconstruction. We compute a feature-based prediction at each padded source position just as at every other retained position. The active-only construction belongs to the N128 text experiments, where padding is removed from the transport support and excluded from pooling.

Writing u(T)u^{(T)} for the stored score-coordinate source, ρ\rho for the quadratic row shift and ΠN\Pi_N for centering over all retained source positions, we fit y=ΠN(u(T)+ρ)y=\Pi_N(u^{(T)}+\rho) and reconstruct from u^=ΠN(ΠN(Xω)−ρ)\hat u=\Pi_N(\Pi_N(X\omega)-\rho). This applies one common gauge shift to all source positions. Centering only active positions and then inserting zero padded potentials would alter their relative column weights, so we do not use that construction. The original fidelity metric includes every source row, and the classifier retains its original max pooling. To address which positions contribute to the error, we also evaluated valid source rows alone and obtained output RMSE 0.003533 for ASAP, 0.004177 for S6 and 0.007730 for the analytic source. The fitted-source benefit remains on the positions containing text.

Q3. Trained Models, Common Implementations and Teacher Seeds

The 144 batch-layer cases are repeated observations from three trained teachers and should not be treated as independent model runs. We now report the teacher-level results explicitly in Table R9, with full-test accuracy evaluated after each replacement propagates its own hidden states through the model. Table R3 supplies the common-MPS timing comparison on teacher 1, including the predictor and every normalization pass, so we measure deployment cost on the trained model itself.

Table R9. Teacher-level IMDb results. Each accuracy uses all 10,000 test examples. Output RMSE averages the 48 frozen batch-layer cases from each teacher.

TeacherS20 accuracy (%)S4 accuracy (%)S6 accuracy (%)ASAP accuracy (%)S6 output RMSEASAP output RMSE
187.3187.3187.3487.360.0049520.003818
287.6387.5687.5687.570.0020570.002009
387.0987.0387.1487.160.0023980.002140

We find lower output error than S6 for each teacher and 99.7733% mean prediction agreement with S20, while reducing teacher-1 full-model latency by 2.32 times in Table R3. The revised main table includes S4, S6 and both fixed sources with their lower latency and larger layer-output errors, so the comparison evaluates the additional fidelity from prediction as well as the saving over the teacher.

Reviewer_wQwWOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors proposed Amortized Doubly-Stochastic Attention via Sliced Dual Projection (ASAP). ASAP is built upon Sinkhorn attention, which make the attention score matrix doubly stochastic instead of unit sums only in the rows. Basically, the authors propose to amortize the sinkhorn iterations used for doubly stochasticity enforcement by learning a query potential f and its dependent key potential g (eq 1). With the learned potential, we can just run amortized inference to make the attention score matrix doubly stochastic. The resulting model outperform other variants of Sinkhorn attention in terms of inference speed and are more or less on par in performance.

Critical strengths
  • the idea seems interesting, i.e. using amortized inference to approximate the sinkhorn iteration for attention
Critical weaknesses
  • it's unclear to me whether the sinkhorn iteration is an actual efficiency problem for sinkhorn attention.
  • there are limited empirical evaluations for how well ASAP performs on nontrivial tasks. Right now The experiments are primarily restricted to text classification in Yelp Review Full / AG News etc and also a bit of image classification like CIFAR-100.
Questions for authors
  • how well does ASAP work in a simple nanoGPT-style pretraining in terms of the achieved perplexity?
3 comments

The complete-model comparisons below cover inference cost on the trained checkpoints and larger-model evidence. We also discuss what a nanoGPT experiment would require.

Whether Normalization Is a Practical Bottleneck

An iteration count by itself does not establish a practical bottleneck. For our S20 IMDb checkpoint, we therefore measured the complete forward through a common implementation, including the surrounding network and source predictor. In Table R4, compilation reduces latency from 1493.66 to 647.89 ms at B32 and from 48.75 to 30.74 ms at B1, retaining 99.75% of the teacher's predictions and 87.36% accuracy against 87.31%. We also tested simply shortening the same checkpoint's loop, as reported in Tables R1 and R3, to measure how much fidelity the fitted source preserves beyond those cheaper alternatives.

In the original Sinkformer IMDb experiment, Section 6.2, three updates suffice with modest training overhead. Our fresh S3/S4 runs show strong low-budget models as well. Published budgets vary across tasks, with Sinkformers studying up to 21 updates on ModelNet40 and ESPFormer, Table 10 using S15 for IMDb. We therefore evaluate the inference cost of the checkpoint being compiled and report fresh low-budget training alongside it.

ESPFormer removes online Sinkhorn iteration, but learns through dense soft-sorting matrices for each slice. Our full-encoder IMDb profile on RTX PRO 6000 Blackwell measures 69.51 GB and 138.82 ms per training example for that soft path at N512 and effective B32, against 12.24 GB and 9.37 ms for S10. We keep the latter training procedure and fit the potential predictor afterward, avoiding the slice-wise dense training representation. The full-model and shared-FlashSinkhorn comparisons then test whether reducing the remaining online normalization pays for prediction.

Evidence Beyond Small Text Classifiers

Our submitted ImageNet-1k replacement uses a DeiT-S/16 teacher fine-tuned from pretrained weights. After calibration on 4096 unlabeled images, default ASAP reaches 77.484% top-1 accuracy against 77.512% on the same 25,000 validation images from one training run. We evaluate the complete classifier, so later layers receive the compiled model's own activations. On the submitted Cats and Dogs model, we also measure a GPU-resident B64 forward reduction from 69.12 to 45.09 ms, with three-teacher mean accuracy of 80.44% against 81.09%.

We also extended the fixed predictors to longer vision inputs, evaluating all 5000 validation images per teacher at N785 and a trained-activation diagnostic at N4097. The complete comparisons appear in Appendix J and our response to qPQN Q3. They supplement the larger-model ImageNet evaluation with fidelity and full-model timing at longer input shapes.

NanoGPT Pretraining and Perplexity

Our present formulation is bidirectional, and adapting it to autoregressive pretraining would require us to make the entire attention output depend only on the prefix. The current column transform sums over source positions, so a triangular key mask alone still lets future tokens affect earlier normalization factors. If exact unit row and column sums are also imposed on a nonnegative lower-triangular matrix, the last column forces its diagonal entry to one, its row has no remaining mass for earlier columns, and induction forces the identity. ESPFormer, page 9 and LOTFormer, Appendix F discuss this same constraint.

We have not evaluated nanoGPT perplexity with a prefix-compatible ASAP variant. Developing one would require a modified or relaxed balancing formulation with prefix-causal outputs. The original Sinkformer translation experiment, Section 6.3 uses Sinkhorn in the encoder while retaining the decoder's causal attention, which is consistent with the bidirectional setting of our current evaluation. The ImageNet and full-model text results above test the scale and deployment of that encoder operator.

Thanks for the update. I understand that moving to decoder-style causal attention requires additional efforts. I'm raising my score in light of the clarification.

Thank you for considering our clarification and updating your assessment. We truly appreciate your thoughtful feedback and the constructive discussion, which helped us clarify and strengthen our paper. Please let us know if any further questions arise during the discussion period.

Best regards, Authors

Reviewer_XSi1OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This work introduces ASAP (Amortized Doubly-Stochastic Attention via Sliced Dual Projection), a train-then-compile method that trains the doubly-stochastic layer with Sinkhorn, then replaces the iterative scaling loop at inference with a fixed sliced-dual operator. The design aims to achieve speed advantages in both training and inference. It learns a lightweight parametric map from exact one-dimensional Kantorovich potentials to the Sinkhorn query-side dual, then reconstructs the attention plan with a two-sided entropic c-transform. This work presents a wide range of experiments to demonstrate the competitiveness of its method.

Critical strengths

This work (1) has a clear and good motivation, (2) with extensive and sufficient experiments.

Critical weaknesses

(1) The baseline comparison in this paper is unfair, and (2) it lacks novelty compared with existing work.

Questions for authors

(1) Could the authors compare ASAP with the Sinkhorn baseline models (S = 2, 4, and 6), and also compare a zero-initialized dual model to which the same K→Q→K correction is subsequently applied? Under identical masking and implementation settings, how do these alternatives perform in terms of attention fidelity, output root-mean-square error (RMSE), downstream accuracy, and latency?

(2) Using the same architecture, training pipeline, and computational resources, what accuracy can be achieved by training Sinkformer models with S = 3 or 4 from scratch? How do these models compare with ASAP in terms of the accuracy–latency trade-off?

(3) In some experiments, although the same number of normalization iterations is used, ASAP runs faster than S = 3 Sinkhorn. What causes this discrepancy? Does this advantage still hold when both use the same kernel function, masking rules, and timing protocol (including sliced-feature computation)?

(4) For IMDb, since using only the key padding mask leads to a transport problem in which the row marginals and active-column marginals are incompatible, could the authors repeat the main comparison experiments using a consistent masking scheme and feasible marginal constraints? How would this affect the teacher model’s row-marginal error, ASAP’s fidelity, and the relative performance of the short-iteration Sinkhorn baselines?

(5) In the unmasked setting, what are the row-marginal and column-marginal errors of the Sinkhorn teacher, ASAP, and the short-iteration Sinkhorn baselines, respectively? How do these errors vary with sequence length and iteration budget?

(6) Appendix I.5 shows that query-based RBF features and two feature expansions involving row log-sums and exponentials have better approximation accuracy than the default sliced potential predictor. Under the same resource budget, when comparing accuracy, inference latency, and memory consumption, what are the specific advantages of sliced potentials relative to these alternatives? Does this trade-off change as sequence length increases?

5 comments

We tested the effects of final normalization, initialization and implementation overhead, and added the from-scratch comparison.

Q1. Short Budgets and Zero-Source Reconstruction

A zero-source baseline needs the same correction as ASAP to establish what the fitted source contributes. We evaluated both zero cost potential and the analytic norm source under KQK, along with column-ending S2/S4/S6 controls. Table R1 reports attention and output errors on the same activations, and Table R3 evaluates complete-model accuracy and latency. On IMDb, the fitted source lowers output RMSE from 0.007806 and 0.005793 for the two fixed sources to 0.002656, while the fixed sources are faster. We observe the same output-fidelity ordering on unmasked vision, so prediction improves on the correction-only alternatives at its measured additional cost.

We specify the coordinates because setting the cost potential to zero is not the same initialization as setting the score potential to zero. With Cij=ρi+κj−sijC_{ij}=\rho_i+\kappa_j-s_{ij} and ρi=∥qi∥2/(2dh)\rho_i=\|q_i\|^2/(2\sqrt{d_h}), the former starts from column-softmax of (s−ρ)/ε(s-\rho)/\varepsilon, while cost source f=ρf=\rho starts from column-softmax of s/εs/\varepsilon. We use ε=1\varepsilon=1 and apply the same remaining row and column transforms to both. The normalization-only source f=ρ−εLSE⁡j(s/ε)f=\rho-\varepsilon\operatorname{LSE}_{j}(s/\varepsilon) over active keys is also covered by S4, since its initial row reduction followed by KQK is exactly that four-update computation.

Q2. Training S3 and S4 From Scratch

We trained S3 and S4 from the same random initialization with the original architecture, 40,000/10,000 split, Adam, B32, 15 epochs and learning-rate schedule, using the final epoch for evaluation. Table R4 compares these models with the existing S20 checkpoint and its compiled replacement on the same hardware.

Table R4. Fresh low-budget training and compilation. The two fresh models share one initialization. All accuracies use the full 10,000-example test set. The four models are timed in a joint MPS evaluation using the resident-input timing procedure of Table R3.

ModelAccuracy (%)S20 agreement (%)B1 msB32 ms
Existing S2087.31100.0048.751493.66
ASAP from that S2087.3699.7530.74647.89
Fresh S387.6293.3712.83338.50
Fresh S487.6993.7215.58428.05

Fresh S3 and S4 achieve higher accuracy and lower latency in this run, supporting low-budget training when a new model is being learned under this recipe. ASAP addresses the additional case in which a trained S20 checkpoint is already available, retaining its weights and 99.75% of its predictions while reducing batch-32 latency by 2.31 times through unlabeled calibration. Training lets the projections and downstream layers adapt to the short operator, while truncating an existing checkpoint does not, which motivates evaluating the frozen replacements in Table R3 as well. Post-training conversion is also studied in LoLCATs, which learns an approximation of pretrained attention before low-rank adaptation. Our potential fit retains the entire backbone, making reuse of the trained model the objective of calibration.

We also included direct training of the two-sided ASAP parameterization in the submitted ImageNet-100 study. With final row normalization in both variants, joint training reaches 63.62% accuracy against 61.28% after compilation from a 61.38% teacher on all 5000 validation images. These are individual 30-epoch runs; the compiled result measures how closely unlabeled calibration retains the selected teacher.

Our submitted text experiments provide further fresh S3/ASAP comparisons at the same epoch budgets, with accuracies of 90.12/90.03% on AG News, 54.45/54.51% on Yelp and 98.21/98.26% on DBpedia14. They use three, three and two paired seeds respectively, selecting the highest observed test accuracy over epochs, while the new IMDb comparison uses the final epoch.

Q3. Why Three-Pass Timings Can Look Faster Than S3

The original Sinkhorn routine computes convergence statistics and extracts a scalar to the host inside the loop, while the fixed ASAP reconstruction does not perform that check. The two routines therefore execute unequal work despite both being described by three updates. We have now compared them through common MPS primitives, including the predictor's projection and sorting cost, and S3 is faster in that comparison.

On the same frozen checkpoint, we measure 371.61 ms for S3 and 714.60 ms for ASAP at B32. Holding the final side fixed as well, the analytic-source KQK control takes 430.37 ms against 714.60 ms for ASAP in Table R3. Prediction adds work at this three-pass budget, while its speedup over S20 comes from replacing most of the longer loop. The common-implementation comparison now attributes that saving to fewer updates, and our shared-FlashSinkhorn evaluation tests the same reduction with fused GPU kernels.

Q4. Compatible Marginals for IMDb

In the original IMDb comparison, we retained all N512 source positions while imposing unit sums only on the active key columns, so the row error to one has the unavoidable lower bound 1−m/N1-m/N. More Sinkhorn updates cannot remove that incompatibility. We repeated calibration and evaluation with probability mass 1/m1/m on each active key, corresponding to attention targets of one per row and N/mN/m per active column. We kept the backbone, slice directions and pooling fixed, and recalibrated the source coefficients on 512 unlabeled inputs propagated through this feasible teacher.

Table R5. Compatible-marginal IMDb. Fidelity uses 256 test inputs through six layers per teacher, with the case-averaged output RMSE and attention estimator of Table R1. Accuracy uses the same 2000 examples per teacher for every method and reports mean +/- sample SD across three teachers. Mean active-column errors to N/m are between 1.23e-6 and 1.31e-6.

MethodOutput RMSEAttention errorRow MAEAccuracy (%)
Shared S20referencereference0.00011287.02 +/- 0.10
S20.0514730.3781990.18247586.58 +/- 0.31
S40.0136250.0933370.03123286.85 +/- 0.35
S60.0055500.0343450.00825287.00 +/- 0.13
Zero cost source, KQK0.0153000.1065110.03790386.75 +/- 0.35
Analytic norm source, KQK0.0115890.0636490.01582386.67 +/- 0.16
ASAP, KQK0.0053780.0353010.01151386.85 +/- 0.10

Under compatible marginals, the learned source still improves output fidelity over both fixed KQK sources and preserves 99.53% of the feasible teacher's predictions. Its output error is slightly below S6, whose attention error, row residual and mean accuracy are better in this comparison. We recalibrate along the feasible teacher's trajectory and propagate each replacement through the classifier, so Table R5 tests the changed normalization beyond a rescaling on fixed activations. Our AG News, DBpedia14 and Yelp teachers also use compatible active-token supports and masked pooling during training.

Q5. Unmasked Errors Across Lengths and Budgets

We measured both marginal errors across lengths and budgets on unmasked vision activations, where the two targets have equal total mass. At native N197 across three teachers, row MAEs for S2/S4/S6/S10/ASAP are 0.201505/0.061195/0.025704/0.007195/0.016030, with mean column errors below 4e-7. The learned source lets three-transform ASAP reduce the row residual below S6, while the S10 teacher remains closer to simultaneous balance. We then evaluated the same checkpoint, images and fitted coefficients at four resolutions to test how that relationship changes with length.

Table R6. Row MAE across resolutions. The same seed-2 checkpoint, 16 images and L64 coefficients are used at every length, with bicubic positional interpolation outside the native resolution. Every displayed method ends in columns, whose mean residuals range from 1.47e-7 to 3.88e-7.

TokensS2S4S6S10ASAP
650.2307140.0703040.0284390.0071970.022358
1970.2185930.0638330.0248790.0058260.015214
4010.2394360.0748340.0313150.0082910.018579
7850.2413470.0768140.0328180.0089730.018993

We observe the same ordering at every tested resolution without refitting. The N4097 diagnostic also has row errors of 0.020438 for ASAP, 0.034615 for S6 and 0.009913 for S10, with column errors below 5.4e-7. Source prediction therefore improves the row residual over S6 across these unmasked inputs, while S10 remains closer to simultaneous balance. Teacher-output fidelity is evaluated alongside these marginal errors because the trained network uses a finite, not fully converged, operator.

Q6. Predictor Accuracy, Memory and Length

The richer dictionaries in Appendix I.5 improve on the sliced default, and parameter count alone is not sufficient to judge their practical value. On the ImageNet teacher, sliced L32 has relative output RMSE 0.067 after KQK, against 0.037 for the norm-plus-rowLSE extension and 0.036 for RBF, at predictor costs of 5.59, 6.96 and 7.57 column-pass units at N197. We chose the compact default for its small closed-form fit and have now added the full-model IMDb comparison in Table R2, with common calibration inputs and reconstruction, to measure accuracy and absolute inference cost.

In the IMDb comparison, the contextual MLP has lower reconstruction error and faster inference than sliced L32, while the sliced fit requires 5.90 seconds of predictor-specific work and 25,344 bytes of numeric state across six layers, against 100.09 seconds and 106,848 bytes for the MLP. We now report peak process RSS for all six learned predictors as well, so the comparison includes measured process memory alongside the stored state in Table R2.

Table R7. Predictor calibration and process memory. Peak RSS is the process high-water mark from a fresh process running five warm full-model forwards at N512/B32, reported in MiB. Fitting time covers predictor-specific feature formation and fitting across six layers, including RBF center setup and all three bandwidth candidates. It excludes shared teacher traversal and training-validation feature preparation, while MLP optimization includes its internal validation. GPU workspace is not measured by RSS.

PredictorFitting time sPeak RSS MiB
Norm0.60333.55
Norm + rowLSE0.91338.22
Sliced L325.90343.64
Sliced + norm + rowLSE6.59349.25
RBF, 32 centers11.00346.05
Context MLP, width 32100.09352.75

To test length dependence, we reused the fitted predictors on N128/256/512 prefixes of the first 32 held-out sequences, recomputing features and teacher activations at each length. Mean per-layer attention-path latencies at B32 are 11.77/39.53/104.15 ms for sliced L32, 5.58/18.81/62.55 ms for the MLP and 5.29/22.59/70.22 ms for RBF, averaging the six layers and including source prediction, reconstruction and output projection but excluding QKV projection. The MLP also has lower pooled output RMSE throughout, at 0.006009/0.004453/0.002420 against 0.012472/0.011017/0.007326 for sliced L32. Its fixed-width prediction avoids sorting, so these results support it for lower-cost repeated inference over the tested range, while sliced regression remains the compact closed-form choice. Appendix J describes the timing procedure and all predictor fits.

Reviewer_qPQNOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes ASAP, a train-then-compile method to accelerate Sinkhorn-based doubly-stochastic attention. It trains a Sinkhorn attention model and then learns a lightweight mapping from sliced one-dimensional Kantorovich potentials to query-side dual potentials using a small unlabeled calibration set. At inference, the predicted potentials are used with fixed entropic c-transforms to reconstruct the attention plan, reducing iterative normalization. Experiments across language and vision benchmarks show that ASAP largely preserves teacher performance while improving inference efficiency, achieving a 5.3× layer-level speedup on IMDb.

Critical strengths
  1. The paper addresses a practical inference bottleneck in Sinkhorn attention with a simple and well-motivated amortization approach. It requires only a small unlabeled calibration set without backbone retraining.
  2. Experiments across multiple language and vision benchmarks demonstrate promising inference speedups with generally small accuracy degradation.
  3. The theoretical analysis and extensive ablations provide useful support for the proposed reconstruction and design choices.
Critical weaknesses
  1. Baseline comparison concerns. Table 1 compares row-ending S=3/5 Sinkhorn with column-ending teacher and ASAP. This mismatch, particularly under padding, may affect the reported teacher-fidelity advantage.

  2. Limited methodological novelty. The benefit of sliced potentials over simpler normalization-based features or existing learned dual initialization methods is not sufficiently established.

  3. Limited practical evaluation. The 5.3× speedup is layer-level, while optimized-kernel speedups are 1.40–2.05×. Full-model evaluation remains limited, and CIFAR-100 accuracy drops from 64.95% to 63.46%.

Questions for authors
  1. Can the authors compare short-budget Sinkhorn baselines using matched final normalization conventions and padding treatment (e.g., S=2/4/6)?
  2. How much benefit do sliced potentials provide over simpler normalization-based features or learned Sinkhorn initialization methods under matched computational budgets?
  3. Could the authors provide end-to-end accuracy–latency comparisons against optimized Sinkhorn baselines on larger models and longer sequences?
3 comments

We have added normalization and initialization controls, compared alternative predictors, and evaluated the replacements through the full classifiers.

Q1. Final Normalization and Padding

The final normalization can affect the reported fidelity gap, especially under padding. In the original IMDb setting, unit sums on NN retained source rows imply total mass NN, while unit sums on mm active key columns imply mass mm. The S3/S5 comparison therefore changed total mass as well as the update budget. We have replaced the main comparison with S2/S4/S6 starting from the teacher's initialization and ending in columns, as the teacher and ASAP do. We also evaluate zero cost potential and the analytic norm source with the same column-row-column reconstruction, denoted KQK, to test whether prediction improves on the correction itself.

Table R1. Frozen-operator fidelity with the same final normalization. Each task uses three teachers and 144 batch-layer cases. IMDb uses eight batches of 32 sequences through six layers per teacher, while native unmasked vision uses eight batches of two images through six layers. Output RMSE averages case-level errors after value mixing and output projection. Attention error averages headwise relative Frobenius ratios for IMDb and whole-case relative Frobenius ratios for vision.

MethodIMDb output RMSEIMDb attention errorVision output RMSEVision attention error
S20.0251490.3809600.1201390.370356
S40.0077670.0938630.0374840.123647
S60.0031350.0344730.0142320.046531
Zero cost source, KQK0.0078060.1069200.0379490.134982
Analytic norm source, KQK0.0057930.0643840.1195760.420332
ASAP, KQK0.0026560.0351940.0104760.043079

With the final constraint held fixed, we find output RMSE below S6 by 15.3% on IMDb and 26.4% on unmasked vision. In the fixed-source comparison, all three methods apply the same transforms, and ASAP improves on both unlearned choices. S6 has slightly lower aggregate attention error on IMDb, while ASAP has lower output error after value mixing and projection, which measures preservation of the representation passed to the next layer.

We also checked the fully occupied IMDb inputs to test whether the gain persists without key padding. On the same 41 sequences per teacher, the mean of teacher-specific pooled output RMSEs is 0.006382 for ASAP, 0.009952 for S6 and 0.013082 for the analytic source, using the estimator in Table R8. The learned-source benefit is therefore present in the unmasked inputs as well as in the controlled padded comparison.

Q2. What Sliced Potentials Contribute

Our source-prediction approach builds on Meta OT, and the sliced regression follows Truong and Nguyen. Thornton and Cuturi also establish the value of data-dependent initialization. We therefore compare normalization features, sliced potentials, RBF features and a contextual network fitted to the same teacher under common KQK reconstruction, so the choice of learned representation can be assessed independently of the correction sequence.

We chose sliced potentials because each feature already describes transport between the two projected token sets, allowing a small coefficient vector to respond to their current geometry. In the submitted feature ablation, this representation has roughly half the output RMSE of raw projections or rank displacement under the same reconstruction. We then fit the source produced by the trained attention layer at its actual stopping budget and score scale. For the optional KL fit, we compare the teacher with the one-key-closure reconstruction and use its realized row marginal in the linear term. This targets the finite computation used by the network, including its residual imbalance, while retaining the dense score kernel that an ESP plan mixture would replace.

Table R2. Predictor quality and complete inference cost. All six predictors use the same 512 calibration and 64 disjoint training-validation sequences from IMDb teacher 1. Fidelity uses 256 held-out sequences and pools squared output errors before taking the root, while accuracy uses all 10,000 test inputs. Full-model MPS timing includes prediction and reconstruction at N512/B32. State includes fitted parameters, fixed buffers and numeric configuration across six layers.

PredictorPooled output RMSEAccuracy (%)Teacher agreement (%)Full model msState bytes
Norm0.01804687.2999.28398.9524
Norm + rowLSE0.01484087.3799.42426.1048
Sliced L320.00773187.3699.75655.1025344
Sliced + norm + rowLSE0.00849187.4099.65687.8525392
RBF, 32 centers0.01305087.4299.49452.9325368
Context MLP, width 320.00268087.3499.93409.10106848

The comparison supports the sliced features over the norm, rowLSE and RBF fits on this teacher. Our contextual MLP has lower reconstruction error and faster inference than sliced L32. The sliced fit uses 32 coefficients per layer and one ridge solve, against 4193 network parameters per layer and 2000 optimization steps for the MLP, with predictor-specific fitting times of 5.90 and 100.09 seconds across six layers. We chose the sliced map for this compact closed-form calibration, while the MLP offers a useful option for applications that can amortize its additional fitting work. Both use the finite teacher as supervision, retaining the trained operator as their target.

Q3. Complete Models and Optimized Implementations

We evaluate every replacement through the full classifier, using common score formation, masking, log-domain normalization and value mixing, with all sliced-feature work included in the timed forward. This lets us compare the cost of preserving the existing S20 model with the cheaper option of truncating that same model.

Table R3. Full IMDb replacement. Accuracy is mean +/- sample SD over three teachers, each evaluated on all 10,000 test examples, and agreement is averaged over those teachers. Latency uses teacher 1 at N512/B32 on Apple M1 Max with 32 GiB unified memory and PyTorch 2.4.1 MPS. Inputs are resident, and timings are medians of three round medians, with five warmups and 15 synchronized forwards per round.

MethodAccuracy (%)Teacher agreement (%)Full model ms
Shared S2087.34 +/- 0.27100.00001658.36
S287.25 +/- 0.1998.0367323.97
S487.30 +/- 0.2799.3500491.78
S687.35 +/- 0.2199.7100615.05
Zero cost source, KQK87.33 +/- 0.2199.3367439.78
Analytic norm source, KQK87.37 +/- 0.2199.5667430.37
ASAP, KQK87.36 +/- 0.2199.7733714.60

We obtain a 2.32-fold full-model reduction at batch 32 and 1.58-fold at batch one, while retaining the teacher's mean accuracy and the lower layer-output errors in Table R1. These measurements include the surrounding network and predictor, extending the submitted attention-path evaluation to complete-model deployment cost. We also use the same streamed FlashSinkhorn kernels for both methods on RTX 6000 Ada, where default ASAP is 1.40 to 2.05 times faster than fused S10 across 4096 to 65,536 tokens. Fusing each update and reducing the number of updates can therefore contribute within the same implementation.

For the longer trained-model comparison, we evaluated all 5000 validation images for each of three Cats and Dogs teachers at 448-pixel resolution, or 785 tokens, without refitting. The fixed L64 predictor retains 80.06 +/- 0.97% accuracy against the teachers' 80.09 +/- 1.02%, compared with 79.17 +/- 0.86% for S6. At B8, the respective full-model latencies are 454.65, 395.30 and 268.26 ms. At 4097 tokens and B1, ASAP takes 768.94 ms against 1559.48 ms for S10, a 2.03-fold reduction, with output RMSE 0.014756 against 0.021350 for S6 on eight images through six layers of one checkpoint. The longer-input gain therefore depends on the deployment shape, with default ASAP slower than shared S10 at N785/B8. Appendix J reports every control at both shapes, and the N4097 timing uses the first resident image. Our ImageNet-1k evaluation addresses larger-model task accuracy, retaining 77.484% against 77.512% on the same 25,000 validation images through the compiled DeiT-S/16 network.

Comments on the paper

2 comments

We would like to extend our thanks to the reviewers for their careful feedback. In this paper we develop ASAP to preserve trained Sinkhorn attention at lower inference cost. Doubly-stochastic attention couples token interactions through a transport plan with prescribed row and column marginals. Sinkformers motivates this prior by observing that learned softmax attention moves toward double stochasticity during training, and reports accuracy improvements on vision and language tasks. Lapenna et al. further observe slower rank decay in attention-path products of trained Sinkhorn models. Deploying a Sinkformer still requires iterative normalization for every input at every attention layer, with repeated row and column reductions over the dense score matrix. ESPFormer removes online Sinkhorn iterations, but its soft-sort training retains dense matrices for every slice. Our full-encoder IMDb profile at 512 tokens and effective batch size 32 measures 5.7 times the peak training memory and 14.8 times the training time per example of S10 Sinkhorn on the same RTX PRO 6000 Blackwell. LOTFormer reduces computation through a small pivot measure, at the cost of restricting the coupling rank. ASAP retains Sinkhorn training and the learned dense scores, then compiles their normalization from unlabeled activations without updating the backbone.

ASAP compiles the attention operator through its transport potentials, which determine how the learned scores are rescaled. Building on Truong and Nguyen's sliced-potential construction and the potential amortization studied in Meta OT, we predict the trained layer's finite source potential from exact one-dimensional transport potentials. Each slice contributes a vector of NN values, producing an N×LN\times L feature matrix without a dense plan for every slice. Fixed entropic cc-transforms reconstruct attention from this prediction and the learned score kernel. The prediction targets the teacher's actual Sinkhorn budget, score scale and mask, so calibration approximates the normalization used by the trained network. In the compact six-layer IMDb model, calibration requires one ridge solve per layer and learns 192 coefficients, without soft-sort backpropagation.

We have expanded the experiments to test whether the fitted source improves on normalization alone, whether its cost is justified in complete models, and how it compares with simpler and richer predictors.

  1. Training cost. The submitted full-encoder IMDb profile measures the cost of obtaining the attention model, not only its final inference operator. At 512 tokens and effective batch size 32 on RTX PRO 6000 Blackwell, ESPFormer Soft uses 69.51 GB peak training memory and 138.82 ms per example, against 12.24 GB and 9.37 ms for S10 Sinkhorn, or 5.7 times the memory and 14.8 times the time. S20 uses 19.77 GB and 16.71 ms per example. ESPFormer's published Cats and Dogs transition also fine-tunes for 40 further epochs while annealing the temperature before hard evaluation (Section 4.4). ASAP retains Sinkhorn training and adds the reported unlabeled potential fit afterward.

  2. Source prediction. The reviewers asked whether the improvement comes from the fitted source or the final normalization. We compare column-ending S2/S4/S6 and two unlearned sources under the same column-row-column (KQK) reconstruction as ASAP. Across three IMDb teachers, output RMSE falls from 0.005793 for the analytic norm source to 0.002656 for ASAP. Relative to S6, ASAP lowers output RMSE by 15.3% on IMDb and 26.4% on unmasked vision. The benefit also appears on fully occupied IMDb inputs and after recalibration with compatible marginals, so it is not confined to the original padded setting. Tables R1, R5 and R8 report both output and attention errors.

  3. Full-model inference. We evaluate all 10,000 IMDb test inputs per teacher, retaining 87.36% mean accuracy against 87.34% and 99.77% mean prediction agreement. With common MPS primitives, teacher-1 full-model latency falls from 1658.36 to 714.60 ms at batch 32, a 2.32-fold reduction including source prediction. The submitted streamed GPU comparison uses the same FlashSinkhorn kernels for both methods and records 1.40- to 2.05-fold speedups over S10, testing the saving when both normalizers use fused implementations.

  4. Predictor choice. The reviewers asked whether simpler features or a learned network would be preferable to the sliced-potential map. We compare six learned predictor families under common calibration and reconstruction, reporting fitting cost, fidelity, complete-model latency and memory in Tables R2 and R7. These comparisons extend the submitted feature ablation and ImageNet predictor study beyond parameter count.

  5. Larger models. The submitted DeiT-S/16 ImageNet-1k replacement retains 77.484% accuracy against 77.512% on the same 25,000 images. Each compiled layer receives the replacement model's own hidden states, so this evaluation includes the accumulated approximation through the network. The full-model Cats and Dogs comparison also records a 1.53-fold GPU speedup with a 0.65-point accuracy gap across three teachers.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.