- Rating
- 2
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
Furuya et al. proved that Transformers can approximate arbitrary continuous vector-valued functions of a token and its context, uniformly over contexts of arbitrary length. The present paper extends this framework to attention matrices between two input sequences. The output is a probability-valued function: either a conditional distribution over target tokens for each source token, or a joint distribution with prescribed marginals equal to the input measures.
The authors study an architecture comprising two independent Transformer encoders, followed by pairwise dot products and either softmax or Sinkhorn normalization. Combining Furuya et al.’s encoder universality theorem with approximation results for these normalized probability distributions, they establish universality for the corresponding classes of continuous relation-valued maps. The guarantee is uniform over input measures and uses Wasserstein distance between joint distributions.
Critical strengths
The problem formulation is coherent and the presentation is generally rather good.
Critical weaknesses
- As far as I understand their results can be obtained by a combination of some standard techniques from outside ML research with the universality result of Furuya et al. 2024. Therefore, I am not sure the contribution is strong mathematically.
- The setting seems very niche and not well motivated (at least to me). This applies both to the problem and the architecture they choose. There are several aspects one could complain about. On the one hand, we could ask if Wasserstein distance is appropriate for length-independent universality as for large input and large attention matrices a small W_d is consistent with large variation on individual rows. On the other hand, their continuity assumption excludes simple positional rules, such as using the first token to switch the attention pattern across the sequence, uniformly over unbounded lengths. Positional embeddings do not resolve this because each token receives mass . The paper could explain this restriction more.
Questions for authors
Would it be possible to strengthen the result to convergence in total variation rather than just Wasserstein distance?