- Rating
- 2
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper studies how attention can represent competition and exclusion among candidates. MarSea applies sparsemax within selected relation sets, with row and column forms for different constraints. The theory analyzes query-local limitations and exact support recovery. Experiments use a 1.5B decoder on retrieval tasks with three seeds. Results improve over ordinary softmax, while gains over softmax-one remain within seed variability.
Critical strengths
The distinction between selection within a row and coordination across queries is useful. The theory helps explain why changing attention scores alone may not provide the desired structural constraints. The row/column comparison and relation-off controls also help separate structural enforcement from behavior learned during training.
Critical weaknesses
The main experimental result is not clearly better than softmax-one. The deployed row form also remains inside the query-local class criticized by the theory, while the column form does not show a clear advantage. The paper therefore offers a stronger theoretical motivation than its experiments support. It remains unclear whether the proposed mechanism provides enough practical benefit to justify its added complexity.
Questions for authors
- Theorem 1 allows a product-form attainable set to lie inside a non-product constraint set. What additional condition rules out satisfying a column capacity through conservative per-query bounds? This matters to the scope of the impossibility claim.
- Given the lack of a clear gain over softmax-one, which reported result best establishes the practical value of MarSea beyond its theoretical contribution?
Thank you for the insights.
Theorem 1: what rules out satisfying a column capacity through conservative per-query bounds? Theorem 1 alone does not, and that case is what "except vacuously" refers to. A query-local scheme can satisfy Σᵢ Aᵢⱼ ≤ c on every input by capping each query's weight on key j at c/n_q. Each attainable set Rᵢ then lies in [0, c/n_q], and the product of the Rᵢ lies inside the constraint.
What rules it out is the non-degenerate replication assumption used in the capacity corollary. Suppose the scheme can, on admissible inputs, give key j a weight of at least a at M queries. In particular, suppose any single query can select key j, which the paper's proposition on local selection grants every query-local scheme. Then the product set contains a point with M coordinates of at least a, and the capacity fails whenever c < M·a. Conservative bounds satisfy the constraint only by giving up selection: no query could place more than c/n_q of its mass on the key it needs. App. J.8 of the revised PDF states this.
The main result is not clearly better than softmax-one. True of the submission. The cause was our implementation of the relation module, not the mechanism (general response). The submitted relation held 8% of the correct candidates at the rows that decide the list; the corrected one holds 99.8% and excludes values already listed (below). With the corrected module, MarSea reaches .988 exact-set on RULER's default layout against .925–.933 for softmax, softmax-one and the submitted version (32 values: .973 against .74–.78), with every seed above every baseline seed.
The deployed row form is query-local; the column form shows no clear advantage.
The corrected relation models a relation across rows. In multi-value listing, the rows that start a value and the values of the queried key stand in a one-to-one relation: each such row copies one value, and each value is to be listed once. This is a constraint across rows, of the non-product form that Theorem 1 is about. The corrected relation models it by a causal route: which values have been listed is read from the generated text, and those values are excluded from the relation of every later row. Row i's attention is therefore a function of what earlier rows emitted. Information passes between rows through the emitted text, not through attention mass.
How this stands with Theorem 1. The theorem assumes that the rows' inputs can be set independently. The causal route removes that independence by construction, so the theorem's impossibility does not bind the mechanism in decoding: the coupling the theorem shows to be necessary is supplied by the decoding loop. Within a single attention operation, with the listed signal taken as given, the row form is still query-local in the sense of Def. 1. And the exclusion is soft (listed keys keep 20% of their base attention), so "no value listed twice" is met in practice, not guaranteed: it holds on all 1,500 evaluation lists of the corrected relation, against .08–.78 repeats per 32-value list for the baselines.
The column form. The theory does not rest on it. The paper gave it too much weight in its presentation, as if it were the way to model a one-to-many relation. Such a relation can be modelled in either form, with the program on the row's slice of keys or on the key's slice of rows; "column form" names the orientation in which the row is the one end. Theorem 1 states the limit of a row-local program and stands whatever the column form's empirical result. As submitted it shows no advantage (.939 on the default layout against .933 for the submitted row form), and its implementation is weaker than the row form's: it marks ownership of a key by attention mass, which is unreliable before a value is emitted. The fair reference for its performance is the row form's machinery with rows and keys exchanged. That run did not finish within the discussion period; we make no claim for it here and will report it in the final version.
Which reported result best establishes practical value? In the submission, the reduction of repeated values at 32 values (from .31 to .21 per list against softmax-one). The result that now establishes it is the corrected relation: about five times fewer failed lists than any baseline on RULER's default layout (9 of 750 against 50–56).
Complexity. The correction adds about 0.4M parameters to a 1.5B-parameter backbone. It costs time at inference: decoding a 4K-token prompt with 32 values takes 14.7 s per list with softmax-one, 16.3 s with the submitted MarSea and 25.8 s with the corrected MarSea (1.75 times softmax-one, same GPU). Against it stand about five times fewer failed lists.
Presentation. We take the score of 1 seriously. The revised PDF of this period adds App. J only; the final version will rewrite Secs. 1–3 with a notation table and a running example.