- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 3
Read review
Paper summary
The paper organizes the expressivity of attention and recurrent architectures through five structural invariants. Under specified architectural and realization assumptions, it establishes necessary conditions and sufficiency up to polylogarithmic factors for four canonical tasks.
Critical strengths
- The goal of organizing the expressivity of different architectures through a common perspective is interesting.
- The width lower bounds and matching constructions for associative recall with single-layer polynomial attention provide concrete quantitative insights under the stated assumptions.
Critical weaknesses
-
The conceptual advance from the unification is insufficiently demonstrated.
The framework largely combines separate obstructions and task-specific constructions for four canonical tasks under different realization conditions. It remains unclear what new insight it provides beyond the individual results and how the contribution differs from the closest prior work. This makes the significance of the central completeness claim difficult to assess. -
The cost comparisons are difficult to interpret.
The measure does not directly account for the complexity of embeddings, readouts, or feature maps, while the restrictions controlling their function-class richness at a fixed budget are unclear. Since these components affect realizability, the paper needs a stronger justification of what the reported cost gaps establish about model complexity.
Questions for authors
Clarification questions
-
Within the results already established in the paper, what additional insight does the unified framework provide beyond the individual obstructions? Please identify one representative result illustrating this benefit and explain its advance over the closest prior work, including relevant differences in assumptions and scope.
-
At a fixed cost budget, to what extent do the stated assumptions constrain the function-class richness of embeddings, readouts, and feature maps? Given that their complexity is not directly counted, what notion of model complexity do the reported cost gaps capture?
Presentation suggestion
A concise comparison table with the closest prior work would help clarify the assumptions, scope, guarantees, and new contributions.