- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper studies action-conditioned predictive learning with causal self-attention. The authors develop identification and convergence guarantees for sparse binary models, analyzing how attention initialization affects higher-order interaction learning. They also draw connections between predictive accuracy and optimal actions through an external planner.
Critical strengths
-
The analysis of how initialization affects the ability to learn higher-order interactions provides some insight into how the attention mechanism benefits the model.
-
The experiments examine several relevant factors, including model width, initialization, and data coverage.
Critical weaknesses
-
My main concern is that the theoretical results apply to several separate, highly simplified settings, and their connection is not strong enough. For example, the sparse model assumes that the relevant information is already provided as fixed binary features, with only one relevant feature per query. The higher-order learning results do not carry over to the multistep planning guarantee. It is therefore difficult to see how these results together address the broader problem of learning predictive models for decision-making.
-
The theoretical requirements are much larger than the models used in the experiments. Appendix U.4 gives sufficient widths of roughly 2.5 million and 75 million head pairs, while the experiments use only 16 to 256.
-
Parts of the manuscript appear inconsistent with the actual results. For example, Appendix P claims contributions involving a “fixed-token-pair finite inverse,” “cache-level reconstruction,” and intervention-specific identification, but I could not find corresponding definitions or results in the paper.
Questions for authors
-
Can the authors show an experiment where the model learns from sampled trajectories and then successfully plans using its predictions?
-
How does the proposed initialization compare with standard random initialization under the same training budget?
-
Can the theoretical bounds be tightened enough to help explain the experimental results?