- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper proposed a Tokaformer that can preserve typed tokens by a linear-recurrent routing before compression after analyzing the information loss in shared additive fusion and theoretically showing its inability to preserve track information. They conduct experiments to support the benefits of identity-preserving fusion and the effectiveness of their method.
Critical strengths
-
They offered a clear theoretical analysis of the information loss in shared additive fusion and its inability to preserve track information, which motivates their Tokaformer natually and smoothly. The method specifically design related mechanism to solve the problem.
-
They did comprehensive experimental validation by conducting experiments across multiple synthetic tasks and two real-world datasets and ablation studies to support their claims.
Critical weaknesses
-
For their proposed method, the idea is highly similar to prior work (paper name: [Climax: A foundation model for weather and climate]), which also preserves specific representations and performs cross interaction before compression. This means the novelty of this work is not enough.
-
Most of improvement are shown on on synthetic tasks while the real-world results show limited advantages for Tokaformer.
Questions for authors
N/A
2 comments
Thanks for your claims and explanation. However, I still have concerns about the novelty and effectiveness of your work. I will maintain my original score.
Response to Reviewer 57PE
Thank you for the review.
W1, similarity to ClimaX. Please see G1. ClimaX's cross-attention pools the variables at each spatial position into one token, and that pooling is the compression step; no interaction across positions happens before it. Tokaformer's routing stage operates across positions while typed tokens are still separate. We have added ClimaX's variable aggregation as a baseline (G1). With its own tuned recipe and no residual it trains, scoring 1.000 on the panel's positive-control cell, but on the task that requires moving information between positions it scores 0.0001 normalised accuracy (chance; n = 30), against 0.79 for Tokaformer on the same data. We accept that the submitted text did not make this distinction clear; the revision states it in the introduction and adds the comparison as Table 2.
W2, synthetic versus real-world evidence. Please see G2. The real-world experiments test the paper's prediction about encoder dependence, and both datasets are consistent with it. The revision presents them as a test of that prediction, and the limitations state that neither dataset measures cross-position retrieval. G2 names real tasks that do; we have not run them.
Presentation. The revision states the distinction of G1 in the introduction, adds the theorem's role to Section 2, and moves the baseline comparison into the main text as Table 2.