acceptodds
Under review as a conference paper at ICLR 2027

Organize, Then Read: Rethinking Token Mixing via Structured State Representation

Abstract

Self attention has become a central token-mixing operator in embodied policies, yet many alternatives primarily address the quadratic cost of pairwise interaction rather than how source information should be organized before retrieval. We introduce Structured State Representation (SSR), an operator based on the principle of organize first, then read. Within each head, learned state centers shared across samples serve as common addresses for source organization. Source tokens are competitively assigned and aggregated into input-dependent states, after which queries selectively retrieve from the resulting compact state bank. This separation yields linear token-mixing complexity for a fixed number of states without constructing dense query–source interaction matrices. Controlled operator-replacement experiments in ACT and Diffusion Policy show that SSR remains competitive with standard attention and improves performance in selected settings, including human demonstrations. Qualitative latent analyses further suggest more progress-consistent representation geometry while preserving high correspondence with Transformer representations. Together, these findings support structured state organization as a viable alternative to direct pairwise attention in embodied policy learning.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Token mixing, Representation learning, Attention Mechanisms, Robot learning

Reviews

Among:
Shape:
1234this paper, Top 17%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.75 over 4 reviews, above 83% of the others.
Reviewer_PTuaOpenReview ↗
Rating
4
Confidence
2
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

The authors introduce Structured State Representation (SSR), a token-mixing operator that organizes source tokens into a compact bank of input-dependent states using learned centers before queries retrieve information from those states. SSR preserves the standard ((Q,K,V)\to Y) interface and reduces token-mixing complexity from quadratic to linear in sequence length for a fixed number of states. It also introduces an inductive bias through shared source organization, which the authors suggest may benefit certain tasks. The method is evaluated within two embodied policy frameworks, ACT and Diffusion Policy, through comparisons with Transformer and Mamba baselines. The experiments examine task performance, training dynamics, representation geometry, and computational characteristics.

Critical strengths

One strength of SSR is its compatibility with the standard attention interface, which allows it to replace attention modules within existing policy architectures. Its linear token-mixing complexity provides a clear computational advantage as sequence length increases. Tables 4 and 5 demonstrate reduced end-to-end FLOPs for projection-free SSR and substantial temporary-memory savings in the evaluated module-level benchmark. These results are promising, although the memory comparison depends on the particular attention implementation used. The policy experiments also demonstrate that SSR can remain competitive and improve performance in selected settings.

Another strength is the analysis of learned representations, which complements the performance comparisons. The visualizations suggest a more continuous organization with respect to episode progress, while high linear CKA indicates substantial representational similarity between SSR and the Transformer. Together, these observations are consistent with the interpretation that SSR changes how information is organized while retaining much of the Transformer’s representational structure. However, they do not directly establish preservation of all task-relevant information.

Critical weaknesses

SSR’s performance gains are task-dependent. In Table 1, SSR-P with center-based readout achieves the highest average success rate, but its improvements are concentrated on Transfer Cube, while it underperforms both the Transformer and Mamba on Bimanual Insertion. Although bolding the highest average is appropriate, this aggregate can obscure the tradeoff between tasks. The results therefore support improvements in selected settings rather than a consistent performance advantage.

The module-level memory comparison also requires qualification. Appendix C states that the Transformer reference uses nn.MultiheadAttention with need_weights=True, whereas SSR uses a TorchInductor-fused implementation. PyTorch recommends setting need_weights=False to enable optimized scaled dot-product attention and achieve the best performance. Consequently, Table 5 compares fused SSR against an attention configuration that requests attention weights, potentially increasing memory consumption unnecessarily. The reported reductions therefore do not establish the same advantage over optimized attention. Although the separate sequence-length benchmark uses SDPA, a corresponding comparison at the ACT configuration used in Table 5 would help establish the practical magnitude of the memory savings.

Questions for authors

1.The related work discusses Set Transformer, Slot Attention, and other approaches that organize information through compact intermediate representations, but these methods are not included in the experimental comparisons. How does SSR compare with these closely related alternatives under matched state capacity and training conditions? Such comparisons would help clarify the benefits of SSR’s specific organization and retrieval mechanism beyond its differences from standard attention and Mamba.

2.The paper identifies the inductive bias introduced by separating source organization from retrieval as its primary interest. However, the results do not clearly isolate the contribution of this design choice. Faster convergence is encouraging, but could also reflect differences in optimization, regularization, or model capacity. What evidence supports attributing the observed improvements specifically to the proposed inductive bias?

3.SSR’s relative performance varies across tasks: it improves some Transfer Cube settings but generally underperforms on ACT Bimanual Insertion and the additional ACT tasks. Can the authors identify conditions under which shared state aggregation is beneficial or detrimental? In particular, is there evidence supporting the hypothesis that aggregation removes fine geometric distinctions needed for precise control, beyond the observed differences in success rates?

Reviewer_YJcXOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Structured State Representation (SSR), which replaces direct pairwise self-attention with a mechanism that first organizes source information into a dynamic state bank and then retrieves information based on queries. The approach shows promise for embodied policies, with preliminary support from complexity analysis, operator-replacement experiments, and ablations. However, its methodological novelty, experimental robustness and generalizability, hybrid-readout rationale, and memory-efficiency claims require further clarification.

Critical strengths
  1. SSR separates source organization from query-conditioned readout while preserving compatibility with existing embodied-policy architectures.
  2. The paper establishes linear token-mixing complexity for a fixed number of states and evaluates computational efficiency and memory usage.
Critical weaknesses
  1. Although the paper discusses ISAB and Slot Attention, the comparison remains largely conceptual. Whether SSR's shared state addresses, distance-based competitive routing, and query-conditioned readout offer substantive advantages remains unclear.
  2. SSR is evaluated mainly on simulated ALOHA tasks, limiting evidence of its generalizability. While it achieves 100% success on Transfer Cube with Diffusion Policy (vs. 97.1% for Transformer and 97.9% for Mamba), all SSR variants underperform Transformer on Square and Transport in supplementary Robomimic ACT experiments, suggesting task-dependent performance. Evaluation across more challenging tasks, varying data scales, demonstration quality, and task complexity is needed to clarify its applicability and limitations.
Questions for authors
  1. SSR averages state centers and state contents to form readout anchors. In the projected variant, these representations derive from independently projected key and value representations. Although the paper describes equal-weight fusion as a design choice, it remains unclear why fixed averaging is preferable and how sensitive performance is to this choice.
  2. Although the paper evaluates sequence-length scaling using Transformer with SDPA, the module-level memory benchmark uses nn.MultiheadAttention with need_weights=True for Transformer, while SSR uses a TorchInductor-fused implementation. Returning attention weights may introduce additional memory overhead and limit the use of optimized attention paths, potentially overstating SSR's memory advantage.
Reviewer_yMP7OpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces Structured State Representation (SSR), a token-mixing operator that aggregates source tokens, via learned shared centers, into a compact set of input-dependent states before query-conditioned retrieval. This yields a dynamic low-rank factorization of token interactions with linear complexity for a fixed state count. The authors evaluate SSR in ACT and Diffusion Policy, reporting competitive performance in selected manipulation tasks alongside computational and representation analyses.

Critical strengths

The method is clearly formulated, with an intuitive distinction between learned addresses and input-dependent state contents. Preserving the standard attention interface enables relatively controlled comparisons within existing policy architectures, with ten paired training seeds in ACT. The authors also provide useful ablations and report negative results rather than presenting only favorable outcomes. The careful distinction between asymptotic computational complexity and practical runtime is also noteworthy.

Critical weaknesses
  1. Novelty and mechanism isolation. The organize-then-read formulation is closely related to prior approaches such as Set Transformer's ISAB and Agent Attention (Han et al., ECCV 2024), both of which support learned intermediate representations. SSR's competitive distance-based routing may provide a useful distinction, but its specific contribution is not adequately isolated. A matched latent-bottleneck baseline or an ablation removing competitive normalization would considerably strengthen their novelty argument.
  2. The M = 1 ablation raises an important concern. With a single state, Equations (6)–(7) reduce to uniform mean pooling over valid source values in the non-causal case, independent of the queries, keys, and learned centers. Yet Table 3 reports performance comparable to M = 4 across all four Insertion settings. This raises the possibility that encoder token mixing contributes relatively little to these tasks, or that multi-state organization adds limited value beyond pooling. Table 1 lends some support to the former possibility: the substantially narrower Transformer-S achieves the highest scripted Insertion success. I would encourage the authors to report M = 1 and identity/no-mixing controls on the main Transfer Cube experiments, where SSR's gains are reported, to determine whether these gains survive the removal of multi-state organization. Assignment-entropy and state-similarity diagnostics would also help establish whether meaningful differentiation emerges among the learned states.
  3. Task-dependent results complicate the empirical conclusions. In Table 1, the default SSR-PF hybrid configuration improves human Transfer Cube success (69.4% versus 52.6%) but underperforms on scripted Insertion (42.8% versus 49.6%). Moreover, all three SSR readout variants have lower Last-5 averages than Transformer on both Robomimic ACT tasks (Table 8). These differences are obscured by the aggregate performance column. The ACT experiments also replace encoder attention only, leaving decoder cross-attention intact, so they do not test SSR as a full replacement for token-level retrieval. I would appreciate paired-seed confidence intervals and further discussion of why SSR benefits certain tasks but degrades on others. The Diffusion Policy results are near saturation, so claims about faster convergence would benefit from a quantitative time-to-threshold analysis across seeds.
  4. Representation-geometry claims need stronger evidence. Figure 3 suggests more progress-consistent SSR representations, but the t-SNE/PCA analyses use only a single seed, and the displayed principal components explain relatively little variance. The high CKA values (0.90–0.99) are also difficult to interpret without a Transformer-versus-Transformer cross-seed reference. Quantitative temporal-consistency measurements or episode-progress probes would provide stronger support for the proposed representational benefits.
Questions for authors

The most valuable additions would be: (i) pooling/no-mixing controls in the main ACT experiments, (ii) diagnostics of learned state differentiation, and (iii) paired-seed uncertainty estimates. A Transformer cross-seed CKA reference should also be feasible using existing checkpoints. A matched inducing-point or learned-agent baseline would be particularly valuable for establishing novelty, although I recognize its greater experimental cost.

Overall, SSR is a coherent and potentially useful operator, but stronger evidence identifying the benefit of its distinctive organization mechanism is needed to establish a sufficiently compelling contribution. I would be receptive to targeted additional evidence during the authors' discussion.

Reviewer_6J3mOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Structured State Representation (SSR), a token-mixing operator that replaces direct pairwise attention with a two-stage organize-then-read mechanism. Source tokens are competitively aggregated into a compact set of learned states, which queries then access. Experiments on ACT and Diffusion Policy demonstrate competitive performance in robotic manipulation tasks, along with reduced memory usage and linear token-mixing complexity.

Critical strengths
  • The organize-then-read formulation is conceptually clear and provides an interesting alternative to direct pairwise attention.
  • SSR achieves linear token-mixing complexity for a fixed number of states, with up to 85.1% reduction in module-level inference memory.
  • The paper evaluates SSR across ACT and Diffusion Policy, includes comparisons with Transformer and Mamba, and provides several architectural ablations.
Critical weaknesses
  • SSR's two-stage aggregation mechanism resembles existing approaches such as Perceiver, Slot Attention, and Set Transformer. While the paper describes architectural differences, the lack of direct comparisons under matched settings makes the practical contribution of these differences unclear.
  • SSR improves performance on Transfer Cube but underperforms on Bimanual Insertion (49.6% for Transformer vs. 46.6% for SSR-P Center). Similar degradation is observed on Robomimic Square (61.44% vs. 53.16% for SSR-P Hybrid), raising concerns about its generalizability across manipulation tasks.
  • Figure 3 provides qualitative t-SNE/PCA visualizations suggesting more continuous representations, but does not directly examine what individual states capture or whether different states learn distinct information. This limits the evidence supporting the proposed structured organization mechanism.
  • Projected SSR slightly increases end-to-end FLOPs compared to Transformer in Diffusion Policy (17.288 vs. 17.167 GFLOPs). Moreover, memory improvements are measured at the module level, while end-to-end policy inference latency is not reported.
Questions for authors
  • How does SSR compare with Perceiver-style latent attention or Slot Attention under matched computational budgets?
  • What explains the performance degradation on Bimanual Insertion and Robomimic tasks? Could aggregation into a small state bank discard fine-grained information?
  • Can the authors provide state assignment visualizations or quantitative measures of state diversity to better validate the proposed organization mechanism?
  • What are the end-to-end inference latency and peak GPU memory of SSR-based policies compared to optimized Transformer baselines?

Comments on the paper

1 comment

Dear Reviewers

The author-reviewer discussion period will close soon. Please read and respond to authors if you have not done yet.

Thank you.

AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.