- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces SELA-TTT, a semantic-conditioning method for TTT-based robot policies. Using an existing linear-attention formulation of TTT, it applies FiLM to effective queries, keys, and residual values, directly conditioning fast-state writes and reads while retaining K–V binding updates for the hidden feature map. Three semantic signals are considered: subtask advancement, remaining goal progress, and atomic-skill execution validity. Experiments cover RoboMME, RMBench, and three real-world task suites, with injection-interface and read–write ablations, historical-label perturbations, imperfect-demonstration ICL, and unseen-failure tests. Semantic labels come from environment state in simulation and a human operator during real-world execution.
Critical strengths
1. Well-isolated interface contribution: Table 16 compares three injection locations with the same semantic information, and effective-QKV modulation consistently wins across all three signals. For subtask advancement, success rises from 32.00% with input-feature FiLM and 43.15% with raw-QKV FiLM to 55.96%. Table 18 further supports joint read–write conditioning (55.96% vs. 48.93% Q-only and 48.26% KV-only). These controls address the obvious concern that the gains come only from additional labels, rather than where and how they enter the policy.
2. Persistent semantic influence is tested directly: Table 1 changes historical labels while keeping replayed observations/actions fixed, then resumes from the same environment state with unchanged current feedback. The resulting performance drops support a lasting effect through the fast state, beyond immediate reactions to feedback. The label-exchange condition also tests whether semantics must correspond to the relevant interaction segments.
Critical weaknesses
1. Unequal feedback in the real-world generalization tests (main concern). In the switching test, negative feedback excludes an otherwise instruction-compatible object; in the continuation test, it changes the required behavior from completing repetitions to stopping early. SELA-TTT receives this information explicitly, while the standard RoboTTT baselines do not. Physical state restoration in the intervention baseline is not an equivalent information channel. Table 3 therefore confounds access to task-changing feedback with the proposed interface’s generalization ability. The same-label controls in Table 16 address this on RoboMME, but are not extended to these real-world tests.
2. Feedback requirements are insufficiently characterized. Environment-state labels in simulation and operator labels on hardware are explicitly disclosed, so their use is not itself a methodological flaw. However, the paper does not quantify operator effort or establish tolerance to delayed, incorrect, or missing feedback. The historical-label perturbations show that semantic correspondence matters; they do not establish robustness to realistic annotation errors. This limits the practical interpretation of the real-world gains.
Questions for authors
-
Can the input-feature and raw-QKV FiLM controls be evaluated on the real-world ICL and unseen-failure tests with identical annotations and online feedback? This would directly test whether the interface advantage extends to the paper’s headline capabilities.
-
Clarify the online annotation protocol: how are labels aligned with action chunks, does execution wait for operator input, and how frequently does the operator change labels?
-
Are delayed/noisy-feedback results available, particularly against the same-label injection controls? If not, state explicitly that the demonstrated benefits assume reliably supplied semantic feedback.
5 comments
Author response to Reviewer Puxm - Part 2
Weakness 2. "the paper does not quantify operator effort or establish tolerance to delayed, incorrect, or missing feedback."
Question 3. "Are delayed/noisy-feedback results available, particularly against the same-label injection controls?"
Response 3. Thank you for requesting results against the same-label controls. Yes. We have added real-world experiments with incorrect, delayed, and missing feedback at three levels each. As shown below, SELA-TTT achieves higher success than Input-feature modulation and Raw-QKV modulation at every tested level, supporting robustness under imperfect feedback.
We apply perturbations when the operator updates a label. The three perturbation types are:
- An incorrect update supplies a randomly selected incorrect label.
- A delayed update is postponed by the specified number of chunks.
- A missing update is omitted.
Input-feature modulation, Raw-QKV modulation, and SELA-TTT use the same perturbation protocol, with 20 episodes per method and level. The updated Appendix F.3.6 details the protocol.
| Perturbation and suite | Levels, in column order | Input-feature modulation (%) | Raw-QKV modulation (%) | SELA-TTT (%) |
|---|---|---|---|---|
| Incorrect updates, Repeated Picking | 10%, 30%, 50% | 20 / 5 / 0 | 45 / 35 / 25 | 65 / 50 / 40 |
| Delayed updates, Repeated Swinging | 2, 4, 6 chunks | 15 / 10 / 0 | 35 / 20 / 15 | 55 / 40 / 25 |
| Missing updates, Chemistry | 10%, 30%, 50% | 25 / 10 / 0 | 25 / 15 / 5 | 45 / 30 / 25 |
The full results are reported in the updated Section 5.3.4, Table 6.
We appreciate the authors’ careful responses. TTT-based robot policies such as RoboTTT compress past interactions into fast weights and use them as context. However, semantically distinct actions are accumulated without distinction. This paper proposes applying semantic modulation to the effective features revealed by expressing KV-binding TTT as an equivalent linear-attention formulation, and supports the effectiveness of this injection point primarily through simulation experiments. RoboTTT updates its fast weights using a self-supervised loss that does not require external semantic labels, whereas this method’s semantic modulation relies on them. Its claims regarding ICL and handling unseen failures are supported only by experiments using human-provided labels, including a small-scale real-world evaluation. This reliance on labeled feedback limits the strength of the evidence and makes it difficult to justify a Strong Accept recommendation.
Author response to Reviewer Puxm - Part 1
Thank you for your positive assessment of our contribution and your constructive feedback. We address your concerns in detail below and hope these responses clarify the concerns you raised.
Response 1. Thank you for highlighting the need for identical semantic feedback. We have added same-feedback comparisons on semantic-aware ICL in Chemistry and unseen switching in Repeated Picking. The results below show higher success for SELA-TTT than the alternative semantic conditioning methods in both settings.
Each condition uses 20 episodes, with matched correction data and post-training budgets. The updated Appendix E.3.1 describes the baseline implementations and training settings.
For semantic-aware ICL, all three modulation methods receive the same imperfect demonstrations, semantic annotations, and online feedback. An error episode reproduces undesired demonstrated behavior and is counted once, following Appendix F.3.4.
These results are integrated into the updated Section 5.3.2, Table 3, with the protocol in Appendix F.3.1. The human demonstration and ICL executions with all three methods are available at the anonymous video page.
For unseen switching, the semantic-token baseline, Input-feature modulation, Raw-QKV modulation, and SELA-TTT receive the same negative feedback excluding the target object. All four methods therefore have access to the changed task requirement. The token baseline retains RoboTTT's history mechanism and receives semantic inputs during both training and inference.
The unseen-switching results are included in the updated Section 5.3.3, Table 5, with the protocol in Appendix F.3.3. Executions with Input-feature modulation, Raw-QKV modulation, and SELA-TTT can be seen at the anonymous video page.
Response 2. Thank you for asking us to clarify the annotation protocol and operator effort.
The updated Appendix F.3 includes this alignment, nonblocking input protocol, and update frequency.