acceptodds
Under review as a conference paper at ICLR 2027

SELA-TTT: Semantically Modulated Test-Time Training for Vision-Language-Action Models

Abstract

Context enables Vision-Language-Action (VLA) models to use past interactions and demonstrations during task execution. Test-Time Training (TTT) represents this information through recurrent fast-weight updates, but conventional KV-binding updates leave implicit how each interaction's semantics should guide its contribution to context. We introduce Semantically Modulated Equivalent Linear Attention for Test-Time Training (SELA-TTT), which uses explicit semantic signals to relate these contributions to task structure. Our method modulates the queries, keys, and values in TTT's equivalent, time-evolving linear-attention formulation, conditioning both context writes and reads. The resulting fast state carries semantic information forward to subsequent decisions. We instantiate this interface with subtask advancement, overall goal progress, and atomic-skill optimality, capturing complementary aspects of task progress and execution validity. Evaluations across simulation benchmarks and real-world task suites show improved success over TTT baselines, with equivalent linear-attention modulation outperforming injection into input features or raw TTT projections. Controlled label perturbations further support the lasting influence of semantics through their correspondence with interaction history. Beyond stronger task performance, SELA-TTT enables semantic-aware in-context learning from imperfect human demonstrations and generalization to previously unseen failures.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Robot Learning, Test-Time Training, Semantic Modulation, In-Context Learning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_PuxmOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces SELA-TTT, a semantic-conditioning method for TTT-based robot policies. Using an existing linear-attention formulation of TTT, it applies FiLM to effective queries, keys, and residual values, directly conditioning fast-state writes and reads while retaining K–V binding updates for the hidden feature map. Three semantic signals are considered: subtask advancement, remaining goal progress, and atomic-skill execution validity. Experiments cover RoboMME, RMBench, and three real-world task suites, with injection-interface and read–write ablations, historical-label perturbations, imperfect-demonstration ICL, and unseen-failure tests. Semantic labels come from environment state in simulation and a human operator during real-world execution.

Critical strengths

1. Well-isolated interface contribution: Table 16 compares three injection locations with the same semantic information, and effective-QKV modulation consistently wins across all three signals. For subtask advancement, success rises from 32.00% with input-feature FiLM and 43.15% with raw-QKV FiLM to 55.96%. Table 18 further supports joint read–write conditioning (55.96% vs. 48.93% Q-only and 48.26% KV-only). These controls address the obvious concern that the gains come only from additional labels, rather than where and how they enter the policy.

2. Persistent semantic influence is tested directly: Table 1 changes historical labels while keeping replayed observations/actions fixed, then resumes from the same environment state with unchanged current feedback. The resulting performance drops support a lasting effect through the fast state, beyond immediate reactions to feedback. The label-exchange condition also tests whether semantics must correspond to the relevant interaction segments.

Critical weaknesses

1. Unequal feedback in the real-world generalization tests (main concern). In the switching test, negative feedback excludes an otherwise instruction-compatible object; in the continuation test, it changes the required behavior from completing repetitions to stopping early. SELA-TTT receives this information explicitly, while the standard RoboTTT baselines do not. Physical state restoration in the intervention baseline is not an equivalent information channel. Table 3 therefore confounds access to task-changing feedback with the proposed interface’s generalization ability. The same-label controls in Table 16 address this on RoboMME, but are not extended to these real-world tests.

2. Feedback requirements are insufficiently characterized. Environment-state labels in simulation and operator labels on hardware are explicitly disclosed, so their use is not itself a methodological flaw. However, the paper does not quantify operator effort or establish tolerance to delayed, incorrect, or missing feedback. The historical-label perturbations show that semantic correspondence matters; they do not establish robustness to realistic annotation errors. This limits the practical interpretation of the real-world gains.

Questions for authors
  1. Can the input-feature and raw-QKV FiLM controls be evaluated on the real-world ICL and unseen-failure tests with identical annotations and online feedback? This would directly test whether the interface advantage extends to the paper’s headline capabilities.

  2. Clarify the online annotation protocol: how are labels aligned with action chunks, does execution wait for operator input, and how frequently does the operator change labels?

  3. Are delayed/noisy-feedback results available, particularly against the same-label injection controls? If not, state explicitly that the demonstrated benefits assume reliably supplied semantic feedback.

5 comments
AuthorsOpenReview ↗

Author response to Reviewer Puxm - Part 1

Thank you for your positive assessment of our contribution and your constructive feedback. We address your concerns in detail below and hope these responses clarify the concerns you raised.

Weakness 1. "Table 3 therefore confounds access to task-changing feedback with the proposed interface’s generalization ability. The same-label controls in Table 16 address this on RoboMME, but are not extended to these real-world tests."

Question 1. "Can the input-feature and raw-QKV FiLM controls be evaluated on the real-world ICL and unseen-failure tests with identical annotations and online feedback? This would directly test whether the interface advantage extends to the paper’s headline capabilities."

Response 1. Thank you for highlighting the need for identical semantic feedback. We have added same-feedback comparisons on semantic-aware ICL in Chemistry and unseen switching in Repeated Picking. The results below show higher success for SELA-TTT than the alternative semantic conditioning methods in both settings.

Each condition uses 20 episodes, with matched correction data and post-training budgets. The updated Appendix E.3.1 describes the baseline implementations and training settings.

For semantic-aware ICL, all three modulation methods receive the same imperfect demonstrations, semantic annotations, and online feedback. An error episode reproduces undesired demonstrated behavior and is counted once, following Appendix F.3.4.

MethodChemistry ICL success (%)Chemistry ICL error episodes
Input-feature modulation30.011
Raw-QKV modulation50.05
SELA-TTT65.04

These results are integrated into the updated Section 5.3.2, Table 3, with the protocol in Appendix F.3.1. The human demonstration and ICL executions with all three methods are available at the anonymous video page.

For unseen switching, the semantic-token baseline, Input-feature modulation, Raw-QKV modulation, and SELA-TTT receive the same negative feedback excluding the target object. All four methods therefore have access to the changed task requirement. The token baseline retains RoboTTT's history mechanism and receives semantic inputs during both training and inference.

MethodUnseen switching success (%), 20 episodes
RoboTTT + DAgger, without semantic input0.0
RoboTTT + DAgger + semantic tokens25.0
Input-feature modulation20.0
Raw-QKV modulation35.0
SELA-TTT50.0

The unseen-switching results are included in the updated Section 5.3.3, Table 5, with the protocol in Appendix F.3.3. Executions with Input-feature modulation, Raw-QKV modulation, and SELA-TTT can be seen at the anonymous video page.


Weakness 2. "the paper does not quantify operator effort"

Question 2. "Clarify the online annotation protocol: how are labels aligned with action chunks, does execution wait for operator input, and how frequently does the operator change labels?"

Response 2. Thank you for asking us to clarify the annotation protocol and operator effort.

  1. Labels are aligned with the executed interaction in each action chunk. At the next policy decision, TTT writes the executed interaction and its associated label into context, then reads the updated state to generate the next chunk. The label therefore affects subsequent predictions.
  2. Execution does not wait for operator confirmation at each boundary. The interface records operator label updates while the robot executes the current action chunk and makes them available for the next policy decision.
  3. The operator changes the label when the semantic state changes, at an estimated frequency of once per 7.7 chunks. The label is retained between updates.

The updated Appendix F.3 includes this alignment, nonblocking input protocol, and update frequency.

AuthorsOpenReview ↗

Author response to Reviewer Puxm - Part 2

Weakness 2. "the paper does not quantify operator effort or establish tolerance to delayed, incorrect, or missing feedback."

Question 3. "Are delayed/noisy-feedback results available, particularly against the same-label injection controls?"

Response 3. Thank you for requesting results against the same-label controls. Yes. We have added real-world experiments with incorrect, delayed, and missing feedback at three levels each. As shown below, SELA-TTT achieves higher success than Input-feature modulation and Raw-QKV modulation at every tested level, supporting robustness under imperfect feedback.

We apply perturbations when the operator updates a label. The three perturbation types are:

  • An incorrect update supplies a randomly selected incorrect label.
  • A delayed update is postponed by the specified number of chunks.
  • A missing update is omitted.

Input-feature modulation, Raw-QKV modulation, and SELA-TTT use the same perturbation protocol, with 20 episodes per method and level. The updated Appendix F.3.6 details the protocol.

Perturbation and suiteLevels, in column orderInput-feature modulation (%)Raw-QKV modulation (%)SELA-TTT (%)
Incorrect updates, Repeated Picking10%, 30%, 50%20 / 5 / 045 / 35 / 2565 / 50 / 40
Delayed updates, Repeated Swinging2, 4, 6 chunks15 / 10 / 035 / 20 / 1555 / 40 / 25
Missing updates, Chemistry10%, 30%, 50%25 / 10 / 025 / 15 / 545 / 30 / 25

The full results are reported in the updated Section 5.3.4, Table 6.

We appreciate the authors’ careful responses. TTT-based robot policies such as RoboTTT compress past interactions into fast weights and use them as context. However, semantically distinct actions are accumulated without distinction. This paper proposes applying semantic modulation to the effective features revealed by expressing KV-binding TTT as an equivalent linear-attention formulation, and supports the effectiveness of this injection point primarily through simulation experiments. RoboTTT updates its fast weights using a self-supervised loss that does not require external semantic labels, whereas this method’s semantic modulation relies on them. Its claims regarding ICL and handling unseen failures are supported only by experiments using human-provided labels, including a small-scale real-world evaluation. This reliance on labeled feedback limits the strength of the evidence and makes it difficult to justify a Strong Accept recommendation.

Reviewer_QwRGOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes SELA-TTT, a method for injecting explicit semantic signals into the Test-Time Training (TTT) layers of a Vision-Language-Action (VLA) policy. Building on the result that KV-binding TTT is equivalent to a time-evolving linear attention (Liu et al., 2026a), the authors identify the effective queries, keys, and values that directly govern fast-state writes and readouts, and apply FiLM modulation to these features conditioned on a semantic signal. Three signals are instantiated: subtask advancement (deviation, correction, or progress), overall goal progress (number of remaining subtasks), and atomic-skill optimality (a signed one-hot over primitives). The method is integrated into the action expert of π0.5 following the RoboTTT design and trained on DAgger-style failure-to-correction trajectories annotated with these signals.

Validation covers RoboMME (16 tasks), RMBench (9 tasks), and three real-world dual-arm task suites. Under matched data and update budgets, SELA-TTT with subtask advancement outperforms RoboTTT trained on the same correction data (55.96% vs. 43.56% on RoboMME, 80.89% vs. 65.67% on RMBench, and 62.5% vs. 40.0% average in the real world). Ablations show that modulating the effective QKV outperforms modulating input features or raw QKV, that joint read-write modulation outperforms Q-only or KV-only variants, and that replaying a fixed history with altered past labels substantially degrades success, indicating that semantics persist through the fast state.

Critical strengths
  • SELA-TTT and RoboTTT + DAgger share the same failure-to-correction data and update budget, and clearly demonstrate that the effective QKV consistently performs best when the same semantic signal is fed into the input feature, raw QKV, and effective QKV, respectively, in interface ablation.
  • The paper verified that linear-attention reconstruction is bitwise identical to the original fast-MLP forward, and label-replay experiments show that semantic effects persist through the fast state rather than through immediate feedback. This provides useful insights into how TTT actually operates within the VLA policy.
Critical weaknesses
  • The signals are computed from simulator state and expert intervention records, and in real-world experiments are supplied online by a human operator, whereas the signal-free baselines receive no comparable information in the unseen-failure tests, the operator’s negative signal effectively acts as an online correction command. Because no learned or estimated signal is evaluated, it is unclear how much of the reported gain would remain in a deployable setting, which limits the practical significance of the claims.
  • The label-replay experiment shows that relabeling a single past interval can collapse success (e.g., SwingXtimes from 78% to 14%), implying high sensitivity to annotation errors that are inevitable with human or learned labelers. Without experiments on noisy, delayed, or estimated signals, the method’s reliability under realistic supervision remains an open and consequential question.
  • No standard deviations or confidence intervals are reported for any result, the real-world ICL and unseen-failure tests use only 20 episodes per condition (roughly ±20 pp 95% intervals at 50% success), and most appendix ablations use 50 episodes per task. In addition, external baselines (e.g., MemER, FrameSamp+Modul, MemoryWAM) are taken from their reported evaluations without the DAgger correction data, so the gains over them mix the effect of the method with the effect of the data.
Questions for authors
  • Can results be presented using semantic signals generated by a trained estimator (e.g., VLM) or controlled label noise (e.g., random flips of varying rates) instead of direct human evaluation? If performance degradation is gradual, the evaluation of practicality is likely to improve significantly.

  • Under data-fitting conditions, the input-feature FiLM using semantic signals (32.00%) is more than 11 percentage points lower than RoboTTT + DAgger without signals (43.56%). Why did performance drop despite the addition of information, and were equivalent initialization and tuning (e.g., zero-init FiLM heads) applied to the alternative interfaces?

  • RoboTTT + DAgger lacks a simple baseline that provides the same signals as text or token inputs. This will clarify whether modulating TTT recurrence is necessary, and it appears to be particularly important in unseen-failure tests where the current baseline receives no signals.

4 comments
AuthorsOpenReview ↗

Notification Regarding the “Critical Strengths” Section

Thank you for your time and thoughtful feedback. We noticed that the two paragraphs under “Critical Strengths” also appear under “Critical Weaknesses,” possibly due to a minor error. Could you please check whether the intended strengths were recorded correctly?

Thanks again and we have carefully considered your concerns and provide point-by-point responses below.

AuthorsOpenReview ↗

Author response to Reviewer QwRG - Part 1

Weakness 1. "Because no learned or estimated signal is evaluated, it is unclear how much of the reported gain would remain in a deployable setting, which limits the practical significance of the claims."

Question 1. "Can results be presented using semantic signals generated by a trained estimator (e.g., VLM) or controlled label noise (e.g., random flips of varying rates) instead of direct human evaluation?"

Response 1. Thank you for suggesting a trained evaluator. We have added a Chemistry evaluation using a learned history-based evaluator. Replacing online human labels yields 50.0% success, preserving a 12.5-percentage-point advantage over RoboTTT + DAgger.

The evaluator uses six transformer layers and a three-class prediction head for subtask advancement (+1), deviation (-1), and correction (0). Its input configuration is aligned with the real-world policy. Agreement with human reference labels is 89.59% at chunk boundaries. The updated Appendix F.3.5 describes the model and label evaluation.

MethodFeedback sourceChemistry success % [95% CI], 40 episodes
RoboTTT + DAggerNo semantic feedback37.5 [24.2, 53.0]
SELA-TTTHuman operator55.0 [39.8, 69.3]
SELA-TTTLearned history evaluator50.0 [35.2, 64.8]

The online results appear in the updated Section 5.3.1, Table 2. Human annotations and evaluator predictions on the same recorded interaction can be seen side by side on the anonymous video page.


Weakness 2. "Without experiments on noisy, delayed, or estimated signals, the method’s reliability under realistic supervision remains an open and consequential question."

Question 1. "Can results be presented using semantic signals generated by a trained estimator (e.g., VLM) or controlled label noise (e.g., random flips of varying rates) instead of direct human evaluation? If performance degradation is gradual, the evaluation of practicality is likely to improve significantly."

Response 2. Thank you for suggesting controlled label noise. We have conducted additional real-world experiments at three levels of incorrect, delayed, and missing feedback. The results below show gradual degradation across the tested levels, with SELA-TTT maintaining higher success than both same-label modulation baselines.

We apply perturbations when the operator updates a label. The three perturbation types are:

  • An incorrect update supplies a randomly selected incorrect label.
  • A delayed update is postponed by the specified number of chunks.
  • A missing update is omitted.

Input-feature modulation, Raw-QKV modulation, and SELA-TTT use the same perturbation protocol, with 20 episodes per method and level. The updated Appendix F.3.6 details the protocol.

Perturbation and suiteLevels, in column orderInput-feature modulation (%)Raw-QKV modulation (%)SELA-TTT (%)
Incorrect updates, Repeated Picking10%, 30%, 50%20 / 5 / 045 / 35 / 2565 / 50 / 40
Delayed updates, Repeated Swinging2, 4, 6 chunks15 / 10 / 035 / 20 / 1555 / 40 / 25
Missing updates, Chemistry10%, 30%, 50%25 / 10 / 025 / 15 / 545 / 30 / 25

The full results are reported in the updated Section 5.3.4, Table 6.


Question 2. "Under data-fitting conditions, the input-feature FiLM using semantic signals (32.00%) is more than 11 percentage points lower than RoboTTT + DAgger without signals (43.56%). Why did performance drop despite the addition of information, and were equivalent initialization and tuning (e.g., zero-init FiLM heads) applied to the alternative interfaces?"

Response 3. Thank you for asking about the performance drop and initialization. We used identical initialization and optimization settings across the compared modulation variants, including zero-initialized modulation output heads that preserve the unconditioned features at initialization. The starting policy, correction data, supervision masks, and post-training budget were also shared. These settings are detailed in Appendix C, Appendix E.1.1, and Table 20.

The benefit of additional semantic information depends on how it is incorporated into the TTT computation. Input-feature modulation changes action-relevant representations before QKV projection, normalization, and the evolving fast-model transformation. These changes can disrupt useful features without producing a corresponding improvement in context writes and reads. SELA-TTT instead applies semantic modulation directly to the effective queries, keys, and values governing these computations. See Table 19 for the matched results and Appendix E.1.1 for the modulation definitions.

AuthorsOpenReview ↗

Author response to Reviewer QwRG - Part 2

Weakness 1. "the signal-free baselines receive no comparable information in the unseen-failure tests"

Question 3. "RoboTTT + DAgger lacks a simple baseline that provides the same signals as text or token inputs. This will clarify whether modulating TTT recurrence is necessary, and it appears to be particularly important in unseen-failure tests where the current baseline receives no signals."

Response 4. Thank you for suggesting a token-input baseline. We have added this baseline on standard Chemistry and unseen switching in Repeated Picking. The token baseline shows that direct semantic input can support feedback-guided switching. Modulating TTT provides an additional performance benefit, with SELA-TTT reaching 50.0% success compared with 25.0% for semantic tokens. On Chemistry, direct input improves success by 5.0 percentage points, and SELA-TTT adds a further 12.5 points.

The baseline retains RoboTTT's register tokens and TTT history mechanism and adds 16 semantic tokens accessible through attention. Semantic conditioning is applied through attention to these tokens, while TTT processes the original register and action tokens. Both methods receive the same semantic signals during training and inference, with matched access to interaction history and past semantic feedback. Training data, initialization, supervision masks, and training budgets are matched, with an added parameter budget comparable to SELA-TTT's modulation modules. The implementation is described in the updated Appendix E.3.1.

MethodChemistry success (%), 40 episodesSwitching success (%), 20 episodes
RoboTTT + DAgger37.50.0
RoboTTT + DAgger + semantic tokens42.525.0
SELA-TTT55.050.0

Both evaluations are included in the updated Sections 5.3.1 and 5.3.3, Tables 2 and 5. Standard Chemistry execution examples are available at the anonymous video page.


Weakness 3. "No standard deviations or confidence intervals are reported for any result, the real-world ICL and unseen-failure tests use only 20 episodes per condition (roughly ±20 pp 95% intervals at 50% success), and most appendix ablations use 50 episodes per task."

Response 5. Thank you for requesting uncertainty estimates. We have added 95% Wilson confidence intervals alongside the success rates, calculated from the success counts and evaluation episode counts. The updated Tables 26, 27, and 30 include intervals for the RoboMME and RMBench results.

The real-world intervals appear for all methods directly in Tables 2, 3, 4, 5, and 6, including the original ICL and unseen-failure evaluations with 20 episodes per condition. The calculation is specified in the updated Appendix F.3.4.


Weakness 3. "In addition, external baselines (e.g., MemER, FrameSamp+Modul, MemoryWAM) are taken from their reported evaluations without the DAgger correction data, so the gains over them mix the effect of the method with the effect of the data."

Response 6. Thank you for raising the effect of correction data. We have additionally post-trained FrameSamp+Modul with the same DAgger correction data and a matched post-training budget. SELA-TTT retains a 13.08-percentage-point advantage in this comparison.

Evaluation covers all 16 RoboMME tasks, with 450 rollouts per task. The updated Section 5.1 includes the baseline and aggregate result in Figure 3. Training details appear in Appendix E.1, with per-task results alongside the other RoboMME results in Table 28.

MethodRoboMME average success (%)
FrameSamp+Modul + DAgger42.88
RoboTTT + DAgger43.56
SELA-TTT55.96
Reviewer_QwRGOpenReview ↗

Thank you for your response.

Thank you for the detailed response. The learned-evaluator study, the token-input baseline, the matched FrameSamp+Modul comparison, and the confidence intervals address several of my concerns and make the paper clearer. I appreciate the care taken in responding to each point. One question remains.

  • In Table 6, the comparison includes only baselines that receive the same labels. Against the signal-free RoboTTT + DAgger results in Figure 6, SELA-TTT appears to match or fall below that baseline at the moderate and high perturbation levels. Could you add this reference to Table 6 and state where the signal stops helping, including where the learned evaluator’s error rate and latency fall?
Reviewer_tZUtOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes SELA-TTT, which uses externally supplied task-progress and execution-validity signals to modulate the writing and reading of fast-weight memory in VLA models through TTT’s equivalent linear-attention formulation. Experiments demonstrate consistent success-rate improvements over the evaluated TTT baselines across RoboMME, RMBench, and real-world manipulation tasks, while injection-interface comparisons on RoboMME show that effective-QKV modulation outperforms input-feature and raw-QKV modulation.

Critical strengths
  1. Semantic modulation of fast-weight memory with consistent empirical gains. The paper proposes a concrete mechanism for using externally supplied task semantics to modulate the writing and reading of episode-level fast-weight memory. It demonstrates consistent improvements over the evaluated TTT baselines across two simulation benchmarks and real-world task suites.

  2. Controlled comparisons of semantic injection interfaces. The paper compares input-feature, raw-QKV, and effective-QKV modulation using the same semantic information. Effective-QKV modulation performs best across all three semantic signal types on RoboMME, providing empirical support for the chosen injection interface.

Critical weaknesses
  1. Insufficient isolation of information advantage and architectural contribution. SELA-TTT benefits from externally provided task-progress and execution-validity signals that are unavailable to the RoboTTT baselines. Although the paper shows that effective-QKV modulation outperforms alternative injection points within TTT under the same semantic signals, it lacks a baseline that directly conditions the VLA policy on the same information. Consequently, it remains unclear how much of the reported performance gain is attributable to privileged semantic feedback rather than the proposed fast-weight modulation mechanism.

  2. Reliance on privileged semantic feedback limits practical applicability. The three semantic signals—subtask advancement, overall goal progress, and atomic-skill optimality—provide critical information about task progress and execution validity that directly informs decision-making. However, SELA-TTT does not infer these signals itself. Instead, they are derived from privileged environment information in simulation or supplied by human operators during real-world execution. Reliably obtaining such feedback in autonomous deployment would require a capable external evaluator or continuous human supervision, substantially limiting the method's practical applicability. The paper also lacks evaluation under imperfect or missing semantic feedback.

Questions for authors
  1. Can the authors compare SELA-TTT against a VLA baseline that receives the same semantic signals as direct policy inputs, rather than through the proposed fast-weight modulation, during both training and inference? The comparison should use matched training data and comparable access to interaction history and past semantic feedback. This would help determine how much of the reported gain is achievable through direct semantic conditioning and what additional benefit the proposed modulation provides.

  2. How robust is SELA-TTT to noisy, delayed, or missing semantic signals? Does its performance advantage persist when feedback is provided by an imperfect evaluator rather than an oracle or human operator?

3 comments
AuthorsOpenReview ↗

Author response to Reviewer tZUt - Part 1

Thank you for recognizing the strengths of our work and for your thoughtful suggestions. We have carefully considered your feedback and provide point-by-point responses below.

Weakness 1. "Consequently, it remains unclear how much of the reported performance gain is attributable to privileged semantic feedback rather than the proposed fast-weight modulation mechanism."

Question 1. "Can the authors compare SELA-TTT against a VLA baseline that receives the same semantic signals as direct policy inputs, rather than through the proposed fast-weight modulation, during both training and inference? The comparison should use matched training data and comparable access to interaction history and past semantic feedback. This would help determine how much of the reported gain is achievable through direct semantic conditioning and what additional benefit the proposed modulation provides."

Response 1. Thank you for raising the distinction between semantic information and fast-weight modulation. We have added a direct semantic-input baseline with matched training data and history access. On Chemistry, direct semantic input improves success by 5.0 percentage points over RoboTTT + DAgger, and SELA-TTT provides a further 12.5-point improvement. On unseen switching, the corresponding improvements are 25.0 and 25.0 points.

The baseline retains RoboTTT's register tokens and TTT history mechanism and adds 16 semantic tokens accessible through attention. Semantic conditioning is applied through attention to these tokens, while TTT processes the original register and action tokens. Both methods receive the same semantic signals during training and inference, with matched access to interaction history and past semantic feedback. Training data, initialization, supervision masks, and training budgets are matched, with an added parameter budget comparable to SELA-TTT's modulation modules. The implementation is described in the updated Appendix E.3.1.

MethodChemistry: success % [95% CI], 40 episodesUnseen switching: success % [95% CI], 20 episodes
RoboTTT + DAgger37.5 [24.2, 53.0]0.0 [0.0, 16.1]
RoboTTT + DAgger + semantic tokens42.5 [28.5, 57.8]25.0 [11.2, 46.9]
SELA-TTT55.0 [39.8, 69.3]50.0 [29.9, 70.1]

These results are included in the updated Sections 5.3.1 and 5.3.3, Tables 2 and 5. Standard Chemistry executions by the token baseline and SELA-TTT can be seen at the anonymous video page.


Weakness 2. "The paper also lacks evaluation under imperfect or missing semantic feedback."

Question 2. "How robust is SELA-TTT to noisy, delayed, or missing semantic signals?"

Response 2. Thank you for raising the concern about feedback robustness. We have added real-world experiments with incorrect, delayed, and missing label updates. As shown in the table below, SELA-TTT achieves higher success than both same-label modulation baselines at every tested level, supporting its robustness under imperfect semantic feedback.

We apply perturbations when the operator updates a label. The three perturbation types are:

  • An incorrect update supplies a randomly selected incorrect label.
  • A delayed update is postponed by the specified number of chunks.
  • A missing update is omitted.

Input-feature modulation, Raw-QKV modulation, and SELA-TTT use the same perturbation protocol, with 20 episodes per method and level. The updated Appendix F.3.6 details the protocol.

Perturbation and suiteLevels, in column orderInput-feature modulation (%)Raw-QKV modulation (%)SELA-TTT (%)
Incorrect updates, Repeated Picking10%, 30%, 50%20 / 5 / 045 / 35 / 2565 / 50 / 40
Delayed updates, Repeated Swinging2, 4, 6 chunks15 / 10 / 035 / 20 / 1555 / 40 / 25
Missing updates, Chemistry10%, 30%, 50%25 / 10 / 025 / 15 / 545 / 30 / 25

The full results are reported in the updated Section 5.3.4, Table 6.

AuthorsOpenReview ↗

Author response to Reviewer tZUt - Part 2

Weakness 2. "Reliably obtaining such feedback in autonomous deployment would require a capable external evaluator or continuous human supervision, substantially limiting the method's practical applicability."

Question 2. "Does its performance advantage persist when feedback is provided by an imperfect evaluator rather than an oracle or human operator?"

Response 3. Thank you for raising the dependence on external feedback. Yes. We have added an evaluation in which a learned history-based evaluator replaces online human semantic labels. SELA-TTT achieves 50.0% success on Chemistry, retaining a 12.5-percentage-point advantage over RoboTTT + DAgger.

The evaluator uses six transformer layers and a three-class prediction head for subtask advancement (+1), deviation (-1), and correction (0). Its input configuration is aligned with the real-world policy. Agreement with human reference labels is 89.59% at chunk boundaries. The updated Appendix F.3.5 describes the model and label evaluation.

MethodFeedback sourceChemistry success % [95% CI], 40 episodes
RoboTTT + DAggerNo semantic feedback37.5 [24.2, 53.0]
SELA-TTTHuman operator55.0 [39.8, 69.3]
SELA-TTTLearned history evaluator50.0 [35.2, 64.8]

The online results appear in the updated Section 5.3.1, Table 2. Human annotations and evaluator predictions on the same recorded interaction can be seen side by side on the anonymous video page.

Reviewer_tZUtOpenReview ↗

Rebuttal Response and Score Update

I thank the authors for their constructive response and additional experiments. The new direct semantic-conditioning baseline, learned-evaluator experiment, and imperfect-feedback evaluations substantively address my main concerns. Some reservations remain about the limited coverage of the new experiments and insufficient detail on the learned evaluator’s training and evaluation protocols. I am therefore raising my score to Weak Accept.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Summary of updates

We sincerely thank the AC and reviewers for their time and thoughtful feedback. Below we summarize how our responses and revisions address the main concerns, followed by the strengths highlighted in the reviews. Revisions are marked in blue, with the principal additional results in the main text.

We address the main concerns in six areas:

  1. Distinguishing semantic information gains from modulation gains. (tZUt W1/Q1, QwRG W1/Q3) We add a semantic-token baseline with the same semantic signals during training and inference, matched data, history access, and training budgets. SELA-TTT outperforms it on standard Chemistry, unseen physical recovery, and unseen switching, demonstrating benefits beyond direct semantic conditioning. Results appear in Tables 2 and 5, with implementation details in Appendix E.3.1.

  2. Establishing the modulation advantage on real hardware. (Puxm W1/Q1 and subsequent concern) We extend Input-feature and Raw-QKV comparisons to all three standard real-world suites, semantic-aware ICL, unseen physical recovery, and unseen switching. SELA-TTT achieves higher success across these settings under the same feedback, supporting the proposed modulation location in standard execution and the evaluated capabilities. Results appear in Figure 6 and Tables 4 and 5.

  3. Replacing online human feedback with a learned evaluator. (tZUt W2/Q2, QwRG W1/Q1, and Puxm’s subsequent concern) We evaluate learned feedback on standard Chemistry and Picking, semantic-aware ICL, and unseen physical recovery. It retains much of the performance achieved with human feedback and improves success over signal-free RoboTTT + DAgger across these settings. Results appear in Section 5.3.2 and Table 3. Appendix F.3.5 describes the evaluator and reports label agreement and inference time.

  4. Characterizing robustness and feedback requirements. (tZUt W2/Q2, QwRG W2/Q1, Puxm W2/Q2/Q3, and subsequent concerns) We test incorrect, delayed, and missing updates at three levels each. SELA-TTT outperforms both same-label modulation baselines at every tested level. Table 6 also includes signal-free RoboTTT + DAgger and identifies where delayed or missing feedback no longer improves success. We clarify chunk alignment, nonblocking input, and the estimated update frequency of once per 7.7 chunks. Results and protocols appear in Section 5.3.5 and Appendix F.3.

  5. Matching DAgger correction data for the memory baseline. (QwRG W3/Q2) We post-train FrameSamp+Modul with matched correction data and training budgets, then evaluate all 16 RoboMME tasks. SELA-TTT retains its advantage when this alternative memory architecture receives the correction data. Results appear in Figure 3 and Table 28. We also clarify matched initialization and optimization settings and explain why Input-feature modulation does not provide the same benefit as effective-QKV modulation.

  6. Expanding evaluation and reporting uncertainty. (QwRG W3 and Puxm’s subsequent concern) We expand standard real-world and capability evaluations to 50 episodes per condition. We add 95% Wilson confidence intervals alongside real-world success rates and SELA-TTT’s per-task simulation results. Updated evaluations appear in Figure 6 and Tables 2–5, with interval calculation in Appendix F.3.4.

The reviewers highlighted strengths at three levels:

  1. Insight: distinguishing the contributions of past interactions through semantics. (tZUt S1, Puxm S2 and subsequent assessment) The reviewers recognized the use of task semantics to guide contextual contributions. Reviewer Puxm also identified the motivation that conventional TTT accumulates semantically distinct interactions without explicitly distinguishing them.

  2. Formulation and model: semantic modulation of context writes and reads. (tZUt S1/S2, Puxm S1) The reviewers recognized the concrete mechanism through TTT’s equivalent linear-attention features and the evidence supporting the chosen modulation location. Reviewer Puxm also noted the benefit of jointly conditioning reads and writes.

  3. Experiments: consistent gains and controlled validation. (tZUt S1/S2, Puxm S1/S2) The reviewers highlighted gains across two simulation benchmarks and real-world suites, controlled same-signal comparisons, and evidence of persistent semantic influence through the fast state and its correspondence with recorded interactions.

We hope these additional experiments and clarifications help the AC and reviewers assess the contribution and practical applicability of SELA-TTT.

Area_Chair_gbMLOpenReview ↗

Please review the authors’ rebuttal and update your assessment

Dear reviewers,

Thank you for your reviews, and thank you to the authors for their responses. Please read the authors’ rebuttal and discuss whether it addresses your main concerns. Please update your review and rating if warranted, and briefly explain your current assessment, including any concerns that remain unresolved. If your assessment is unchanged, please explain why.

Thank you for your continued engagement.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.