acceptodds
Under review as a conference paper at ICLR 2027

Temporal Contrastive Test-Time Training for Robotic Manipulation

Abstract

Robot memory faces two directions in time: the past determines what can be known now, while the future reveals what was worth knowing. Test-time training (TTT) gives this asymmetry an optimization form: the inner objective determines what writes into memory, while outer task losses determine which updates are useful. However, common TTT objectives derive self-supervision primarily from the current observation, providing no direct objective for modeling relations across observed history. We introduce Temporal Contrastive Test-Time Training (TC–TTT), a contrastive inner objective that augments reconstruction by distinguishing the current representation from historical alternatives conditioned on an earlier observation. We further study how to select historical observations for temporal comparison, examining how sampling stride and candidate count affect policy performance. In a simplified model, TC–TTT adds a velocity-like term to the reconstruction gradient, making first-order temporal change explicit. Across nine RMBench tasks, TC–TTT improves mean success from 31.8% to 51.8% over reconstruction-based TTT under the same 77M-parameter Diffusion Policy and action-gradient window. These results identify the inner objective as an important design axis for online robot memory and show that historical contrast can substantially improve control without extending the action-gradient horizon.

Primary area:
applications to robotics, autonomy, planning
Keywords:
robot learning, test-time training, contrastive learning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_rhQPOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

TC-TTT adds a second term to the inner loss of a TTT memory: besides reconstruction, the key from s steps back must retrieve the current value against K earlier values at the same stride. The analysis collapses every key and query onto one unit vector, leaving the new term as the gap between the current feature and its history mean, read as a derivative at small s. On nine RMBench tasks with per-task 77M Diffusion Policies, mean success goes 31.8 to 51.8 at a one-update gradient window, above reconstruction-only TTT at G = 16 (40.7). Stride helps, candidate count peaks then falls, and offline write replacement supports the derivative reading only at s = 1.

Critical strengths
  1. Table 2 is the useful part. Backbone and training budget fixed, the outer gradient window saturates near G = 16 while the inner history still pays at G = 1.
  2. Honest reporting. Table 6 keeps the 83 to 96 percent policies in view, Table 7 includes the configurations that lose, and the velocity reading is shown failing at large stride.
Critical weaknesses
  1. 51.8 is the best of nine configurations in Table 7b, picked on the rollouts it is reported on. The spread runs 28.7 to 51.8, and (1, 8) sits below the 31.8 baseline. λ, τ and ε stay frozen; reconstruction gets no comparable search. Against the G = 16 baseline, Swap Blocks alone (24 to 80) supplies about 6 of the 11.1 points. One seed, one checkpoint, 100 rollouts: binomial error near 50 percent is already 5 points, and Table 7b swings further, Shell Game Table at 60, 37, 81 across s = 1, 8, 16. Five validation demonstrations are held out, so select on those, run two or three seeds, and report what survives.
  2. The TC term changes two things at once: the input moves from k_t to k_{t-s}, and negatives appear. A version without negatives, ‖f_W(k_{t-s}) − v_t‖², would say which one matters. The comparison is not state-matched either, since TC-TTT caches sK values the baseline lacks. And H_t includes t − s, so v_{t-s} becomes a negative against its own key, undoing what reconstruction wrote a step earlier. λ arbitrates that, unremarked.
  3. Proposition 1 puts all keys and queries on one unit vector, which removes the addressing problem the method depends on. The Eq. 22 remainder scales as s²(K+1)(2K+1)/12, about 1.2e4 at s = K = 16, against a leading term of 136. Table 3 agrees: 2.60 percent at s = 1, 378.63 percent at s = 8. A small-stride sanity check, not an account of the reported setting.
  4. The gains miss the tasks the framing is about. M(1) goes 41.4 to 69.6 at G = 1 while M(n) stops at 29.5, Cover 19, Press 7. I suspect the write rather than the timing: a 96 by 96 frame becomes a 512-d feature and then a 64-d key and value, so the block's original slot competes with the whole scene for room, which is roughly what Figure 1b claims. A spatially selective front end may be complementary, for instance the fixation-based spatially dynamic computation of AdaptiveNN (Wang et al., Emulating human-like adaptive vision for efficient and flexible machine visual perception, Nature Machine Intelligence 2025). Not required here. But probing W_t⁺ for the task-relevant latent would show what the objective bought. Minor

Table 2 gives 49.2 for TC at s = 1, K = 32 where Table 7a gives 49.1. The introduction says 31.7, the abstract 31.8. No timing or per-step cost anywhere, which matters at K = 64.

Questions for authors

See Weakness

5 comments
AuthorsOpenReview ↗

Response to Reviewer rhQP (1/2)

We sincerely thank the reviewer for the detailed assessment and specific suggestions for strengthening the evidence supporting our claims. We appreciate your recognition of the controlled comparisons and our transparent reporting of unsuccessful configurations and the limitations of the velocity interpretation. Your comments on configuration selection, statistical reliability, matched controls, and the assumptions underlying our analysis are particularly helpful. Below, we respond point by point to these concerns, as well as your questions about representation capacity, numerical consistency, and computational costs.

W1: configuration selection and seeds. We agree that one seed and a configuration chosen on the reported rollouts are not a sufficient basis for the headline, and that the gains are uneven: Swap Blocks contributes about 6.2 of the 11.1 points over G=16G = 16, while the other eight tasks average about +5.5. We have now separated selection from evaluation. We selected the objective parameters of every method using held-out validation diffusion loss (45 training and 5 held-out demonstrations per task), with nine candidates per method, seed 42, 20k updates and terminal EMA weights. For Reconstruction, we varied η0\eta_0 and the outer learning rate; for the two history controls, we varied η0\eta_0 and λ\lambda; and for TC, we evaluated its original (η0,λ,τ,ϵ)=(0.1,0.3,0.1,0.05)(\eta_0, \lambda, \tau, \epsilon) = (0.1, 0.3, 0.1, 0.05) together with eight one-at-a-time alternatives. The original TC parameters had been chosen early in development based on a small number of pilot runs and kept fixed in subsequent experiments. The validation-based search described above selects the same original configuration, with the three ϵ\epsilon values tied. All methods share (s,K)=(16,16)(s, K) = (16, 16). We then trained the selected configurations with seeds 101/202/303 and evaluated them on 50 scenarios per task, shared across methods and seeds:

Success (%), 3 seedsSelected settingAll nine tasks
Reconstruction, G=1G = 1η0=0.03\eta_0 = 0.03, LR 5×10−55 \times 10^{-5}26.7 ± 1.8
Reconstruction, G=16G = 16 (paper setting)η0=0.1\eta_0 = 0.1, LR 10−410^{-4}38.7 ± 2.2
+ cross-time regressionη0=0.03\eta_0 = 0.03, λ=0.3\lambda = 0.333.3 ± 3.1
+ historical reconstructionη0=0.03\eta_0 = 0.03, λ=0.1\lambda = 0.133.6 ± 0.4
+ TC (ours)original49.3 ± 1.3

TC remains 10.6 points above G=16G = 16 reconstruction and above both history controls.

W2: negatives, state matching and vt−sv_{t-s}. The table includes the no-negative regression ∥fW(kt−s)−vt∥2\lVert f_W(k_{t-s}) - v_t\rVert^2 you propose. All four memory objectives keep the same cache of sK=256sK = 256 detached keys and values, so storage is matched, and historical reconstruction also uses that cache actively. You are also right that HtH_t contains t−st-s: vt−sv_{t-s} is a negative against its own key and pushes against the preceding reconstruction write, with λ\lambda arbitrating between the two. λ\lambda was among the parameters searched for TC and for the regression control, and the original value was selected.

W3: Proposition 1. This is correct. With a common address the proposition removes the addressing problem, and at s=K=16s = K = 16 its remainder exceeds the leading term by a factor of s(2K+1)/6≈88s(2K+1)/6 \approx 88. Table 3 agrees: the local slope implied by the proposition matches the native write at s=1s = 1 (+2.6% action MSE) but not at s=16s = 16 (+365.4%), where replacements that keep the change over the whole sampled window come much closer (endpoint secant +38.8%).

AuthorsOpenReview ↗

Response to Reviewer rhQP (2/2)

W4: what W+W^+ stores and whether the action uses it. We followed your suggestion and probed W+W^+. At one decision point per task, we replayed the same 50 expert demonstrations (45 training, 5 held-out) through every checkpoint (five methods × three seeds). The probed variables are the block's original mat in Put Back Block after the button press, whether one or three transfers are complete in Swap Blocks, and which cover to open first in Cover Blocks. A linear probe decodes all three from W+W^+ under every method, including G=1G = 1 reconstruction, with near-ceiling balanced accuracy (90–100%). The decoding is easy, though: the current joint state alone reaches 67.6% in Put Back Block (chance 25%) and 96% in Swap Blocks (chance 50%), and only in Cover Blocks are the current inputs at chance, although W+W^+ still decodes the variable there.

We therefore tested whether the action follows W+W^+. At the same decision points we transplant W+W^+ from a donor demonstration of the same checkpoint, whose current frame closely matches the recipient's but whose variable differs. The recipient's current input and all diffusion noise stay fixed. We measure how far the next 8-step action chunk moves toward the donor's expert action, as the projection of the action change onto the donor-minus-recipient expert difference (0 = unchanged, 1 = the donor's expert action):

Shift toward donor (closed-loop success %)Put Back BlockSwap BlocksCover Blocks
Reconstruction, G=1G = 10.38 (54.7)0.14 (17.3)0.00 (0.0)
+ historical reconstruction0.52 (71.3)0.25 (22.0)0.00 (4.0)
+ cross-time regression0.53 (81.3)0.30 (24.0)0.01 (5.3)
Reconstruction, G=16G = 160.54 (89.3)0.47 (36.7)0.06 (2.7)
+ TC (ours)0.85 (98.7)0.99 (83.3)0.04 (19.3)
TC minus G=16G = 16 [95% CI]+0.30 [0.24, 0.38]+0.51 [0.29, 0.73]−0.02 [−0.09, 0.05]

On Put Back Block and Swap Blocks, the action follows the memory most strongly under TC. The five methods also exhibit the same descriptive ordering by action shift and closed-loop success, although the differences among the three intermediate methods are within seed variability.

Interestingly, Cover Blocks exhibits a different pattern. Although the relevant variable is decodable from $W^+$ under every method, transplanting $W^+$ produces little change in the next 8-step action chunk. In particular, TC does not show a larger action shift than $G = 16$ reconstruction (0.04 vs. 0.06; difference: −0.02, 95% CI [−0.09, 0.05]), despite achieving higher closed-loop success (19.3% vs. 2.7%). This is an interesting observation, suggesting that the ability to represent information in memory does not necessarily translate into a stronger immediate influence on action, and that the relationship between memory representation, action utilization and task success may depend on the task.

We believe this is an interesting direction for further investigation rather than evidence for a specific bottleneck. More broadly, it raises the question of how memory mechanisms interact with the spatial representations provided to them. In this respect, spatially selective front ends such as AdaptiveNN may complement our approach by improving the selection of task-relevant spatial information. We will discuss this possibility as a direction for future work.

Minor points. We will make Table 2 consistent with Table 7a (49.2 vs 49.1) and the introduction consistent with the abstract (31.7 vs 31.8). On cost, TC adds 0.9 ms per observation over reconstruction on an A800 (99.5 vs 98.6 ms amortized, within timing noise). The TC write itself takes 1.6 ms against 0.7 ms for reconstruction and scales as O(d2+Kd)O(d^2 + Kd) per observation.

Thank you again for the detailed review and for the support.

Dear Reviewer rhQP,

Thank you again for your detailed review and for supporting the paper. In our two-part response, we followed several of your suggestions. We now select hyperparameters on held-out validation loss and evaluate every method with three seeds. We also added the no-negative regression control, and we probed and transplanted W+W^+ to test whether the action follows the stored variable.

If any point is still unclear, we are happy to clarify it before the discussion closes on November 18 (AoE).

Best regards, The authors

Reviewer_yoj1OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper adds temporal contrastive learning to the online memory updates of a Diffusion Policy. It pairs past observations with current representations and uses historical representations as negatives. Experiments on nine simulated RMBench tasks show improved success without extending the action-gradient window. The paper also examines temporal sampling and provides a local interpretation of the memory update.

Critical strengths
  1. The improvement over the matched reconstruction baseline is substantial. Average success increases from 31.8% to 51.8% with the same policy architecture and action-gradient window. Comparisons with longer-window reconstruction baselines further support the value of the proposed update.

  2. The authors make a useful effort to understand why the update works. The derivation and write-replacement experiments provide evidence beyond success rates. The paper also reports where the local velocity interpretation breaks down, helping readers understand the limits of the explanation.

Critical weaknesses
  1. The current comparison changes both temporal pairing and the loss. It shows that the full modification helps, but does not isolate the contribution of historical negatives. Could the authors compare against regression using the same past input and current target? This would clarify what contrast adds beyond temporal supervision. It is a direct test of the paper’s main contribution.

  2. How should we understand the gains of the strongest model? The velocity interpretation works best at short strides, while the strongest policy uses s=16. The replacement tests change only one update, then return to the original memory trajectory. Could the authors apply the replacement throughout an episode and measure closed-loop success? This would better test whether the explanation holds for memory over longer periods.

  3. All policy evaluations are conducted on nine simulated RMBench tasks, with a separate policy trained for each task. No real-world experiments are reported. Could the authors add selected tasks from another benchmark, or a small real-world evaluation? This would help assess whether the gains extend beyond the current setting.

Questions for authors

See weakness.

5 comments
AuthorsOpenReview ↗

Response to Reviewer yoj1

We sincerely thank the reviewer for the thoughtful review and constructive suggestions. We appreciate your recognition of the substantial improvement over reconstruction-based TTT and our effort to understand the update mechanism. Your suggestions motivated additional experiments to isolate the contribution of contrastive negatives, examine repeated write interventions in closed-loop execution, and evaluate performance beyond RMBench. We present these results below and discuss their implications and limitations.

W1: contrast beyond temporal pairing. We ran the regression you suggest, from the past key kt−16k_{t-16} to the current value vtv_t with raw MSE and no negatives, together with a second control that reconstructs up to 16 cached historical pairs. Both are added on top of current-frame reconstruction. All methods share G=1G = 1, (s,K)=(16,16)(s, K) = (16, 16), the architecture, the detached key/value cache and the training budget, and each method's hyperparameters were selected on validation loss. Results are three training seeds per method on the same 50 scenarios per task:

Same cache and update budgetNine-task success (%)TC minus row, pp [95% CI]
Reconstruction, G=1G = 126.7 ± 1.8+22.7 [+18.0, +27.6]
+ cross-time regression33.3 ± 3.1+16.0 [+10.9, +20.9]
+ historical reconstruction33.6 ± 0.4+15.8 [+12.2, +19.3]
+ TC (ours)49.3 ± 1.3–
Reconstruction, G=16G = 1638.7 ± 2.2+10.6 [+6.2, +15.5]

Temporal supervision alone adds about 7 points, and contrasting against the agent's own history adds another 16. We will add this comparison to Sec. 4.1 of the revised version.

W2: the explanation over a whole episode. The velocity-based interpretation is supported only at small strides, and Table 3 evaluates single-write substitutions, after which the system returns to the native memory trajectory. We therefore also perturbed the write at every step of closed-loop episodes. Success fell to nearly zero across all settings we tested, including those with small strides.

We interpret this result as reflecting the sensitivity of closed-loop execution to repeated memory perturbations. Unlike a single-step intervention, repeated perturbations can alter the memory state at each step, affecting subsequent actions and, in turn, the observations and states encountered later in the episode. These deviations can propagate and accumulate through the closed-loop dynamics, leading to substantial performance degradation.

In the revised version, we will restrict the velocity-based interpretation in the abstract and Contribution 1 to small strides and present the mechanism underlying the results at $s = 16$ as an open question in the limitations.

W3: beyond nine simulated tasks. We evaluated TC on the four Counting tasks of RoboMME (Dai et al., 2026): BinFill, PickXTimes, SwingXTimes and StopCube. They require tracking how many times an event has occurred. TC keeps its RMBench configuration with no search on RoboMME, and uses three training seeds, 100k updates, terminal EMA weights and 50 evaluation episodes per task. The reference is the Diffusion Policy baseline reported by the benchmark authors:

RoboMME Counting, success (%)BinFillPickXTimesSwingXTimesStopCubeMean
Diffusion Policy (reported by RoboMME)32.0 ± 2.72.7 ± 3.66.7 ± 1.80.0 ± 0.010.3
TC (ours, 3 seeds)38.0 ± 3.543.3 ± 4.221.3 ± 3.10.0 ± 0.025.7 ± 2.6

TC outperforms the reported Diffusion Policy baseline on three of the four tasks and matches it on StopCube (0.0%). Its mean success rate is 25.7%, compared with 10.3% for the reported baseline, corresponding to a gain of 15.4 percentage points based on the displayed values. This indicates that the improvement carries over to a second benchmark and to counting-type memory. We will add these results to the appendix of the revised version.

Ethics flag. The review marks the paper for ethics review without naming a concern. All experiments use simulation and public benchmark data, and the use of AI assistance is disclosed in the AI-use statement. We are glad to address any specific concern.

We hope the added controls, the closed-loop analysis and the RoboMME results address your concerns, and would be grateful if you would consider raising your score.

Dear Reviewer yoj1,

We would like to follow up on our response. It includes the regression control you proposed (past key to current value, raw MSE, no negatives), closed-loop experiments that perturb the write at every step of an episode, and results on the four RoboMME Counting tasks. On those tasks TC reaches a mean success of 25.7%, against 10.3% for the Diffusion Policy baseline reported by the benchmark authors. We also replied to the ethics flag: all experiments are in simulation and use public benchmark data.

As the discussion period ends on November 18 (AoE), may we ask whether these additions address your main concerns? If so, we hope you might consider updating your score. If a specific ethics concern lies behind the flag, please let us know and we will respond to it directly.

Best regards, The authors

AuthorsOpenReview ↗

Follow-up on Our Response

Dear Reviewer yoj1,

As the discussion period is nearing its deadline, we just wanted to kindly bring our previous response to your attention once more.

We sincerely hope our additional experiments and clarifications, including those regarding the ethics flag, have addressed your concerns. If you have any further thoughts or feedback, we would greatly appreciate hearing from you.

Thank you again for your time and consideration!

Best regards, The Authors

Reviewer_biwTOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
3(+1)
Read review
Paper summary

The paper proposes Temporal Contrastive Test-Time Training (TC–TTT), a self-supervised objective for updating an online memory in robotic manipulation. In addition to reconstructing the current key–value pair, TC–TTT uses an earlier key to identify the current value against historical alternatives. The history stride and candidate count can vary without extending the action-gradient window. For a linear matrix memory, the authors derive the exact update and, under simplifying assumptions, a history-dependent correction with a velocity-like interpretation in representation space. Across nine simulated tasks using task-specific Diffusion Policies, the best configuration reaches 51.8% mean success, versus 31.8% for reconstruction-only TTT at the same one-update gradient window and 40.7% for reconstruction with a longer window. The paper also studies history sampling and offline write-replacement interventions.

Critical strengths
  1. The central empirical result is substantial and reasonably well controlled. TC–TTT improves mean success from 31.8% to 51.8% over reconstruction-only TTT with the same architecture and one-update gradient window, and exceeds the reconstruction baseline with a sixteen-update gradient window (40.7%).

  2. The method clearly separates history used to supervise the current memory update from the horizon of outer action gradients. The exact matrix-update derivation, restricted local analysis, and implementation details make the proposed mechanism understandable and reproducible in principle.

Critical weaknesses
  1. The experiments do not isolate the contribution of contrastive discrimination. The proposed objective simultaneously introduces historical inputs, cross-time prediction, and negative candidates. There is no matched baseline using an earlier key to regress to the current value without negatives, or reconstructing sampled historical pairs with the same cache and update budget. Consequently, the results establish the benefit of the combined history-aware objective more clearly than the specific benefit of contrastive learning.

  2. The proposed velocity-like interpretation does not adequately explain the strongest configuration. The local-slope replacement increases action MSE by just 2.60% at stride 1 but by 365.37% at stride 16, where success is highest; removing the temporal write at stride 16 increases MSE by 124.58%. These interventions are offline and single-step. In addition, no multi-seed training uncertainty is reported, and evaluation covers nine simulated tasks with separately trained policies.

Questions for authors
  1. How does TC–TTT compare with a matched cross-time regression objective, ‖Wk(t−s) − v(t)‖², without contrastive negatives, and with historical key–value reconstruction using the same cache, memory, and update budget?

  2. Were multiple training seeds used for the main reconstruction and TC–TTT results? Please report run-to-run variability and any paired evaluation uncertainty available.

  3. What explains the benefit at stride 16, where the local-slope replacement performs substantially worse than removing the temporal write? Can this be studied with repeated closed-loop write interventions?

  4. What are the per-observation inference latency and complete memory costs, including the historical feature cache, relative to reconstruction-only TTT?

4 comments
AuthorsOpenReview ↗

Response to Reviewer biwT

We sincerely thank the reviewer for the careful evaluation and constructive feedback. We appreciate your recognition of the empirical gains and the distinction between historical supervision and the action-gradient horizon. Below, we address your concerns about matched controls, training variability, the stride-16 interpretation, and inference costs with additional experiments and clarifications.

Q1 / W1: what contrast adds beyond cross-time supervision. We added the two requested controls. Both retain current-frame reconstruction and add a history term: cross-time regression λ∥Wkt−16−vt∥2\lambda\lVert W k_{t-16} - v_t\rVert^2 (raw MSE, no negatives), or historical reconstruction, the mean reconstruction loss over up to 16 cached pairs (kt−16j,vt−16j)(k_{t-16j}, v_{t-16j}). All methods share the architecture, detached 256-key/256-value cache, training budget, and one combined update per observation; the G=1G = 1 methods use (s,K)=(16,16)(s, K) = (16, 16). Hyperparameters were selected on held-out validation diffusion loss. Each row averages three seeds (101/202/303), evaluated on the same 50 scenarios per task:

Same cache and update budgetNine-task success (%)TC minus row, pp [95% CI]
Reconstruction, G=1G = 126.7 ± 1.8+22.7 [+18.0, +27.6]
+ cross-time regression33.3 ± 3.1+16.0 [+10.9, +20.9]
+ historical reconstruction33.6 ± 0.4+15.8 [+12.2, +19.3]
+ TC (ours)49.3 ± 1.3–
Reconstruction, G=16G = 1638.7 ± 2.2+10.6 [+6.2, +15.5]

Both history controls gain ~7 points over reconstruction. TC adds a further ~16 points over either and remains 10.6 points above reconstruction trained with 16-step backpropagation through time.

Q2: seeds and uncertainty. The submission reported one seed per configuration; each method above now has three. ± is the SD across seed-level nine-task means, and 95% CIs come from a paired bootstrap over seeds and, within each task, shared scenarios. Since every method/seed uses the same 50 scenarios per task, all comparisons are paired. Absolute values differ slightly from the paper's single-seed, 100-rollout results (49.3 here vs. 51.8 there).

Given rebuttal time, we prioritized the additional controls and multi-seed evaluations; the main results tables are not yet fully updated. We will update them in the revision with three-seed results and standard deviations.

Q3 / W2: why stride 16 helps. We agree the velocity-based interpretation does not explain the strongest setting. Table 3 supports it at s=1s = 1 (local slope increases action MSE by 2.6%), but not at s=16s = 16 (365.4%). At s=16s = 16, alternatives preserving change over the entire sampled window are closer to the native write: endpoint secant and full-window slope increase action MSE by 38.8% and 57.4%, respectively, vs. 365.4% for local slope and 124.6% without the write. Thus, at large strides, the write may capture changes accumulated over the sampled history rather than local velocity.

We also perturbed the write at every step in closed-loop episodes. Success fell to nearly zero across all tested settings, including small strides. This likely reflects sensitivity to repeated memory perturbations: unlike a single-step intervention, they alter memory at each step, affecting subsequent actions and later observations/states. These deviations can propagate and accumulate through closed-loop dynamics, degrading performance.

In the revision, we will restrict the velocity-based interpretation in the abstract and Contribution 1 to small strides, and present the mechanism at s=16s = 16 as an open question in the limitations.

Q4: latency and memory. Persistent state is 144.75 KiB: a 16 KiB fast matrix, a 128 KiB cache of 256 detached keys/values, and 0.75 KiB of masks (~0.05% of FP32 model weights, 293.8 MiB). For dimension dd and cache length NN, persistent storage is 4(d2+2Nd)4(d^2 + 2Nd) bytes plus masks; TC write costs O(d2+Kd)O(d^2 + Kd) per observation. All methods run in one process on an A800 (FP32, batch 1, 100 DDPM steps), replaying five held-out Put Back Block traces for 30 interleaved rounds. The last column measures write-only latency on identical inputs (d=64d = 64, K=16K = 16).

A800, FP32, batch 1Observation-only p50 (ms)Amortized (ms/observation)Write alone (ms)
Diffusion Policy (no memory)2.895.1–
Reconstruction, G=1G = 15.698.60.72
+ cross-time regression6.098.10.91
+ historical reconstruction6.398.81.21
+ TC (ours)6.999.51.59

Action calls take ~735 ms for every method because diffusion sampling dominates. Across paired rounds, TC is 0.9 ms/observation slower than G=1G = 1 reconstruction (within the 3.0 ms round-to-round SD) and 4.4 ms (4.6%) slower than Diffusion Policy without memory.

We hope these results address the contrastive-term isolation, seed variability, and stride-16 questions, and that the reviewer will consider raising the score.

Dear Reviewer biwT,

Thank you again for your review. Following your suggestion, our response adds the two controls that keep the same cache and update budget but replace the contrastive term with a non-contrastive history term (cross-time regression and historical reconstruction). It also reports three training seeds per method with paired bootstrap intervals, the size of the persistent memory state, and latency on an A800. TC remains about 16 points above both controls and costs 0.9 ms more per observation than reconstruction.

The author–reviewer discussion closes on November 18 (AoE). If these results answer your questions about the contrastive term, the seed variability and the stride-16 setting, we would be grateful if you took them into account in your final rating. If anything is still unclear, we will reply as quickly as we can before the deadline.

Best regards, The authors

Reviewer_biwTOpenReview ↗

Response

I thank the authors for running the additional experiments. I can now see that across three seeds, TC–TTT significantly outperformed both cross-time regression and historical reconstruction by about 16 percentage points. This addresses my main concerns and I am updating my score as a consequence. I also appreciate the honesty about the velocity-based interpretation and the stride size as an open question.

Thanks!

AuthorsOpenReview ↗

Thank You for Your Reassessment

Thank you for your time and thoughtful feedback. We appreciate your consideration of our additional experiments and are glad that they have addressed your concerns. Thank you again for your constructive suggestions!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.