- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
TC-TTT adds a second term to the inner loss of a TTT memory: besides reconstruction, the key from s steps back must retrieve the current value against K earlier values at the same stride. The analysis collapses every key and query onto one unit vector, leaving the new term as the gap between the current feature and its history mean, read as a derivative at small s. On nine RMBench tasks with per-task 77M Diffusion Policies, mean success goes 31.8 to 51.8 at a one-update gradient window, above reconstruction-only TTT at G = 16 (40.7). Stride helps, candidate count peaks then falls, and offline write replacement supports the derivative reading only at s = 1.
Critical strengths
- Table 2 is the useful part. Backbone and training budget fixed, the outer gradient window saturates near G = 16 while the inner history still pays at G = 1.
- Honest reporting. Table 6 keeps the 83 to 96 percent policies in view, Table 7 includes the configurations that lose, and the velocity reading is shown failing at large stride.
Critical weaknesses
- 51.8 is the best of nine configurations in Table 7b, picked on the rollouts it is reported on. The spread runs 28.7 to 51.8, and (1, 8) sits below the 31.8 baseline. λ, τ and ε stay frozen; reconstruction gets no comparable search. Against the G = 16 baseline, Swap Blocks alone (24 to 80) supplies about 6 of the 11.1 points. One seed, one checkpoint, 100 rollouts: binomial error near 50 percent is already 5 points, and Table 7b swings further, Shell Game Table at 60, 37, 81 across s = 1, 8, 16. Five validation demonstrations are held out, so select on those, run two or three seeds, and report what survives.
- The TC term changes two things at once: the input moves from k_t to k_{t-s}, and negatives appear. A version without negatives, ‖f_W(k_{t-s}) − v_t‖², would say which one matters. The comparison is not state-matched either, since TC-TTT caches sK values the baseline lacks. And H_t includes t − s, so v_{t-s} becomes a negative against its own key, undoing what reconstruction wrote a step earlier. λ arbitrates that, unremarked.
- Proposition 1 puts all keys and queries on one unit vector, which removes the addressing problem the method depends on. The Eq. 22 remainder scales as s²(K+1)(2K+1)/12, about 1.2e4 at s = K = 16, against a leading term of 136. Table 3 agrees: 2.60 percent at s = 1, 378.63 percent at s = 8. A small-stride sanity check, not an account of the reported setting.
- The gains miss the tasks the framing is about. M(1) goes 41.4 to 69.6 at G = 1 while M(n) stops at 29.5, Cover 19, Press 7. I suspect the write rather than the timing: a 96 by 96 frame becomes a 512-d feature and then a 64-d key and value, so the block's original slot competes with the whole scene for room, which is roughly what Figure 1b claims. A spatially selective front end may be complementary, for instance the fixation-based spatially dynamic computation of AdaptiveNN (Wang et al., Emulating human-like adaptive vision for efficient and flexible machine visual perception, Nature Machine Intelligence 2025). Not required here. But probing W_t⁺ for the task-relevant latent would show what the objective bought. Minor
Table 2 gives 49.2 for TC at s = 1, K = 32 where Table 7a gives 49.1. The introduction says 31.7, the abstract 31.8. No timing or per-step cost anywhere, which matters at K = 64.
Questions for authors
See Weakness
5 comments
Response to Reviewer rhQP (2/2)
W4: what stores and whether the action uses it. We followed your suggestion and probed . At one decision point per task, we replayed the same 50 expert demonstrations (45 training, 5 held-out) through every checkpoint (five methods × three seeds). The probed variables are the block's original mat in Put Back Block after the button press, whether one or three transfers are complete in Swap Blocks, and which cover to open first in Cover Blocks. A linear probe decodes all three from under every method, including reconstruction, with near-ceiling balanced accuracy (90–100%). The decoding is easy, though: the current joint state alone reaches 67.6% in Put Back Block (chance 25%) and 96% in Swap Blocks (chance 50%), and only in Cover Blocks are the current inputs at chance, although still decodes the variable there.
We therefore tested whether the action follows . At the same decision points we transplant from a donor demonstration of the same checkpoint, whose current frame closely matches the recipient's but whose variable differs. The recipient's current input and all diffusion noise stay fixed. We measure how far the next 8-step action chunk moves toward the donor's expert action, as the projection of the action change onto the donor-minus-recipient expert difference (0 = unchanged, 1 = the donor's expert action):
| Shift toward donor (closed-loop success %) | Put Back Block | Swap Blocks | Cover Blocks |
|---|---|---|---|
| Reconstruction, | 0.38 (54.7) | 0.14 (17.3) | 0.00 (0.0) |
| + historical reconstruction | 0.52 (71.3) | 0.25 (22.0) | 0.00 (4.0) |
| + cross-time regression | 0.53 (81.3) | 0.30 (24.0) | 0.01 (5.3) |
| Reconstruction, | 0.54 (89.3) | 0.47 (36.7) | 0.06 (2.7) |
| + TC (ours) | 0.85 (98.7) | 0.99 (83.3) | 0.04 (19.3) |
| TC minus [95% CI] | +0.30 [0.24, 0.38] | +0.51 [0.29, 0.73] | −0.02 [−0.09, 0.05] |
On Put Back Block and Swap Blocks, the action follows the memory most strongly under TC. The five methods also exhibit the same descriptive ordering by action shift and closed-loop success, although the differences among the three intermediate methods are within seed variability.
Interestingly, Cover Blocks exhibits a different pattern. Although the relevant variable is decodable from $W^+$ under every method, transplanting $W^+$ produces little change in the next 8-step action chunk. In particular, TC does not show a larger action shift than $G = 16$ reconstruction (0.04 vs. 0.06; difference: −0.02, 95% CI [−0.09, 0.05]), despite achieving higher closed-loop success (19.3% vs. 2.7%). This is an interesting observation, suggesting that the ability to represent information in memory does not necessarily translate into a stronger immediate influence on action, and that the relationship between memory representation, action utilization and task success may depend on the task.
We believe this is an interesting direction for further investigation rather than evidence for a specific bottleneck. More broadly, it raises the question of how memory mechanisms interact with the spatial representations provided to them. In this respect, spatially selective front ends such as AdaptiveNN may complement our approach by improving the selection of task-relevant spatial information. We will discuss this possibility as a direction for future work.
Minor points. We will make Table 2 consistent with Table 7a (49.2 vs 49.1) and the introduction consistent with the abstract (31.7 vs 31.8). On cost, TC adds 0.9 ms per observation over reconstruction on an A800 (99.5 vs 98.6 ms amortized, within timing noise). The TC write itself takes 1.6 ms against 0.7 ms for reconstruction and scales as per observation.
Thank you again for the detailed review and for the support.
Dear Reviewer rhQP,
Thank you again for your detailed review and for supporting the paper. In our two-part response, we followed several of your suggestions. We now select hyperparameters on held-out validation loss and evaluate every method with three seeds. We also added the no-negative regression control, and we probed and transplanted to test whether the action follows the stored variable.
If any point is still unclear, we are happy to clarify it before the discussion closes on November 18 (AoE).
Best regards, The authors
Response to Reviewer rhQP (1/2)
We sincerely thank the reviewer for the detailed assessment and specific suggestions for strengthening the evidence supporting our claims. We appreciate your recognition of the controlled comparisons and our transparent reporting of unsuccessful configurations and the limitations of the velocity interpretation. Your comments on configuration selection, statistical reliability, matched controls, and the assumptions underlying our analysis are particularly helpful. Below, we respond point by point to these concerns, as well as your questions about representation capacity, numerical consistency, and computational costs.
W1: configuration selection and seeds. We agree that one seed and a configuration chosen on the reported rollouts are not a sufficient basis for the headline, and that the gains are uneven: Swap Blocks contributes about 6.2 of the 11.1 points over G=16, while the other eight tasks average about +5.5. We have now separated selection from evaluation. We selected the objective parameters of every method using held-out validation diffusion loss (45 training and 5 held-out demonstrations per task), with nine candidates per method, seed 42, 20k updates and terminal EMA weights. For Reconstruction, we varied η0 and the outer learning rate; for the two history controls, we varied η0 and λ; and for TC, we evaluated its original (η0,λ,τ,ϵ)=(0.1,0.3,0.1,0.05) together with eight one-at-a-time alternatives. The original TC parameters had been chosen early in development based on a small number of pilot runs and kept fixed in subsequent experiments. The validation-based search described above selects the same original configuration, with the three ϵ values tied. All methods share (s,K)=(16,16). We then trained the selected configurations with seeds 101/202/303 and evaluated them on 50 scenarios per task, shared across methods and seeds:
TC remains 10.6 points above G=16 reconstruction and above both history controls.
W2: negatives, state matching and vt−s. The table includes the no-negative regression ∥fW(kt−s)−vt∥2 you propose. All four memory objectives keep the same cache of sK=256 detached keys and values, so storage is matched, and historical reconstruction also uses that cache actively. You are also right that Ht contains t−s: vt−s is a negative against its own key and pushes against the preceding reconstruction write, with λ arbitrating between the two. λ was among the parameters searched for TC and for the regression control, and the original value was selected.
W3: Proposition 1. This is correct. With a common address the proposition removes the addressing problem, and at s=K=16 its remainder exceeds the leading term by a factor of s(2K+1)/6≈88. Table 3 agrees: the local slope implied by the proposition matches the native write at s=1 (+2.6% action MSE) but not at s=16 (+365.4%), where replacements that keep the change over the whole sampled window come much closer (endpoint secant +38.8%).