acceptodds
Withdrawn from ICLR 2027

Test-Time Optimization Is Not Test-Time Memorization: Auditing Neural Memory on Egocentric Video

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

A test-time memory writes a stream into adaptive parameters by descending a self-supervised loss during the forward pass. This recipe can succeed as optimization while failing as memorization, and the training loss cannot tell the two apart. We audit a Titans-style MLP memory over frozen DINOv2 features. We first formalize when a recall objective cannot identify memory use, through a sufficient condition for a write-free global optimum and a Bayes-risk gap that states when causal history can help beyond the query. Trained for recall on i.i.d. image sequences, the memory approaches that optimum. The loss falls from 1.24 to 0.12 while the write rate contracts 200-fold, the final state loses input dependence, and retrieval drops below that of the same architecture at random initialization. Egocentric video adds a second shortcut. Frozen features remain correlated twenty seconds apart, and a causal convolution in the key exposes recent features to the query. Under a past-future windowed objective, the standard diagnostics look positive, and controls explain each of them away. At the default horizon, a copy of the oldest feature in the key window matches the far-past gain, a stateless filter with five coefficients per offset matches or beats the model at every offset, and the seen/unseen AUC ranges from 0.23 to 0.92 depending on which video is written. With a horizon beyond the key window, the model outperforms the filter twelve to seventeen seconds back while relying on its own state, yet its trace of individual items stays weak. The seen/unseen AUC detects neither case. We distill the audit into the controls that any storage claim about a test-time memory should report.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
test-time training, test-time memorization, neural memory, associative memory, fast weights, evaluation methodology, degenerate objectives, shortcut learning, egocentric video, self-supervised learning, video representation learning

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_aq6POpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper states that whether a Titans-style test-time memory, a 2-layer MLP updated online by gradient steps on an associative loss over frozen DINOv2 features, actually stores stream content. There are two main empirical findings: 1. IID ImageNet features with a recall objective whose key and value come from the same feature. 2. Egocentric video (Nymeria) with a past/future windowed objective. The paper ends with a minimum set of controls for storage claims about test-time memories.

Critical strengths

The paper names concrete shortcut paths: the key's receptive field, the static scene component, and the write gate collapsing while the loss still falls. It then shows that commonly reported diagnostics can be fully explained by these paths. The diagnostics in question are inner loss, gate magnitude, past/future asymmetry, and seen/unseen AUC.

The state-swap control (rather than only disabling writes, which moves the readout off-distribution), the receptive-field copy, the leave-one-video-out stateless filter, and the rotation over truth videos are good.

Critical weaknesses

The IID recall objective derives key and value from the same feature. Proposition 1 then predicts a write-free optimum by design, and associative-recall benchmarks are built to avoid exactly this; Titans itself is not trained this way.

This paper never shows a configuration where the trained memory passes the proposed audit, e.g. a decoupled-key/value recall objective (MQAR-style) or a synthetic stream with planted long-lag dependencies. Without that, readers cannot tell whether the audit discriminates or whether this training pipeline simply never learns to write.

Each configuration is trained once, with one seed. The probe fixture has five videos, from one dataset, with one encoder and a 4.2M-parameter memory.

The only stateless baseline is a 5-coefficient linear filter over the key window. A natural stateful but non-learned baseline is missing: an exponential moving average or running per-video mean of past features with a fitted decay. Figure 2 shows that much long-range similarity is a static per-video component, and the τ = 10 model "helps the readout without storing items". That pattern is exactly what a running-mean state would produce, so the paper's one positive result cannot yet be attributed to anything beyond scene averaging.

Questions for authors

Can you provide a positive control: a training objective or synthetic stream (decoupled keys/values, or planted dependencies beyond the key window) under which the same architecture and training pipeline passes your audit, with item-level traces and a beneficial state swap? This would show that the audit can support your claims.

For the τ = 10 models, how does a non-learned stateful baseline (EMA or running mean of past features with a decay fit on four videos, read out by the same per-offset linear heads) compare with the memory at j = −7 … −10?

Are the five fixture videos excluded from the 1065 training sequences? If not, how much does training-set overlap affect the seen/unseen and write-effect measurements?

How sensitive are the conclusions to the seed?

Reviewer_S2fbOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies the problem of whether test-time optimization actually produces retrievable memory. Its experiments focus on using Titans-style MLP over frozen DINOv2 features. On egocentric video, it uses query-only baselines, state swaps, and written/unwritten comparisons to examine apparent evidence of memory. The default direct-memory model is matched or outperformed by a simple query-only filter, it shows that a longer horizon improves over the tested filter, whereas item-level retrieval remains weak under the reported probes.

Critical strengths
  1. This paper studies the important problem of retrievable memory in TTO, identifies limitations in existing evaluations.

  2. The query-only comparison is interesting. For direct memory at the default horizon, copying the oldest feature in the key window largely removes the apparent far-past advantage, it shows why the current-feature baseline is insufficient.

  3. The independent-image experiment gives a concrete failure case: recall loss falls while the write rate and input dependence of the final state become very small. This supports the point that low recall loss alone does not establish online storage in this setup.

Critical weaknesses
  1. I find this paper extremely difficult to read. The critical logical flow is incoherent and the key information is spread across the entire paper.

  2. The experiments only covers one encoder and Titans-style memory variants. I think each frame is oversimplified, which is reduced to a single DINOv2 CLS vector, and then averaged over 50 frames by default. The audit is not extended to other TTT architectures, it remains unclear how well the empirical findings generalize beyond this particular setup.

  3. The main video evaluation uses only five probe videos and one training seed per configuration. The gain intervals in Table 1 also resample correlated windows rather than independent videos. The authors acknowledge these limitations, but stronger evidence is still needed to establish conclusive results.

  4. I am not sure the low retrieval scores mean the memory did not retain the items. The probe compares written items with nearby unwritten tokens from the same video, but their features may be too similar for this test to clearly distinguish them.

Questions for authors

Please respond to the weaknesses I listed above. Besides,

  1. I wonder if the main findings still hold with at least one alternative encoder or a spatial-token representation? And Which conclusions are expected to generalize to other TTT architectures?

  2. Can the video probes reliably recognize a memory known to retain the queried items under the same video correlations?

Reviewer_1xxFOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper examines whether successful test-time optimization demonstrates that an adaptive memory retains useful information. It formalizes conditions under which a recall objective admits a write-free solution and when history can improve prediction beyond the query. Experiments with a Titans-style memory over frozen DINOv2 features show that recall training can largely eliminate writing and that stateless baselines match short-horizon video prediction. A model trained with a longer horizon outperforms a simple stateless filter, although direct item-retrieval probes remain weak. The proposed audit combines query-only baselines, state interventions, content-matched retrieval tests, and evaluation across independent streams.

Critical strengths

The paper tackles an important evaluation problem: improved prediction alone does not establish that online updates stored useful information. The receptive-field-matched controls are useful here. In the default video setting, the apparent benefit over copying the current feature nearly disappears when the baseline copies the oldest feature available to the query instead. The longer-horizon experiment provides a positive contrast. Together, these controls offer a useful way to distinguish the contribution of adaptive memory from information already available through the query.

Critical weaknesses

My main concern is the limited connection between the selected experiments and practical memory evaluation. The recall experiment trains on past keys and values derived from the same input and empirically reaches a nearly write-free solution. In the default direct-memory video setting, three of the four past targets are already within the query's receptive field. These experiments usefully demonstrate how an objective can fail to test storage, but say less about memory use under objectives with fewer opportunities for such shortcuts. The paper partly addresses this concern with the separately trained Δ=10\Delta=10 model, which improves over the stateless filter by 0.1110.111 at the farthest past offset. Together with the weak item-retrieval results, this distinguishes predictive benefit from direct associative recall. But the main temporal prediction task still reaches only ten tokens into the past. I recognize that this is a diagnostic case study rather than a demonstration of long-context performance. Even within that scope, I am not yet convinced that the selected tasks establish the audit's usefulness in representative settings with longer temporal dependencies.

Second, the robustness and representativeness of the evaluation are difficult to assess. The principal video audit uses five videos from Nymeria, frozen DINOv2 features, and one training seed per configuration. Continuous video is a reasonable application, but the choice of this particular egocentric setting needs more justification. It remains unclear how much its temporal and scene correlations account for the observed shortcuts. The offset-weighting experiments do not replicate the same configuration across independent seeds. The gain intervals in Table 1 resample windows rather than independent videos, which the authors acknowledge understates uncertainty. Without per-video temporal prediction gains, I cannot tell whether the Δ=10\Delta=10 advantage is consistent across streams. The reported variation in seen/unseen AUC concerns a different diagnostic and does not answer this question.

Questions for authors

The Δ=10\Delta=10 experiment provides some evidence of a benefit beyond the query's recent receptive field. Can the authors provide a video retrieval evaluation with dependencies longer than the current ten-token horizon, where a later query requires past information not directly available in its recent receptive field? Please include an appropriate query-only baseline and explain what practical memory use the task represents.

Can you report the paired far-past gain of the Δ=10\Delta=10 model over the stateless filter separately for each evaluation video, with uncertainty assessed across videos rather than windows? Is the reported +0.111+0.111 advantage consistent across streams, or concentrated in particular videos?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.