- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper tries to address how reconstructive writeback changes the future quality of its individual memories. It uses response operators to predict these changes and controlled interventions to examine the effects of structural routing and retrieval conditions. Experiments are performed over synthetic memory systems, visual writable memory, and a focal layer of a pretrained test-time training (TTT) model.
Critical strengths
-
The paper tries to address the limitation of evaluating writable memory solely via present recall and aggregate scores.
-
The matched experiments isolate the effect of retrieval conditions by keeping the written updates and stored traces identical. This helps distinguish changes in stored information from changes in how that information is retrieved.
-
The authors clearly report negative results and limitations, including cases where simpler baselines explain the findings or the proposed predictions fail. This makes the conclusions easier to assess.
Critical weaknesses
-
The main experiment shows that changing the retrieval operator changes future responses despite matching selected present measurements. However, the paper needs to explain more clearly what general insight about memory dynamics follows from this result.
-
The prediction experiment tests whether a response estimated using one set of reconstruction errors can predict the effects of another set. However, removing the part of the predictor that depends on error direction barely changes performance. This suggests that the strong results mainly reflect effects shared across errors. Since the stored memories and update order remain fixed within each comparison, it is unclear whether the predictor works when these conditions change.
-
Full propagation is used to track how each update affects later memory states. However, a simpler predictor that accounts for how much earlier updates decay over time performs almost as well. This baseline is missing from the later tests of changes across memories and unseen error directions, so those results do not establish the additional benefit of full propagation.
-
The TTT experiment tests whether the predictions remain useful in a pretrained model. Here, the predictor fails to estimate the actual changes in memory scores. After the average change is removed, it can partly identify which items fare better or worse than others, but the size of these differences remains unreliable. The result therefore supports relative comparisons more than reliable prediction of future scores.
-
The experiments do not establish whether the audit can identify meaningful memory failures. The main causal result comes from one constructed system, visual-memory accuracy remains perfect, and the TTT evaluation measures reconstruction at a single layer. The paper therefore does not establish whether the reported score changes help identify actual recall failures or poorer performance on the model’s main task.
-
The writing in the paper is difficult to follow. I understand Generative AI tools were used to refine the paper but I feel it hampers the readability and comprehensibility of the paper. The content switches between memory systems with different update rules and quality scores, making it hard to connect the experiments to the main claim. A simpler problem statement, clearer descriptions of the experimental setups, and a direct explanation of what each result establishes would improve readability.
Questions for authors
-
What insight about memory dynamics can be inferred from the controlled retrieval experiment, and what evidence suggests it applies beyond the constructed system?
-
Does the predictor remain reliable when the stored memories or update order change, given that the error-direction component contributes little to its performance?
-
How does full propagation compare with the decay-weighted baseline in the tests of changes across memories and unseen error directions?
-
Why does the native TTT predictor fail to estimate actual score changes despite retaining some ability to rank the changes across items?
-
Do the predicted score changes correspond to actual recall errors or poorer performance on the model’s main task?
-
Could you provide a concise overview linking each experiment’s memory system, update rule, and quality measure to the claim it supports?
Thank you to the authors for the detailed response and reporting additional experiments. My assessment is as follows -
The additional experiments address several specific questions. My main concerns about prediction under changing memory conditions and the connection to actual task failures remain. I therefore maintain my original rating of 1.