acceptodds
Under review as a conference paper at ICLR 2027

SPACE: Sparse Predictive Attractor via Counterfactual Eviction for Streaming Video Memory

Abstract

Fixed-capacity streaming video memory requires repeated eviction decisions whose effects accumulate over time. Yet existing policies are evaluated primarily in terms of retained information or downstream accuracy, leaving how repeated updates alter the futures supported by memory largely unexamined. We define a memory's predictive state as the future representations supported by its retained history and formulate eviction as counterfactual control over transitions in this space. We introduce SPACE (Sparse Predictive Attractor via Counterfactual Eviction), which uses a frozen multi-horizon JEPA to predict the future representations induced by alternative eviction actions. Counterfactual utility identifies future-useful alternatives, while slow predictive-basin geometry determines when to correct avoidable drift and when to adapt to sustained predictive change, without online parameter updates. We further introduce MABS-Bench, which evaluates future-task sufficiency, within-regime predictive stability, transition responsiveness, and perturbation recovery under matched causal streams and memory budgets. Across multiple video datasets, SPACE yields consistent improvements in dataset-native task performance while reducing predictive-state drift.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Streaming Video Memory, Predictive Representation Learning, Long-Horizon Video Understanding, Attractor Dynamics

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_VQopOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This work addresses streaming memory management by proposing a memory eviction method based on future predictive utility and semantic stability. This paper also introduces a new benchmark for evaluating memory utility and predictive-state stability.

Critical strengths
  1. This work addresses an important problem: how to retain information that supports future prediction under a fixed memory budget. The problem is well motivated and practical.
  2. Evaluating the complete memory state after each candidate eviction is a promising approach to capturing the future utility of stored information, beyond relying solely on past observations or the properties of individual memory elements.
Critical weaknesses
  1. The proposed method relies on an offline utility bank constructed using future observations from the training distribution. During deployment, counterfactual utility is estimated by retrieving similar experiences from this bank. However, the relationship between retrieved experiences and future utility may not remain reliable under distribution shifts or in previously unseen domains. This assumption limits the practical applicability of the proposed method and requires further discussion or experimental validation.
  2. The improvements on Assembly101 are relatively small compared with the baselines. These results provide limited evidence of the effectiveness of the proposed method, particularly given its additional complexity and computational overhead. Stronger empirical evidence is needed to justify the proposed design.
  3. The methodology is difficult to follow because the motivation and role of each component are not clearly connected to the problem being addressed. The introduction of numerous technical details and new terms further obscures the overall design logic. The authors should explain the problem addressed by each component and establish a consistent line of reasoning that connects these components to the proposed method.
Questions for authors

Is it possible to evaluate the future utility of individual memory slots at test time, at least retrospectively using observed future data? Such an analysis could help demonstrate whether the proposed method preferentially retains high-utility memory elements and provide further insight into how the memory eviction mechanism works.

7 comments
AuthorsOpenReview ↗

Response to Reviewer VQop [1/3]

We thank the reviewer for recognizing the importance of memory retention for future prediction and the promise of evaluating the complete memory state after each candidate eviction.

1. [W1] Reliability under distribution shifts

Thank you for raising the question of how the Utility Bank performs under distribution shift. We use future observations from training streams to construct utility targets. During evaluation, the controller queries the frozen bank using information available at each update.

a. Task and override outcomes under distribution shift

We kept the bank, predictor, and thresholds fixed across all four settings. An assembly configuration refers to the toy being assembled, identified by its Assembly101 toy ID. A configuration is seen if the Utility Bank contains training examples with that toy ID and unseen if the ID is absent from the bank. The observation stride determines how often the controller receives an observation. At native stride, the controller receives every observation in the standard input stream. At doubled stride, it receives every other observation from that stream, halving the input rate. We count prediction horizons in observations received by the controller.

Evaluation settingRecordings / participantsAction NLL gain vs. Reservoir ↑Participant 95% CIRealized override utility gain ↑
Seen configurations, native stride33 / 9+0.002449[0.001803, 0.003224]+0.16759 ± 0.28153
Unseen configurations, native stride14 / 7+0.001028[0.000761, 0.001544]+0.26568 ± 0.25622
Seen configurations, doubled stride33 / 9+0.001747[0.000838, 0.002050]+0.28787 ± 0.21862
Unseen configurations, doubled stride14 / 7+0.000923[0.000729, 0.001118]+0.27407 ± 0.30832

Action NLL gain measures the paired reduction in loss over the full stream. Realized override utility measures the hindsight gain of the action SPACE actually chose over the Reservoir action at the same memory state, using equal weights on normalized one-step and 32-step prediction costs. With the bank, predictor, and thresholds frozen, SPACE retains positive Action NLL gains on unseen configurations at both strides, supporting generalization to the tested shifts.

b. Utility ranking and gate performance under the same shifts

To test whether the bank continues to rank useful actions under shifts, we sampled up to eight Reservoir decision states per recording and seed and scored all 17 actions using observed futures. Sign accuracy checks the direction of gain over Reservoir. Spearman measures agreement with the hindsight ranking, and Hit@k checks whether the hindsight-best action appears among the top k predictions. Gate precision is the fraction of admitted alternatives with positive hindsight gain.

Evaluation settingUtility sign accuracy (%) ↑Spearman ↑Hit@1 (%) ↑Hit@3 (%) ↑Gate precision (%) ↑
Seen configurations, native stride59.52 ± 0.760.3185 ± 0.054215.21 ± 3.0226.05 ± 4.7964.44 ± 5.91
Unseen configurations, native stride60.86 ± 1.13+0.3203 ± 0.029315.18 ± 2.0827.98 ± 2.9355.42 ± 3.83
Seen configurations, doubled stride60.62 ± 2.16+0.3149 ± 0.035913.21 ± 1.1625.86 ± 2.6764.76 ± 7.22
Unseen configurations, doubled stride65.41 ± 2.60+0.3152 ± 0.070913.74 ± 1.8725.51 ± 2.4555.54 ± 8.68

Mean utility-ranking correlations remain around 0.315–0.320 across all four conditions. On unseen configurations, doubling the stride raises sign accuracy from 60.86% to 65.41%, while gate precision stays near 55.5% and Hit@1 changes from 15.18% to 13.74%. Higher sign accuracy can result from correctly identifying more harmful candidates, while Hit@1 depends on ranking the best removal first. These diagnostics capture different aspects of utility estimation, alongside a positive stream-level predictive gain under the combined shift.

AuthorsOpenReview ↗

Response to Reviewer VQop [2/3]

2. [W2] Improvement relative to complexity

We agree that the headline Assembly101 Action NLL reduction from 3.6315 to 3.6291 is modest given the additional storage and computation. We therefore tested whether SPACE's predictive gains persist on reserved recordings and additional datasets, whether simpler utility estimators can match those gains, and how much of the gain remains with a smaller Utility Bank.

a. Predictive gains on reserved recordings and additional datasets

On reserved Assembly101 recordings with the original thresholds and learned assets frozen, SPACE achieves a paired Action NLL gain of +0.001865 (participant-clustered 95% CI [0.000961, 0.002755]), with positive gains for 8 of 9 participants. SPACE also lowers mean Action NLL and yields a positive paired H64 gain over Reservoir on each additional dataset.

DatasetMean Action NLL gain over ReservoirPaired H64 gain (mean ± seed SD)
Breakfast+0.0285+0.0328 ± 0.0091
IKEA ASM+0.0798+0.0877 ± 0.0051
EPIC-KITCHENS-100+0.0230+0.0433 ± 0.0094

SPACE’s predictive gains extend beyond the original threshold-selection split and persist across three additional datasets. The full cross-dataset results are provided in Tables 8–10 of Appendix D in the revised PDF.

b. Comparison with simpler utility estimators

We trained Ridge (α=1), a 64/32-unit ReLU MLP, and histogram GBDT (80 rounds, up to 15 leaves) on the same training descriptors and counterfactual targets. With inherited guards, each estimator replaces the retrieved utility mean and retains bank-derived uncertainty, support, and the other gates. The bank-free variants use a fixed threshold on the predicted utility advantage. Both decision rules compute the candidate descriptors.

Estimator and decision ruleAction NLL (mean ± seed SD) ↓NLL gain (×10⁻³) ↑H64 gain (×10⁻³) ↑Override rate (%)
Reservoir3.6423 ± 0.02280.00000.00000.000
SPACE retrieval3.6405 ± 0.0211+1.7725+5.40611.022
Ridge with inherited guards3.6428 ± 0.0234−0.5534−1.28350.747
MLP with inherited guards3.6423 ± 0.0248+0.0101+0.49504.280
GBDT with inherited guards3.6426 ± 0.0226−0.2836−0.24740.080
Ridge with bank-free point decisions3.6658 ± 0.0236−23.4800−42.48208.203
MLP with bank-free point decisions3.6533 ± 0.0214−10.9800−3.974031.590
GBDT with bank-free point decisions3.6503 ± 0.0218−8.0570−13.61006.000

SPACE retrieval yields the largest mean Action NLL and H64 gains, while all three bank-free point variants increase Action NLL relative to Reservoir. The tested simpler estimators therefore fall short of reproducing SPACE’s predictive benefits, even when they retain the original guards.

c. Predictive gains and cost at smaller bank sizes

In a separate Assembly101 ablation, we formed nested bank subsets stratified by training participant and source policy (FIFO or Reservoir). We evaluated each bank size on the same 30 validation recordings from five participants with three matched policy seeds, keeping the predictor, thresholds, and slot count fixed. We report paired gains over Reservoir, tensor storage, and time per online policy update.

ConfigurationBank tensors (MiB)Action NLL ↓NLL gain (×10⁻³) ↑Participant 95% CI (×10⁻³)H64 gain (×10⁻³) ↑Override rate (%)Policy time (ms/update)
Reservoir0.03.6423 ± 0.02280.0000 ± 0.0000—0.00000.0000.77
SPACE, 12.5% bank92.53.6411 ± 0.0219+1.1732 ± 0.9945[0.6384, 1.7914]+3.26480.6307.87
SPACE, 25% bank185.03.6404 ± 0.0227+1.9318 ± 0.4695[0.9171, 2.9450]+4.20630.7598.07
SPACE, full bank740.03.6405 ± 0.0211+1.7725 ± 1.7283[0.0139, 3.4228]+5.40611.0229.66

The 25% bank cuts tensor storage by 75% (740 to 185 MiB) and policy time by 16.5% (9.66 to 8.07 ms per update) relative to the full bank. It retains 78% of the full bank's H64 gain and a positive Action NLL gain of +0.001932 over Reservoir.

A quarter-size bank preserves positive Action NLL and H64 gains while reducing storage and update time, providing a practical way to tune SPACE’s computation–storage trade-off.

AuthorsOpenReview ↗

Response to Reviewer VQop [3/3]

3. [W3] Clearer explanation in the revised PDF

Thank you for suggesting that we connect each component to the problem it solves. We have revised the Method section to clarify each component’s role in the decision to follow or override Reservoir.

  1. Evaluate candidate removals. The frozen predictor forecasts the futures supported by each resulting memory set.
  2. Screen for utility. The Utility Bank estimates which candidate changes are likely to improve prediction over the Reservoir action. The guards use retrieval support and uncertainty to screen these estimates.
  3. Stabilize predictions within a regime. The slow basis and basin prototype guide the choice among admissible changes toward a stable predictive state.
  4. Adapt at boundaries. When the current basin becomes unreachable, SPACE follows the Reservoir action and updates the basin prototype more quickly.

The revised presentation explains when SPACE follows Reservoir, what evidence warrants an override, and how the decision changes at a boundary.

4. [Q1] Retrospective utility of individual slots

Thank you for suggesting that we evaluate individual slots retrospectively with observed futures. We measure a slot’s predictive utility by evaluating how removing it affects future predictions given the rest of the memory.

At 960 sampled memory states from one policy seed, we enumerate all 17 removal actions at K=16, including discarding the incoming observation. We compare each candidate with the Reservoir action at the same state and score both using the same observed future. The cost equally weights training-normalized one-step and 32-step prediction terms, and the rollout branches share subsequent random choices. The override-level analysis uses the 209 executed SPACE overrides with complete oracle targets reported in Table 14 of the revised PDF. For each override, we compare the chosen action with the Reservoir action at the same memory state.

Evaluation levelDiagnosticSPACEReference or 95% CI
Candidate-levelPairwise Sign Accuracy ↑61.98%51.29% majority
Candidate-levelHit@1 ↑13.53%5.88% random
Candidate-levelHit@3 ↑28.29%17.65% random
Override-levelBeneficial Override Fraction ↑58.85%[55.06%, 62.99%]
Override-levelActual Override Gain ↑+0.4166[0.1188, 0.9328]

In the context of the remaining memory, SPACE identifies beneficial removals more accurately than the reference baselines, and its actual overrides improve retrospective predictive utility on average.

Reviewer_Vzi9OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes SPACE, a fixed-capacity streaming video memory controller that treats eviction as sparse counterfactual control over a predictive state. At each full-memory update, a frozen multi-horizon JEPA evaluates all K+1 eviction candidates. A retrieval-based utility bank, built offline from training streams with future observations, screens utility-admissible candidates, and a slow predictive basis with an adaptive basin prototype decides whether to override the Reservoir default or trigger boundary release. The paper also introduces MABS-Bench for evaluating future-task sufficiency, within-regime drift, boundary selectivity, and perturbation recovery.

Critical strengths
  • Evaluating evictions by how they change the futures supported by the retained set, rather than by item-level recency, novelty, or importance, is a meaningful conceptual perspective.
  • The asymmetric design addresses the stability–plasticity trade-off in a structured, interpretable way.
  • MABS-Bench jointly measures future-task sufficiency, within-regime drift, boundary responsiveness, and perturbation recovery, with experiments on various datasets.
  • The paper includes adapted recent baselines, component ablations, participant-level paired inference, normalization audit, retrospective oracle validation of the utility bank, an independently trained audit evaluator, and drift analysis beyond the slow space. Limitations are disclosed candidly.
Critical weaknesses

Major Concerns:

  • The improvement on the primary task is far smaller than seed-to-seed variation and indistinguishable from existing baselines on some metrics. Per-update latency is not the issue. The real burden is the large utility bank, the offline enumeration of all candidate actions, and candidate evaluation that scales with memory size. The paper does not show that the gains justify this machinery.
  • The controller thresholds are tuned on the same Assembly101 validation split used to report the main results. The statistical tests are therefore post-selection measures rather than independent confirmatory evidence, as the authors acknowledge.
  • The same JEPA predictions both drive the controller and define the drift and selectivity metrics, so the evaluation is coupled to the method itself. Under a separately trained audit evaluator, the effects largely disappear or fall within noise, and both evaluators share the same TSM features. The absolute drift values and the differences between methods are tiny. The paper does not show that differences of this size are behaviourally meaningful, which weakens the “attractor” framing.
  • The utility bank depends on offline targets built from future observations in training streams. Retrospective validation shows the estimator is only modestly better than chance, and a substantial share of SPACE’s actual overrides turn out to be harmful in hindsight. How the estimator behaves under stream dynamics that differ from the training distribution, and how sensitive it is to bank size or coverage, are not analysed. It is therefore unclear whether this core component would remain reliable in new settings.
Questions for authors
  • Can the authors decompose the gains by the 3.44% of updates where SPACE overrides Reservoir, and compare under matched compute and storage budgets?
  • How does utility-bank coverage or size affect action selection on unseen streams, and do simpler utility estimators give a similar trade-off?
  • Can the authors provide held-out evidence that excludes the effect of Assembly101 threshold selection, uncertainty estimates for the cross-dataset results, and an independent evaluator built on a different visual representation?
6 comments
AuthorsOpenReview ↗

Response to Reviewer Vzi9 [1/4]

We thank the reviewer for recognizing the predictive-state view of eviction, the interpretable stability–adaptation design, and the breadth of MABS-Bench and its audits.

1. [W1 & Q1 & Q2] Bank size, coverage, and candidate costs

We thank the reviewer for raising the cost of the Utility Bank, which prompted us to test how much of SPACE's predictive gain we could retain with fewer bank states and narrower participant coverage. SPACE enumerates candidate actions offline to build the bank and evaluates K+1 candidates at each full-memory update. To isolate the effect of bank size, we used nested subsamples stratified by training participant and FIFO/Reservoir source while holding the predictor, thresholds, and memory-slot count fixed. We measured tensor storage and policy time per update.

ConfigurationBank tensors (MiB)Action NLL ↓NLL gain (×10⁻³) ↑Participant 95% CI (×10⁻³)H64 gain (×10⁻³) ↑Override rate (%)Policy time (ms/update)
Reservoir0.03.6423 ± 0.02280.0000 ± 0.0000—0.00000.0000.77
SPACE, 12.5% bank92.53.6411 ± 0.0219+1.1732 ± 0.9945[0.6384, 1.7914]+3.26480.6307.87
SPACE, 25% bank185.03.6404 ± 0.0227+1.9318 ± 0.4695[0.9171, 2.9450]+4.20630.7598.07
SPACE, 50% bank370.03.6409 ± 0.0219+1.4078 ± 1.0140[0.1232, 2.6692]+3.60290.8708.58
SPACE, full bank740.03.6405 ± 0.0211+1.7725 ± 1.7283[0.0139, 3.4228]+5.40611.0229.66

At the same slot count, the 25% bank retains 78% of the full bank's H64 gain while cutting bank tensor storage from 740 to 185 MiB (75%) and policy time from 9.66 to 8.07 ms per update (16.5%). Its Action NLL gain over Reservoir is +0.001932 (95% CI [0.000917, 0.002945]). Even the 12.5% bank retains a positive Action NLL gain (95% CI [0.000638, 0.001791]).

We separately varied the number of training participants represented in a bank of 896 states and evaluated each variant on the same held-out streams.

ConfigurationBank statesAction NLL ↓NLL gain (×10⁻³) ↑Participant 95% CI (×10⁻³)H64 gain (×10⁻³) ↑Override rate (%)
Reservoir03.6423 ± 0.02280.0000 ± 0.0000—0.00000.000
SPACE, full bank37123.6405 ± 0.0211+1.7725 ± 1.7283[0.0139, 3.4228]+5.40611.022
SPACE, 7 of 29 participants8963.6410 ± 0.0222+1.3045 ± 0.7016[0.5847, 1.9472]+3.50410.695
SPACE, 14 of 29 participants8963.6411 ± 0.0219+1.1643 ± 1.3336[0.1352, 2.0743]+3.53640.716
SPACE, 29 of 29 participants8963.6410 ± 0.0227+1.2730 ± 0.5233[0.5531, 1.9975]+3.79040.738

Narrower coverage slightly reduces override frequency. Reducing bank size lowers storage and update time while preserving positive predictive gains, and mean NLL gains remain similar across 7 to 29 training participants at a fixed bank size. A quarter-size bank therefore preserves positive NLL gains both overall and at H64 while reducing storage and update time, and the fixed-size comparison shows similar mean NLL gains with as few as seven training participants.

AuthorsOpenReview ↗

Response to Reviewer Vzi9 [2/4]

2. [W1 & Q2] Improvement magnitude and simpler utility estimators

We agree that the Assembly101 Action NLL reduction from 3.6315 to 3.6291 is modest. So we test whether simpler utility estimators can match those gains. To test the role of SPACE retrieval on Assembly101, we trained Ridge (α=1), a 64/32-unit ReLU MLP, and histogram GBDT (80 rounds, up to 15 leaves) on the same training descriptors and counterfactual targets. With inherited guards, each estimator replaces the retrieved utility mean and retains bank-derived uncertainty, support, and the other gates. Bank-free point decisions use a fixed threshold on the predicted utility advantage. Both variants compute the candidate descriptors, and all gains are relative to Reservoir in this estimator experiment.

Estimator and decision ruleAction NLL (mean ± seed SD) ↓NLL gain (×10⁻³) ↑H64 gain (×10⁻³) ↑Override rate (%)
Reservoir3.6423 ± 0.02280.00000.00000.000
SPACE retrieval3.6405 ± 0.02111.7725+5.40611.022
Ridge with inherited guards3.6428 ± 0.0234−0.5534−1.28350.747
MLP with inherited guards3.6423 ± 0.0248+0.0101+0.49504.280
GBDT with inherited guards3.6426 ± 0.0226−0.2836−0.24740.080
Ridge with bank-free point decisions3.6658 ± 0.0236−23.4800−42.48208.203
MLP with bank-free point decisions3.6533 ± 0.0214−10.9800−3.974031.590
GBDT with bank-free point decisions3.6503 ± 0.0218−8.0570−13.61006.000

With the same guards, retrieval yields the largest mean Action NLL and H64 gains. All three bank-free point variants increase Action NLL relative to Reservoir. The tested simpler utility estimators, Ridge, MLP, and GBDT estimators, therefore fall short of reproducing SPACE's predictive gains, even when they retain the original bank-derived guards.

3. [W2 & Q3] Threshold selection and cross-dataset uncertainty

We agree that inference on the original Assembly101 validation split is post-selection. We evaluated both methods on reserved recordings excluded from threshold selection, keeping the original thresholds and learned assets fixed. Across three seeds and nine participants, SPACE obtains Action NLL 3.580925 ± 0.025777, versus 3.582790 ± 0.025717 for Reservoir. The paired gain is +0.001865, with a participant-clustered 95% CI of [+0.000961, +0.002755]. Eight of nine participants had positive gains. The positive gain therefore persists on data excluded from threshold selection.

We report mean ± SD across seeds for Action NLL and paired H64 gain over Reservoir on each additional dataset.

DatasetReservoir NLLSPACE NLLPaired H64 gain
Breakfast2.6472 ± 0.01672.6187 ± 0.0274+0.0328 ± 0.0091
IKEA ASM2.5689 ± 0.01802.4891 ± 0.0255+0.0877 ± 0.0051
EPIC-KITCHENS-1004.3469 ± 0.02374.3239 ± 0.0307+0.0433 ± 0.0094

The full cross-dataset results will appear later in Tables 8–10 of Appendix D in the revised PDF.

AuthorsOpenReview ↗

Response to Reviewer Vzi9 [3/4]

4. [W3 & Q3] Independent evaluation and behavioral meaning of drift

We thank the reviewer for asking whether the gains persist with a different visual representation and whether drift corresponds to observable changes in prediction behavior. In a matched-camera replication, we rebuilt the TSM controller using the original training recipe and thresholds. We trained a separate DINOv2-based evaluator with a slot MLP, causal GRU, and action decoder on the training recordings. It scores the frozen memory event IDs after the controller has made its decisions, using a different visual representation and independently trained weights.

Across 57 recordings and 10 participants, with three policy and three evaluator seeds, SPACE reduces audit Action NLL from 3.4848 ± 0.0003 for Reservoir to 3.4829 ± 0.0020. The paired NLL gains at H16 and H64 are +0.005957 ± 0.000151 and +0.005031 ± 0.000125, respectively.

MethodAudit Action NLL ↓H16 gain (×10⁻³) ↑H64 gain (×10⁻³) ↑Audit drift gain (×10⁻⁵) ↑Audit boundary selectivity ↑
Reservoir3.4848 ± 0.000340.0000 ± 0.00000.0000 ± 0.00000.0000 ± 0.00000.59775 ± 0.00061
Utility-only3.4912 ± 0.00204−5.1556 ± 0.9719−4.3252 ± 1.5135+4.9008 ± 0.74550.59682 ± 0.00115
State-only3.5342 ± 0.00348−4.8197 ± 0.3838−3.5662 ± 0.3501+5.6550 ± 0.42100.59654 ± 0.00059
SPACE3.4829 ± 0.00202+5.9570 ± 0.1507+5.0310 ± 0.1254+5.5195 ± 0.25640.59898 ± 0.00038

Using the same evaluator, we compared H1 and H4 predictions between adjacent observations with unchanged current and future action labels. We also evaluated H1 predictions over the 16 updates following an action boundary.

AnalysisMetricReservoirUtility-onlyState-onlySPACE
H1 StableJSD (×10⁻⁶) ↓23.560 ± 0.023323.340 ± 0.027423.049 ± 0.021823.098 ± 0.0153
Dropout / 100 pairs ↓0.7232 ± 0.02490.7094 ± 0.01030.8043 ± 0.01860.6951 ± 0.0156
Action NLL ↓3.0559 ± 0.00073.0673 ± 0.00363.1316 ± 0.00513.0519 ± 0.0030
Latent Drift (×10⁻³) ↓1.8444 ± 0.00051.8388 ± 0.00721.8354 ± 0.00331.8343 ± 0.0027
H4 StableJSD (×10⁻⁶) ↓20.401 ± 0.077620.250 ± 0.069320.054 ± 0.009920.013 ± 0.0946
Dropout / 100 pairs ↓0.6827 ± 0.01710.6731 ± 0.00440.6703 ± 0.02100.6671 ± 0.0200
Action NLL ↓3.1592 ± 0.00103.1711 ± 0.00293.2315 ± 0.00423.1544 ± 0.0023
Latent Drift (×10⁻³) ↓2.0531 ± 0.00062.0477 ± 0.00902.0482 ± 0.00382.0404 ± 0.0032
Boundary16-update Action NLL ↓3.56779 ± 0.00123.57298 ± 0.00173.62054 ± 0.00333.56487 ± 0.0034
Support Reached (%) ↑14.75 ± 0.0114.78 ± 0.0214.66 ± 0.0214.80 ± 0.01

SPACE's gain persists under the independent DINOv2 evaluator. On stable pairs, lower prediction drift accompanies fewer true-label Top-5 dropouts and lower H1 NLL. SPACE also achieves lower H1 NLL over the post-boundary window.

5. [W4 & Q1] Distribution shifts and override attribution

We agree that the Utility Bank should be tested on streams that differ from those used to build it, and that its overrides should be assessed against the Reservoir actions they replace.

a. Downstream Action NLL at override updates

In the headline Assembly101 evaluation, we compared SPACE and Reservoir at matched updates and grouped the paired gains by SPACE's override flag. Positive gains indicate lower NLL for SPACE. Each weighted contribution is the group mean multiplied by its share of updates.

SPACE update groupShare of full-memory updatesMean Action NLL gainWeighted Action NLL contributionMean H64 gainWeighted H64 contribution
Override3.44%+9.025e-03+3.105e-04+3.4498e-02+1.187e-03
Other updates96.56%+2.165e-03+2.091e-03+5.8587e-03+5.657e-03

The mean gain at override updates is 4.17× the gain at other updates for Action NLL and 5.89× for H64. The split shows where gains appear along the paired trajectories. An override can also affect later losses by changing the memory available at subsequent updates.

Reviewer_b2dLOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

SPACE uses a frozen multi-horizon JEPA, an offline counterfactual-utility bank, and predictive-state stabilization to control eviction in fixed-capacity video memory. Its MABS-Bench evaluates future prediction, stability, boundary adaptation, and perturbation recovery across four datasets.

Critical strengths
  • Separating utility admissibility from stabilization addresses the stability–adaptation trade-off; component ablations support combining both signals.
  • Shared streams and predictors, three complete seeds, additional datasets, and independent-evaluator audits strengthen the evidence.
Critical weaknesses
  • Assembly101 thresholds are selected on the same validation split used for headline results; statistical inference is therefore post-selection. Cross-dataset results lack uncertainty estimates
  • Action NLL improves from 3.6315 to 3.6291, while updates take 9.87ms and require substantial utility-bank storage. These feel very marginal.
Questions for authors

The authors should consider discussing/citing some other memory compression in video generation papers, including:

  • Long-term Consistent World Simulation with Memory. (NeurIPS 2025)
  • Mixture of Contexts for Long Video Generation. (ICLR 2026)
  • SANA-Video: Efficient Video Generation with Block Linear Diffusion Transformer. (ICLR 2026)
  • Pretraining Frame Preservation in Autoregressive Video Memory Compression. (arXiv 2026)
  • Moga: Mixture-of-groups attention for end-to-end long video generation. (ICLR 2026)
  • StreamMem: Query-Agnostic KV Cache Memory for Streaming Video Understanding. (arXiv 2025)
2 comments
AuthorsOpenReview ↗

Response to Reviewer b2dL

We thank the reviewer for recognizing the separation of utility admissibility and stabilization, and the value of our shared-stream evaluation, component ablations, and independent audits.

1. Threshold selection and independent confirmation

We agree that inference on the original threshold-selection split is post-selection. We therefore evaluated reserved recordings excluded from threshold selection, freezing the original thresholds, predictor, Utility Bank, and other learned assets, with no retuning. Across three seeds and nine participants, SPACE achieves Action NLL 3.580925 ± 0.025777, versus 3.582790 ± 0.025717 for Reservoir. The paired gain is +0.001865, with a participant-clustered 95% CI of [+0.000961, +0.002755], and is positive for 8/9 participants. The positive gain therefore persists on data excluded from threshold selection.

2. Cross-dataset uncertainty and improvement magnitude

We report mean ± SD across seeds for Action NLL and for the paired H64 gain over Reservoir on each dataset.

DatasetReservoir NLLSPACE NLLPaired H64 gain
Breakfast2.6472 ± 0.01672.6187 ± 0.0274+0.0328 ± 0.0091
IKEA ASM2.5689 ± 0.01802.4891 ± 0.0255+0.0877 ± 0.0051
EPIC-KITCHENS-1004.3469 ± 0.02374.3239 ± 0.0307+0.0433 ± 0.0094

SPACE improves mean NLL/H64 over Reservoir on all three datasets. The full cross-dataset results will appear later in Tables 8–10 of Appendix D in the revised PDF.

We agree that the Assembly101 improvement in Table 1 is modest in absolute terms (3.6315 to 3.6291 in Action NLL), and that the 9.87 ms update time and additional bank storage matter. The gain persists on reserved Assembly101 recordings with the thresholds frozen. The three additional datasets also show larger mean NLL reductions over Reservoir and positive paired H64 gains. In the estimator ablation, replacing the retrieved utility mean with Ridge, MLP, or GBDT while keeping the original guards yielded a best mean NLL gain of +0.0000101, compared with +0.001773 for retrieval.

In a separate Assembly101 ablation, we varied the size of the Utility Bank using nested subsets stratified by training participant and source policy (FIFO or Reservoir). We evaluated each size on the same 30 validation recordings from five participants, holding the predictor, controller thresholds, and memory-slot count fixed.

ConfigurationBank tensors (MiB)Action NLL ↓NLL gain (×10⁻³) ↑Participant 95% CI (×10⁻³)H64 gain (×10⁻³) ↑Override rate (%)Policy time (ms/update)
Reservoir0.03.6423 ± 0.02280.0000 ± 0.0000—0.00000.0000.77
SPACE, 12.5% bank92.53.6411 ± 0.0219+1.1732 ± 0.9945[0.6384, 1.7914]+3.26480.6307.87
SPACE, 25% bank185.03.6404 ± 0.0227+1.9318 ± 0.4695[0.9171, 2.9450]+4.20630.7598.07
SPACE, full bank740.03.6405 ± 0.0211+1.7725 ± 1.7283[0.0139, 3.4228]+5.40611.0229.66

Retaining 25% of the bank reduced tensor storage from 740 to 185 MiB and policy time from 9.66 to 8.07 ms per update, while yielding a paired Action NLL gain of +0.001932 over Reservoir. Reservoir took 0.77 ms under the same timing protocol. We see SPACE as an option for streaming applications that prioritize long-horizon prediction under a fixed memory-slot budget and can accommodate the additional update time.

3. Related work in the revised PDF

Thank you for pointing out these relevant works and for helping us broaden the related-work coverage. We will expand the revised PDF to discuss and cite all the suggested works—WorldMem, Mixture of Contexts, SANA-Video, Pretraining Frame Preservation, MoGA, and StreamMem. We will clarify the relationship between their approaches to long-video memory and SPACE's control of discrete evictions for future-prediction sufficiency.

I do not have many problem with this paper and intend to keep my score. That being said, I am not an expert on this area, so my assessment is with of confidence of 2.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response [1/2]

1. "The improvement is marginal."

We thank the reviewers for recognizing the novelty of our method, and we agree that the absolute performance improvement of SPACE is modest relative to its additional computation and storage. Our claim is instead that the added components provide consistent predictive-memory benefits that are not reproduced by the tested simpler alternatives. Compared with Reservoir, the mean Action NLL gains are +0.0285 on Breakfast, +0.0798 on IKEA ASM, and +0.0230 on EPIC-KITCHENS-100. Mean paired H64 gains range from +0.0328 to +0.0877 across these three datasets. The reserved Assembly101 evaluation with frozen thresholds and the independent DINOv2 audit each show an Action NLL gain of about +0.0019.

We tested Ridge, MLP, and GBDT on the same training descriptors and counterfactual targets. When replacing the retrieved utility mean while retaining the original bank-derived guards, their Action NLL gains over Reservoir are −0.000553, +0.0000101, and −0.000284, respectively, compared with +0.001773 for SPACE retrieval. All three bank-free point variants increase Action NLL relative to Reservoir. The tested simpler estimators therefore do not reproduce SPACE’s predictive gains.

The storage cost is also tunable: reducing the Utility Bank to 25% (185 MiB) or 12.5% (92.5 MiB) still retains positive NLL gains of +0.00193 and +0.00117, respectively. SPACE does not require an additional end-to-end controller training stage, since its slow basis and Utility Bank are constructed offline after freezing the shared predictor.

To assess whether the gains depend on the controller’s own predictor, we scored frozen memory decisions using a separately trained DINOv2-based evaluator with a different visual representation. SPACE retains positive Action NLL and H64 gains under this evaluator. It also reduces true-label Top-5 dropouts on stable observation pairs and H1 NLL during the post-boundary window, linking predictive-state stability to observable prediction behavior.

We therefore view SPACE as a complementary predictive-state-control formulation with modest but systematic gains and an adjustable computation–storage trade-off, and we hope this memory-eviction perspective can be explored more broadly in memory-augmented models.

2. Threshold selection and statistical independence

We agree that evaluating a controller on the same validation data used for threshold selection can raise concerns about statistical independence. We therefore performed an additional confirmation experiment using held-out reserved recordings that were not used for threshold selection, freezing the original thresholds, predictor, Utility Bank, and other learned assets, with no retuning.

Across three policy seeds and nine participants, SPACE achieves an Action NLL of 3.5809 ± 0.0258, compared with 3.5828 ± 0.0257 for Reservoir, corresponding to a gain of +0.00187. The participant-clustered 95% CI is [+0.000961, +0.002755], and the gain is positive for 8/9 participants. SPACE’s predictive gain therefore persists on recordings excluded from threshold selection, providing independent confirmation with the original thresholds and learned assets frozen.

AuthorsOpenReview ↗

General Response [2/2]

3. Utility Bank reliability under distribution shifts

We thank the reviewers for raising the question of whether the Utility Bank remains useful when stream content and observation rates change. We froze the bank, predictor, and controller thresholds and tested three shifts within Assembly101. Unseen configurations correspond to toy IDs absent from the training bank. Doubled observation stride retains every other observation from the standard input stream, with prediction horizons counted in received observations.

SPACE achieves paired Action NLL gains over Reservoir of +0.001028 on unseen configurations, +0.001747 with doubled stride, and +0.000923 when both shifts occur together. The participant-clustered 95% CIs are above zero in all three shifted settings, including [+0.000729, +0.001118] under the combined shift. Utility-ranking Spearman correlations remain around 0.315–0.320 across the four evaluation conditions, supporting the bank’s ability to rank candidate actions under the tested shifts.

Together, these results support SPACE’s robustness under the tested distribution shifts, with predictive benefits preserved even when assembly configuration and observation rate change simultaneously.

4. Retrospective utility of memory-slot decisions

We thank the reviewers for suggesting a retrospective assessment of individual memory slots and the overrides SPACE executes. We measure a slot’s conditional predictive utility through the effect of removing it at a fixed decision state, using observed futures.

At 960 sampled memory states from one policy seed, we evaluated all 17 removal actions at K=16, including discarding the incoming observation. Utility combines equally weighted, training-normalized one-step and 32-step prediction costs. SPACE achieves utility-sign accuracy of 61.98%, compared with 51.29% for the majority-sign baseline, and Hit@1 of 13.53%, compared with 5.88% for random ranking.

Among 209 executed overrides with complete retrospective targets, 58.85% improve utility over the Reservoir action at the same memory state (participant-clustered 95% CI [55.06%, 62.99%]). The mean local utility gain is +0.4166 (95% CI [+0.1188, +0.9328]) under this normalized prediction cost.

These audits show that SPACE identifies beneficial removals more accurately than the reference baselines, and its executed overrides improve local predictive utility on average.

5. Revisions to the manuscript

We thank the reviewers for their suggestions and have incorporated the following changes into the revised PDF.

  • Related work. Following Reviewer b2dL’s suggestions, we have added and discussed all six recommended papers on video memory and compression.
  • Method presentation. Following Reviewer VQop’s feedback, we have revised the Method section to clarify each component’s motivation and role, connecting them through the decision to follow or override Reservoir at a full-memory update.
  • Experimental appendix. We have updated the full cross-dataset tables with uncertainty estimates and incorporated the additional rebuttal experiments, including their settings, detailed results, and analyses.

The revised PDF makes the method’s decision process clearer and documents the additional experimental evidence, with revisions marked in blue.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.