- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
MeMix proposes a training-free state preservation mechanism for recurrent streaming 3D reconstruction models. After computing the candidate state and current-frame readout, the method measures compatibility between candidate state tokens and the mean projected image feature, freezing a small fraction of old state entries while allowing the rest to follow the native update rule. The authors apply this masking strategy across direct, soft-gated, and covariance-aware updates (built around the CUT3R state family) and evaluate it on long-horizon 3D reconstruction and pose estimation benchmarks with minimal runtime overhead.
Critical strengths
-
The proposed intervention is plug-and-play—requiring no retraining or extra parameters—and the before/after comparisons are cleanly set up using identical checkpoints, inputs, and evaluators. It adds almost no computational overhead while delivering clear error reductions on long sequences (such as the 1,000-view TTSA3R benchmark) and generalizing to covariance-aware updates like FILT3R.
-
I really appreciate the thorough sanity checks and honest reporting of limitations. The authors rule out trivial explanations by comparing against random masks, periodic updates, reverse allocation, stale masks, and continuous write-strength sweeps, while using scene-level bootstrap resampling rather than treating correlated frames as independent. The paper is also upfront about short-horizon trade-offs and very long-horizon failure cases instead of cherry-picking only positive results.
Critical weaknesses
-
Because all tested update rules (CUT3R, TTT3R, TTSA3R, FILT3R) rely on the shared CUT3R pretrained backbone and state representation, it remains unclear whether this compatibility heuristic generalizes to other recurrent 3D architectures. Moreover, the feature-compatibility score is purely empirical, and the theoretical analysis—while transparent about its assumptions—does not guarantee downstream reconstruction gains. In practice, hard-freezing old state entries can become a liability if the initial state was poorly estimated or the scene is dynamic, which likely contributes to the regressions observed on very long trajectories. In addition, the write-strength sweep suggests that strong update damping works nearly as well, meaning exact zero-write preservation is not strictly required.
-
A few experimental inconsistencies and missing scripts need to be cleared up. First, the default TUM numbers (even for the unmodified baseline) do not match between Table 3 and the write-strength ablation, which needs an explanation of whether different sequence subsets or settings were used. It is also unclear whether the default preservation ratio was tuned on a held-out split or directly on the test benchmarks. Second, while the provided code covers the base CUT3R/TTT3R/TTSA3R evaluations, the scripts and commands needed to reproduce the FILT3R experiments, the 5-scene 3,000-view stress test, and the paired bootstrap intervals are missing. Finally, because ATE is computed after prefix-specific Sim(3) alignment, it is fair across methods but should not be over-interpreted as maintaining a globally consistent metric scale over time.
Questions for authors
-
On which specific sequences and frame horizons do the longer-run regressions occur, and do they stem from locking in noisy early states or actual scene changes?
-
How was the default preservation ratio chosen—was it tuned on a separate validation split?
-
Why do the baseline and default MeMix numbers on TUM differ between Table 3 and the write-strength sweep?
-
Could you provide the exact code scripts and commands for the FILT3R integration, the 3,000-view experiment, and the paired bootstrap resampling, as well as clarify which sequences were used to compute the 79.3% retained-alignment statistic?
4 comments
Q3 TUM numbers · Q2 choice of p = 60 · Q4 scripts and the 79.3% statistic · Sim(3) · W2 empirical score
TUM numbers in Table 3 vs Table 5 (Q3). No sequence subset differed: both tables use the same eight streams, 7,023 views, and evaluator. Table 5's TUM column came from a separate, earlier run. Its λ = 0 and λ = 1 arms, mathematically identical to MeMix and TTSA3R, gave 0.0793 and 0.0945 instead of 0.0767 (Table 3) and 0.0951 (Appendix A4). We reran the TUM sweep on the released code path: its λ = 0 and λ = 1 trajectories are byte-identical to the MeMix and TTSA3R runs on all eight streams, and Table 5's TUM column is replaced:
| λ | 0 | 0.1 | 0.25 | 0.5 | 1 |
|---|---|---|---|---|---|
| TUM ATE (new column) | 0.0767 | 0.0851 | 0.0919 | 0.0876 | 0.0951 |
| Δ vs λ = 0, 95% CI | — | [−0.0009, 0.0210] | [−0.0000, 0.0355] | [0.0042, 0.0197] | [0.0065, 0.0319] |
The conclusions are unchanged: the new column orders the five λ values exactly as the old one did, no mean moves by more than 0.0026 m (a seventh of the TTSA3R − MeMix difference, 0.0184 m), and the per-stream slope of ATE on λ is positive on all eight streams (Appendix A9). On the released code path, the TUM MeMix and TTSA3R trajectories have since been reproduced byte-for-byte in three separate batches, and Table 5's 7-Scenes column matches Table 3 and Section 4.1 exactly. Only that one run deviated: its trajectories differ from the released-path runs by about 10⁻⁶ m at the first frame, and the difference accumulates along each stream; we could not trace its origin.
The default p = 60 (Q2). No separate validation split was used: p = 60 was fixed during development, alongside the short-sequence budget sweep of Appendix A3, and kept unchanged across benchmarks, update rules, and horizons, so the experiments added afterwards (1,000-view 7-Scenes, ScanNet through 4,000 views, FILT3R, Mem3R, and all controls) test a pre-fixed value. Post hoc sensitivity (* = paired interval against p = 60 excludes zero):
| Budget p | TUM ATE | 7-Scenes Acc / Comp | FILT3R ScanNet ATE |
|---|---|---|---|
| none (TTSA3R; FILT3R) | 0.0951* | 0.0892* / 0.0292* | 0.2886* |
| 30 | 0.0845* | 0.0725* / 0.0221 | 0.2529* |
| 60 (default) | 0.0767 | 0.0625 / 0.0202 | 0.2372 |
| 120 | 0.0830* | 0.0613 / 0.0272 | 0.2261 |
Every budget has lower mean error than the unmodified rule; p = 30 is worse than p = 60 on all three benchmarks (on 7-Scenes, in accuracy), and p = 120 is worse on TUM, with intervals against p = 60 that include zero on the other two. Section 4 states the provenance; Appendices A4 and A9 give the contrasts.
Scripts (Q4). The updated supplement contains the covariance-consistent FILT3R integration, the ScanNet 3,000-view runners and analyzers, the paired stream bootstrap (100,000 resamples, fixed seed), the Mem3R integration with its TUM and ScanNet runners, and the runners, protocols, and analyzers behind every new control and rerun (gate-ranked protection, the gate-overlap trace, uniform damping alone and with MeMix, the full Figure 3 rerun, and the Table 2 rerun on 7-Scenes at 500 views), with commands in the README; the 4,000-view scripts are included.
The 79.3% statistic (Q4). All eight TUM-dynamics streams (fr3 sitting/walking × halfsphere/rpy/static/xyz), first min(1000, available) frames, 7,023 frames in total. On 79.3% of these frames, the mean old-state compatibility of the 60 protected locations exceeds that of the 708 updated locations, with a positive mean difference in all eight streams. Appendix A4 now states the comparison and this cohort.
Per-prefix Sim(3). Section 4 now states this explicitly: per-prefix Sim(3) ATE measures trajectory-shape error at each horizon, not the consistency of one global metric scale over time.
Empirical score (W2). The claims rest on measurement and intervention, not on the analysis: the revision measures what the score selects and tests each element of the rule. The score is the agreement of a location's proposed new content with the current frame, and it splits exactly (Eq. 10) into the old content's agreement plus that of the proposed change. On 7,023 TUM frames, the 64 top-ranked locations exceed the 64 bottom-ranked by 2.12 within-frame SDs of old-state alignment (8/8 streams; 95% CI [2.01, 2.22]; Appendix A4), and a passive trace shows that the update rule would rewrite the 60 protected locations strongly (87th/85th native-gate percentile on TUM/ScanNet; Appendix A10). Each of the following matched interventions is worse than MeMix in TUM ATE with an interval excluding zero: reverse, random, periodic, old-state scoring, stale masks, and now the model's own gate ranking (the last also in 7-Scenes accuracy and 65-scene ScanNet ATE).
Thanks for the detailed response, I have no more questions. And I will preserve my positive rating.
Thank you for reading our response and for your support.
W1 shared backbone · Q1 / W3 hard-freezing and long-run regressions · W4 exactness
Thank you for the careful check and the supportive assessment. In short: applied unchanged to Mem3R, a separately trained model with its own learned gate, MeMix lowers ScanNet ATE by 10.5% on 65 scenes (interval excluding zero). The Table 5 TUM numbers, the choice of p, the scripts, Sim(3), and the empirical score are in our next comment.
Beyond the shared CUT3R checkpoint (W1). The four update rules we test are structurally different (direct replacement, two soft gates, and a Kalman gain with covariance), and on the same 65 ScanNet scenes MeMix lowers long-horizon ATE under all four with every paired interval excluding zero (CUT3R/TTT3R/TTSA3R 50/65, 54/65, 58/65 scenes for the 400–1,000-view mean; FILT3R 54/65 at 1,000 views; Appendix A9). To go beyond the shared checkpoint, we tested Mem3R (code released after our submission). It writes its 768-token state through a learned channel-wise gate and adds a fast-weight memory for camera tracking; it is initialized from CUT3R's weights and trained further with the encoder frozen. MeMix sets that gate to zero at the 60 protected tokens. We used the authors' public code and 64-view checkpoint. As a check of this setup, our runs on the eight TUM streams against the values reported by the Mem3R authors (ATE, m):
Applied unchanged, MeMix lowers Mem3R's mean ATE on both benchmarks (Δ = Mem3R − Mem3R+MeMix):
On ScanNet the interval excludes zero at every prefix from 300 views on; on TUM's eight streams the interval includes zero. MeMix thus transfers on ScanNet to a separately trained model with its own learned gate (Section 4.2; Appendix A11); recurrent models not derived from CUT3R remain untested.
Hard-freezing and long-run regressions (Q1 / W3). Where and when. Through 3,000 views MeMix lowers ATE on all five long ScanNet scenes at each evaluated horizon (15/15 scene–horizon cells; scene 0747 is tied at 3,000 views). The regressions come later, on the three scenes that extend to 4,000 views: scene 0747 is worse from 3,100 views on (by up to 44.4%, at 3,700 views) and turns the three-scene mean into a loss after 3,400 views (mean ATE at most +9.4%, at 3,700 views; +4.8% at 4,000), while the other two scenes keep lower ATE through 3,900 views. The per-scene table is in our second reply to H7Yz and in Appendix A5 (Table 7).
Cause. Not scene change: the regression occurs on static ScanNet scenes, while on TUM's dynamic sitting/walking streams MeMix has lower ATE than TTSA3R on all eight at every horizon from 800 to 1,000 views. Against a locked-in noisy early state the evidence is indirect: scene 0747 is 37.8% better with MeMix at 1,000 views and 24.5% better at 2,000 before it turns after 3,000, and the protected set is reselected every frame (a fixed initial mask is worse, ATE 0.0979 vs 0.0767). Releasing the protection after 3,000 writes does not recover the ATE benefit (three-scene mean 0.0714 m higher at 4,000 views than with continued protection; higher on 2 of 3 scenes), so protection late in the stream does not explain the ATE reversal, although the release does remove most of a late rise in rotational RPE (Appendix A5). Weaker suppression from the start (λ = 0.1) keeps the three-scene mean below TTSA3R at every evaluated horizon (−4.7% at 4,000 views; −6.6% vs exact zero's −10.3% at 3,000). With three scenes, the cause is not identified.
Exactness (W4). What MeMix adds is the choice of locations: at the same budget, protecting the 60 locations the update rule itself would write most strongly is worse than MeMix in TUM ATE, both 7-Scenes errors, and 65-scene ScanNet ATE (0.332 vs 0.300 m), all intervals excluding zero. What the selected locations need is strong suppression; exact zero is its parameter-free default, not a unique optimum, as Section 4.4 and the Table 5 caption now state: on 7-Scenes, the λ = 0.1 interval against exact zero includes zero (accuracy Δ −0.0025 [−0.0070, 0.0017]) while the full native write is worse (+0.0267 [0.0154, 0.0392]); on TUM, the intervals for λ = 0.1 and 0.25 include zero and λ = 0.5 and 1 are worse (next reply; Appendix A9).