- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
SpaceCache is a training-free approach for improving the inference efficiency of pretrained 3D point cloud backbones such as Utonia and LitePT-B by caching and reusing tokens in consecutive frames. The authors observe that spatial regions at different timesteps have consistent semantic representations. Thus, tokens of previously observed regions can be skipped in some encoder stages, and their features are reused from the cache. The experiments show improved inference efficiency with small drops in segmentation accuracy.
Critical strengths
- The proposed approach is simple, intuitive, and does not require additional training.
- It achieves meaningful inference speedups while mostly preserving segmentation performance.
- The paper provides an interesting initial investigation into when and how intermediate features can be reused across point cloud streams to improve efficiency.
- The implementation is provided in the supplementary material.
Critical weaknesses
- The speed-up improvements are modest in most settings (20%). Given the additional cache management and dependency on temporal correspondences, the practical benefits appear somewhat limited.
- The evaluated token-merging baselines use redundancy within individual frames, whereas SpaceCache benefits from inter-frame redundancy. This requires accurate poses to establish correspondences between frames. The sensitivity to pose estimation errors and the cost of obtaining these poses should be considered when assessing the practical efficiency of the approach.
- The ScanNet experimental results are not similar to current state-of-the-art, as the trained models use complete fused scenes rather than individual RGB-D observations. The baseline segmentation accuracy of 50.63 mIoU is thus low. It would be valuable to evaluate a model trained or fine-tuned directly on single-view RGB-D point clouds.
- The ScanNet experiments are conducted on only 10 validation scenes. This is a relatively small subset of the dataset (312 validation scenes).
- The related work could be expanded to discuss temporal computation reuse in video transformers, particularly Eventful Transformers: Leveraging Temporal Redundancy in Vision Transformers (ICCV 2023), which explores a related motivation in 2D vision. In addition, Volume Transformer: Revisiting Vanilla Transformers for 3D Scene Understanding (ECCV 2026) is a relevant recent 3D backbone.
Questions for authors
- As I understand it, cache-hit tokens are removed from the computation within the selected transformer blocks, and their cached features are restored afterward. However, removing a substantial fraction of tokens should also change the neighborhood available to the remaining tokens, both in sparse convolutions and attention. I would expect this to affect their representations, even if the cached features themselves remain semantically consistent. Could the authors explain why the impact on segmentation accuracy is relatively small? An analysis of the feature changes or prediction accuracy of cache-miss tokens would be interesting.
- Have the authors considered applying SpaceCache to the most recent state-of-the-art Volume Transformer (ECCV 2026)? Since this model uses global attention with quadratic complexity in the token count, I would expect potentially larger computational benefits from skipping cached tokens. Evaluating such a backbone could also help demonstrate the generality of the proposed approach.
- It would be useful to understand which semantic classes are most affected by feature reuse. Although Table 3 provides an analysis of dynamic and static regions, a per-class IoU comparison could reveal whether particular categories, especially small or dynamic objects, suffer disproportionately.
- SpaceCache achieves substantially larger speedups with LitePT-B than with Utonia on Waymo. What architectural differences explain this behavior?
- What motivated the use of the self-supervised Utonia backbone rather than a stronger supervised baseline such as Point Transformer V3 or a fine-tuned Utonia model?
8 comments
Response to Reviewer 5yff (2)
W1 The speed-up improvements are modest in most settings (20%). Given the additional cache management and dependency on temporal correspondences, the practical benefits appear somewhat limited.
TL;DR. The lower end of our speedup (1.25×) is indeed the ~20% the reviewer notes, but it is the largest among training-free methods (1.25–1.56× vs. 0.92–1.14×). Cache management costs only about 3 ms per frame, and the pose costs little (W2). Revision: Sec. 3.4 (L305–312); Sec. 4.2 (L476–478); Sec. 4.3 (L482–491); Table 11 (App. E.3, L1232–1265).
Answer. We address the reviewer's three concerns in turn: the size of the speedup, the cost of cache management, and the dependency on temporal correspondence.
(1) Size of the speedup. We agree that 1.25× at the lower end, a 20% latency reduction, looks modest in absolute terms. We clarify below why this gain is still meaningful.
Relative to alternatives, SpaceCache is faster than every training-free baseline. At the default setting (θ=0.85 on Waymo and 0.98 on nuScenes, L=8), SpaceCache reaches 1.25–1.56×. ToMe, ALGM, and GitMerge3D reach only 0.92–1.14×, at equal or lower accuracy (Figure 6). In this setting, 20% is therefore not a small gain.
The speedup can also be combined with other methods. SpaceCache reuses features across frames, so merging or quantization within a frame can further speed up the uncached parts (Stages 0–2 and the decoder) and the cache-miss tokens. We leave this to future work.
The FLOPs reduction leaves further room. At the default setting, SpaceCache removes 28.7–45.7% of the FLOPs. If latency fell in proportion, this would give 1.40–1.84×. The measured 1.25–1.56× is lower, and kernel fusion or CUDA graphs can narrow this gap.
(2) Cost of cache management. Cache management adds only about 3 ms per frame, which is negligible. This covers all cache operations: coordinate transformation, binary-search lookup, feature assembly, and the Commit Gate update. We keep the cost low by fusing the Commit Gate update into fewer kernel launches and host reads.
(3) Dependency on temporal correspondence. The pose is already provided by driving stacks. As discussed in W2, driving stacks estimate the pose at 50–200 Hz, faster than the 10–20 Hz LiDAR, so SpaceCache reads an available pose for every frame. Even with a LiDAR-only pose or 0.2 m / 0.2° pose noise, mIoU drops by at most 0.34 points (W2).
Q4 SpaceCache achieves substantially larger speedups with LitePT-B than with Utonia on Waymo. What architectural differences explain this behavior?
Q5 What motivated the use of the self-supervised Utonia backbone rather than a stronger supervised baseline such as Point Transformer V3 or a fine-tuned Utonia model?
TL;DR. SpaceCache's speedup follows how each backbone distributes its computation across stages, and LitePT-B is the supervised backbone in our evaluation. Revision: Sec. 4 (L360–361); App. E.5 (L1308–1311); Table 13 (App. E.5, L1296–1332).
Answer. LitePT-B places 90% of its FLOPs in the cached Stages 3–4, against 64% in Utonia, so SpaceCache speeds it up more (1.56× vs. 1.38× on Waymo). Our Utonia is the pretrained encoder with a fine-tuned decoder, which the revised Sec. 4 now states.
Why LitePT-B gains more than Utonia (Q4). LitePT-B places attention only in Stages 3–4 (90% of FLOPs), whereas Utonia uses transformer blocks across every stage and the decoder (64% in Stages 3–4). With similar Stage-3 token hit rates (53% vs. 49%), SpaceCache removes 45.7% vs. 33.5% of FLOPs, giving 1.56× vs. 1.38× on Waymo.
The Utonia we use (Q5). We freeze the pretrained Utonia encoder and fine-tune a transformer decoder per dataset (now stated in revised Sec. 4), pairing one large-scale pretrained backbone with one efficient supervised backbone (LitePT-B, higher mIoU in Table 1). Fully supervised PTv3 results are in Table 13 (App. E.5).
Response to Reviewer 5yff (3)
W3 The ScanNet experimental results are not similar to current state-of-the-art, as the trained models use complete fused scenes rather than individual RGB-D observations. The baseline segmentation accuracy of 50.63 mIoU is thus low. It would be valuable to evaluate a model trained or fine-tuned directly on single-view RGB-D point clouds.
W4 The ScanNet experiments are conducted on only 10 validation scenes. This is a relatively small subset of the dataset (312 validation scenes).
TL;DR. With the revised evaluation, SpaceCache runs 1.24–1.25× faster than the backbone, while every baseline stays at or below 1.08×. Revision: Sec. 4 and Table 2 (L321–323, L378–388, L415–420); App. C.1 (L1028–1043).
Answer. We agree with both comments. We now train LitePT-B on ScanNet's single-view split and evaluate it on the full validation set (all 312 scenes). On this setup, the backbone reaches 65.91 mIoU, and SpaceCache remains the fastest method (Table R-3).
Revised setup.
- Training. We train LitePT-B from scratch on ScanNet's official single-view split (
scannet_frames_25k, 19,466 RGB-D frames from the 1,201 training scans) with the paper's recipe. Each depth frame is back-projected and mapped to the world frame by its camera pose, as in our streams. - Evaluation. All 312 validation scenes at 3 Hz (53,193 frames), with every second depth pixel in each direction.
Table R-3. Revised Table 2: comparison on indoor streams (ScanNet, all 312 validation scenes, LitePT-B trained on single RGB-D frames, SpaceCache with L = 5). Each method is shown at its fewest-FLOPs setting within 1-point and 2-point mIoU drops. Latency (ms) is the GPU kernel time on a Jetson AGX Orin, averaged over 50 representative frames. *: 2D method applied to point tokens. †: without the adaptive patch partition. ‡: same setting as within the 1-point drop.
| Method | ≤1.0 pt mIoU↑ | ≤1.0 pt Latency↓ | ≤1.0 pt GFLOPs↓ | ≤1.0 pt Speed↑ | ≤2.0 pt mIoU↑ | ≤2.0 pt Latency↓ | ≤2.0 pt GFLOPs↓ | ≤2.0 pt Speed↑ |
|---|---|---|---|---|---|---|---|---|
| LitePT-B | 65.91 | 39.1 | 47.4 | 1.00× | 65.91 | 39.1 | 47.4 | 1.00× |
| + ToMe* | 64.93 | 38.0 | 32.5 | 1.03× | 63.99 | 37.2 | 30.7 | 1.05× |
| + ALGM* | 65.15 | 37.1 | 38.2 | 1.06× | 64.16 | 36.3 | 37.2 | 1.08× |
| + GitMerge3D† | 65.12 | 42.8 | 43.2 | 0.91× | 64.42 | 42.6 | 42.8 | 0.92× |
| + SpaceCache (Stage 3) | 64.97 | 31.6 | 28.0 | 1.24× | 64.97‡ | 31.6‡ | 28.0‡ | 1.24ׇ |
| + SpaceCache (Stage 3,4) | 64.92 | 32.0 | 27.2 | 1.22× | 64.51 | 31.4 | 26.0 | 1.25× |
‡ Stage-3 stays within 1 point at its largest saving (θ=0), so the 2-point column repeats it.
SpaceCache is the only method above 1.1× in kernel time. It removes 42.6% / 45.3% of the FLOPs (1 / 2 points), against 31.4% / 35.4% for the best baseline (ToMe).
The revised Sec. 4 uses this setup for all indoor results and reports Table R-3 as Table 2, and App. C.1 describes the single-frame model.
Response to Reviewer 5yff (1)
We sincerely thank the reviewer for their insightful comments on our manuscript. We hope we have addressed their concerns and questions regarding the paper and hope they will reconsider their rating. Unless marked as "original", all table, section, equation, and line references below follow the revised PDF.
TL;DR. Our evaluation already uses drifting poses, and SpaceCache stays robust because its finite lifetime L limits drift error (6.6 cm over 4 sec) and larger noise lowers the hit rate (≤0.34 mIoU drop at 5× real error); for pose cost, driving stacks already provide 50–200 Hz poses, and even a CPU LiDAR-only KISS-ICP pose preserves mIoU at ≤4.4 ms overhead. Revision: Sec. 4.2 (L476–478); Table 11 (App. E.3, L1232–1265).
Answer. Our results have already reflected pose drifts. SpaceCache is robust to both pose drift/noise and the cost of pose estimation, as discussed below:
1) Sensitivity to pose noise and drift.
Our dataset pose already drifts. Waymo provides a separate per-frame offset to align poses with its map, and this offset reaches 64 cm within a 20-sec segment. All results in the paper use the uncorrected, drifting pose.
Drift does not accumulate in the cache. A cache entry lives for at most L=8 frames (4 sec), so only the relative pose over 4 sec matters. Over 4 sec, the median drift of the dataset pose is 6.6 cm, well below a 0.4 m Stage-3 region.
Pose noise lowers the hit rate so misaligned tokens are recomputed. Following collaborative perception [1, 2], we inject independent Gaussian noise on x,y (σt) and yaw (σr) of every frame (Table R-1). At 0.2 m / 0.2° (already 5× the real 4 sec error), mIoU drops by only 0.34 and 0.15. As noise grows larger, misaligned tokens miss the cache—the token hit rate of LitePT-B diminishes from 53% to 32%—so SpaceCache computes more tokens and accuracy degrades gradually instead of collapsing.
Table R-1. Pose noise on Waymo, full validation set. SpaceCache with Gaussian noise of the given standard deviation on x, y, and the yaw of every frame pose. Pose error: median (p95) error of the relative pose over 4 sec against the map-aligned pose of Waymo. Hit: cache-hit fraction of Stage 3.
2) Cost of obtaining poses. Driving stacks already estimate poses online (e.g., Apollo [3] at 200 Hz with 5–10 cm RMS error; Autoware's EKF localizer at 50 Hz), so SpaceCache can simply reads an available pose per frame. We conducted an additional experiment with a conservative pose setting where every relative pose comes from LiDAR alone without IMU, GNSS, or map, replacing the dataset pose with CPU LiDAR-only KISS-ICP [4] running in parallel with the GPU network (Table R-2).
Table R-2. SpaceCache with the dataset pose and the KISS-ICP pose on Waymo (Xeon Gold 6326 + RTX A6000; 4 sec pose error 7.4 cm with KISS-ICP vs. 6.6 cm with the dataset pose). KISS-ICP runs on every 10 Hz sweep on 16 CPU threads, in parallel with the network (19.6 ms per sweep alone). Latency (ms) is the network time averaged over frames and includes any wait for the pose. Stage 3: mean time (ms) until a frame reaches Stage 3. ICP: mean (p99) time (ms) of KISS-ICP per sweep. Waiting: fraction of the frames that wait for the pose. Wait: wait time (ms) averaged over all frames.
Accuracy is unchanged and latency rises by at most 4.4 ms (3.5%): SpaceCache needs the pose only at Stage 3, so LitePT-B (40.7 ms to Stage 3) waits briefly on 36.5% of keyframes while Utonia (90.5 ms) never waits.
The revised App. E.3 adds both experiments (Table 11).
[1] Xu et al., V2X-ViT: Vehicle-to-Everything Cooperative Perception with Vision Transformer, ECCV 2022.
[2] Lu et al., Robust Collaborative 3D Object Detection in Presence of Pose Errors, ICRA 2023.
[3] Wan et al., Robust and Precise Vehicle Localization Based on Multi-Sensor Fusion in Diverse City Scenes, ICRA 2018.
[4] Vizzo et al., KISS-ICP: In Defense of Point-to-Point ICP – Simple, Accurate, and Robust Registration If Done the Right Way, RA-L 2023.