acceptodds
Under review as a conference paper at ICLR 2027

SpaceCache: Intermediate Feature Reuse for Efficient Semantic Segmentation on Point Cloud Streams

Abstract

Point-cloud streams are central to autonomous driving and robotics, yet existing segmentation networks process every frame from scratch, although consecutive frames largely overlap. We observe that the intermediate features of point-cloud transformers remain semantically consistent across frames for the same spatial region, especially in deep stages. Building on this, we present SpaceCache, a training-free feature cache that reuses these features across frames. Tokens of previously observed regions skip a sequence of transformer blocks and take their cached features, only the remaining tokens are computed, and the full token layout is restored for dense prediction. Lightweight cache management keeps reused features reliable by limiting each entry’s age and evicting low-confidence entries. Unlike token merging, SpaceCache selects tokens at negligible cost and runs its cache operations once per encoder stage. On both outdoor and indoor benchmarks, such as nuScenes, Waymo, and ScanNet, with two transformer backbones of Utonia and LitePT-B, SpaceCache achieves up to 1.65× speedup with less than 1-point mIoU drop and consistently outperforms token-merging methods.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Point-cloud Stream, Point-cloud Semantic Segmentation, Feature Caching, Efficient Inference, Temporal Redundancy

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_5yffOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

SpaceCache is a training-free approach for improving the inference efficiency of pretrained 3D point cloud backbones such as Utonia and LitePT-B by caching and reusing tokens in consecutive frames. The authors observe that spatial regions at different timesteps have consistent semantic representations. Thus, tokens of previously observed regions can be skipped in some encoder stages, and their features are reused from the cache. The experiments show improved inference efficiency with small drops in segmentation accuracy.

Critical strengths
  • The proposed approach is simple, intuitive, and does not require additional training.
  • It achieves meaningful inference speedups while mostly preserving segmentation performance.
  • The paper provides an interesting initial investigation into when and how intermediate features can be reused across point cloud streams to improve efficiency.
  • The implementation is provided in the supplementary material.
Critical weaknesses
  • The speed-up improvements are modest in most settings (20%). Given the additional cache management and dependency on temporal correspondences, the practical benefits appear somewhat limited.
  • The evaluated token-merging baselines use redundancy within individual frames, whereas SpaceCache benefits from inter-frame redundancy. This requires accurate poses to establish correspondences between frames. The sensitivity to pose estimation errors and the cost of obtaining these poses should be considered when assessing the practical efficiency of the approach.
  • The ScanNet experimental results are not similar to current state-of-the-art, as the trained models use complete fused scenes rather than individual RGB-D observations. The baseline segmentation accuracy of 50.63 mIoU is thus low. It would be valuable to evaluate a model trained or fine-tuned directly on single-view RGB-D point clouds.
  • The ScanNet experiments are conducted on only 10 validation scenes. This is a relatively small subset of the dataset (312 validation scenes).
  • The related work could be expanded to discuss temporal computation reuse in video transformers, particularly Eventful Transformers: Leveraging Temporal Redundancy in Vision Transformers (ICCV 2023), which explores a related motivation in 2D vision. In addition, Volume Transformer: Revisiting Vanilla Transformers for 3D Scene Understanding (ECCV 2026) is a relevant recent 3D backbone.
Questions for authors
  • As I understand it, cache-hit tokens are removed from the computation within the selected transformer blocks, and their cached features are restored afterward. However, removing a substantial fraction of tokens should also change the neighborhood available to the remaining tokens, both in sparse convolutions and attention. I would expect this to affect their representations, even if the cached features themselves remain semantically consistent. Could the authors explain why the impact on segmentation accuracy is relatively small? An analysis of the feature changes or prediction accuracy of cache-miss tokens would be interesting.
  • Have the authors considered applying SpaceCache to the most recent state-of-the-art Volume Transformer (ECCV 2026)? Since this model uses global attention with quadratic complexity in the token count, I would expect potentially larger computational benefits from skipping cached tokens. Evaluating such a backbone could also help demonstrate the generality of the proposed approach.
  • It would be useful to understand which semantic classes are most affected by feature reuse. Although Table 3 provides an analysis of dynamic and static regions, a per-class IoU comparison could reveal whether particular categories, especially small or dynamic objects, suffer disproportionately.
  • SpaceCache achieves substantially larger speedups with LitePT-B than with Utonia on Waymo. What architectural differences explain this behavior?
  • What motivated the use of the self-supervised Utonia backbone rather than a stronger supervised baseline such as Point Transformer V3 or a fine-tuned Utonia model?
8 comments
AuthorsOpenReview ↗

Response to Reviewer 5yff (1)

We sincerely thank the reviewer for their insightful comments on our manuscript. We hope we have addressed their concerns and questions regarding the paper and hope they will reconsider their rating. Unless marked as "original", all table, section, equation, and line references below follow the revised PDF.


W2 The evaluated token-merging baselines use redundancy within individual frames, whereas SpaceCache benefits from inter-frame redundancy. This requires accurate poses to establish correspondences between frames. The sensitivity to pose estimation errors and the cost of obtaining these poses should be considered when assessing the practical efficiency of the approach.

TL;DR. Our evaluation already uses drifting poses, and SpaceCache stays robust because its finite lifetime LL limits drift error (6.6 cm over 4 sec) and larger noise lowers the hit rate (≤0.34 mIoU drop at 5× real error); for pose cost, driving stacks already provide 50–200 Hz poses, and even a CPU LiDAR-only KISS-ICP pose preserves mIoU at ≤4.4 ms overhead. Revision: Sec. 4.2 (L476–478); Table 11 (App. E.3, L1232–1265).

Answer. Our results have already reflected pose drifts. SpaceCache is robust to both pose drift/noise and the cost of pose estimation, as discussed below:

1) Sensitivity to pose noise and drift.

Our dataset pose already drifts. Waymo provides a separate per-frame offset to align poses with its map, and this offset reaches 64 cm within a 20-sec segment. All results in the paper use the uncorrected, drifting pose.

Drift does not accumulate in the cache. A cache entry lives for at most L=8L = 8 frames (4 sec), so only the relative pose over 4 sec matters. Over 4 sec, the median drift of the dataset pose is 6.6 cm, well below a 0.4 m Stage-3 region.

Pose noise lowers the hit rate so misaligned tokens are recomputed. Following collaborative perception [1, 2], we inject independent Gaussian noise on x,yx, y (σt\sigma_t) and yaw (σr\sigma_r) of every frame (Table R-1). At 0.2 m / 0.2° (already 5× the real 4 sec error), mIoU drops by only 0.34 and 0.15. As noise grows larger, misaligned tokens miss the cache—the token hit rate of LitePT-B diminishes from 53% to 32%—so SpaceCache computes more tokens and accuracy degrades gradually instead of collapsing.

Table R-1. Pose noise on Waymo, full validation set. SpaceCache with Gaussian noise of the given standard deviation on x, y, and the yaw of every frame pose. Pose error: median (p95) error of the relative pose over 4 sec against the map-aligned pose of Waymo. Hit: cache-hit fraction of Stage 3.

PosePose error (cm)Utonia mIoU↑Utonia Hit (%)LitePT-B mIoU↑LitePT-B Hit (%)
Dataset6.6 (28.0)66.904972.2153
+ noise 0.2 m, 0.2°37.1 (75.6)66.754271.8744
+ noise 0.4 m, 0.4°70.8 (148.2)66.233769.6337
+ noise 0.6 m, 0.6°105.5 (221.4)65.533368.7432

2) Cost of obtaining poses. Driving stacks already estimate poses online (e.g., Apollo [3] at 200 Hz with 5–10 cm RMS error; Autoware's EKF localizer at 50 Hz), so SpaceCache can simply reads an available pose per frame. We conducted an additional experiment with a conservative pose setting where every relative pose comes from LiDAR alone without IMU, GNSS, or map, replacing the dataset pose with CPU LiDAR-only KISS-ICP [4] running in parallel with the GPU network (Table R-2).

Table R-2. SpaceCache with the dataset pose and the KISS-ICP pose on Waymo (Xeon Gold 6326 + RTX A6000; 4 sec pose error 7.4 cm with KISS-ICP vs. 6.6 cm with the dataset pose). KISS-ICP runs on every 10 Hz sweep on 16 CPU threads, in parallel with the network (19.6 ms per sweep alone). Latency (ms) is the network time averaged over frames and includes any wait for the pose. Stage 3: mean time (ms) until a frame reaches Stage 3. ICP: mean (p99) time (ms) of KISS-ICP per sweep. Waiting: fraction of the frames that wait for the pose. Wait: wait time (ms) averaged over all frames.

BackboneDataset mIoU↑KISS-ICP mIoU↑Dataset Latency↓KISS-ICP Latency↓Stage 3ICPWaiting (%)Wait
Utonia66.9066.93316.6318.690.532.1 (58.1)0.00.00
LitePT-B72.2172.38126.4130.840.738.3 (72.7)36.53.45

Accuracy is unchanged and latency rises by at most 4.4 ms (3.5%): SpaceCache needs the pose only at Stage 3, so LitePT-B (40.7 ms to Stage 3) waits briefly on 36.5% of keyframes while Utonia (90.5 ms) never waits.

The revised App. E.3 adds both experiments (Table 11).

[1] Xu et al., V2X-ViT: Vehicle-to-Everything Cooperative Perception with Vision Transformer, ECCV 2022.
[2] Lu et al., Robust Collaborative 3D Object Detection in Presence of Pose Errors, ICRA 2023.
[3] Wan et al., Robust and Precise Vehicle Localization Based on Multi-Sensor Fusion in Diverse City Scenes, ICRA 2018.
[4] Vizzo et al., KISS-ICP: In Defense of Point-to-Point ICP – Simple, Accurate, and Robust Registration If Done the Right Way, RA-L 2023.

AuthorsOpenReview ↗

Response to Reviewer 5yff (2)

W1 The speed-up improvements are modest in most settings (20%). Given the additional cache management and dependency on temporal correspondences, the practical benefits appear somewhat limited.

TL;DR. The lower end of our speedup (1.25×) is indeed the ~20% the reviewer notes, but it is the largest among training-free methods (1.25–1.56× vs. 0.92–1.14×). Cache management costs only about 3 ms per frame, and the pose costs little (W2). Revision: Sec. 3.4 (L305–312); Sec. 4.2 (L476–478); Sec. 4.3 (L482–491); Table 11 (App. E.3, L1232–1265).

Answer. We address the reviewer's three concerns in turn: the size of the speedup, the cost of cache management, and the dependency on temporal correspondence.

(1) Size of the speedup. We agree that 1.25× at the lower end, a 20% latency reduction, looks modest in absolute terms. We clarify below why this gain is still meaningful.

Relative to alternatives, SpaceCache is faster than every training-free baseline. At the default setting (θ=0.85 on Waymo and 0.98 on nuScenes, L=8), SpaceCache reaches 1.25–1.56×. ToMe, ALGM, and GitMerge3D reach only 0.92–1.14×, at equal or lower accuracy (Figure 6). In this setting, 20% is therefore not a small gain.

The speedup can also be combined with other methods. SpaceCache reuses features across frames, so merging or quantization within a frame can further speed up the uncached parts (Stages 0–2 and the decoder) and the cache-miss tokens. We leave this to future work.

The FLOPs reduction leaves further room. At the default setting, SpaceCache removes 28.7–45.7% of the FLOPs. If latency fell in proportion, this would give 1.40–1.84×. The measured 1.25–1.56× is lower, and kernel fusion or CUDA graphs can narrow this gap.

(2) Cost of cache management. Cache management adds only about 3 ms per frame, which is negligible. This covers all cache operations: coordinate transformation, binary-search lookup, feature assembly, and the Commit Gate update. We keep the cost low by fusing the Commit Gate update into fewer kernel launches and host reads.

(3) Dependency on temporal correspondence. The pose is already provided by driving stacks. As discussed in W2, driving stacks estimate the pose at 50–200 Hz, faster than the 10–20 Hz LiDAR, so SpaceCache reads an available pose for every frame. Even with a LiDAR-only pose or 0.2 m / 0.2° pose noise, mIoU drops by at most 0.34 points (W2).


Q4 SpaceCache achieves substantially larger speedups with LitePT-B than with Utonia on Waymo. What architectural differences explain this behavior?

Q5 What motivated the use of the self-supervised Utonia backbone rather than a stronger supervised baseline such as Point Transformer V3 or a fine-tuned Utonia model?

TL;DR. SpaceCache's speedup follows how each backbone distributes its computation across stages, and LitePT-B is the supervised backbone in our evaluation. Revision: Sec. 4 (L360–361); App. E.5 (L1308–1311); Table 13 (App. E.5, L1296–1332).

Answer. LitePT-B places 90% of its FLOPs in the cached Stages 3–4, against 64% in Utonia, so SpaceCache speeds it up more (1.56× vs. 1.38× on Waymo). Our Utonia is the pretrained encoder with a fine-tuned decoder, which the revised Sec. 4 now states.

Why LitePT-B gains more than Utonia (Q4). LitePT-B places attention only in Stages 3–4 (90% of FLOPs), whereas Utonia uses transformer blocks across every stage and the decoder (64% in Stages 3–4). With similar Stage-3 token hit rates (53% vs. 49%), SpaceCache removes 45.7% vs. 33.5% of FLOPs, giving 1.56× vs. 1.38× on Waymo.

The Utonia we use (Q5). We freeze the pretrained Utonia encoder and fine-tune a transformer decoder per dataset (now stated in revised Sec. 4), pairing one large-scale pretrained backbone with one efficient supervised backbone (LitePT-B, higher mIoU in Table 1). Fully supervised PTv3 results are in Table 13 (App. E.5).

AuthorsOpenReview ↗

Response to Reviewer 5yff (3)

W3 The ScanNet experimental results are not similar to current state-of-the-art, as the trained models use complete fused scenes rather than individual RGB-D observations. The baseline segmentation accuracy of 50.63 mIoU is thus low. It would be valuable to evaluate a model trained or fine-tuned directly on single-view RGB-D point clouds.

W4 The ScanNet experiments are conducted on only 10 validation scenes. This is a relatively small subset of the dataset (312 validation scenes).

TL;DR. With the revised evaluation, SpaceCache runs 1.24–1.25× faster than the backbone, while every baseline stays at or below 1.08×. Revision: Sec. 4 and Table 2 (L321–323, L378–388, L415–420); App. C.1 (L1028–1043).

Answer. We agree with both comments. We now train LitePT-B on ScanNet's single-view split and evaluate it on the full validation set (all 312 scenes). On this setup, the backbone reaches 65.91 mIoU, and SpaceCache remains the fastest method (Table R-3).

Revised setup.

  1. Training. We train LitePT-B from scratch on ScanNet's official single-view split (scannet_frames_25k, 19,466 RGB-D frames from the 1,201 training scans) with the paper's recipe. Each depth frame is back-projected and mapped to the world frame by its camera pose, as in our streams.
  2. Evaluation. All 312 validation scenes at 3 Hz (53,193 frames), with every second depth pixel in each direction.

Table R-3. Revised Table 2: comparison on indoor streams (ScanNet, all 312 validation scenes, LitePT-B trained on single RGB-D frames, SpaceCache with L = 5). Each method is shown at its fewest-FLOPs setting within 1-point and 2-point mIoU drops. Latency (ms) is the GPU kernel time on a Jetson AGX Orin, averaged over 50 representative frames. *: 2D method applied to point tokens. †: without the adaptive patch partition. ‡: same setting as within the 1-point drop.

Method≤1.0 pt mIoU↑≤1.0 pt Latency↓≤1.0 pt GFLOPs↓≤1.0 pt Speed↑≤2.0 pt mIoU↑≤2.0 pt Latency↓≤2.0 pt GFLOPs↓≤2.0 pt Speed↑
LitePT-B65.9139.147.41.00×65.9139.147.41.00×
+ ToMe*64.9338.032.51.03×63.9937.230.71.05×
+ ALGM*65.1537.138.21.06×64.1636.337.21.08×
+ GitMerge3D†65.1242.843.20.91×64.4242.642.80.92×
+ SpaceCache (Stage 3)64.9731.628.01.24×64.97‡31.6‡28.0‡1.24ׇ
+ SpaceCache (Stage 3,4)64.9232.027.21.22×64.5131.426.01.25×

‡ Stage-3 stays within 1 point at its largest saving (θ=0), so the 2-point column repeats it.

SpaceCache is the only method above 1.1× in kernel time. It removes 42.6% / 45.3% of the FLOPs (1 / 2 points), against 31.4% / 35.4% for the best baseline (ToMe).

The revised Sec. 4 uses this setup for all indoor results and reports Table R-3 as Table 2, and App. C.1 describes the single-frame model.

Reviewer_2rmaOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents SpaceCache, a training-free acceleration method for streaming point-cloud semantic segmentation. It maps tokens across frames to shared spatial regions to reuse intermediate features at deeper encoder stages. Cache-hit tokens bypass all blocks within a stage, and the full token layout is restored before subsequent processing. The cache is managed using entry lifetime and prediction confidence. Experiments on Waymo, nuScenes, and ScanNet evaluate the accuracy–efficiency trade-off. The main contribution is a stage-level feature reuse mechanism based on spatial correspondence, with cache operations confined to stage boundaries to reduce management overhead.

Critical strengths
  1. The method requires no additional training, and the reported speedups support its practical value. SpaceCache can be directly applied to the pretrained Utonia and LitePT-B backbones. Table 1 reports speedups across both backbones on two outdoor datasets. For example, with FlashAttention disabled, it reduces LitePT-B latency on Waymo from 307 ms to 186 ms, achieving a 1.65× speedup with a 0.71-percentage-point drop in mIoU.

  2. The design directly reduces computation and cache-management overhead. Cache-hit tokens skip all blocks within the selected stage, reducing sparse convolution, attention, and FFN computation. Cache operations run once per stage, avoiding repeated per-block selection and aggregation. Figure 7 and Appendix D break down runtime, policy overhead, and memory usage. The timings include cache updates. These analyses help explain how FLOPs reductions translate into latency gains.

Critical weaknesses
  1. The outdoor evaluation uses ground-truth labels to filter cache entries, and the impact remains unclear. Appendix B.1 and the supplementary code describe a label-based rule for newly computed candidate regions: a region is excluded from the cache if any input token within it has the ground-truth ignore label. Although ignored points do not contribute to mIoU, this rule changes the features reused in subsequent frames. It may therefore affect predictions on valid points and inference speed. Ground-truth labels are unavailable at deployment, and the nuScenes ignore category extends beyond points near the ego vehicle. Filtering a fixed area around the ego vehicle therefore cannot be assumed equivalent to the current label-based rule. I appreciate the authors’ disclosure of this procedure. Please report results with this rule disabled or replaced by label-free filtering to assess whether the accuracy and speed advantages hold without ground-truth labels. This is my main concern regarding the evaluation protocol.

  2. The relationship to existing caching methods needs clarification.
    SpaceCache uses cross-frame feature caching, but its primary comparisons are against within-frame token merging. These experiments show advantages over the selected merging methods. However, they do not clarify how spatial association, stage-level reuse, and update criteria improve on existing caching strategies.

    The paper discusses EVE and TempNet, which rely on learned modules, but gives limited attention to caching methods that require no additional training. For example, Eventful Transformers [1] studies selective token updates and feature reuse across video frames, including settings without additional training. ToCa [2] performs training-free token-wise feature reuse across diffusion timesteps. Its cache selection considers reuse counts since the last recomputation and spatial distribution.

    Please clarify which mechanisms have precedents, what new challenges point-cloud streams introduce, and why token merging is the primary comparison. Table 3 supports the effectiveness of several design choices within SpaceCache. A targeted caching baseline would help support any further claim of superiority over existing caching strategies.

  3. Speedups with FlashAttention enabled require further evidence.
    Appendix C.2 states that FlashAttention is disabled throughout the main latency comparisons to accommodate the merging baselines. Both backbones support FlashAttention, and the authors state that SpaceCache is compatible with it. Please compare accuracy and latency for each backbone with and without SpaceCache, with FlashAttention enabled. The paper reports memory usage for Utonia on Waymo under this setting, but not the corresponding latency gains.

    [1] Dutson et al. Eventful Transformers: Leveraging Temporal Redundancy in Vision Transformers. ICCV, 2023.

    [2] Zou et al. Accelerating Diffusion Transformers with Token-wise Feature Caching. ICLR, 2025.

Questions for authors

Please clarify the conditions required by SpaceCache and its applicability to other architectures. The design assumes that token coordinates remain unchanged within each stage. Under this condition, could it extend to certain sparse convolutional and KPConv backbones? The current evaluation covers only two Transformer backbones. Please discuss the adaptations required and any limitations of these extensions.

7 comments
AuthorsOpenReview ↗

Response to Reviewer 2rma (1)

We sincerely thank the reviewer for their insightful comments on our manuscript. We hope we have addressed their concerns and questions regarding the paper and hope they will reconsider their rating. Unless marked as "original", all table, section, equation, and line references below follow the revised PDF.


W1 The outdoor evaluation uses ground-truth labels to filter cache entries, and the impact remains unclear. Appendix B.1 and the supplementary code describe a label-based rule for newly computed candidate regions: a region is excluded from the cache if any input token within it has the ground-truth ignore label. Although ignored points do not contribute to mIoU, this rule changes the features reused in subsequent frames. It may therefore affect predictions on valid points and inference speed. Ground-truth labels are unavailable at deployment, and the nuScenes ignore category extends beyond points near the ego vehicle. Filtering a fixed area around the ego vehicle therefore cannot be assumed equivalent to the current label-based rule. I appreciate the authors' disclosure of this procedure. Please report results with this rule disabled or replaced by label-free filtering to assess whether the accuracy and speed advantages hold without ground-truth labels. This is my main concern regarding the evaluation protocol.

TL;DR. SpaceCache does not always need ground-truth labels: label-free filtering keeps the same fixed region close to the ego vehicle out of the cache and matches label-based filtering within 0.35 mIoU and 0.02× speedup. Revision: App. B.1 (L907–912); Table 10 (App. E.2, L1211–1227).

Answer. We thank the reviewer for raising this point. We use ignore labels to exclude points near the ego vehicle from the cache, avoiding potentially unreliable feature reuse in these regions. This is because, under the default training setup, these points carry the ignore label, so the backbone never receives supervision on these regions.

The ignore label (label-based rule) is simply a convenient, dataset-defined proxy for this region; any label-free rule that prevents near-vehicle points from being cached serves the same purpose.

For example, on Waymo, each point carries the ID of the LiDAR that captured it, and the points close to the ego vehicle come from the four short-range LiDARs. Label-free filtering therefore can exclude a point from the cache if its LiDAR ID belongs to one of the four short-range LiDARs. On nuScenes, the points inside a fixed box around the ego vehicle can be ignored from the cache. Table R-1 compares the label-based and label-free rules, showing only negligible differences in accuracy and speedup.

Table R-1. Inference without ground-truth labels on outdoor streams (flash-attention, RTX A6000). SpaceCache uses the θ of Table 1. The regions of the points that carry the ignore label are excluded by that label (Label-based) or by the LiDAR ID and the ego-vehicle box (Label-free).

MethodWaymo mIoU↑Waymo Speed↑nuScenes mIoU↑nuScenes Speed↑
Utonia67.191.00×77.351.00×
+ SpaceCache (Label-based)66.701.39×76.581.24×
+ SpaceCache (Label-free)66.751.38×76.461.26×
LitePT-B72.301.00×80.021.00×
+ SpaceCache (Label-based)71.581.64×79.211.26×
+ SpaceCache (Label-free)71.231.65×79.241.27×

The revised App. B.1 states that the results of Sec. 4 take these points from the ignore labels, and App. E.2 finds them without labels (Table 10).

AuthorsOpenReview ↗

Response to Reviewer 2rma (2)

W2-Part1 The relationship to existing caching methods needs clarification. SpaceCache uses cross-frame feature caching, but its primary comparisons are against within-frame token merging. These experiments show advantages over the selected merging methods. The paper discusses EVE and TempNet, which rely on learned modules, but gives limited attention to caching methods that require no additional training. For example, Eventful Transformers [1] studies selective token updates and feature reuse across video frames, including settings without additional training. ToCa [2] performs training-free token-wise feature reuse across diffusion timesteps. Its cache selection considers reuse counts since the last recomputation and spatial distribution. Please clarify which mechanisms have precedents, what new challenges point-cloud streams introduce, and why token merging is the primary comparison.

TL;DR. Generic caching components have 2D precedents, but our global spatial association handles moving point-cloud tokens, which 2D caching cannot; token merging is the primary baseline as the main training-free alternative. Revision: Sec. 1 (L52–53, L76); Sec. 5 (L515–521); App. B.5 (L987–996).

Answer. We thank the reviewer for this question and address its three parts below.

(1) Which mechanisms have precedents. At a high level, SpaceCache shares generic caching components with 2D methods: deciding which tokens to recompute (e.g., the gate in ET [1] and DeltaCNN [3]) and periodically refreshing stale features (e.g., ToCa's reuse-count-based refresh [2]).

These components are common to any feature cache. SpaceCache's core contributions lie elsewhere: (i) spatial association via global coordinates, which addresses the new challenge discussed below, and (ii) stage-level reuse that skips cache-hit tokens across a block range rather than gating each layer.

(2) New challenges point-cloud streams introduce. Unlike images, where each token's reuse candidate is fixed by the pixel grid and 2D caching only needs to decide whether to reuse (e.g., ET's input-difference test or ToCa's attention ranking), point-cloud tokens carry a spatial position as well as a feature, and that position changes every frame as occupied voxels vary in count and location with sensor motion.

One must therefore first find which previous token corresponds to the current one, a step that 2D caching does not address. While 2D caching could be extended with feature-based matching, this requires costly per-layer comparisons; SpaceCache instead uses a global position-based lookup that identifies the same physical region directly (see W2-Part2 for a comparison with feature-matching baselines).

(3) Why token merging is the primary comparison. Training-free acceleration of point-cloud backbones has mainly relied on token merging (ToMe, ALGM, GitMerge3D), which reduces redundancy within a frame. SpaceCache opens a different axis, feature reuse across frames, and shares token merging's training-free setting on the same backbones, unlike EVE and TempNet (which require training) or 2D caching (which assumes modification for point clouds; compared in W2-Part2).

The revised Sec. 1 and Sec. 5 cover ET, DeltaCNN, and ToCa and contrast their fixed-grid assumption with sparse, moving point-cloud streams.

[1] Dutson et al., Eventful Transformers: Leveraging Temporal Redundancy in Vision Transformers, ICCV 2023.
[2] Zou et al., Accelerating Diffusion Transformers with Token-wise Feature Caching, ICLR 2025.
[3] Parger et al., DeltaCNN: End-to-End CNN Inference of Sparse Frame Differences in Videos, CVPR 2022.

AuthorsOpenReview ↗

Response to Reviewer 2rma (3)

W2-Part2 However, they do not clarify how spatial association, stage-level reuse, and update criteria improve on existing caching strategies. Table 3 supports the effectiveness of several design choices within SpaceCache. A targeted caching baseline would help support any further claim of superiority over existing caching strategies.

TL;DR. SpaceCache achieves 1.56× speedup with 0.84 GB, vs. 0.80–0.93× with 7.82–12.06 GB for ET-style layer-level reuse, and its confidence filtering adds +1.07 mIoU over lifetime-only caching at matching speed. Revision: Sec. 3.3 (L267–296); Sec. 4.1 and Table 3 (L390–403, L421–431); Sec. 4.2 and Table 4(a) (L432–444, L461–469); App. B.5 (L983–996).

Answer. We clarify below how each of the three mechanisms improves on existing caching strategies, and add a targeted layer-level caching baseline (Table R-2).

(1) Spatial association. 2D caching methods decide reuse from features: ET compares each token's input across frames, which requires computing distances over high-dimensional features and storing the previous inputs, and ToCa ranks tokens by attention scores, which requires materializing the attention map at every layer.

SpaceCache instead matches tokens through their global coordinates, so each reuse decision reduces to a lightweight coordinate lookup with no feature comparison or stored inputs for comparison. Our Reuse Agreement analysis (Sec. 2; Fig. 2c) confirms that this cheaper position-based criterion remains reliable in point-cloud streams.

(2) Stage-level reuse. Layer-level caching decides reuse and stores features at every layer. We test this with two ET-style layer-level baselines (Table R-2): a direct port of ET that compares tokens at the same ego-frame position (local), and a variant that shares SpaceCache's global matching but still reuses at the layer level with per-layer feature comparisons (global). Both must cache tensors at all 45 gated layers of Stages 3–4 (12.06 / 7.82 GB), and their per-layer overhead outweighs the saved FLOPs (0.80–0.93×).

Note that the local variant is even slower, because under ego-motion the same ego-frame position quickly covers a different physical region, which lowers temporal overlap and thus the cache-hit rate.

SpaceCache decides once per stage and caches only stage outputs, achieving 1.56× with 0.84 GB.

Table R-2. Layer-level reuse and SpaceCache on Waymo with LitePT-B (full validation set, flash-attention, RTX A6000). Layer-level reuse follows Eventful Transformers without training (App. B.5). It matches tokens in the local or global coordinate system and recomputes a token only when its input changes by more than τ times its cached input. Hit: cache-hit fraction of the Stage-3 tokens. ΔFLOPs: total FLOPs change relative to the backbone.

MethodmIoU↑Hit (%)ΔFLOPs (%)Speed↑Mem (GB)↓
LitePT-B72.30––1.00×0.97
+ Layer-level reuse (local, τ = 0.8)72.3723−13.30.80×12.06
+ Layer-level reuse (global, τ = 0.8)72.9841−27.00.89×7.83
+ Layer-level reuse (global, τ = 1.0)72.4252−32.80.93×7.82
+ SpaceCache72.2153−45.71.56×0.84

(3) Update criteria. Unlike confidence-agnostic ET and ToCa, which only decide when to recompute tokens and store every computed result regardless of prediction quality, SpaceCache's Commit Gate filters features after computation via decoder confidence and enforces a finite lifetime LL (Sec. 3.3). This both bounds cache growth as the sensor traverses 3D space and removes uncertain or stale entries; our ablation in Table 4(a) shows that, at a matching speed, adding confidence filtering raises mIoU from 65.84 (lifetime only, L=4L = 4, 1.24×) to 66.91 (L=8L = 8, 1.25×), +1.07 points.

The revised Sec. 4.1, Table 3, and App. B.5 add this layer-level reuse (ET) comparison alongside input-level reuse (EVE).

Reviewer_ShJ4OpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes SpaceCache, a training-free method that speeds up point-cloud transformer segmentation on streams by reusing intermediate features across frames. A preliminary study shows that deeper encoder stages have both higher cross-frame token overlap and higher "Reuse Agreement". SpaceCache maps each token to a global voxel region index using the ego pose. Within a selected stage, tokens whose region is already cached skip all transformer blocks of that stage and take the cached feature. After prediction, a Commit Gate inserts new features and evicts cached ones based on decoder confidence and a bounded lifetime L. On Waymo, nuScenes and a ScanNet stream built from RGB-D video, with Utonia and LitePT-B backbones, SpaceCache reaches up to 1.65× speedup within a 1-point mIoU drop. It consistently beats ToMe, ALGM and GitMerge3D, with about 5 ms overhead and about 70 MiB of cache.

Critical strengths
  1. The method plugs into pretrained backbones without retraining. That matters now that point-cloud encoders are pretrained at scale, and it should be easy to adopt.
  2. The observation that token coordinates are fixed within a stage lets one lookup per stage cover every block, unlike merging methods that re-score at every block. The preliminary study gives a clear rationale for targeting deep stages.
  3. FLOPs by component, latency by operation type and peak memory explain why SpaceCache converts FLOP savings into wall-clock gains better than merging. This is more careful than most efficiency papers.
Critical weaknesses
  1. All baselines (ToMe, ALGM, GitMerge3D) are single-frame merging methods, and GitMerge3D runs without its adaptive partition. EVE, which the paper calls closest to its work, is not compared, nor are simple alternatives such as propagating previous predictions via ego-motion or using a smaller backbone at matched accuracy. Temporal feature reuse in video (e.g., Eventful Transformers, DeltaCNN, Deep Feature Flow) is not discussed.
  2. The Commit Gate skips regions containing ground-truth ignore-label points (Appendix B.1), which uses labels at inference. Operating points are picked on the evaluation set, with θ ranging from 0.8 to 0.99 (Table 4) despite a stated default of 0.85. Latency is measured with dense attention rather than FlashAttention, which may overstate the gains.
  3. Within a 1-point drop, speedup is 1.18–1.34× for Utonia and 1.18× indoors; "up to 1.65×" is the best case. Stages 0–2 and the decoder always run in full, and Utonia still takes 537 ms per frame on Waymo.
  4. Outdoor streams use 2 Hz labeled keyframes instead of native 10–20 Hz sweeps, and all experiments use dataset poses with no robustness test to pose noise or drift.
  5. The default setting loses 1.8 points on dynamic tokens (Table 3a), but there is no per-class IoU for pedestrians or cyclists and no check of error accumulation over long sequences.
Questions for authors
  1. Does the Commit Gate read ground-truth ignore labels at inference (Appendix B.1)? What are the results if it is replaced by a fixed, label-free region around the ego vehicle?
  2. What are the speedup and mIoU with one fixed θ (e.g., 0.85) and L across all datasets and backbones, rather than the per-setting fastest point within 1 point?
  3. What are the backbone and SpaceCache latencies with FlashAttention on Waymo and nuScenes? How does the speedup change?
9 comments
AuthorsOpenReview ↗

Response to Reviewer ShJ4 (1)

We sincerely thank the reviewer for their insightful comments on our manuscript. We hope we have addressed their concerns and questions regarding the paper and hope they will reconsider their rating. Unless marked as "original", all table, section, equation, and line references below follow the revised PDF.


W1-Part1 All baselines (ToMe, ALGM, GitMerge3D) are single-frame merging methods, and GitMerge3D runs without its adaptive partition.

TL;DR. With its adaptive partition, GitMerge3D is at most 0.01× faster and less accurate in every setting (by up to 16.7 points on Waymo), so we compare against the version without it. Revision: Table 6 (App. B.4, L950–967).

Answer. We reported GitMerge3D without its adaptive partition because that version is stronger. We now run GitMerge3D with and without its adaptive patch partition at the same merge ratio on every setting (Table R-1).

Table R-1. GitMerge3D with and without its adaptive patch partition at the same merge ratio r = 0.5. Speed is measured in latency on Waymo and nuScenes (RTX A6000) and in kernel time on ScanNet (Jetson AGX Orin). †: without the adaptive patch partition (the version in our tables).

MethodWaymo mIoU↑Waymo Speed↑nuScenes mIoU↑nuScenes Speed↑ScanNet mIoU↑ScanNet Speed↑
Utonia67.171.00×77.291.00×––
+ GitMerge3D† (r = 0.5)65.641.05×74.031.00×––
+ GitMerge3D (r = 0.5)63.640.89×73.350.86×––
LitePT-B72.821.00×80.041.00×65.911.00×
+ GitMerge3D† (r = 0.5)72.051.07×74.921.06×65.480.91×
+ GitMerge3D (r = 0.5)55.371.08×74.871.07×65.460.91×

The adaptive partition is not better. It either loses more accuracy, because it merges low-energy patches too aggressively (Waymo), or is at most 0.01× faster, because computing the energy score offsets the savings from merging (nuScenes, ScanNet).

AuthorsOpenReview ↗

Response to Reviewer ShJ4 (2)

W1-Part2 EVE, which the paper calls closest to its work, is not compared, nor are simple alternatives such as propagating previous predictions via ego-motion or using a smaller backbone at matched accuracy.

TL;DR. Input-level reuse (propagating previous predictions via ego motion), the training-free alternative closest to EVE, is either barely faster (1.05–1.06× with 5 cm voxels) or loses up to 10.4 points (20 cm voxels), while SpaceCache stays within 0.3 points at 1.38–1.56×. Revision: Sec. 4.1 and Table 3 (L390–403, L421–431); App. B.5 (L974–982); Table 13 (App. E.5, L1316–1332); Table 17 (App. E.9, L1420–1431).

Answer. EVE trains an additional module to compensate for input pruning, so a direct comparison on the same off-the-shelf backbones is not possible without retraining. Instead, we compare SpaceCache with the two alternatives the reviewer suggests: propagating previous predictions via ego motion and a smaller backbone at matched accuracy.

(1) Input-level reuse (propagating previous predictions via ego motion). We find that it cannot keep both accuracy and speed (Table R-2). In this alternative implementation, each point's prediction is stored in a voxel of the global coordinate system, and the backbone runs only on points whose voxel does not hold a previous prediction. We vary the voxel size from 5 to 20 cm, with the same L=8L = 8, θ=0.85\theta = 0.85 and label-free rule as SpaceCache.

Table R-2. Input-level reuse and SpaceCache on Waymo (full validation set, flash-attention, RTX A6000). Input-level reuse matches points within voxels of 5, 10, or 20 cm. Hit: cache-hit fraction of the input points for input-level reuse and of the Stage-3 tokens for SpaceCache.

MethodUtonia mIoU↑Utonia Hit (%)Utonia Speed↑LitePT-B mIoU↑LitePT-B Hit (%)LitePT-B Speed↑
Backbone67.19–1.00×72.30–1.00×
+ Input-level reuse (5 cm)64.67241.06×71.25261.05×
+ Input-level reuse (10 cm)61.20411.19×67.77441.16×
+ Input-level reuse (20 cm)56.79541.42×64.54581.38×
+ SpaceCache66.90491.38×72.21531.56×

For this input-level reuse, voxel size sets the accuracy-speed trade-off: small voxels give fine-grained reuse that preserves accuracy but yields only a 1.05–1.06× speedup, while large voxels reuse the same label to every point in a voxel and lose up to 10.4 mIoU points. In contrast, SpaceCache reuses only Stage-3/4 features and recomputes every point's label in the decoder, staying within 0.3 points of the baseline at a 1.38–1.56× speedup.

(2) Smaller backbone at matched accuracy. A smaller backbone and SpaceCache are complementary rather than competing (Table R-3). LitePT-S runs faster than LitePT-B with SpaceCache (71 vs. 128 ms), but it is 0.70 mIoU below LitePT-B and requires training a separate model, whereas SpaceCache keeps LitePT-B within 0.09 points without training. SpaceCache also speeds up LitePT-S by 1.30× (71 → 54 ms) at a 0.19-point drop, so the two can be combined for the fastest setting.

Table R-3. SpaceCache on LitePT-B and LitePT-S on Waymo (θ = 0.85, L = 8, label-free rule, flash-attention, RTX A6000). mIoU is measured on the full validation set. GFLOPs and latency (ms) are the mean per frame over the frames of representative scenes. Hit: cache-hit fraction of the Stage-3 tokens. Speed: speedup against the same backbone without SpaceCache.

MethodmIoU↑Hit (%)GFLOPs↓Latency↓Speed↑
LitePT-B72.30–23542001.00×
+ SpaceCache72.215312791281.56×
LitePT-S71.60–606711.00×
+ SpaceCache71.4152339541.30×
AuthorsOpenReview ↗

Response to Reviewer ShJ4 (3)

W1-Part3 Temporal feature reuse in video (e.g., Eventful Transformers, DeltaCNN, Deep Feature Flow) is not discussed.

TL;DR. Video feature reuse (ET, DeltaCNN, DFF) relies on a fixed pixel grid, whereas point-cloud tokens change position every frame; we now discuss these methods in Sec. 1 and Sec. 5. Revision: Sec. 1 (L52–53, L76); Sec. 5 (L515–521); Sec. 4.1 and Table 3 (L390–403, L421–431); App. B.5 (L983–996).

Answer. We thank the reviewer for pointing out these works.

(1) Relation to video feature reuse. These methods share SpaceCache's goal of exploiting cross-frame redundancy, but all rely on a fixed 2D pixel grid: ET [1] gates tokens by their input difference at the same pixel index, DeltaCNN [2] propagates sparse activation deltas of changed pixels through conv layers, and DFF [3] warps key-frame features with a learned optical-flow network. Because the grid fixes each token's reuse candidate, these methods only need to decide whether to reuse.

(2) Why they do not transfer directly. Point-cloud tokens are sparse occupied voxels whose count and positions change every frame with sensor motion, so one must first find which previous token corresponds to the current one. SpaceCache resolves this with a global-coordinate lookup computed once per stage, whereas grid-based methods would require per-layer feature checks and caches; DFF additionally requires training a flow network. Indeed, an ET-style port to point clouds is slower than the original backbone (0.80–0.93× vs. 1.56× for SpaceCache; revised Table 3).

We add this discussion to the revised Sec. 1 and Sec. 5, and the ET-style comparison to Sec. 4.1, Table 3, and App. B.5.

[1] Dutson et al., Eventful Transformers: Leveraging Temporal Redundancy in Vision Transformers, ICCV 2023.
[2] Parger et al., DeltaCNN: End-to-End CNN Inference of Sparse Frame Differences in Videos, CVPR 2022.
[3] Zhu et al., Deep Feature Flow for Video Recognition, CVPR 2017.


W2-Part1 The Commit Gate skips regions containing ground-truth ignore-label points (Appendix B.1), which uses labels at inference.

Q1 Does the Commit Gate read ground-truth ignore labels at inference (Appendix B.1)? What are the results if it is replaced by a fixed, label-free region around the ego vehicle?

TL;DR. SpaceCache does not need ground-truth labels: label-free filtering excludes the same region near the ego vehicle from the cache and matches label-based filtering within 0.35 mIoU and 0.02× speedup. Revision: App. B.1 (L907–912); Table 10 (App. E.2, L1211–1227).

Answer. In the submitted version, the Commit Gate uses ignore labels, but only as a convenient proxy for the region near the ego vehicle, not for any semantic information.

Under the default training setup, points near the vehicle carry the ignore label, so the backbone never receives supervision there and their features are unreliable to reuse. Any label-free rule that excludes this region serves the same purpose.

On nuScenes, we exclude points inside a fixed box around the ego vehicle, as the reviewer suggests; on Waymo, we exclude points from the four short-range LiDARs, which mainly cover this region, using each point's sensor ID.

As shown in Table R-4, the label-free rules match the label-based ones within 0.35 mIoU and 0.02× speedup. The revised App. B.1 clarifies the role of the ignore labels, and App. E.2 (Table 10) reports the label-free results.

Table R-4. Inference without ground-truth labels on outdoor streams (flash-attention, RTX A6000). SpaceCache uses the θ of Table 1. The regions of the points that carry the ignore label are excluded by that label (Label-based) or by the LiDAR ID and the ego-vehicle box (Label-free).

MethodWaymo mIoU↑Waymo Speed↑nuScenes mIoU↑nuScenes Speed↑
Utonia67.191.00×77.351.00×
+ SpaceCache (Label-based)66.701.39×76.581.24×
+ SpaceCache (Label-free)66.751.38×76.461.26×
LitePT-B72.301.00×80.021.00×
+ SpaceCache (Label-based)71.581.64×79.211.26×
+ SpaceCache (Label-free)71.231.65×79.241.27×

The revised App. B.1 clarifies that the results in Sec. 4 identify this region using the ignore labels, and App. E.2 (Table 10) reports the results with the label-free rules.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response to All Reviewers and the AC

We thank the reviewers and the AC for their time and constructive feedback. During the discussion phase, all three reviewers (ShJ4, 2rma, 5yff) confirmed that our rebuttal resolved their major concerns. We summarize the key strengths, core contribution, and major revisions below.

Strengths noted by the reviewers.

  • Training-free, plug-and-play applicability to off-the-shelf pretrained backbones, delivering meaningful inference speedups while largely preserving segmentation accuracy (ShJ4, 2rma, 5yff).
  • Preliminary analysis of when intermediate features can be reused across point-cloud streams, providing a clear rationale for targeting deep stages (ShJ4, 2rma, 5yff).
  • Low-overhead stage-level reuse that skips deep-stage blocks via a single spatial lookup per stage, supported by principled FLOPs, latency, and peak memory breakdowns explaining the wall-clock gains (ShJ4, 2rma).

Core contribution. SpaceCache accelerates semantic segmentation on 3D point-cloud streams without retraining. Based on our key observation that deeper encoder stages exhibit higher cross-frame token overlap and feature consistency (Sec. 2.2), SpaceCache reuses deep-stage features across frames, bypassing all blocks in these stages via a single lightweight global-coordinate lookup per stage.

Major revisions.

  • Label-free inference & FlashAttention (ShJ4, 2rma): Ego-region rules without ground-truth labels and FlashAttention verification (Tables 9–10).
  • Baselines & generality (ShJ4, 2rma, 5yff): Input/layer-level reuse, adaptive GitMerge3D, and five additional backbones (Tables 3, 6, 13).
  • ScanNet benchmark (5yff): Single-view RGB-D training evaluated on all 312 validation scenes (Table 2).
  • Streaming robustness (ShJ4, 5yff): Native frame rates, pose noise/KISS-ICP, long drives, and per-class IoU (Tables 11–12, 14–15).
  • Cache-miss tokens (5yff): Cache-miss tokens stay within their class feature clusters and keep their accuracy despite losing cache-hit neighbors (Sec. 3.3, Table 16).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.