acceptodds
Withdrawn from ICLR 2027

Faster3D: Exploring the Roles of Latent Tokens for Faster Image-to-3D Generation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Flow-based image-to-3D models repeatedly predict velocities for all latent tokens during sampling, although the need for recomputation can vary across tokens. Image-to-3D generation is primarily concerned with the sparsity problem that the surrounding space of the formed object always remains empty. Our analysis of TRELLIS sparse-structure generation shows that non-surface tokens associated with empty regions generally have smaller velocity magnitudes and directional changes than surface tokens associated with occupied regions. These observations motivate selective velocity reutilization. However, omitting selected tokens from self-attention can degrade geometry because their representations still provide context for other tokens. In this paper, we propose Faster3D, a training-free acceleration framework with two components. The Temporal Token Decoupling which instructs the reutilization of velocity predictions, and the Cached Context Restoration retains cached keys/values at their original positions for sampling. These two components reduce redundant velocity computation while preserving the attention context provided by selected tokens. Experiments on the Toys4K dataset show that Faster3D consistently accelerates diverse image-to-3D backbones by 1.30×–1.95× speedups in the targeted structure or geometry sampling stage, with geometry and appearance metrics remaining close to the full-compute baselines. These results highlight context-preserving token-wise velocity reutilization as an effective technique for image-to-3D accelerations.

Primary area:
generative models
Keywords:
Image-to-3D Generation, Training-Free Acceleration, Flow Matching, Velocity Reutilization, KV Caching

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_hFqrOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a training-free approach, Faster3D, to improve the image-to-3D models. It employs temporal token decoupling (TTD) to selectively reuse token velocities and cached-content restoration (CCR) to preserve cached attention keys/values. The reported speedups in the target structure or geometry sampling stage vary from 1.30x to 1.95x for five backbones.

Critical strengths
  1. Tokens associated with empty space may need fewer velocity evaluations while still providing useful attention context. This distinction is compelling.
  2. Reusing kv at their original positions and periodically refreshing all tokens forms a plausible practical system.
  3. Results across five backbones make this more convincing than a single-model optimization, although each backbone requires separate calibration.
Critical weaknesses
  1. Calibration needs clarification. Dataset separation, threshold construction, calib cost, and generalization are insufficiently specified.
  2. Reported speedups cover only the targeted sampling stage, while end-to-end pipeline latency, stage-level timings, and additional cache memory would make the practical gain easier to assess.
  3. In tab.1, the gains over Faster3DCache are fairly modest. eg. on TRELLIS, it achieves almost the same speedup and small improvements, w/o a overhead discussion. It's hard to judge the proposed method.
Questions for authors
  1. Does calibration use any evaluation objects or closely related assets?
  2. How accurately does activity predict velocity-reuse error across timesteps and backbones?
  3. Does CCR outperform spending the same runtime on additional fresh token evaluations?
  4. How much of refresh’s benefit comes from fresh velocities versus fresh K/V?
  5. What are the end-to-end latency, memory overhead, and worst failure cases?
2 comments

Thanks for your positive feedback that our paper identifies a compelling distinction, that we provide convincing evidence, and that our method Faster3D forms a plausible practical system across diverse backbones. Below we respond to each of your concerns.

Q1&W1: In Appendix B.1, we clarify that all calibration sets are disjoint from Toys4K-210 and that configurations are fixed before testing. Table 5 in Appendix B.3 reports the calibration data source, sample count, and threshold construction for each backbone. As shown in the table below, we test two different calibration images randomly sampled from the disjoint Toys-4k and 3D Arena datasets. The main metrics suggest that Faster3D generalizes across calibration sets.

Calibration imagesTest objectsv-IoU ↑TFLOPs
Original examples21060.65138.03
3D Arena21061.54133.91
Disjoint Toys4K21060.72124.88

Q5&W2: The 3D asset generation models generally consist of geometry generation and texture generation stages, thus requiring different acceleration techniques due to the architectural differences. We also argue that the geometry generation is building a strong and important basis for the texture generation stage. There are also numerous 3D models such as Hi3DGen and TripoSG that focus solely on geometry generation, while existing acceleration method Fast3Dcache focuses on geometry synthesis. Our Faster3D follows the same scope of Fast3Dcache, and all reported speedups refer specifically to geometry sampling.

Below, we report the end-to-end geometry generation speedups and GPU memory as you mentioned. Our method Faster3D achieves large ratios of speedups at the cost of necessary increased memory usage.

Regarding failure cases, we did not observe severe visual degradation relative to the full-compute baseline. Nevertheless, velocity reuse remains an approximation and does not guarantee identical geometric outputs. Figures 1, 7, and 12–16 provide numerous qualitative comparisons across different backbones for your reference.

Backbone / methodTFLOPsTime (s)GPU memory (GiB)
TRELLIS / baseline244.241.0885.357
TRELLIS / Faster3D138.030.6156.107
TRELLIS.2 / baseline303.251.4693.724
TRELLIS.2 / Faster3D215.151.0425.130
Hunyuan3D-2.1 / baseline1824.167.0776.875
Hunyuan3D-2.1 / Faster3D861.603.3428.187
TripoSG / baseline627.022.7663.874
TripoSG / Faster3D412.451.8194.531
Step1X-3D / baseline704.7615.1296.770
Step1X-3D / Faster3D536.1911.5108.176

W3: First, the improvements of geometry generation are not always well reflected in metrics. Fast3Dcache loses substantial geometric detail at a sampling speed comparable to our method. By comparison, Faster3D more closely preserves the completeness of geometric structures and fine-grained details. Please refer to the Fig.1, Fig.7 and Fig.12-16 for comparisons. Also, noticeable metric improvements and robustness to diverse backbones can be observed on vIoU, Ulip-2 and Uni3D in Table 1.

Q2: As the table below, we compute the velocity-reuse MSE of Faster3D and Fast3Dcache on TRELLIS with Toys4K-210 as the dataset. These suggest that activity-prediction velocity-reuse could more accurately approximate the velocities of the selected tokens. More results are reported in the additional Appendix C.6.

MethodStep 7Step 12Step 17Step 22
Faster3D0.013480.029050.063750.000996
Fast3Dcache0.569620.1600111.379150.004553

Q3: To study the importance of CCR mechanism, we disable the CCR and only apply the TTD technique. This operation allows for additional fresh token computations while degrading the performance, as demonstrated below with lower vIoU, CD and F-score metrics.

ConfigurationTime(s)TFLOPsFresh token-forwardsCD ↓vIoU (%) ↑F-score (%) ↑
Faster3D, α=31.4832138.025101,8470.0277760.6596.00
TTD-only, α=2.321.4689136.699104,8830.0310557.2994.82

Q4: To test the benefit of refresh, we compare refreshing only velocities, only K/V, both, or neither, with token masks and refresh steps fixed on 210 Toys4k objects. As shown below, refreshing K/V alone provides a larger gain than refreshing velocities alone. Both components are beneficial, and refreshing both achieves the best metrics performance.

What is refreshedCD ↓F-score ↑
Neither0.0281395.52
Velocity only0.0278695.88
K/V only0.0277996.01
Velocity and K/V0.0277796.04

Thank you again for your time and thoughtful feedback. With two days remaining in the discussion period, please let us know if any questions or concerns remain after our responses. We would be happy to provide further clarification.

Reviewer_RrMTOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Faster3D, a training-free acceleration framework for flow-based image-to-3D generation. The method is motivated by the observation that latent tokens associated with non-surface regions generally exhibit smaller velocity magnitudes and temporal variations than surface-associated tokens.

Based on this observation, the authors introduce two components: (1) Temporal Token Decoupling (TTD), which estimates token activity from the predicted clean latent and selectively reuses velocity predictions for low-activity tokens; and (2) Cached Context Restoration (CCR), which preserves the cached keys and values of reused tokens to maintain the attention context for actively computed tokens. Periodic full-token evaluations are employed to mitigate cache staleness.

Critical strengths
  1. Effective training-free acceleration across multiple image-to-3D backbones. The proposed framework achieves consistent acceleration across five different image-to-3D architectures without requiring additional model training. The reported speedups range from approximately 1.30× to 1.95× in the targeted sampling stages, while geometry and appearance metrics remain close to those of the full-computation baselines.

  2. Convincing empirical evidence for the importance of preserving attention context. A particularly valuable aspect of this work is the distinction between reusing a token's velocity prediction and preserving its contribution to the attention context. The controlled interventions in Appendix C.3 provide evidence supporting this distinction. With reused velocities held fixed, restoring cached K/V substantially reduces prediction errors on actively computed tokens and improves terminal occupancy agreement.

Critical weaknesses
  1. The proposed framework largely builds upon existing token-wise caching techniques. More importantly, Appendix C.4 shows that adding K/V restoration to Fast3Dcache already substantially improves geometry quality. Appendix C.5 further indicates that TTD provides only modest improvements over existing token selection strategies under comparable runtime. These results raise questions about the significance of TTD as an independent contribution.

  2. The proposed framework largely builds upon existing token-wise caching techniques. More importantly, Appendix C.4 shows that adding K/V restoration to Fast3Dcache already substantially improves geometry quality. Appendix C.5 further indicates that TTD provides only modest improvements over existing token selection strategies under comparable runtime. These results raise questions about the significance of TTD as an independent contribution.

Questions for authors
  1. Given the results in Appendix C.5, what is the main advantage of TTD over existing velocity-based token selection methods?
  2. Could the authors report end-to-end generation speedups and additional GPU memory consumption introduced by K/V caching?
2 comments

Thanks for your positive feedback that Faster3D serves as an effective training-free acceleration method across multiple image-to-3D backbones, and that we provide convincing empirical evidence for the importance of preserving attention context. Below we respond to each of your concerns.

W1&W2: We mentioned that you had the identical weakness comments for our paper, we assume that you made a mistake while copying your notes from another place. Therefore we merge these two points into one to address your concerns as below.

First, We agree that token reuse and K/V caching are established techniques. However, our contribution first lies in the observation that tokens at different spatial locations exhibit different velocity dynamics. Our experiments show that a large number of non-surface tokens can skip fresh velocity computation, yet they still provide meaningful context for other tokens. Our observation and proposal are native to 3D geometry generative models, which distinguishes from previous techniques. Based on the above observation, TTD identifies which tokens can reuse their velocities at each timestep, while CCR retains their K/V and controls when to refresh them.

Then, for your concern about the results in Appendix C.4, we agree that directly applying K/V caching can already improve the performance of Fast3Dcache. However, as you already mentioned, this K/V caching slows down the inference speed of 3D geometry generation from 1.338x to 1.259x. This phenomenon indicates the non-reasonable selection of cached tokens is lagging the acceleration performance. To overcome this issue, our proposed TTD identifies tokens associated with extraneous spatial regions. Afterwards, our Faster3D brings back the speed-up ratio to 1.342x.

Last, regarding Appendix C.5, we also apply CCR to Fast3Dcache selector and that results in its close performance to our method. Retaining the K/V of skipped tokens improves Fast3Dcache performance, supporting our claim that preserving context matters for 3D geometry generation. In the updated main text, we included the original Fast3Dcache for your reference in Table 10 and Figure 12.

Q1: Existing velocity-based method, Fast3Dcache selects tokens based on temporal velocity stability. By comparison, our method Faster3D is based on token activities in which we analyze token behavior from a 3D-native perspective. Our advantage is that we reduce computation on non-surface tokens while preserving the normal sampling trajectories of surface tokens. In Appendix C.5, we also apply CCR to Fast3Dcache, which improves its performance and makes the advantage of TTD less apparent. In the new version of our paper, we included the original Fast3Dcache in Appendix C.5 for your reference.

We also conducted additional experiments to measure the root mean square (RMS) error of reused velocities across the channels of each token. The first table shows that, at the same token reuse budget, Faster3D selects more non-surface tokens and achieves a lower RMS error on 210 Toys-4k objects. The second table shows that Faster3D also achieves a lower RMS reuse error across different backbones. These results highlight advantages of TTD in token selection and velocity reuse.

Selected by one method onlyToken–stepsSurface-related (%)Next-step error ↓
TTD384,17016.210.16795
Fast3Dcache384,17046.150.17460
BackboneTTD reuse error ↓Fast3Dcache reuse error ↓
TRELLIS0.040210.04336
TRELLIS.20.063380.07037
Hunyuan3D-2.10.071530.08279

Q2: The 3D assets generation includes three main sources of time consumptions, including the geometry generation, the texture generation, and the 3D latent decoding. In this paper, we aim to accelerate the end-to-end geometry generation stage following previous 3D geometry methods (Fast3Dcache, TripoSG, Hi3DGen, etc.). We target at accelerating the texture generation stage as future work due to its diverse architecture design from the geometry generation. Below, we report the end-to-end geometry generation speedups and GPU memory as you mentioned. Our method Faster3D achieves large ratios of speedups at the cost of necessary increased memory usage.

Backbone / methodTFLOPsTime (s)GPU memory (GiB)
TRELLIS / baseline244.241.0885.357
TRELLIS / Faster3D138.030.6156.107
TRELLIS.2 / baseline303.251.4693.724
TRELLIS.2 / Faster3D215.151.0425.130
Hunyuan3D-2.1 / baseline1824.167.0776.875
Hunyuan3D-2.1 / Faster3D861.603.3428.187
TripoSG / baseline627.022.7663.874
TripoSG / Faster3D412.451.8194.531
Step1X-3D / baseline704.7615.1296.770
Step1X-3D / Faster3D536.1911.5108.176

Thank you again for your time and thoughtful feedback. With two days remaining in the discussion period, please let us know if any questions or concerns remain after our responses. We would be happy to provide further clarification.

Reviewer_PVJKOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Faster3D, a training-free method for accelerating the structure or geometry sampling stage of image-to-3D generators. It selects latent tokens whose velocities can be reused, retains their cached keys and values as attention context for freshly computed tokens, and periodically refreshes the caches. Experiments on Toys4K-210 report 1.30×–1.95× speedups in the targeted stage across five backbones, with quality generally close to full computation

Critical strengths
  1. The paper identifies a useful distinction between reusing a token’s velocity and retaining its attention context. The controlled K/V interventions provide convincing evidence that removing the context can harm active-token predictions and final structure.
  2. The evaluation covers five backbones and includes component ablations and a comparison with Fast3Dcache. At comparable Stage-I speed on TRELLIS, Faster3D preserves geometry substantially better than Fast3Dcache.
Critical weaknesses
  1. The methodological advance is fairly incremental: token-wise reuse and K/V caching are established ideas. Appendix C.5 compares the token selectors under shared CCR and comparable runtime. TTD modestly improves CD and PSNR over Fast3Dcache’s selection strategy, but shows no clear advantage in vIoU, F-score, or LPIPS. This limits the evidence for the proposed selector as a substantial independent contribution.
  2. The efficiency evaluation covers the targeted sampling stage rather than complete 3D asset generation. Downstream stages remain, while per-layer K/V caches add memory. Without end-to-end latency and peak-memory measurements, the practical benefit is difficult to assess. The five backbones also use separate offline calibrations.
Questions for authors
  1. What are the end-to-end generation speedups and peak GPU memory changes for each backbone, including the K/V caches?
  2. At matched runtime, how does Faster3D compare with a simple reduction in sampling steps?
  3. How sensitive are quality and speed to the choice of calibration examples and generation seed?
2 comments

Thanks for your positive feedback that our paper identifies a useful distinction, that we provide convincing evidence, and that our method Faster3D preserves geometry substantially. Below we respond to each of your concerns.

W1: We agree that token reuse and K/V caching are well-established techniques widely used for model acceleration. However, these techniques are not our contributions. Instead, our key contribution lies in identifying a geometry-aware distinction between surface and non-surface tokens in 3D generation. Furthermore, our analysis reveals that a large proportion of non-surface tokens can skip fresh velocity computation while still providing meaningful context for other tokens. Based on these findings, we leverage established token reuse and K/V caching techniques to achieve efficient acceleration.

Regarding Appendix C.5, we also apply CCR to Fast3Dcache selector and that results in its close performance to our method. Retaining the K/V of skipped tokens improves Fast3Dcache performance, supporting our claim that preserving context matters for 3D geometry generation. In the updated main text, we included the original Fast3Dcache for your reference in Table 10 and Figure 12.

Q1&W2: The 3D asset generation models generally consist of geometry generation and texture generation stages, thus requiring different acceleration techniques due to the architectural differences. We also argue that the geometry generation is building a strong and important basis for the texture generation stage. There are also numerous 3D models such as Hi3DGen and TripoSG that focus solely on geometry generation, while existing acceleration method Fast3Dcache focuses on geometry synthesis. Our Faster3D follows the same scope of Fast3Dcache, and all reported speedups refer specifically to geometry sampling.Below, we report the end-to-end geometry generation speedups and GPU memory on Toys4K-210. Our Faster3D achieves large ratios of speedups at the cost of necessary increased memory usage for K/V caches.

While applying Faster3D to accelerate 3D generations, each 3D model requires only one time offline calibration due to the difference in token representations and sampling schedules. We calibrate on separate examples and use the same settings for all 210 test objects without per-input tuning to ensure fair comparisons.

Backbone / methodTFLOPsTime (s)GPU memory (GiB)
TRELLIS / baseline244.241.0885.357
TRELLIS / Faster3D138.030.6156.107
TRELLIS.2 / baseline303.251.4693.724
TRELLIS.2 / Faster3D215.151.0425.130
Hunyuan3D-2.1 / baseline1824.167.0776.875
Hunyuan3D-2.1 / Faster3D861.603.3428.187
TripoSG / baseline627.022.7663.874
TripoSG / Faster3D412.451.8194.531
Step1X-3D / baseline704.7615.1296.770
Step1X-3D / Faster3D536.1911.5108.176

Q2: To address your concerns, we evaluate the model with different numbers of reduced sampling steps (10, 15, and 20) on Toys4K-210.With similar TFLOPs (Full, 15 steps vs. Faster3D), Faster3D yields better performance compared with the 15-step baseline. That indicates naive reduction on sampling steps is not a reliable acceleration approach for these 3D geometry models. Our method Faster3D achieves better trade-offs between acceleration and generation qualities.

ScheduleTFLOPsCD ↓vIoU (%) ↑F-score (%) ↑
Full, 25 steps244.240.0278561.0395.93
Full, 20 steps199.830.0281660.5695.89
Full, 15 steps149.870.0285460.4695.86
Full, 10 steps99.910.0287760.1195.79
Faster3D, 25 steps138.030.0277760.6595.99

Q3: To understand the sensitivity to the choice of calibration sets and generation seeds, we ablate several options and the resutls shown in the table below.We test different seeds, and the results of IoU/FLOPs remain stable. Also, we change calibration examples from the original TRELLIS calibration set as used in the main paper, to Toys-4k and 3D Arena datasets. Both new alternative calibration sets maintained high vIoU while reducing Stage I FLOPs, with a modest drop in vIoU compared to the original calibration set.

Calibration (42 images)SeedvIoU ↑TFLOPs
Original examples4260.65138.03
3D Arena4261.54133.91
Disjoint Toys4K4260.72124.88
Original examples060.77137.84
Original examples123460.22137.98

Thank you again for your time and thoughtful feedback. With two days remaining in the discussion period, please let us know if any questions or concerns remain after our responses. We would be happy to provide further clarification.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.