acceptodds
Under review as a conference paper at ICLR 2027

RETIRE3D — WHERE RESOLUTION LIVES: LOCAL SCALE COMPLETION FOR COARSE-TO-FINE 3D GENERATION

Abstract

Coarse-to-fine 3D generators commonly advance predicted surface regions through a shared finest generation stage. This uniform allocation conflicts with a basic property of 3D geometry: geometric complexity is highly non-uniform in space. Broad, smooth surfaces can often be determined at a coarse scale, whereas thin structures, sharp boundaries, holes, and junctions demand finer precision. The benefit of additional fine-stage computation can therefore vary substantially across regions. We identify this conflict as a mismatch between uniform computational resolution and local geometric demand, and formulate Local Scale Completion: fine-scale computation should follow remaining local geometric demand. We introduce Retire3D, a training-free framework that instantiates this principle through spatially selective evaluation within the high-resolution stage, retaining all latent coordinates. Retire3D uses coarse-stage signals to initialize a fixed evaluation partition, while all fine-scale latent states continue to evolve and provide attention context. History-based propagation supplies unevaluated velocities, and residual checks schedule full reevaluations. The complete spatial representation enters the native downstream pipeline. Experiments on Direct3D-S2 and TRELLIS.2 assess end-to-end efficiency and fidelity to native generation. On Direct3D-S2, Retire3D achieves a native-geometry fidelity score of .927 versus .634 for backbone-adapted Fast3Dcache, a 46.2% relative improvement. With temporal caching, Retire3D achieves a 1.528× end-to-end speedup over native generation on TRELLIS.2.

Primary area:
generative models
Keywords:
3D generation, coarse-to-fine generation, adaptive computation, local refinement, inference acceleration, sparse computation, training-free acceleration

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_pG8KOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Retire3D is a training-free framework for accelerating coarse-to-fine 3D generation through spatially selective evaluation in high-resolution stages. It uses coarse-stage geometry, trajectory changes, and uncertainty to establish a fixed partition of active and retired tokens. All latent states continue to evolve: active tokens receive fresh network evaluations, while retired tokens use history-based velocity predictions and provide approximate attention context through embeddings rebuilt from their current states. Residual checks trigger full reevaluations. Experiments on Direct3D-S2 and TRELLIS.2 assess reconstruction quality, fidelity to native outputs, and end-to-end efficiency, alongside ablations and composition with temporal caching.

Critical strengths
  1. A practical approach to selective computation while maintaining the full spatial state. Retire3D reduces fresh evaluations without removing latent coordinates or freezing retired regions. Maintaining their evolution and contextual participation addresses an important difficulty in spatially selective 3D sampling. The refresh ablations further show that evaluation timing matters: on Direct3D-S2, residual-triggered refresh improves native fidelity over a uniform schedule at nearly identical query cost.

  2. An informative evaluation of the quality–efficiency trade-off. The paper complements ground-truth reconstruction metrics with native-output fidelity and structural statistics, revealing geometric changes that conventional reconstruction scores can obscure. End-to-end timing includes selection, prediction, probes, and refresh overhead. The explicit reporting of negative ablations, missing measurements, and failure cases makes the demonstrated benefits and limitations easier to assess.

Critical weaknesses
  1. The contribution of coarse-guided spatial allocation remains insufficiently established. The random-allocation control in Appendix D shows mixed results: G/C/U scoring decreases native F-score by 1.57 percentage points on Direct3D-S2 while improving it by 0.80 points on TRELLIS.2. Although this comparison uses an earlier configuration, a corresponding controlled evaluation of the final configuration is missing. Moreover, increasing the retirement fraction improves fidelity on Direct3D-S2, raising the question of whether the benefits primarily arise from state propagation and full refreshes rather than identifying local refinement demand. This distinction matters because coarse-guided allocation is central to the paper’s contribution.

  2. The qualitative evidence provides limited coverage of the reported performance. The visual comparisons repeatedly feature a small set of objects, and the appendix clarifies that these examples are selected rather than randomly sampled. Reusing cases helps comparisons across settings, but offers limited evidence of robustness across the broader evaluation set. Aggregate metrics alone also do not reveal how often substantial local structural damage occurs or how severe the worst cases are. The paper does disclose failures, which is valuable; nevertheless, a broader, systematically selected visual evaluation, together with per-object fidelity distributions and structural failure rates, is needed to establish how representative the displayed results are.

Questions for authors
  1. Contribution of coarse-guided allocation. Could you provide a controlled comparison of G/C/U-guided allocation against random allocation using the final configuration on the held-out cohort, with matched query budgets or end-to-end latency? In particular, does coarse-guided allocation consistently improve fidelity across both backbones, and how much benefit remains relative to a simplified approach using only state propagation, probes, and full refreshes?
  2. Representativeness of the qualitative results. The qualitative comparisons repeatedly use a small set of selected objects. How representative are these examples of the full evaluation cohort? Could you summarize the per-object fidelity distribution and the frequency of substantial local structural failures, and provide additional examples selected by a predefined rule—for instance, random samples and cases near the lower fidelity quantiles? This would help assess whether the reported average performance reflects consistent behavior or masks a subset of severe failures.
3 comments
AuthorsOpenReview ↗

Round 1 — Response to Reviewer pG8K

Thank you for the specific requests.

W1 / Q1 — Contribution of coarse-guided allocation.

  • Data. Your reading is correct: most of the benefit comes from state propagation and refresh placement (+32 to +42 pp, Table G2). G/C/U vs. random under matched budgets: +0.8 pp on TRELLIS.2 (10 of 12 objects, CI [+0.31, +1.28]), −1.6 pp on Direct3D-S2.
  • Takeaway. State propagation and refresh are the method's contribution, not something separate from it. They are what allow spatially selective evaluation to keep the native shape: spatial methods without them reach Fnat=.242F_{\rm nat}=.242–.634, and with them even random masks reach .810 / .856 on the control objects. The coarse score is a component on top. It tracks refinement demand (precision .758 vs. .649, Table G3), and the gap to the oracle shows room for better scores, which the framework can use directly.
  • On the retirement-fraction trend. On Direct3D-S2, FnatF_{\rm nat} rises from .829 to .953 as ρ\rho goes from .80 to .99, while the executed query fraction falls from .830 to .779. The trend is therefore not a budget effect. It agrees with Table G2: fidelity is carried by state propagation and refresh, and the framework stays faithful with a small active set.
  • Next. The evaluation you describe: final configuration, 50 held-out objects per backbone, matched budgets and schedules; G/C/U vs. random (several seeds) vs. propagation + probes + refreshes only. We will restate the contributions in this order in the revision.

W2 / Q2 — Per-object distribution and structural failures.

  • Data: per-object FnatF_{\rm nat} (main comparison).
BackboneMethodMedianp10Min# below .8# below .5
Direct3D-S2Retire3D.958.783.511110
Direct3D-S2Fast3Dcache.643.496.3769511
TRELLIS.2Retire3D.904.753.465152
TRELLIS.2Fast3Dcache.751.611.335704
  • Data: structural failures (thresholds fixed for this response; 99 / 98 objects with a native reference).
CriterionRetire3D (D3-S2 / TRELLIS.2)Fast3Dcache (D3-S2 / TRELLIS.2)
Largest-component share drops > 5 pp0 / 621 / 37
≥ 10 added components of ≥ 1,000 faces1 / 1756 / 70
  • Takeaway. The mean reflects consistent behavior. The median is above the mean on both backbones, and on Direct3D-S2 Retire3D's 10th percentile (.783) exceeds Fast3Dcache's 90th (.766). Loss of the main body does not occur on Direct3D-S2 and is rare on TRELLIS.2.

Q2 — Examples by a predefined rule.

  • Rule, per backbone: 5 objects drawn at random with a fixed seed, plus the 5 objects with the lowest Retire3D FnatF_{\rm nat} (including tv_002 and glass_047), each shown for native, Retire3D, and the baselines under one camera protocol.
  • Next. Plates to follow.

Thank you for the clarification and the additional per-object statistics. These results substantially alleviate my concern about the representativeness of the reported performance. I also agree that state propagation and refresh placement are legitimate contributions of the framework.

My main remaining concern is the practical value of coarse-guided spatial allocation. The reported improvement over random allocation is small on TRELLIS.2 (+0.8 pp) and negative on Direct3D-S2 (−1.6 pp). The refinement-demand precision indicates that the score captures a relevant signal, but does not establish that using this signal improves the final quality–efficiency trade-off. Similarly, the retirement-fraction trend supports the effectiveness of propagation and refresh with a small active set, rather than demonstrating the benefit of the coarse-guided score itself.

The promised controlled evaluation under the final configuration is therefore important. If random allocation achieves comparable or better results under matched budgets and refresh schedules, this would support the propagation-and-refresh framework more strongly than the coarse-guided allocation mechanism. This would not invalidate the overall approach, but would require a substantive adjustment to the contribution claims and a clearer justification for retaining the scoring component.

Regarding structural reliability, the improvements over Fast3Dcache are encouraging. However, 17 of 98 TRELLIS.2 objects meet the added-component criterion, so the severity of the remaining failures still deserves examination. The planned random and worst-case visual comparisons would help assess whether these defects materially affect usability and whether local structural damage is adequately captured by the reported connectivity metrics.

For now, I am retaining my current rating while awaiting the promised controlled experiments and visual comparisons. I am open to revising it if the additional evidence establishes a convincing quality–efficiency advantage and supports a clear, appropriately scoped statement of the method’s contributions.

AuthorsOpenReview ↗

Round 2 — Response to Reviewer pG8K

Thank you. We measured the severity of the remaining failures directly, and ran the controlled experiment you asked for.

Severity of failures, and whether component counts capture it

Data: connectivity vs. directly measured damage (TRELLIS.2, 100-object main comparison, Retire3D).

  • 22 objects trip a connectivity criterion.
  • 18 objects show severe visible damage.
  • Only 6 objects are in both groups.

Takeaway. Your doubt is confirmed. Component counts do not identify local damage: they flag objects whose surface is intact apart from small detached fragments, and they miss local surface errors. We therefore report the direct measure for every method (definitions in the general update).

Data: objects with severe damage, 50 held-out objects per backbone.

Direct3D-S2 visibleDirect3D-S2 surfaceTRELLIS.2 visibleTRELLIS.2 surface
Retire3D32718
Fast3Dcache, nearest / published budget11 / 3919 / 318 / 4319 / 43
Fast-SAM3D48405047
RAS483600

Plates. Rule: 5 random and 5 lowest objects per backbone, every method, with missing- and extra-surface maps.

  • 4 of our 5 lowest TRELLIS.2 objects have visible damage below 2%. Their low F_nat comes from small fragments and small surface offsets.
  • On the same objects, Fast3Dcache scatters debris and Fast-SAM3D loses large surfaces.
  • All 20 regenerated objects match the main table within .02.

Takeaway.

  • Severe local damage is rare with Retire3D on Direct3D-S2 (3 of 50) and common for the spatial alternatives (11–48).
  • On TRELLIS.2, Retire3D matches the most frugal Fast3Dcache budget and is far ahead of the others, except RAS, which uses 1.75× the memory.
  • With per-layer context for retired tokens, severe visible damage falls to 1 of 49 on both backbones (an analysis result; its timing is not yet measured).

Quality–efficiency advantage

Data: Direct3D-S2, matched speed.

F_natSevere visible damage
Retire3D.9293 / 50
Fast3Dcache, best budget.84111 / 50
RAS.41148 / 48
Fast-SAM3D≤ .21648 / 50
  • Retire3D vs. the best Fast3Dcache budget: +8.8 pp [6.9, 10.8].
  • The accelerated stage is 1.29× / 1.32× faster.
  • On a single-call (CFG-off) sampler, Retire3D keeps 97–99% of that speedup and its fidelity (.976 / .934).
  • In the accelerated stage, removing CFG alone gives 1.99×; with Retire3D added it is 2.64× (Direct3D-S2).

Takeaway. Within spatial acceleration, Retire3D occupies an operating point no other method reaches, and it multiplies with CFG distillation. We scope the claim to that.

The allocation control, and the contribution statement

Setup. Final configuration, 50 held-out objects per backbone, same refresh rule. G/C/U and the random partitions execute identical queries (verified per object); ρ = .99 and propagation-only execute fewer.

Direct3D-S2TRELLIS.2
Coarse score (G/C/U).929.869
Random partition, 3 seeds.943.873
ρ = .99*.953.898
Propagation + probes + refresh only (no score-selected tokens)*.938.893

* Objects that cannot run these settings are excluded rather than counted:

  • ρ = .99 runs on 50 / 42 objects (Direct3D-S2 / TRELLIS.2);
  • the propagation-only setting runs on 32 / 32 objects;
  • on the remaining objects, the query-subset kernel either needs more query rows than the probes provide (Direct3D-S2) or would fall back to dense execution (TRELLIS.2).

Takeaway. As you anticipated, the coarse score does not improve on a random partition.

Your Round-1 question was how much benefit remains with propagation, probes and refreshes alone. The answer is all of it (.938 / .893). The framework holds with every partition we tested, and fidelity rises as the active set shrinks.

Following your suggestion:

  • the revision presents the propagation-and-refresh framework as the method;
  • the score becomes one replaceable initialization of the partition;
  • Figure 1 is presented as evidence of spatial redundancy.

This replaces the earlier 12-object TRELLIS.2 figure (+0.8 pp).

Reviewer_eToHOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

Retire3D is a training-free method for accelerating coarse-to-fine 3D generation. It uses coarse-stage geometry, trajectory changes, and uncertainty to select which high-resolution tokens receive fresh network evaluations. Other tokens continue updating through velocity prediction and remain available as attention context, with full evaluations triggered by prediction residuals. Experiments show end-to-end speedups of 1.062× on Direct3D-S2 and 1.225× on TRELLIS.2, reaching 1.528× when combined with TaylorSeer. Ablations show that state updates and refresh timing are important, while the benefits of coarse-guided allocation vary across backbones.

Critical strengths
  • The evaluation reports end-to-end runtime, memory, reconstruction quality, and fidelity to native outputs across two backbones.
  • The Direct3D-S2 ablation shows that residual-based refresh improves fidelity over uniform refresh at nearly the same query cost.
  • The supplementary material clearly reports implementation details, missing results, failure cases, and settings where the proposed choices do not outperform simpler alternatives.
Critical weaknesses
  • The quality–speed trade-off is not convincing. Direct3D-S2 achieves only a 1.062× speedup. Adding Retire3D to TaylorSeer gives an 8.6% incremental speedup, but native fidelity drops from 0.926 to 0.828, with visible structural losses such as missing panels and shelf slats.
  • Coarse-guided allocation has mixed support. At matched query budgets, it performs worse than random allocation on Direct3D-S2 and only slightly better on TRELLIS.2 in the supplementary study.
  • The state-maintenance ablations change latent updates and attention context together, so they do not isolate the benefit of the proposed K/V construction.
Questions for authors
  1. At the same speedup, how does Retire3D compare with tuned caching baselines or simply reducing the number of sampling steps?
  2. Why does coarse-guided allocation underperform random allocation on Direct3D-S2? What evidence links the proposed scores to actual local refinement needs?
  3. Can you isolate the effect of the K/V construction by keeping latent updates and refresh schedules fixed?
  4. Why use first-order prediction when zeroth-order holding gives better native fidelity on both backbones?
  5. How many main-evaluation objects were used during development?
4 comments
AuthorsOpenReview ↗

Round 1 — Response to Reviewer eToH

Thank you for the direct assessment.

W1 — Quality–speed trade-off.

  • Data: same goal, same speed range, very different fidelity (Direct3D-S2, 100 objects).
Fnat↑F_{\rm nat}\uparrowC×↓C_\times\downarrowObjects with Fnat<.8F_{\rm nat}<.8Speed
Fast3Dcache.63467.7951.081×
Fast-SAM3D.24240.8991.158×
RAS.297235.598 of 981.036×
Retire3D.9272.26111.062×
  • Data: where the 1.062× comes from.
Direct3D-S2TRELLIS.2
Share of runtime in the accelerated stages (3 profiled objects)24–37%65–89%
Executed DiT FLOPs vs. native0.78×0.67×
Per-object speedup, smallest → largest0.99× → 1.24×0.98× → 1.33×
Correlation of speedup with log token countr = .76r = .98
  • Takeaway. The end-to-end number on Direct3D-S2 is limited by the pipeline (refinement, decoding, and meshing are untouched), not by the method. The gain grows with token count, which is the direction high-resolution 3D generation is moving.
  • On the composition. The +8.6% (95% CI [7.2%, 9.9%]) is gained on top of a cache that already removed 43% of the queries. 86.6% of the added components have fewer than 100 faces, and the median largest-component share is 99.5%. Some objects lose a local part, such as the panel you noted; these are shown in the appendix.

Q1 — Tuned caches and fewer steps at the same speedup.

  • Data. TaylorSeer alone 1.407×; with Retire3D 1.528×, at the same memory.
  • Takeaway. Temporal methods and Retire3D act on different axes and stack.
  • Next. A speed-matched table: fewer fine-stage steps, TaylorSeer and EasyCache at several budgets, Fast3Dcache at our speed, Retire3D alone and composed.

W2 / Q2 — Allocation on Direct3D-S2; what links the scores to refinement need?

  • Data. The link: the score selects blocks already determined by the coarse result with precision .758 vs. a base rate of .649, above chance on all 61 objects (Table G3). The effect on final fidelity under matched budgets: +0.8 pp on TRELLIS.2 (10 of 12 objects), −1.6 pp on Direct3D-S2 (2 of 12), against +32 to +42 pp for state maintenance and refresh placement (Table G2).
  • Takeaway. The score tracks where fine computation is unnecessary, but its effect on final fidelity is small. The large effects come from state maintenance and refresh placement, which are what make spatial selection usable at all: prior spatial methods without them reach .242–.634.
  • Next. 50 held-out objects, final configuration, with a no-allocation variant.

W3 / Q3 — Isolating K/V.

  • Data. At the same query budget: .869 with per-step propagation and embedding K/V vs. .446–.522 without. Full-depth K/V with frozen states: .458 at full cost.
  • Next. K/V source varied alone in the main sampler.

Q4 — Why first-order?

  • Data. TRELLIS.2 face ratio on three objects: .926 / .796 / .808 with holding vs. .997 / .883 / .946 with first-order. Held-out face-ratio deviation: .077 vs. .061. FnatF_{\rm nat}, holding vs. first-order: .917 vs. .869 on TRELLIS.2; .939 vs. .929 on Direct3D-S2. Direct3D-S2 face-ratio deviation: .011 vs. .014.
  • Takeaway. On TRELLIS.2 holding erodes the surface and first-order keeps it; holding gives higher point-wise fidelity. Same queries, same speed. We will report both.

Q5 — Development objects.

  • Data. 15 development objects, all inside the 100-object main cohort (Appendix B). The 50-object ablation cohort is disjoint.
FnatF_{\rm nat}Direct3D-S2: 100 → 85 objectsTRELLIS.2: 100 → 85 objects
Retire3D.927 → .931.882 → .884
Fast3Dcache.634 → .639.747 → .752
  • Takeaway. Removing all development objects changes nothing.

Thank you for the detailed response and additional analyses. I appreciate the authors’ efforts to address my concerns. However, my main concern remains structural integrity: missing panels and shelf slats are unacceptable losses for the modest speedup reported. Connected-component statistics do not address these failures. The presented examples also focus on simple objects, providing insufficient evidence of robustness on complex geometry. Existing CFG distillation and step-distillation methods have demonstrated 2× or greater acceleration while preserving high generation quality, albeit with additional training. Overall, I remain unconvinced by the quality–speed trade-off and maintain my score.

AuthorsOpenReview ↗

Round 2 — Response to Reviewer eToH (1/2)

Thank you for the follow-up. You are right that component statistics do not show missing parts. We therefore measured local damage directly (definitions in the general update), including on the composition your examples come from.

Retire3D alone

Data: objects with severe damage, 50 held-out objects per backbone.

Direct3D-S2 visibleDirect3D-S2 surfaceTRELLIS.2 visibleTRELLIS.2 surface
Retire3D3 / 502 / 507 / 5018 / 50
Fast3Dcache, nearest budget1119819
Fast3Dcache, published budget39314343
Fast-SAM3D (published budget)48405047
RAS (published budget)48 / 4836 / 4800

Median visible damage on Direct3D-S2:

Retire3DFast3Dcache (nearest / published)Fast-SAM3DRAS
0.2%0.8% / 4.5%8.4%47%

Against the nearest Fast3Dcache budget, Retire3D has less damage on 41 of 50 objects.

Takeaway.

  • On Direct3D-S2, severe local damage is the exception for Retire3D and the rule for the other spatial methods.
  • On TRELLIS.2, Retire3D is on par with Fast3Dcache's most frugal budget and far below its other budgets and Fast-SAM3D. RAS has no severe cases there, but uses 1.75× our memory.
  • The damage can be reduced inside the framework. With per-layer context for retired tokens (the K/V experiment, see the reply to JuM8), it falls to 1 of 49 objects on each backbone. Its timing has not been measured yet, so we report it as an analysis of the context choice, not as the deployed configuration.

The composition (your sink_009 example)

Data: TRELLIS.2, the 50-object composition cohort.

SpeedSevere visibleMedian visible
TaylorSeer i21.43×4 / 500.2%
TaylorSeer i2 + Retire3D1.55×15 / 501.0%
TaylorSeer i31.58×11 / 500.7%
TaylorSeer i41.71×17 / 501.3%

Composition vs. TaylorSeer i3 (nearly the same speed), paired by object:

  • visible damage: −0.1 pp [−1.3, 0.9];
  • missing surface: −1.2 pp [−3.2, 0.6];
  • extra surface: −3.6 pp [−6.6, −1.1].

Two objects named in our appendix (visible damage, composition vs. i3):

  • sink_009: 4.2% vs. 1.7%;
  • piano_034: 7.7% vs. 14.1%.

Takeaway. The losses you point out are real. Going from 1.43× to 1.55× adds severe damage on about one object in five. Reaching the same speed by tightening TaylorSeer instead costs about as much: visible and missing surface show no significant difference, and the composition adds less extra surface. The cost comes with the higher speed, not from Retire3D. The appendix shows these maps for every method, sink_009 included.

Complex objects

Data: Direct3D-S2, our 5 lowest objects (selected by rule). F_nat per method:

Retire3DFast3DcacheFast-SAM3DRAS
closet_007.665.517.344.229
fridge_001.693.457.128.167
stove_009.701.541.157.203
cake_023.704.594.334.204
tv_002.511.706.141.230

Takeaway. On our own worst, complex objects, Retire3D keeps more of the native shape than every spatial alternative on 4 of 5. The tv_002 fragment is shown as it is, not averaged away.

AuthorsOpenReview ↗

Round 2 — Response to Reviewer eToH (2/2)

Where the time goes

Data.

Direct3D-S2TRELLIS.2
Speedup of the stage Retire3D accelerates1.29×1.32×
Change in all other stages+0.1%+0.8%
End-to-end, small / mid / large objects1.00 / 1.07 / 1.10×1.09 / 1.28 / 1.30×

Data: temporal methods at our speed (the table promised in Round 1).

Retire3DFewer stepsTaylorSeerEasyCache
Direct3D-S2 @1.08×.929.973.963.982
TRELLIS.2 @1.21×.869.947.910.992

Takeaway.

  • The end-to-end figure is bounded by the stages Retire3D does not touch. Inside its own stage it saves about a quarter of the time, and the gain rises with object size (r = .96).
  • Among spatial token-reduction methods, Retire3D is the only one that keeps the native shape: +8.8 pp [6.9, 10.8] over the best Fast3Dcache budget.
  • Temporal methods skip whole steps. They act on a different quantity and, at this low speedup, reach higher fidelity on their own. Retire3D works inside the steps they keep, so the two can be combined.

Distillation

Single-call sampler. We turned CFG off in the accelerated stage, so that the sampler makes one call per step, as a CFG-distilled model does. Fidelity is measured against that sampler's own output on 50 objects. Timing was run on H200, and every ratio is taken within the same jobs.

Direct3D-S2TRELLIS.2
Retire3D stage speedup, CFG on → single call1.37× → 1.33×1.29× → 1.28×
F_nat on the single-call sampler.976 [.962, .986].934 [.898, .961]
Accelerated stage: removing CFG alone1.99×1.34×
Accelerated stage: removing CFG + Retire3D2.64×1.71×
End-to-end: removing CFG alone → + Retire3D1.12× → 1.16×1.20× → 1.40×

On TRELLIS.2, objects whose main body loses more than 5% go from 3 to 7 of 50.

Short schedule (10 steps on Direct3D-S2, 8 on TRELLIS.2):

  • the stage speedup holds: 1.38× / 1.17×;
  • fidelity holds on TRELLIS.2 (.888) but drops on Direct3D-S2 (.791).

Takeaway.

  • CFG distillation removes calls; Retire3D removes tokens inside each call. On a single-call sampler Retire3D keeps 97–99% of its speedup and its fidelity, so the gains multiply. In the stage we accelerate, the 2× from removing CFG becomes 2.64×.
  • Step distillation leaves fewer steps to share the work, and on Direct3D-S2 fidelity suffers. We state this as a limit.
  • Retire3D needs no training. The revision positions it as complementary to distilled samplers, and makes no claim about any specific distilled checkpoint.
Reviewer_JuM8OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces Retire3D, a training-free framework that accelerates coarse-to-fine 3D generation by evaluating only part of the high-resolution tokens on a given step. Coarse-stage signals set a fixed active/retired partition; retired tokens keep evolving through history-based velocity prediction and stay in the attention context; residual checks decide when to reevaluate everything. The full latent is passed to the native decoder, so the output geometry is still produced by the original pipeline.

Critical strengths
  1. The motivation is well grounded: the paper goes after spatially uneven refinement demand, which is a different lever from the temporal redundancy that most acceleration work exploits.
  2. The design holds together. Less frequently evaluated tokens are neither dropped nor frozen — they continue to move, and they continue to contribute context, which is where naive token-skipping usually breaks.
  3. The empirical results are concrete: 1.062x on Direct3D-S2 and 1.225x on TRELLIS.2 end to end, and composing with TaylorSeer pushes it from 1.407x to 1.528x, showing the two axes really are complementary.
  4. The ablations are worth reading, particularly on state maintenance and refresh placement. I appreciate that the authors report where behavior changes with the backbone instead of presenting one uniform story.
Critical weaknesses
  1. Coarse-guided allocation is the least established part of the story. Figure 1 uses an oracle ordering, which is not the score the method actually deploys, and the gain over random partitions turns out to depend on the backbone. I would like to see the comparison done under matched query budgets.
  2. It is hard to tell which component is doing the work. The state-maintenance ablation runs in a separate sampler and, as the authors note, does not isolate the K/V reconstruction choice inside the main sampler.
  3. The trade-off claim needs a matched comparison. Combining Retire3D with TaylorSeer degrades the reported quality metrics: native fidelity drops from 0.926 to 0.828, and the component-count ratio rises from 1.02 to 10.26. Whether that is a better point than simply tightening an existing baseline's budget is not resolved by the current tables.
Questions for authors
  1. Does coarse-guided allocation consistently outperform random allocation under matched query budgets and refresh schedules?

  2. Why is first-order prediction preferred over velocity holding?

  3. Does Retire3D outperform budget-adjusted baselines at matched speed or matched fidelity?

3 comments
AuthorsOpenReview ↗

Round 1 — Response to Reviewer JuM8

Thank you for the constructive review.

W1 / Q1 — Coarse-guided allocation vs. random, matched budgets.

  • Data. The Appendix D control is budget-matched (same executed queries, same refresh schedule). G/C/U vs. random: +0.8 pp on TRELLIS.2 (10 of 12 objects, 95% CI [+0.31, +1.28]); −1.6 pp on Direct3D-S2. State maintenance and refresh placement: +32 to +42 pp (Table G2). The score does track refinement demand: precision .758 vs. base rate .649 on 61 objects (Table G3).
  • Takeaway. The contribution is making spatially selective evaluation work without breaking geometry; that is where the 30–40 pp come from. The score is a component on top: it helps consistently on TRELLIS.2 and not on Direct3D-S2. The framework is not sensitive to it: with random masks it still reaches Fnat=.810F_{\rm nat}=.810 / .856 on the control objects.
  • Next. The same control with the final configuration on 50 held-out objects per backbone, plus a no-allocation variant. In the revision we will state the contributions in this order and make explicit, in the text and caption of Figure 1, that the oracle ordering is a diagnostic and not the deployed score.

W2 — Which component does the work.

  • Data (Direct3D-S2, 50 held-out objects). The first table uses the separate sampler described in the paper, so its scores are compared within that table; the second uses the main sampler.
Retired-state variantFnatF_{\rm nat}Query fraction
Frozen retired states.446.642
Update only at refresh.522.642
Full-depth K/V, frozen states.4581.000
Per-step propagation + embedding K/V (ours).869.642
Refresh placement (same count, same cost)FnatF_{\rm nat}C×C_\times
Late.60917.03
Uniform.8815.22
Residual-triggered (ours).9292.01
  • Takeaway. Two components carry the result: retired tokens keep evolving and stay in context, and refreshes are placed by the residual rule. More computation alone does not help (.458 at full budget).
  • Next. K/V source varied alone in the main sampler, with latent updates and refresh schedule fixed.

W3 / Q3 — The composed point vs. a tighter baseline budget.

  • Data (TRELLIS.2, 50 objects).
SpeedAccelerated-stage queriesAttention FLOPsMemory
TaylorSeer (i=2i=2)1.407×57.1% of native—7.5 GiB
+ Retire3D1.528×49.0%−19.5%7.4 GiB
  • Takeaway. A step cache skips whole evaluations; Retire3D shrinks the evaluations that remain. They remove different work, so the gain stacks. The rise in C×C_\times (1.02 → 10.26) is mostly small debris: 86.6% of the added components have fewer than 100 faces, the median object gains 3 components of ≥1,000 faces, the median largest-component share is 99.5%, and the face ratio is unchanged (1.021 vs. 1.022). A minority of objects lose a local part (e.g., a panel on sink_009).
  • Next. TaylorSeer at intervals 3 and 4 on the same 50 objects with the same timing protocol, for a speed-matched comparison.

Q2 — Why first-order prediction?

  • Data.
TRELLIS.2Face ratio, 3 dev objects (→1)Face-ratio deviation, 50 held-out ↓FnatF_{\rm nat}, 50 held-out ↑
Velocity holding.926 / .796 / .808.077.917
First-order.997 / .883 / .946.061.869

On Direct3D-S2 the two are close on both criteria (FnatF_{\rm nat} .929 vs. .939; deviation .014 vs. .011).

  • Takeaway. With holding, retired regions lag the trajectory and the decoded surface erodes on TRELLIS.2; first-order prediction compensates. First-order keeps surface extent, holding scores higher on point-wise fidelity. Both run the same queries at the same speed, so the framework works with either. We will report the main table with both.

Thank you for the detailed response. I will keep my current rating and wait for the promised new experiments to make my final judgment. I also find the comments from the other two reviewers valuable, which point out many issues that need to be addressed. I am not changing my score 3 for now, but it leans toward 2 until all reviewers’ concerns are properly resolved.

AuthorsOpenReview ↗

Round 2 — Response to Reviewer JuM8

Thank you again. The promised experiments are complete.

Q3 — Matched speed against budget-adjusted baselines

Direct3D-S2. Fast3Dcache at 5 budgets; RAS and Fast-SAM3D at 3 each.

F_natSpeedSevere visible damage
Retire3D.9291.078×3 / 50
Fast3Dcache, best budget.8411.085×11 / 50
RAS.4111.078×48 / 48
Fast-SAM3D.2161.189×48 / 50

Damage for RAS and Fast-SAM3D is measured at their published budgets.

Retire3D vs. the best Fast3Dcache budget: +8.8 pp [6.9, 10.8].

TRELLIS.2. The picture is mixed:

  • Fast3Dcache's most frugal budget reaches similar fidelity (.882 vs. .869) at a higher speed (1.34× vs. 1.20×).
  • RAS reaches higher fidelity, at 1.75× the memory.

Our matched-speed claim is therefore made for Direct3D-S2. Temporal methods are reported as a separate, complementary axis (see the reply to eToH).

Q1 — Allocation under matched budgets

Under the final configuration, the coarse score does not outperform random, and propagation, probes and refreshes alone keep the full benefit (table in the reply to pG8K). The revision presents the score as a replaceable initialization, and Figure 1 as a diagnostic of spatial redundancy.

W2 / Q2 — K/V source and prediction order

Both are now varied alone in the main sampler. Full tables are in the revised ablation section.

  • Context. With partition, updates and refresh rule fixed, context from cached per-layer hidden states raises F_nat to .955 / .904 (from .929 / .869), while executing fewer queries because fewer refreshes trigger (Q .758 vs. .790; .671 vs. .707).
    • On TRELLIS.2, C× falls from 25.6 to 1.07.
    • Objects with severe visible damage fall from 3 / 7 to 1 / 1 of 49.
    • A coarse first-layer rebuild gives no benefit on Direct3D-S2.
    • Its timing has not been measured yet; the revision reports it as an analysis of the context choice, and the main table keeps the deployed configuration.
  • Prediction order. The framework works with either order at the same speed. Holding gives higher F_nat on 100 objects (.943 / .917 vs. .927 / .882). The face-ratio difference reported in Round 1 is not significant at this scale. The main table reports both.

W3 — Composition vs. a tighter TaylorSeer budget

At nearly the same speed as TaylorSeer i3 (1.55× vs. 1.58×), the composition:

  • keeps the surface extent much better: face-ratio deviation .080 vs. .219;
  • adds less extra surface: −3.6 pp [−6.6, −1.1];
  • has comparable visible damage: −0.1 pp [−1.3, 0.9].

F_nat is +4.2 pp; interpolated to exactly equal speed, the difference is within noise.

Composing is therefore at least as good as tightening the cache, and better on surface extent. The full table is in the revised composition section.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Round 1 — General response (to all reviewers)

We thank the reviewers for their careful reading. The paper makes three contributions. We give the evidence for each, then show where the reviewers' questions sit.

Table G1. Contributions and evidence.

ContributionEvidenceReviews
1. Problem. Fine-stage 3D generation as a local computation-allocation problem, motivated by spatially uneven refinement benefitA median of 65.6% of fine blocks are already determined by the coarse result (61 objects). With the right ordering, 59% of regions can be substituted at F=0.98F=0.98 (Figure 1)Well grounded (JuM8)
2. Framework. Retire3D: retirement initialized from coarse evidence, retired states that keep evolving and stay in context, residual-triggered refreshFnatF_{\rm nat} .927 vs. .242–.634 for other query-reduction methods at 1.04–1.16× on Direct3D-S2. State maintenance +35 to +42 pp; refresh placement +32 ppHolds together (JuM8, pG8K); questions concern individual choices inside it
3. Evaluation. Two generators, end-to-end cost, ground-truth metrics, fidelity to native output, composition with temporal cachingGround-truth F-score differs by under 2 points across methods while C×C_\times ranges from 2.26 to 235.5; native fidelity exposes this. Composition: +8.6% speed (95% CI [7.2%, 9.9%]), −19.5% attention FLOPsInformative (eToH, pG8K)

Table G2. Size of each effect. The reviews question three choices inside the framework: the allocation score, the prediction order, and the K/V source (isolation in progress). Their measured effects are far smaller than the effects that define the framework.

FnatF_{\rm nat} ↑Face-ratio deviation ↓Source
State maintenance (framework)+35 to +42 pp (.869 vs. .446–.522).033 vs. .063–.204Direct3D-S2, 50 held-out objects
Refresh placement (framework)+32 pp (.929 vs. .609).014 vs. .333Direct3D-S2, 50 held-out objects
Allocation score, G/C/U vs. random−1.6 pp (Direct3D-S2), +0.8 pp (TRELLIS.2)—12 objects, Appendix D
Prediction order, first-order vs. holding−1.0 pp (Direct3D-S2), −4.8 pp (TRELLIS.2)TRELLIS.2: .061 vs. .077; Direct3D-S2: .014 vs. .01150 held-out objects

Table G3. The spatial opportunity and the coarse score.

EvidenceResult
Fine blocks already determined by the coarse result (61 objects, one-fine-voxel tolerance)median 65.6% (range 55–73%)
Oracle ordering (Figure 1)59% of regions substituted at F=0.98F=0.98; random fails far earlier
Coarse score selects determined blocks (61 objects, offline)precision .758 vs. base rate .649; lift 1.17, above 1 on all 61 objects (1.10–1.26)

Takeaway. The framework carries the result, and its component choices are replaceable: with a random partition or with velocity holding it still keeps the native shape. We ask that the paper be assessed on the three contributions in Table G1. Each component question is answered with data below.

New in this round: results without development objects, per-object distributions, structural-failure counts, the speedup breakdown, and the evidence behind first-order prediction.

In progress (held-out cohort, final configuration):

  1. G/C/U vs. random vs. no-allocation, matched query budgets, 50 objects per backbone.
  2. Speed-matched table: fewer sampling steps, TaylorSeer, EasyCache, Fast3Dcache at multiple budgets.
  3. Main comparison with both prediction orders.
  4. K/V source isolated in the main sampler.
  5. Qualitative examples chosen by a fixed rule.
AuthorsOpenReview ↗

Round 2 — General update (to all reviewers)

We thank the reviewers for the follow-ups. All evaluations announced in Round 1 are complete, with the final configuration. In addition, we now measure local damage directly, and we test Retire3D on single-call (distillation-style) and short-schedule samplers.

1. Retire3D is the only spatial acceleration method that keeps the native shape at its speed. Direct3D-S2, with the budgets of Fast3Dcache (5 budgets), RAS and Fast-SAM3D (3 each) adjusted toward our speed:

F_natSpeed
Retire3D.9291.08×
Fast3Dcache, best budget.8411.08×
RAS.4111.08×
Fast-SAM3D≤ .216—

2. Local damage, measured directly, is rare. On Direct3D-S2, 3 of 50 held-out objects show severe visible damage with Retire3D, against 11–48 for the spatial alternatives. The worst cases are shown with missing- and extra-surface maps.

Definitions (fixed before measurement; Appendix B of the revision):

  • Visible damage: the largest connected region where the rendered surface departs from the native one (silhouette mismatch, or normals more than 30° apart), as a share of the native silhouette, maximized over four fixed views.
  • Surface damage: the largest contiguous region of missing or extra surface at the F_nat tolerance, as a share of the surface.
  • An object is severe if either region exceeds 2%.

3. Where Retire3D acts, it removes about a quarter of the time. The accelerated stage is 1.29× / 1.32× faster. Stages we do not touch change by under 1%. The gain grows with object size.

4. It multiplies with CFG distillation. On a single-call sampler, which is the call structure of a CFG-distilled model:

  • Retire3D keeps 97–99% of its stage speedup;
  • its fidelity to that sampler's own output is .976 / .934;
  • in the accelerated stage, removing CFG alone gives 1.99×, and adding Retire3D gives 2.64× (Direct3D-S2).

Contributions, as revised:

  1. The spatial-redundancy finding in coarse-to-fine 3D generation.
  2. A framework that makes spatially selective evaluation preserve the native geometry, through per-step state propagation and residual-triggered refresh.
  3. An evaluation by fidelity to the native output, local damage and end-to-end cost.

Corrections to Round 1.

  • Under the final configuration on 50 objects, the coarse score does not outperform a random partition. On TRELLIS.2 the difference is −0.4 pp, replacing +0.8 pp measured on 12 objects with an earlier configuration.
  • The first-order face-ratio difference on TRELLIS.2 does not hold on 100 objects.

The revision uses the new numbers.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.