acceptodds
Withdrawn from ICLR 2027

Orch3D: Global Compute Orchestration for Multi-Stage 3D Generation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Modern 3D generators are increasingly multi-stage, making efficient orchestration of their growing inference computation critical for practical deployment. However, existing acceleration methods largely optimize individual stages or operators in isolation, overlooking dependencies across both the generative pipeline and sparse execution. We find that stage-wise optima are non-compositional: changing an upstream schedule can reverse downstream schedule preferences, showing that computation must be optimized over the complete generation process. Based on this finding, we introduce Orch3D, a global compute orchestration framework built on two complementary components: Pareto Compute Optimizer (PaCO), which jointly optimizes cross-stage compute allocation and within-stage sampling schedules through allocation-conditioned Pareto search, and Packed Sparse Window Attention (PSWA), a structure-preserving sparse execution scheme that converts irregular window partitions into reusable packed attention schedules while preserving the original attention semantics. PaCO determines where and how much computation should be spent, while PSWA reuses window structure and cached boundaries to reduce the cost of retained computation. Under an equal 90-minute search budget, complete-policy joint search attains feasible deployment policies in every evaluated search seed, whereas the evaluated factorized procedure does not; frozen joint policies also improve reference-model fidelity on held-out scenes. PSWA adds semantics-preserving execution acceleration; together Orch3D reaches 1.895×–3.776× end-to-end speedup after transferring frozen policies to SA3DAO without retuning.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Multi-stage Sampling Optimization, Inference acceleration and orchestration, 3D Inference Acceleration

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 5 reviews, above 27% of the others.
Reviewer_mPyCOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates how to accelerate inference in multi-stage 3D generative models by optimizing the entire generation pipeline. It introduces two components:

  1. PaCO. It jointly searches for compute allocations across stages and sampling schedules within each stage, seeking Pareto-optimal trade-offs between generation quality and computational cost.

  2. PSWA. It organizes irregular sparse attention windows into reusable packed attention schedules and caches window boundaries, reducing execution overhead while preserving the original attention semantics.

Critical strengths
  1. the studied topic is very important.

  2. 3D generation is not easy. This paper presents a substantial amount of work.

  3. Combining 3D generation and MOO is new(at least for me).

Critical weaknesses
  1. The writing styles look like CVPR/ICCV, in ML papers, placing a figure before abstract is not very ususal.

  2. Too many bullets, such as make some concluding text bold and put them in front of a para. You'd better to use more words to analyze or tell your methods clearly, not to conclude them too much.

  3. line 284, it is unusual to directly put specific numbers into the main method.

  4. too many important things are referred to app. This makes the writing feels somewhat disjointed and lacks a smooth flow.

  5. pareto compute optimizer is lack of theoretical analysis.

  6. The relationship between PACO and PSWA is not very strong. Those two algs. seem pretty independent.

Questions for authors
  1. for the lex order, if two var.s are very close, they are deemed as approximately equal, will lex order in your work figure this issue out?

  2. consider using some packages in a multiobjective software called libmoon and conduct some baseline experiments.

  3. line 317, NSGA2 style search is too old. Can you provide a more advanced MOEA based method?

  4. table 1, why only numbers 3, 6, 12 are chosen, whose number is a bit small. How is its general ability.

  5. what is the main reason for the efficiency improvement? By algorithm or by implement? is efficiency a major contribution of this work?

  6. Is it possible to learn the entire PF/PS, called pareto set/front learning?

  7. how is a scaling law against number (m) objectives? is it O(m)?

  8. How much more time is needed to find N PO solutions instead of just one? Does it take O(N) times as long?

Reviewer_RB69OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces Orch3D, a framework for accelerating multi-stage 3D generation. It integrates PaCO, which jointly optimizes sampling budgets across stages and timestep placement within each stage, with PSWA, which improves the execution efficiency of sparse-window attention. The primary end-to-end experiments are performed on SAM 3D Objects.

Critical strengths
  1. The joint optimization of stage-wise sampling budgets and timestep placement is well motivated. The observed downstream preference reversals provide evidence that schedules optimized independently for individual stages may lead to suboptimal combinations.

  2. Using measured end-to-end latency to evaluate policies, rather than relying solely on model-evaluation counts, aligns the optimization objective with practical deployment requirements.

  3. The reported quality–latency trade-offs on SAM 3D Objects are encouraging. Experiments with fixed policies also help distinguish the execution-level gains attributable to PSWA.

Critical weaknesses

1. The full evaluation centers on a single backbone, providing insufficient evidence of broader applicability.

This is my principal concern. Although Orch3D is positioned as a framework for global compute orchestration in multi-stage 3D generation, its complete search procedure, component ablations, and end-to-end quality–latency evaluation are largely restricted to SAM 3D Objects. It is therefore difficult to determine whether the gains arise from generally applicable methodological improvements or depend substantially on this model’s specific stage organization, sampling redundancy, and sparse-attention implementation.

I recognize that the appendix includes experiments on TRELLIS. However, these primarily show that identical model-evaluation counts can translate into different actual latencies. They do not establish the end-to-end gains of deploying the complete Orch3D framework on TRELLIS and thus cannot replace a full evaluation on a second backbone.

Likewise, transferring frozen policies from LVIS to SA-3DAO provides evidence of cross-dataset generalization for the same backbone, but does not demonstrate transferability across distinct generation architectures.

2. Evidence across models remains limited for the paper’s central motivation.

The preference reversals identified on SAM 3D Objects support optimizing upstream and downstream schedules jointly. By contrast, the corresponding experiment on TRELLIS is deemed inconclusive under the authors’ predefined resolution criterion. While this does not imply that the phenomenon is absent, it leaves its prevalence across different models an open question.

Moreover, establishing the existence of cross-stage coupling is distinct from validating the advantages of PaCO’s particular search strategy. Pure global search reportedly recovers the same tier winners as PaCO at B64, while the contribution of certain refinement components remains unclear. Comparisons on another backbone would help determine whether the proposed search design delivers consistent practical benefits.

3. The transferability of PaCO and PSWA warrants separate evaluation.

PaCO could, in principle, be applied to other multi-stage generators, but the evaluated search space is defined around two stages and 24 predefined allocation configurations. PSWA, meanwhile, relies on reusable sparse-window partitions and token permutations. The two components consequently impose different architectural requirements and entail different adaptation efforts.

The paper should specify which aspects can be transferred directly, which require model-specific configuration or implementation, and whether each component continues to provide gains on a different backbone.

Questions for authors

See Weaknesses.

Reviewer_qd5hOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces Orch3D, which accelerates multi-stage 3D generation at the sampling-schedule and operator-execution levels through PaCO and PSWA. PaCO accounts for dependencies between the two generation stages by jointly searching cross-stage step allocations and within-stage timestep placements. PSWA improves computational efficiency through packed execution and metadata reuse while preserving window-attention semantics.

Critical strengths

The paper offers a valuable insight: acceleration strategies should be evaluated globally, accounting for the combined effects of both generation stages.

Critical weaknesses
  1. Limited justification for PaCO’s search strategy. PaCO relies on heuristic exploration and local refinement within a predefined configuration space, without guaranteeing global optimality or estimating the remaining optimality gap. Moreover, its B64 results match those of pure global exploration, leaving the benefit of its additional search mechanisms unclear.

  2. Limited methodological novelty of PSWA. PSWA primarily combines token reordering, packed variable-length attention, and metadata reuse. These are established implementation techniques supported by existing FlashAttention interfaces. The paper should clarify its technical contribution beyond applying these techniques to sparse 3D window attention.

Questions for authors
  1. Can the authors provide theoretical support for PaCO, such as conditions under which it can recover the globally optimal step configuration, or bounds on its optimality gap under a finite search budget?

  2. What are PSWA’s specific technical innovations in sparse attention computation?

Reviewer_ULfNOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces Orch3D for multi-stage 3D generation. It combines PaCO, which jointly searches stage-wise computation allocations and timestep schedules, with PSWA, which packs sparse attention windows into variable-length attention calls. The method reports 3.776× speedups, with the balanced policy achieving 3.28× acceleration.

Critical strengths
  1. The paper carefully shows that independently selected stage schedules may not compose into the best end-to-end policy.
  2. PSWA is a practical optimization that preserves the original masked-attention computation while providing consistent additional speedups.
  3. The frozen-policy transfer from LVIS to SA-3DAO and the reporting of multiple geometry improve the credibility of the evaluation.
Critical weaknesses
  1. The experiments justify complete-policy search, but do not establish the value of PaCO itself. The confidence intervals for both allocation memory and the latency anchor cross zero, and only 1 of 101 latency-anchor offspring becomes a new online incumbent.

  2. The search-baseline comparisons are not fully controlled. Fair NSGA-II is matched by logical proposal slots rather than fresh rollouts or GPU time. The aggregate PaCO–NSGA-II result is also statistically unresolved.

  3. The factorized baseline is structurally disadvantaged. It spends approximately 90–93% of its budget evaluating stage-wise references before testing complete pairs.

  4. Generality remains limited. Full PaCO–PSWA validation is conducted only on SAM 3D Objects. TRELLIS is used only for the equal-NFE latency observation, not for end-to-end policy search or quality evaluation.

Questions for authors
  1. Under an equal number of distinct complete-policy evaluations, does PaCO outperform Pure Global and Fair NSGA-II at both B64 and B128?

  2. Given that the sequential conditional selector recovers the joint optimum in Table 10, should the claim be narrowed from the necessity of complete-policy search to the failure of fixed independent stage rankings?

  3. Do the authors have existing evidence that the complete PaCO–PSWA pipeline transfers to TRELLIS, rather than only the equal-NFE latency observation?

Reviewer_wyNLOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Orch3D speeds up multi-stage 3D generation by jointly optimizing sampling steps across stages and making sparse attention more efficient. Its key insight is that optimizing each stage separately can hurt the overall result. It reports 3.28× faster generation, with all seven evaluated quality metrics within 5% of the original model.

Critical strengths
  • Meaningful practical result: reports 3.28× faster generation while keeping degradation below 5% on all seven evaluated quality metrics.
  • Clear motivation: demonstrates that optimizing stages separately can produce inferior overall schedules.
  • Useful controlled experiments: separates scheduling gains from attention-execution gains and evaluates frozen schedules on another dataset without retuning.
Critical weaknesses
  • PaCO’s added value is unclear: pure global search matches its selected results at the default budget; confidence intervals for the memory improvements include zero.
  • The factorized baseline is questionable: it spends 90–93% of its budget before evaluating complete stage combinations, which limits what its failure establishes.
  • PSWA’s novelty needs scrutiny: the paper should clarify its advance over existing packed/variable-length attention implementations.
  • Limited generality and timing comparability: the main deployment evaluation uses one generator and 100 objects; cross-method timings are not paired on the same GPU.
Questions for authors

In page 22, Appendix D.3, it has this peculiar line:

The pre-amendment study observed 0/32 tier-winner changes under one-repeat timing on the measured early pool; the amendment requires a final full-pool stability recheck before submission. 

This reads very similar to my conversation with my LLM agent. Did you do a final full-pool stability recheck before submission? Why or why not?

If you don't think a final full-pool stability recheck is necessary, then why did you include this sentence in the paper? If you do think it is necessary, then why are you submitting intentionally incomplete work to ICLR?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.