acceptodds
Withdrawn from ICLR 2027

DesignOrbit

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Early concepts for buildings, pavilions, and public installations often exist as a single sketch or image, yet judging them requires seeing their unseen sides and how they sit in their surroundings. We study generative design in context: from one image and a text description, we sample alternative designs as loop-closed orbit videos, each proposing plausible unseen sides while keeping the design and its surroundings consistent across the full orbit. We introduce DesignOrbit, which learns this behavior without any real architectural orbit videos. Procedural cuboid buildings rendered along known camera paths, combined with loop-closed background inpainting, provide initial supervision. A structure-from-motion verifier then selects the model's own generations for self-training based on frame registration and camera coverage, and later supplies preference pairs for direct preference optimization, extending training beyond the procedural shapes while increasing camera coverage. On architecture, camera coverage rises from to , against for the strongest baseline, while diversity in unseen rear views is retained. Although trained only on synthetic building videos, the model also generates orbits for sketches, playgrounds, ships, interiors, and urban scenes without further fine-tuning. In a study with 50 participants, 80% with a background in architecture, design, or 3D visualization and over half with five or more years of experience, DesignOrbit was preferred over the baselines in 91% of ratings that expressed a preference.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Video Generation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_ZHPoOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

A nice fine-tuned model that allow designers to "rotate" an input image as if it is made in 3D.

It is nice because artist sketches are in 2D, yet we'd like to see them from different angles. This method preserves the creative expression of the artist, and having AI to augment it to different view points.

Critical strengths

It works.

nice initial synthetic data generation pipeline and the following verifiable reward during an RL phase. very clever.

Critical weaknesses

In their human study, they don't ask the designers to use the tool in the manner of Figure 2.

In a proper study, they should get desginers who actually spend time drawing a novel design, along with a prompt, and have their tools make it "rotatable". Then, they can ask the designers some in-depth questions on if the tool was useful in any way, and what are some rooms for improvement.

More technically, since the designers did not give the initial sketch by hand, we do not know if the proposed method handles out of domain generalization well. For example, if I give it a very rough pen/paper sketch, would the proopsed method even work? Can it turn a concept sketch like this into rotatable form? https://thereader.mitpress.mit.edu/frank-gehry-and-the-art-of-drawing/

Ultimately, for a project that's so intimately related to a designer's self expression, this project failed to capture the human interaction in its studies. Its participants are reduced to robotically labeling either they preferred A or B. I find this extremely dehumanizing and disappointing.

Questions for authors

Do a proper human study where you give designers creative prompts and have them actually create a novel design in 2D, using whichever tools they like. Then, run this through your tool to make it rotatable in 3D. Have designers play with your tools for 1 hour in free-form, and get detailed feedbacks from them. Report these findings as well.

It would cost you 40 hours of labeling time (assuming you hire 40 designers), which would be very cheap in comparison to fine-tun a 14B model. It would do your work justice.

Do you have an IRB for your subject study?

Reviewer_YWYYOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes generating orbit videos from a concept image and a text description to help designers explore other sides of a design shown in a single image. The method has three training stages: training on procedurally generated cuboid buildings, self-training on filtered model-generated videos, and preference-based training to improve camera coverage. The paper evaluates the method through baseline comparisons, stage-wise ablations, examples across different input types, and a user study with 50 participants. DesignOrbit achieves an high orbit pass rate on architectural inputs, higher than baselines reported in the main paper. Overall, the work may have potential value for presenting early concept designs and comparing design alternatives.

Critical strengths

The main paper is not easy to follow, but from the supp material and web page, we can see the orbit videos are impressive in comparison with other methods. The concept of seeing unseen sides of a concept sketch or an architecture image may have potential value for early stage design in the work flow.

To solve the above issues, the strategy used in this paper is well concerned: adopt the synthetic data for the video finetunning due to limited real videos; SfM-based verification for curating synthetic data, and direct preference optimization for the final generation.

Critical weaknesses
  1. unclear motivation As the original idea for early concept design in design workflow such as architectural design, this proposal did not follow the declared workflow, but only a refined video generation model. There is no such support for orbit alternative, client review, and design update in current pipeline. If only for the early design with alternative views, maybe the multiview image generation is suitable rather than a video generation. The comparisons with the solutions among consistent image and video generation approaches can be compared. The result evaluation did not touch any elements of such design work flow. The user study invited 50 participants "80% with a background in architecture, design, or 3D visualization", actually a diverse backgrounds rather than design. The study is for final generated results, not design workflow as depicted in Fig 2.

  2. unclear framework This proposed framework is fine-tuned video generation with user preference, but the important figure Fig.3 did not show the main contributions, but with some details techniques, it makes readers difficult to understand. The actual model is Wan2.1-FLF2V-14B. Even the paper added a good story of design workflow, but fundamentally it is different work. We can see the improvement on the base models, but it is not surprising with such fine-tunning and preference optimization, I did not see the core technical contributions for such research topic.

Questions for authors
  1. There is no details of synthetic orbit data. From the main paper and supp material, only one rendered example was shown. How about the other procedural models and in-painting results. The are lots of existing procedural models in CG tools such as blender, why the paper (Tsai & Hariharan, 2025) was selected? I understand the limited data of real orbit video of buildings, but we still can see lots of drone camera data and the other real data, I am wondering what will be the results fine-tunned with real data?

  2. camera coverage rises from 28.77◦ to 342.13◦ in abstract. Table 2 shows the max of 337.33◦. Sec 4.3 reported "we find that coverage rises from 318◦ to 342◦", which data is true?

  3. Sec 5.1 lists lots of baseline methods, but Table 1 only reported part of them. As an independent paper, the paper should be self-included without the app material.

  4. For evaluation, no objective evaluation with metrics were given.

  5. Sec 5.4 loses detailed discussion, and Table 6 should be Table 3.

Reviewer_6bGtOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper introduces “generative design in context”: from a single concept image and a text prompt, generate loop-closed 360° orbit videos that propose plausible unseen sides of a design while preserving its surroundings. DesignOrbit fine-tunes Wan2.1-FLF2V-14B without any real orbit videos, in three stages: 1.SFT on procedural cuboid buildings with split-and-condition background inpainting. 2.Self-training on the model’s own generations, filtered by a COLMAP-based verifier (all frames registered, coverage ≥351°). 3.DPO with pairs ranked by camera coverage C(x). The paper reports coverage rising from 28.77° to 342.13° on architecture, and a 50-participant user study in which DesignOrbit wins 91% of ratings that express a preference.

Critical strengths

1.The problem is well motivated and practically relevant. Early design review needs multi-view visualization in context, and existing 3D generators discard the scene. 2.The pipeline is data-efficient. About 600 clips and 350 preference pairs take the model from zero verifier-passing base samples to high coverage. Bootstrapping from synthetic data and then self-training with a geometric verifier is an interesting recipe. 3.The training pipeline also transfers to HunyuanVideo (Table 3), so it is not tied to one backbone. Qualitative results extend beyond the training domain (sketches, playgrounds, ships, interiors).

Critical weaknesses

W1. The verifier is the foundation of the method, but its reliability is never validated. Data filtering, DPO scoring and the main quantitative evaluation all rely on the same rule-based COLMAP verifier, yet the paper reports neither its agreement with human judgment (e.g. precision/recall) nor any analysis of its failure cases. There are five specific problems:

  • It cannot see the design itself. The verifier does not mask the foreground. With a texture-rich background, a video can register successfully, and pass, even when the central design deforms. The orbit center is fitted from camera positions rather than the design’s location, so a trajectory circling a point offset from the design can still score highly.
  • It has little discriminative power. 90% of Stage II generations register all frames, so the registration check does almost nothing. The only effective filter is coverage C(x), which only reflects how far the camera rotates. Known SfM failure modes are not discussed. Symmetric buildings and repeated textures can cause wrong poses and false accepts, while low-texture surfaces such as glass facades and white walls can cause registration failures and false rejects. These are common in architecture.
  • The reward can be gamed. C(x) measures neither fidelity to the input image nor design quality. Used as the DPO reward, it may push the model to complete the orbit at the expense of the design, and the paper provides no evidence ruling this out.
  • The model is evaluated on its own optimization target. C(x) is both the training signal and the main evaluation metric. The only other geometric metric, foreground point count, is weak: more points does not mean more consistent geometry, and the count depends on texture. The quantitative gains therefore do not show that geometric quality actually improves; they may simply reflect overfitting to this metric. The paper lacks independent metrics such as first-to-last-frame loop-closure error, multi-view consistency (e.g. MEt3R), similarity to the input image, video quality (FVD) and diversity. The abstract’s claim that rear-view diversity is “retained” rests only on the user study.

W2. Training and test data are underspecified, so the generalization claims lack support.

  • The source of training inputs is unspecified. The main text never says where the concept images and prompts for Stage II and III come from. Test inputs are produced by an LLM prompt → T2V model → first-frame pipeline. If training inputs come from the same pipeline, the training and test distributions overlap heavily, and the paper does not state whether the two are strictly disjoint. The test-set size is not reported, and there is no quantitative evaluation on real photos or real sketches.
  • Sampling scale and acceptance rates are missing. The main text reports only the roughly 300 clips that survive filtering. Without the total number of samples, it is impossible to judge how strict the verifier is or how costly self-training is. Training-data diversity is not described, so the source of the cross-domain ability is unclear. The category and style distribution of the free-form buildings is not described, and no control experiment separates how much of the out-of-domain ability comes from the base model’s prior versus the proposed training. Outside the training domain, the margin over DimensionX is negligible: Cov≥300° is 74.6% vs. 74.5%, foreground points are 1878 vs. 1792, and neither comparison has variance. The claim of outperforming all baselines beyond the training domain is overstated.

W3. Key numbers are inconsistent across the paper.

  • The abstract and Sec. 4.3 report coverage of 342.13° / 342°, while Table 2 reports an architecture average of 337.33°. Table 2’s column headers appear swapped: the “Ave.” column holds percentages and the “Cov≥300°” column holds degrees. One Table 3 entry (55.30%) exactly matches Stable Virtual Camera’s figure in Table 2.
  • Table 3’s base-model row reports 48.40% of samples at ≥300°, which contradicts both the claim that the base model “never” completes an orbit and its 28.77° average coverage.
  • The stage configuration in Table 3 is unclear, and the text cites a non-existent “Table 6”.
Questions for authors

(W1) Can you report the verifier’s precision/recall on a human-annotated subset? Do the conclusions hold when coverage is computed on the foreground only? Can you add metrics independent of C(x) to show that DPO does not sacrifice fidelity to the input? (W2) Where do the Stage II and III inputs come from? How many videos were sampled at each stage, and what were the acceptance rates? How large is the test set, and how is it separated from the training inputs? (W3) Which evaluation sets do 342.13° and 337.33° correspond to? Please clarify Table 2’s headers and the training configuration of each row in Table 3.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.