acceptodds
Withdrawn from ICLR 2027

ScalingEdit: Scaling Real-World 3D Object Editing with Self-Supervised Training

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

3D object editing aims to rotate, scale, and translate a target object while preserving its identity and surrounding scene. Scaling up this capability is hindered by the scarcity of real-world editing pairs with geometric supervision. Existing multi-view videos offer diverse object observations and camera poses, but camera motion changes both the object viewpoint and the background, creating a mismatch with scene-preserving editing. To address these challenges, we introduce ScalingEdit, a self-supervised framework that turns these observations into training data for geometry-controlled object editing. Specifically, we formulate object editing as a foreground transformation within a shared scene context. We construct scene-aligned training pairs by compositing the source foreground onto the completed target-view background while retaining the unmodified target image as supervision. This construction aligns training with the scene-preserving editing task and enables learning from real multi-view observations without dedicated captures of object manipulation. A geometry-conditioning module encodes relative camera pose and image-plane displacement to provide explicit control over object rotation, scaling, and translation. Experiments on ObjectMover-A and our RotationBench demonstrate improvements over the evaluated baselines in appearance fidelity and geometric accuracy. Our data-scaling experiments further demonstrate the potential to improve geometric object editing by increasing real-world multi-view supervision.

Primary area:
generative models
Keywords:
Image edit, Object edit

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_X6puOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This is mainly a data-centric approach for 3D-aware image editing. The authors want to turn existing real-world multi-view videos into useful editing pairs, and use these pairs to adapt a pretrained editor. The final output is still an edited 2D image. So the main motivation seems to be making geometric editing easier to train at scale. The main contribution is the training pair construction. They put the source foreground onto the completed target-view background, while keeping the real target image as supervision. And This approach can be easily scaled up.

Critical strengths
  1. There is some useful evidence here. The real-versus-UE comparison supports the proposed data pipeline as a whole, and increasing the training set from 10k to 40k improves performance. I would not say the data construction is completely unvalidated.

  2. The method gives promising rotation and translation results against the evaluated baselines, and the scaling examples also look reasonable, although scaling is mainly evaluated qualitatively. The category-disjoint rotation test and the fixed-step data-scaling study are positive points.

  3. The overall formulation is relatively simple, and using a pretrained model with lightweight adaptation seems practical. It may also be adaptable to newer backbones, but that is a possibility rather than something demonstrated in this paper. The writing is generally organized and easy to follow.

Critical weaknesses
  1. Compared with pipelines involving mesh reconstruction, rendering and harmonization, this could reduce some preprocessing steps and avoid errors in synthesized targets. But I did not find a direct comparison that clearly shows the advantage over closely related approaches such as Ctrl&Shift, 3D-Fixup or ObjectMover (Already mentioned in main paper but not compared).

  2. Current ablations mostly remove geometry conditioning or the reference image. They do not directly isolate the main data-construction idea. Another important detail from the supplementary code(this is pointed out by LLM tools): the actual base model is Wan2.1-VACE-14B, with LoRA on the VACE branch. Comparing against VACE-1.3B does not really convince me in ablation part. The paper does not establish a clear cost advantage over previous methods. Data preparation cost and inference latency are still missing.

  3. Learning editing from videos, keeping real frames as targets, and separating object changes from background/camera changes all have related precedents, especially in Magic Fixup, 3D-Fixup and BlenderFusion. The particular combination here may still be useful. But the architectural novelty looks limited, and the main additional contribution seems to be a simpler data pipeline. With the current evidence, it reads somewhat more like a practical pipeline report.

  4. (Minor) In Figure 2, the x-axis already uses labels like “10k” and “40k”, while the axis title also says “×1k”. This looks redundant and should be fixed. The appendix contents also omit Section D and its subsections.

Questions for authors
  1. Where the improvement comes from? (scene alignment/real target images/the object/viewpoint coverage/pretrained model ...)

  2. It would help to explain exactly what is improved comparing with previous work: data quality(already done part of), preparation cost, downstream model performance, or some combination of these.

  3. A VACE-14B baseline and a controlled comparison using the same backbone would be quite important.

Reviewer_oUJeOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

ScalingEdit converts object-centric multi-view video (MVImgNet2.0) into supervision for scene-preserving object editing. It pastes the source-view foreground onto the target view with the object removed (ObjectClear), and supervises with the unmodified target frame. A Wan2.1-14B backbone (rank-32 LoRA) is conditioned on an 8D relative-pose descriptor, a reference image, and an editing mask. The paper reports gains on ObjectMover-A and a new 200-pair RotationBench, plus a 10k→40k data-scaling study.

Critical strengths
  1. Scalable data recipe. Real target frames serve as supervision without per-sample 3D reconstruction or rendering. This is the clearest difference from Ctrl&Shift, whose real-data targets are rendered meshes harmonized into the scene.
  2. Real-image evaluation on held-out categories.
  3. Monotonic real-data scaling, while the synthetic (Unreal Engine) curve plateaus.
  4. Geometry conditioning is shown to matter within the model. Restoring it improves PSNR from 20.94 to 22.38 dB and Pose MAPE from 72.58% to 28.01% (Table 3).
Critical weaknesses
  1. The core contribution is not isolated. Ablations (Sec. 4.5) only remove conditioning components. Missing comparisons, at matched backbone and compute:

    • raw (I_S → I_T) pairs;
    • source-background inputs with a masked loss;
    • Ctrl&Shift-style rendered targets.

    Without these, gains from the data construction cannot be separated from the 14B backbone. The first two need no new preprocessing, since they reuse the existing 40k pairs; a reduced-scale run (e.g., 10k pairs) would suffice for the rebuttal. The third can be addressed through published benchmark results.

  2. Inherited design is not attributed. The following closely follow Ctrl&Shift (ICLR 2026) and should be credited in Sec. 3.3 and Sec. 4:

    • the 8D descriptor (axis-angle, relative translation, 2D image-plane shift);
    • a Fourier-encoded MLP producing 8 cross-attention tokens of width 4096;
    • the reference image plus an estimated target mask;
    • the Pose MAPE and Object IoU metrics, whose precedent should be acknowledged.

    Magic Fixup's precedent (inputs constructed from video, supervised by real target frames) also deserves direct contrast. A closely related prior work is missing entirely: Zhang et al., "Geometric Image Editing via Effects-Sensitive In-Context Inpainting with Diffusion Transformers" (ICLR 2026; code at PRIS-CV/GeoEdit). It addresses translation, rotation, and scaling with explicit shadow and lighting modeling, and introduces a 120k-pair geometric-editing dataset (RS-Objects). It should be cited, contrasted, and distinguished from the cited GeoEdit (He et al., ECCV 2026), which shares its name.

  3. Comparisons are incomplete or not comparable.

    • (a) ObjectMover-A. Published results under the original protocol are omitted: ObjectMover reports 25.27 dB / DINO 85.07, and Ctrl&Shift reports 28.69 dB / DINO 88.07. ScalingEdit's 29.02 dB exceeds Ctrl&Shift by only 0.33 dB. Re-runs of the same baseline differ across papers by 0.5–2.6 dB (e.g., ObjectMover 25.27 vs 24.06; Paint-by-Example 20.83 vs 23.43), so the 0.33 dB gap does not support a superiority claim without a verified common protocol. The paper's baseline identity scores also disagree with independent reports: MagicFixup's DINO is 0.581 here vs 78.49 (ObjectMover paper) and 80.90 (RoPEMover, target crop). App. B.3 defines a generic image-pair score and separately describes the original ObjectMover crop protocol, but does not state which one produced Table 1. If it is the original protocol, ScalingEdit trails both ObjectMover and Ctrl&Shift on DINO (0.737 vs 0.851 / 0.881, on the same 0–1 scale) and DreamSim (0.239 vs 0.142 / 0.075). If it is not, Table 1 cannot be compared with published results.
    • (b) RotationBench control asymmetry. Only 3DIT receives a 3D rotation input. DragAnything and VACE receive only a 2D trajectory or box, and Nano-Banana receives yaw/pitch as text. VACE-14B, from the same Wan2.1 family, should replace VACE-1.3B.
    • (c) Missing rotation-capable editors. GeoDiffuser has public code and has already been evaluated on ObjectMover-A by other work, so it is rebuttal-feasible. Diffusion Handles, BlenderFusion, and 3D-Fixup are cited but not compared; comparisons where code is available would strengthen the paper.
    • (d) Resolution penalty. Full-image PSNR penalizes baselines run at lower native resolution: 3DIT at 256², DragAnything at 320×576, versus 768×432 here.
    • (e) Suggested qualitative comparison. Current closed image editors such as ChatGPT Images and Reve support region-based edits and handle moving and rotating objects well. A qualitative comparison on a subset of RotationBench and ObjectMover-A, using region-based prompts to move or rotate the object, would show whether explicit pose conditioning beats state-of-the-art general editors.
  4. Train/inference gap and physics.

    • Source-shadow removal is not explicitly supervised. Cast shadows and reflections generally lie outside the extracted foreground, and the foreground is pasted onto a background intended to be cleaned of object and effects. The construction therefore provides no paired supervision for removing a source-location shadow or reflection, which real inputs often contain. Please evaluate cases where such effects are prominent.
    • Camera motion is not object rotation under fixed light. With a static object and a moving camera, the light–object relationship never changes, so the supervision corresponds to lights rotating together with the object. A physical rotation under fixed scene illumination changes the object's shading and its cast shadow. Please distinguish learning cross-view appearance from demonstrating physically consistent rotation, particularly at larger angles.
    • Scale is controlled through mask size. A scale request only resizes the estimated target mask and is not part of the geometry descriptor (Sec. 3.3, A.1). Scale accuracy is not reported separately, and apparent resizing is not distinguished from physical 3D scaling.
  5. The RotationBench protocol is unclear. Source and target views differ by 20–67° of camera rotation, yet full-image PSNR is computed against I_T.

    • If the input is the raw I_S, PSNR mainly measures background viewpoint mismatch; all methods fall within 19.6–22.4 dB.
    • If the input is C(B_T, F_S), target-view information leaks into the input, and comparison with baselines given I_S is unfair.
    • It is not stated how the estimated mask M̂ is obtained. If it comes from M_T, Object IoU is compromised.

    Further issues:

    • 199 of 200 pairs fall in one rotation band, so there is no difficulty breakdown.
    • Pose MAPE ignores roll and is unbounded for small deltas (3DIT: 299.87%).
    • The provenance of the "images collected online" is unclear.
  6. The scaling study is under-specified. Every data size uses a fixed budget of 62.5k steps (batch 64, ≈4M samples). That reasonably tests whether more diverse data helps at fixed compute. The open questions:

    • Are the subsets nested?
    • Does a larger pair count add objects and categories, or more views of the same objects?
    • Are the real and UE sets matched in object diversity and transformation distribution? The UE dataset (assets, size, construction) is undocumented.
    • How variable are results across seeds or sampled subsets? Results are single-run, and the gain is modest (+0.65 dB).

    The evidence supports "this real-data pipeline scales better than this UE pipeline in this setting," not a general claim that synthetic supervision saturates.

  7. The cost–quality tradeoff is unreported and confounded.

    • Scale gap. The 14B backbone is about 10× Ctrl&Shift (1.3B plus a control branch) and most baselines (VACE-1.3B and SD/SVD-era editors).
    • Missing costs. App. C reports training cost (4×H100 for 30 h) and 28 denoising steps. Inference latency, peak memory, and per-pair preprocessing cost (COLMAP, segmentation, ObjectClear) are not reported.
    • What the ablation does and does not show. The ablation establishes that geometry conditioning helps within ScalingEdit. It does not isolate the scene-aligned data construction or explain the advantage over smaller baselines. The "w/o Geometry" variant already beats every RotationBench baseline on PSNR (20.94 vs ≤20.75), so the PSNR margin over baselines likely owes much to backbone scale.
Questions for authors
  1. What do the data-construction ablations in W1 show under matched backbone and compute?
  2. On RotationBench, what exactly are I_ctx and M̂ at test time, for ScalingEdit and for each baseline?
  3. Which ObjectMover-A protocol was used (crop vs full image, resolution, source removal for insertion baselines)? Please report results under the original protocol next to ObjectMover and Ctrl&Shift.
  4. How does ScalingEdit compare with VACE-14B, GeoDiffuser, and Diffusion Handles on RotationBench?
  5. Can you show the training intermediates (I_S, I_T, M_T, B_T, Ĩ) and inference cases under strong directional light, including removal of the source shadow? Please also confirm that ObjectClear removes the target's shadows and reflections, since any residue would leak target-view lighting into the input.
  6. Can you report angular (degrees), translation, and scale errors separately, results binned by rotation magnitude, and answer the scaling-study questions in W6?
  7. What are per-image latency and peak memory for locally runnable methods on a common GPU, API latency and model versions for closed models, and the preprocessing cost per training pair? How does the method perform on a 1.3B backbone?
Reviewer_eNUTOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces a training method for 3 kinds of 3D object movements and claims to be scalable. It also gives a benchmark to evaluate rotation. The experiments were focused on comparisons with traditional 3D editing pipelines.

Critical strengths

It has a plain and clear description of the method, and the arrangement of text and visualizations is easy to follow.

Critical weaknesses
  1. Limited scope and value. This paper focuses on a restricted set of movements. Considering the broad capabilities of current general-purpose image editing methods, which are also able to get in camera conditions, a model trained for a narrow task set needs a stronger support rather than a self-constructed test set for its value.

  2. The ablation focused on modules in the model architecture, but for a self-supervised method, the most important ablation should isolate the contribution of your training signal. Due to the limited time in rebuttal period, it is hard to get a controlled comparison with varying supervision construction.

  3. The self-supervised pipeline relies on supervised-pretrained models. In particular, ObjectClear for object masking, Wan2.1-14B for backbone. It should be noted that how these pretrained components affect the the core self-supervised training pair construction.

Questions for authors
  1. Given the availability of modern general-purpose image generation models and languge model for 3D modeling and editing workflows, what concrete problem does ScalingEdit address? The demonstrated scope is limited to simple movements like object translation, scaling and rotation, which are basically linear transformations, while at the same time we can change the object's material or physics properties in an image with modern methods.

  2. The efficiency of this method should be disclosed. It uses a Wan2.1-14B backbone and 28 denoising steps, which would not be a low inference cost. Please provide end-to-end latency per edit and peak GPU memory.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.