acceptodds
Under review as a conference paper at ICLR 2027

EditHero: A Benchmark for Long-Horizon Part-Level 3D Editing and Vibe Modeling

Abstract

3D editing methods are typically evaluated one edit at a time, although creating an asset requires a sequence of revisions. A method must follow each new instruction while preserving earlier edits and untouched regions, yet this ability has not been systematically measured. We introduce EditHero, to our knowledge the first benchmark for long-horizon 3D editing from image and natural-language instructions covering both geometry and texture. A deterministic assembly engine produces the target after every edit, and manual review verifies each sequence. Evaluating four 3D editing methods and six large language models (LLMs) reveals that conventional methods often miss the requested changes and alter regions meant to be preserved. Most LLMs that edit through code follow instructions more accurately, and all of them preserve unchanged regions well, but their time and token costs limit interactive use. We release the synthesis engine and edit sequences to support research on reliable iterative 3D editing.

Primary area:
datasets and benchmarks
Keywords:
3D Generation, 3D editing, LLM, agent

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_FWrYOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

EditHero proposes a benchmark for evaluating long-chain, part-level 3D editing, addressing the limitation of isolated edit evaluation. It includes 485 chains and 2965 edits across 255 objects, with corresponding ground truth after each edit. Introduce evaluation using a self-rollout protocol and region-specific metrics.

Critical strengths
  1. The strongest contribution is reconstructing every intermediate state, enabling reliable evaluation. This directly targets the limitation of isolated edit benchmarks
  2. The use of self-rollout is intuitive and can measure error propagation well over multiple sequential edits. Also separation of edited and unchanged parts solved inflated valued which may occur while evaluating.
  3. Manual intervention while generating JSON enhances the reliability of the benchmark.
Critical weaknesses
  1. All the conventional methods are built on top of TRELLIS latent. This represents a significant limitation in diversity when compared with the benchmark. The results are meaningful at the system level, but an identity reconstruction baseline would help establish how much degradation is specifically attributable to editing.
  2. Code-editing agents operate directly on part-structured meshes with iterative render-and-compare refinement, while conventional editors do not have equivalent capabilities or inference budgets. The LLM evaluation also uses only 55 hand-selected chains. These differences limit how strongly the results support a general advantage for code-based editing. Additionally, with a median chain length of six, the longer-horizon claims would benefit from more direct analysis.
Questions for authors
  1. How much of the preservation degradation comes from TRELLIS encoding/decoding itself? Could the authors report an identity reconstruction baseline?
  2. Do the code-editing agents receive semantically named part nodes? How does their performance change without this part decomposition?
  3. Could the authors clarify whether instructions and target renders are revealed one turn at a time, and report performance separately on the held-out views?
  4. Could the authors provide per-turn results for the LLM agents and a more detailed analysis of chains beyond 10 turns?
  5. Is there a particular reason why only 55 chains are evaluated on LLMs? Can you report on a larger subset, as now it’s less than 15% of the set?
3 comments

Thank you for the helpful and thorough review. We address each of your questions and comments below.

Q1. Reconstruction loss

How much of the preservation degradation comes from TRELLIS encoding/decoding itself? Could the authors report an identity reconstruction baseline?

The 4 TRELLIS-based editors encode the source asset once, propagate their latent state across turns, and decode the output at each turn. We evaluated both single encode–decode and repeated encode–decode reconstruction on the 55-chain set:

MethodDelivered / TriedIF ↑CCprev ↑F_W ↑LPIPS_E ↓LPIPS_W ↓PSNR_W ↑
No-op (original GT source)416/4160.001.000.750.5670.15418.2
TRELLIS single-pass reconstruction416/4160.010.980.700.5820.19316.5
TRELLIS repeated reconstruction416/4160.020.860.640.6150.24114.6

Single-pass reconstruction maintains a CCprev of 0.976, well above all 4 conventional editors (0.732–0.900), indicating that most preservation loss arises from the editing updates rather than a single encoder–decoder pass.

For reference, repeated encode–decode reconstruction yields a CCprev of 0.864, illustrating error accumulation when meshes are repeatedly re-encoded.

Q2. Semantic part names and merged meshes

Do the code-editing agents receive semantically named part nodes? How does their performance change without this part decomposition?

Yes, the original source meshes include semantic node labels (e.g., slot_helmet and slot_boot_left).

To examine how strongly agents depend on semantic labels and part decomposition, we evaluated Opus 5.5 and Sol 6 on 10 chains (82 turns) after stripping semantic names (retaining geometry parts). We also tried merging the initial parts into a single mesh for agents. All runs used medium reasoning under the turn-by-turn sandbox protocol. While scores decrease slightly without semantic names or part separation, LLM agents still substantially outperform conventional editors:

MethodDelivered / TriedIF ↑CCprev ↑F_W ↑LPIPS_E ↓LPIPS_W ↓PSNR_W ↑
Opus 5.5, semantic names82/820.690.990.880.2220.05527.2
Opus 5.5, semantic names removed82/820.600.980.860.2670.06726.2
Opus 5.5, merged mesh82/820.560.990.820.2770.07025.7
Sol 6, semantic names82/820.470.970.780.3310.09325.0
Sol 6, semantic names removed82/820.460.960.760.3430.10224.6
Sol 6, merged mesh82/820.430.950.750.3740.11624.1
PartFlow82/820.100.880.650.5650.20616.0
Nano3D82/820.240.900.710.4680.15317.9
3DEditFormer82/820.210.820.600.5060.19116.2
VoxHammer73/820.310.780.550.5400.21715.3

Q3. Turn-by-turn inputs and held-out views

Could the authors clarify whether instructions and target renders are revealed one turn at a time, and report performance separately on the held-out views?

Turn-by-turn inputs. In the original workspace, the full sequence of instructions and target renders was accessible. To strictly enforce causal turn-by-turn execution within the rebuttal timeframe, we re-evaluated 3 representative models—Astra, Sol 6, and Opus 5.5—across all 55 chains in a sealed sandbox: at each turn, the agent receives only the current instruction and target image and submits a single output mesh that serves as the next turn's input, with future instructions and renders kept strictly hidden. All 3 agents completed all 55 chains (416 turns):

MethodDelivered / TriedIF ↑CCprev ↑F_W ↑LPIPS_E ↓LPIPS_W ↓PSNR_W ↑
Astra, original (Table 3)416/4160.520.980.810.2890.07825.5
Astra, sandbox416/4160.580.980.820.2510.06826.1
Sol 6, original (Table 3)416/4160.420.970.790.3440.09324.6
Sol 6, sandbox416/4160.510.980.810.2840.08025.2
Opus 5.5, original (Table 3)416/4160.620.980.820.2360.06526.7
Opus 5.5, sandbox416/4160.670.980.840.2120.05727.1

Sandbox performance matches or even slightly exceeds the original workspace results, confirming that sequential editing does not rely on access to future turns much. We will incorporate this strict turn-by-turn protocol into the revised paper.

Held-out views. Appendix K (Table 14) already reports held-out view metrics for the 4 conventional editors, and Figures 15 and 16 visualize LLM outputs from non-conditioning views. Here we provide the corresponding quantitative results for all 6 LLMs using the PSNR and LPIPS metrics of Table 14. Subscript 0 denotes the conditioning view, and 1–3 denotes held-out views:

MethodPSNR₀ ↑PSNR₁–₃ ↑LPIPS₀ ↓LPIPS₁–₃ ↓
Astra, original25.325.60.0760.079
Astra, sandbox26.026.10.0640.069
Sol 6, original24.524.60.0910.094
Sol 6, sandbox25.725.10.0700.084
Opus 5.5, original26.826.60.0580.067
Opus 5.5, sandbox27.726.90.0490.060
Fable 5.125.325.40.0750.082
GLM 5.320.821.40.1410.134
DeepSeek 4.1 Flash23.924.10.1020.108

Performance remains consistent between conditioning and held-out views. We will add these quantitative tables to the revision.

Q4. Per-turn results and longer chains

Could the authors provide per-turn results for the LLM agents and a more detailed analysis of chains beyond 10 turns?

Following Figures 4 and 8 and Table 8, we will include per-turn LLM results under the sandbox protocol in the revised manuscript. The table below mirrors the format of Table 8:

Metric / Method1234567
Chains55555554504635
IoU ↑
Astra0.9070.8420.8110.7830.7760.7660.767
Sol 60.8940.8220.7910.7600.7460.7400.734
Opus 5.50.9090.8580.8230.8100.8000.7920.786
IF ↑
Astra0.5770.5130.5350.5720.5530.6670.688
Sol 60.5010.4250.5030.5260.5050.5720.647
Opus 5.50.6730.6780.5950.6780.6360.7500.671
CC₀ ↑
Astra0.9860.9720.9600.9530.9530.9590.964
Sol 60.9780.9540.9480.9360.9250.9320.915
Opus 5.50.9730.9660.9540.9610.9640.9600.958

For chains exceeding 10 turns, all 6 LLMs and the 3 sandbox reruns completed the same 11-, 15-, and 19-turn chains. Following Table 9, we report first-turn → last-turn mean scores:

ModelSettingIoU ↑: first → lastCC₀ ↑: first → last
AstraOriginal0.97 → 0.660.99 → 0.63
AstraSandbox0.97 → 0.710.99 → 0.65
Sol 6Original0.96 → 0.600.99 → 0.60
Sol 6Sandbox0.97 → 0.670.99 → 0.64
Opus 5.5Original0.97 → 0.700.99 → 0.65
Opus 5.5Sandbox0.96 → 0.770.98 → 0.97
Fable 5.1Original0.97 → 0.610.98 → 0.62
GLM 5.3Original0.93 → 0.570.95 → 0.60
DeepSeek 4.1 FlashOriginal0.97 → 0.570.99 → 0.58

On the 3 chains longer than 10 turns, Opus 5.5 achieves the highest final IoU (0.77) in the sandbox.

Q5. Why 55 chains? Can the evaluation be expanded?

Is there a particular reason why only 55 chains are evaluated on LLMs? Can you report on a larger subset, as now it’s less than 15% of the set?

We used 55 chains for two main reasons: the sample size already produces stable comparative conclusions, and evaluating all 6 LLMs on additional chains is computationally expensive.

Notably, each LLM performs 416 edits across 55 chains (2,496 edits total across the 6 LLMs), surpassing the per-model edit scale of benchmarks, such as BlenderGym (245 editing tasks) and EditBench3D (240 scene–edit pairs). Table 12 details the token and runtime costs.

To verify that our conclusions are robust to subset size and selection, we performed two additional checks:
1. Selection stability: Randomly sampling 20 of the 55 chains 1,000 times (evaluating all 6 LLMs on the identical sample per draw) yields high rank stability:

MetricMean Spearman rank correlation with full 55Mean absolute score change
IF0.9460.0360
CCprev0.9210.0056
CC₀0.9150.0148

2. Out-of-sample evaluation: Evaluating Astra and Sol 6 on 20 additional unseen chains under the turn-by-turn sandbox protocol preserves the same Astra–Sol 6 ordering on IF, CCprev, and F_W:

ModelIF: original 55 (sandbox) → additional 20 (sandbox)CCprevCC₀F_W
Astra0.5774 → 0.59720.9847 → 0.98320.9627 → 0.93740.8246 → 0.8484
Sol 60.5106 → 0.53080.9799 → 0.98120.9380 → 0.93900.8057 → 0.8387

W1. Backbone diversity

All the conventional methods are built on top of TRELLIS latent. This represents a significant limitation in diversity when compared with the benchmark.

Using a shared TRELLIS backbone allows us to isolate the editing mechanism from differences in base generator quality; moreover, TRELLIS is a widely adopted state-of-the-art 3D backbone, and few alternative backbones natively support both single-view conditioning and textured mesh editing.

To further broaden backbone diversity, we additionally evaluated ShapeUP, a recent method built on Step1X-3D (a non-TRELLIS generator) with SDXL-based texture synthesis, across all 55 chains and 416 self-rollout turns (conditioned on the current target image and previous exported mesh):

MethodDelivered / TriedIF ↑CCprev ↑F_W ↑LPIPS_E ↓LPIPS_W ↓PSNR_W ↑
ShapeUP416/4160.150.480.190.6730.39211.0

ShapeUP achieves an IF below VoxHammer, Nano3D, and 3DEditFormer (and above PartFlow), confirming that sequential instruction following and content preservation remain challenging across distinct 3D generative backbones.

W2. Representation and inference budget

Code-editing agents operate directly on part-structured meshes with iterative render-and-compare refinement, while conventional editors do not have equivalent capabilities or inference budgets.

Our benchmark evaluates each method under its standard operating setup, with full token and runtime cost breakdowns detailed in Table 12.

We completely agree that combining the complementary strengths of LLMs and generative models is a promising path forward: LLMs excel at high-level reasoning, spatial planning, and instruction following, whereas generative models excel at synthesizing fine-grained geometry and rich textures. As our benchmark shows, LLM-based editors already lead in instruction following and content preservation while struggling primarily with detailed part synthesis—suggesting that a hybrid architecture combining both paradigms is a natural and exciting next step.

Reviewer_2JwZOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

This paper introduces EditHero, a benchmark for sequential 3D editing that evaluates whether methods can follow instructions while preserving earlier edits and unaffected regions. It provides exact targets for each editing step and measures instruction following and content preservation. Experiments show that conventional 3D editors struggle with both requirements and accumulate errors over successive edits, while LLM-based code-editing agents perform better but have higher latency.

Critical strengths
  1. Thorough experiments and informative visualizations. The paper evaluates both conventional 3D editors and LLM-based code-editing agents, supported by useful baselines and diagnostic experiments. The ground-truth reset analysis helps isolate error accumulation, while the step-by-step visualizations clearly illustrate missed edits and unintended changes to preserved regions.

  2. Well-designed evaluation metrics. The paper separates instruction following from content preservation, addressing weaknesses in whole-object similarity scores. Measuring the required change relative to a method’s own previous output is thoughtful. Tracking preservation against both the previous output and the original object also captures immediate disruption and cumulative drift.

Critical weaknesses
  1. Despite the extensive experiments, the analysis offers limited insight beyond documenting performance differences. The main finding( repeated generative editing accumulates errors and explicit code-based modifications better preserve untouched regions) is largely expected and points to no direction for future improvements.

  2. The evaluation scale is limited, particularly for the LLM-based methods, which are tested on only 55 manually selected sequences.

Questions for authors
  1. Could the authors clarify the significance of the findings and the new insights gained from this benchmark? In particular, what actionable lessons do the results provide for designing more reliable sequential 3D editing methods?

  2. Could the authors expand the evaluation, particularly the LLM subset and longer edit sequences, and report the sample distributions across object categories, edit operations, and sequence lengths? Please also explain how the evaluation samples were selected and how representative they are of the full benchmark.

7 comments

We thank the reviewer for the thoughtful feedback and constructive suggestions. We have added further analysis and experiments to address your questions.

W1. The main finding……points to no direction for future improvements & Q1. What do we learn from the benchmark?

Could the authors clarify the significance of the findings and the new insights gained from this benchmark? In particular, what actionable lessons do the results provide for designing more reliable sequential 3D editing methods?

Our overarching goal is to advance sequential 3D editing toward practical workflows for artists in game and film production, where assets routinely undergo iterative revisions. We hope EditHero encourages the graphics and vision communities to explore LLM-based editing in realistic multi-turn workflows, particularly given the rapid progress of LLMs in 3D content creation.

For the first time, EditHero quantitatively characterizes the behavioral gap between code-editing agents and conventional editors across long-horizon 3D editing, spanning both geometry and texture modifications.

Beyond error accumulation, our findings highlight a clear complementary divide: code agents excel at preserving untouched regions (Table 3), yet struggle when synthesizing detailed new geometry. Specifically, while removal IF is high (0.88–0.99), addition and replacement reach only 0.10–0.49 and 0.17–0.51, respectively. This points directly toward combining an LLM's strengths in semantic reasoning and spatial planning with a generative model's strength in synthesizing high-frequency geometry and textures.

Concretely, an agent can localize target parts, track edit history, and specify editable regions, while a generative backbone synthesizes fine geometric and texture details with seamless transitions inside those boundaries—similar in spirit to approaches like Code World Model that pair coding agents with generative priors. EditHero provides the intermediate ground-truth states needed to systematically evaluate such hybrid 3D editors.

Additionally, Table 10 offers a second actionable insight: Nano3D's turn-7 IoU improves from 0.534 under self-rollout to 0.725 when conditioned on the ground-truth previous state with the identical instruction and target. This underscores the value of intermediate state verification and error correction between turns rather than propagating errors forward.

We will highlight our motivation and these design takeaways more prominently in the Introduction and Discussion.

W2 & Q2. Selection, distributions, and expanded evaluation

Could the authors expand the evaluation, particularly the LLM subset and longer edit sequences, and report the sample distributions across object categories, edit operations, and sequence lengths? Please also explain how the evaluation samples were selected and how representative they are of the full benchmark.

We selected 55 chains for two primary reasons: (1) this sample size already yields statistically stable method rankings, and (2) evaluating all 6 frontier LLMs—particularly Astra and Fable—incurs substantial computational and API cost.

Notably, each LLM completes 416 edits across 55 chains (2,496 edits across all 6 LLMs), which exceeds the per-model edit scale of prior benchmarks, including BlenderGym (245 editing tasks) and EditBench3D (240 scene–edit pairs). Table 12 details the token and runtime costs.

Selection stability. To verify statistical stability, we drew 1,000 random subsets of 20 chains from the original 55 chains, evaluating all 6 LLMs on the exact same subsets in each draw:

MetricMean rank correlation with full 55Mean absolute score change
IF0.9460.0360
CCprev0.9210.0056
CC₀0.9150.0148

Method rankings remain highly consistent across random subsets.

For the original 55 chains, we first defined quotas across edit operations and their combinations, then randomly sampled chains within each group to cover all 6 object families by human eyes. We do so because we want to ensure content diversity and covering. To evaluate generalization beyond this set, we further tested Astra and Sol 6 on 20 additional chains sampled programmatically using operation-group quotas, a fixed seed (20389), and distinct host assets under the turn-by-turn sandbox protocol. We will include these sampling details and results in the revised paper:

ModelIF: original 55 (sandbox) → additional 20 (sandbox)CCprevCC₀F_W
Astra0.5774 → 0.59720.9847 → 0.98320.9627 → 0.93740.8246 → 0.8484
Sol 60.5106 → 0.53080.9799 → 0.98120.9380 → 0.93900.8057 → 0.8387

Both sets follow the turn-by-turn sandbox protocol, faithfully modeling sequential editing in practice.

Dataset distributions. Below we compare the full 485-chain dataset against the original 55-chain and additional 20-chain subsets (categories and lengths are counted per chain; operations are counted per turn):

Chain lengthFull 485Original 55Additional 20
2–5 turns41.6%16.4%40.0%
6–10 turns54.6%78.2%55.0%
>10 turns3.7%5.5%5.0%
Operation, counted per turnFull 485Original 55Additional 20
Add34.3%30.0%31.0%
Remove18.9%17.8%19.0%
Replace22.1%28.6%23.0%
Retexture24.7%23.6%27.0%
Object categoryFull 485Original 55Additional 20
Characters18.4%7.3%15.0%
Robots17.9%7.3%30.0%
Animals & creatures9.7%5.5%0.0%
Vehicles18.6%20.0%10.0%
Objects21.0%36.4%30.0%
Small scenes & structures14.4%23.6%15.0%

While Figure 3 illustrates the distributions of the full benchmark, we will add these subset comparison tables to the revised manuscript.

Short versus long chains. We further compared all 6 LLMs and 4 conventional editors on short chains (52 chains of 3–10 turns) versus long chains (3 chains of >10 turns: 11, 15, and 19 turns) within the 55-chain set (rank 1 indicates best performance):

MethodIF rank: ≤10 → >10CCprev rank: ≤10 → >10CC₀ rank: ≤10 → >10F_W rank: ≤10 → >10
Opus 5.51 → 12 → 21 → 21 → 1
Fable 5.12 → 63 → 33 → 33 → 3
Astra3 → 21 → 12 → 12 → 2
Sol 64 → 34 → 54 → 44 → 4
DeepSeek 4.1 Flash5 → 45 → 45 → 55 → 5
VoxHammer6 → 910 → 1010 → 1010 → 10
Nano3D7 → 57 → 77 → 77 → 7
GLM 5.38 → 76 → 66 → 66 → 6
3DEditFormer9 → 89 → 99 → 99 → 9
PartFlow10 → 108 → 88 → 88 → 8

Spearman rank correlations between short-chain and long-chain rankings are 0.794 for IF, 0.988 for CCprev, 0.988 for CC₀, and 1.000 for F_W. Method rankings are broadly similar across the two groups, with larger shifts in IF. All 6 LLMs outperform all 4 conventional editors on CCprev, CC₀, and F_W in both regimes, with identical F_W rankings across short and long chains. We will include this analysis in the revision.

Thank you for your additional analysis and results. I now have some additional questions based on the insights and dataset distribution newly provided.

  1. Is the ranking stable across different subsets? Will some models excel in some subsets while perform badly in other?
  2. In your newly provided insights, you mentioned "combining an LLM's strengths in semantic reasoning and spatial planning with a generative model's strength in synthesizing high-frequency geometry and textures." Based on my current knowledge, most vlms tend to render the object and do inspections based on rendered images and do edit after this, do you think this will make vlms perform badly on situations related to internal structures or very small parts?
  3. (This question is something I'm thinking recently, just for discussion, has nothing to do with rating) Frontier vlms now evolve so fast that they can even make videos with pure code now. Demos trying to use vlms to operate software like zbrush to make detailed monster meshes have also appeared. Do you think we still need 3d-specific models in the future?

I will raise my score after your answers. Good luck!

Reviewer_TKmcOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Long-horizon 3D editing is an important and challenging task and has real-world applications, as a real 3D artist makes continuous edits. Existing methods focus on single edits and fail to handle multi-step and long-horizon edits. The authors introduce EditHero, a new benchmark of part-level edit chains with exact ground truth.

The main contributions of this work are as follows:

  • EditHero Benchmark: EditHero contains 485 chains and 2965 edits across 255 hosts. Each chain has 2 to 30 turns, with a median of six; the median instruction is 17 words long.
  • Sequential Evaluation Framework: A self-rollout evaluation setting (where turn kk takes the model's output from turn k−1k-1).
  • Comprehensive Model Benchmarking: Benchmarks both non-agentic latent-space 3D editors (PartFlow, Nano3D, 3DEditFormer, VoxHammer) and six LLM/VLM code agents.

Their key finding is that conventional methods struggle with instructions from the first turn. In contrast, LLM/VLM coding agents follow instructions better than the conventional methods.

Critical strengths
  • [S1] Robust Ground-Truth Creation: Using an image-guided generative model to create an edited pair results in the 3D model containing unintended changes. Hence, the authors construct this dataset via part assembly, similar to programmatically generated benchmarks, with a separate mechanism for addition/replacement and texture editing. Further, every edit action is logged at each step; the exact look of the object and the identity of every edited part are known at any point in time. This allows the authors to evaluate specific, localized regions accurately rather than guessing. Additionally, a human reviewer flags wrong part semantics, orientation or any ambiguity at each step.
  • [S2] Unique Evaluation protocol: The sequential self-rollout protocol measures error accumulation in 3D editing. The authors explain this in more detail in Appendix H, and Tab. 10 shows that by feeding the ground-truth previous state instead of the model's own output, Nano3D's IoU improves to 0.725 (GT input), compared with 0.534 under self-rollout for step 7.
  • [S3] Rigorous Experimentation: The authors perform exhaustive experimentation with both conventional baselines (4 methods) and LLM/VLM coding agents (6 methods). Further, the paper reports execution coverage and failure categories per method in Tab. 5, runtime and memory cost (Tab. 12), and whole-object metrics on the self-rollout runs (Tab. 7) in the Appendix.
Critical weaknesses
  • [W1] Missing Human User Study: The authors report different metrics (Appendix D) for evaluation. However, a comprehensive user study will correlate these metrics with human judgment as well.
Questions for authors
  • [Q1] CCprev\text{CC}_\text{prev} is low for conventional methods compared to LLM coding agents (Tab. 3). Can the authors report this metric for conventional methods with no edit, just a forward pass of the encoder and decoder?
  • [Q2] How many human reviewers were used? What was their background? Were any specific instructions provided to the human reviewers?
3 comments

Thank you for the constructive and encouraging review. We address each of your points in detail below.

W1. Human study

However, a comprehensive user study will correlate these metrics with human judgment as well.

To validate the metrics proposed in our paper against human perception, we conducted a user study with 8 participants who rated instruction following (IF) and content consistency (CC) on a 1–5 scale in a blind evaluation (without method names).

Across 496 ratings covering all 100 chain–method pairs, human ratings and the metrics proposed in our paper give similar rankings of the 10 methods, yielding Spearman rank correlations of 0.879 for IF, 0.952 for CCprev, and 0.952 for CC₀. We will include this user study in the revised manuscript.

Q1. Reconstruction without editing

Can the authors report this metric for conventional methods with no edit, just a forward pass of the encoder and decoder?

Yes. We conducted a non-edit reconstruction test (single-pass) using the TRELLIS 1 source-SLAT encoder and native mesh/Gaussian decoders, followed by textured mesh export, and will include these results in the revision:

MethodDelivered / TriedIF ↑CCprev ↑F_W ↑LPIPS_E ↓LPIPS_W ↓PSNR_W ↑
No-op (original GT source)416/4160.001.000.750.5670.15418.2
TRELLIS single-pass reconstruction416/4160.010.980.700.5820.19316.5

Single-pass reconstruction achieves a CCprev of 0.976, substantially higher than all 4 conventional editors (0.732–0.900).

We emphasize that the 4 conventional editors in our paper pass the latent state directly across turns; thus, each evaluated output undergoes at most one encoding and one decoding pass, without re-encoding decoded meshes.

For completeness, we also include repeated encode–decode reconstruction below, which illustrates cumulative degradation when meshes are repeatedly re-encoded:

MethodDelivered / TriedIF ↑CCprev ↑F_W ↑LPIPS_E ↓LPIPS_W ↓PSNR_W ↑
TRELLIS repeated reconstruction416/4160.020.860.640.6150.24114.6

Q2. Human reviewers

How many human reviewers were used? What was their background? Were any specific instructions provided to the human reviewers?

The benchmark was verified by 3 human reviewers with backgrounds in computer graphics and computer vision and hands-on experience in 3D and game-production pipelines.

They were instructed to ensure that text prompts were accurate and sufficiently specific to drive the intended edit, and that part placements and orientations appeared visually natural. In addition, they manually corrected inaccurate placements or orientations, removed redundant or low-quality turns, and verified diversity across material edits and object additions/removals.

Thanks to the authors for addressing my concerns. I maintain my original rating.

Thank you for your response and for taking the time to review our rebuttal. We will incorporate the promised analyses into the revision.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.