- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
EditHero proposes a benchmark for evaluating long-chain, part-level 3D editing, addressing the limitation of isolated edit evaluation. It includes 485 chains and 2965 edits across 255 objects, with corresponding ground truth after each edit. Introduce evaluation using a self-rollout protocol and region-specific metrics.
Critical strengths
- The strongest contribution is reconstructing every intermediate state, enabling reliable evaluation. This directly targets the limitation of isolated edit benchmarks
- The use of self-rollout is intuitive and can measure error propagation well over multiple sequential edits. Also separation of edited and unchanged parts solved inflated valued which may occur while evaluating.
- Manual intervention while generating JSON enhances the reliability of the benchmark.
Critical weaknesses
- All the conventional methods are built on top of TRELLIS latent. This represents a significant limitation in diversity when compared with the benchmark. The results are meaningful at the system level, but an identity reconstruction baseline would help establish how much degradation is specifically attributable to editing.
- Code-editing agents operate directly on part-structured meshes with iterative render-and-compare refinement, while conventional editors do not have equivalent capabilities or inference budgets. The LLM evaluation also uses only 55 hand-selected chains. These differences limit how strongly the results support a general advantage for code-based editing. Additionally, with a median chain length of six, the longer-horizon claims would benefit from more direct analysis.
Questions for authors
- How much of the preservation degradation comes from TRELLIS encoding/decoding itself? Could the authors report an identity reconstruction baseline?
- Do the code-editing agents receive semantically named part nodes? How does their performance change without this part decomposition?
- Could the authors clarify whether instructions and target renders are revealed one turn at a time, and report performance separately on the held-out views?
- Could the authors provide per-turn results for the LLM agents and a more detailed analysis of chains beyond 10 turns?
- Is there a particular reason why only 55 chains are evaluated on LLMs? Can you report on a larger subset, as now it’s less than 15% of the set?
3 comments
Q3. Turn-by-turn inputs and held-out views
Could the authors clarify whether instructions and target renders are revealed one turn at a time, and report performance separately on the held-out views?
Turn-by-turn inputs. In the original workspace, the full sequence of instructions and target renders was accessible. To strictly enforce causal turn-by-turn execution within the rebuttal timeframe, we re-evaluated 3 representative models—Astra, Sol 6, and Opus 5.5—across all 55 chains in a sealed sandbox: at each turn, the agent receives only the current instruction and target image and submits a single output mesh that serves as the next turn's input, with future instructions and renders kept strictly hidden. All 3 agents completed all 55 chains (416 turns):
| Method | Delivered / Tried | IF ↑ | CCprev ↑ | F_W ↑ | LPIPS_E ↓ | LPIPS_W ↓ | PSNR_W ↑ |
|---|---|---|---|---|---|---|---|
| Astra, original (Table 3) | 416/416 | 0.52 | 0.98 | 0.81 | 0.289 | 0.078 | 25.5 |
| Astra, sandbox | 416/416 | 0.58 | 0.98 | 0.82 | 0.251 | 0.068 | 26.1 |
| Sol 6, original (Table 3) | 416/416 | 0.42 | 0.97 | 0.79 | 0.344 | 0.093 | 24.6 |
| Sol 6, sandbox | 416/416 | 0.51 | 0.98 | 0.81 | 0.284 | 0.080 | 25.2 |
| Opus 5.5, original (Table 3) | 416/416 | 0.62 | 0.98 | 0.82 | 0.236 | 0.065 | 26.7 |
| Opus 5.5, sandbox | 416/416 | 0.67 | 0.98 | 0.84 | 0.212 | 0.057 | 27.1 |
Sandbox performance matches or even slightly exceeds the original workspace results, confirming that sequential editing does not rely on access to future turns much. We will incorporate this strict turn-by-turn protocol into the revised paper.
Held-out views. Appendix K (Table 14) already reports held-out view metrics for the 4 conventional editors, and Figures 15 and 16 visualize LLM outputs from non-conditioning views. Here we provide the corresponding quantitative results for all 6 LLMs using the PSNR and LPIPS metrics of Table 14. Subscript 0 denotes the conditioning view, and 1–3 denotes held-out views:
| Method | PSNR₀ ↑ | PSNR₁–₃ ↑ | LPIPS₀ ↓ | LPIPS₁–₃ ↓ |
|---|---|---|---|---|
| Astra, original | 25.3 | 25.6 | 0.076 | 0.079 |
| Astra, sandbox | 26.0 | 26.1 | 0.064 | 0.069 |
| Sol 6, original | 24.5 | 24.6 | 0.091 | 0.094 |
| Sol 6, sandbox | 25.7 | 25.1 | 0.070 | 0.084 |
| Opus 5.5, original | 26.8 | 26.6 | 0.058 | 0.067 |
| Opus 5.5, sandbox | 27.7 | 26.9 | 0.049 | 0.060 |
| Fable 5.1 | 25.3 | 25.4 | 0.075 | 0.082 |
| GLM 5.3 | 20.8 | 21.4 | 0.141 | 0.134 |
| DeepSeek 4.1 Flash | 23.9 | 24.1 | 0.102 | 0.108 |
Performance remains consistent between conditioning and held-out views. We will add these quantitative tables to the revision.
Q4. Per-turn results and longer chains
Could the authors provide per-turn results for the LLM agents and a more detailed analysis of chains beyond 10 turns?
Following Figures 4 and 8 and Table 8, we will include per-turn LLM results under the sandbox protocol in the revised manuscript. The table below mirrors the format of Table 8:
| Metric / Method | 1 | 2 | 3 | 4 | 5 | 6 | 7 |
|---|---|---|---|---|---|---|---|
| Chains | 55 | 55 | 55 | 54 | 50 | 46 | 35 |
| IoU ↑ | |||||||
| Astra | 0.907 | 0.842 | 0.811 | 0.783 | 0.776 | 0.766 | 0.767 |
| Sol 6 | 0.894 | 0.822 | 0.791 | 0.760 | 0.746 | 0.740 | 0.734 |
| Opus 5.5 | 0.909 | 0.858 | 0.823 | 0.810 | 0.800 | 0.792 | 0.786 |
| IF ↑ | |||||||
| Astra | 0.577 | 0.513 | 0.535 | 0.572 | 0.553 | 0.667 | 0.688 |
| Sol 6 | 0.501 | 0.425 | 0.503 | 0.526 | 0.505 | 0.572 | 0.647 |
| Opus 5.5 | 0.673 | 0.678 | 0.595 | 0.678 | 0.636 | 0.750 | 0.671 |
| CC₀ ↑ | |||||||
| Astra | 0.986 | 0.972 | 0.960 | 0.953 | 0.953 | 0.959 | 0.964 |
| Sol 6 | 0.978 | 0.954 | 0.948 | 0.936 | 0.925 | 0.932 | 0.915 |
| Opus 5.5 | 0.973 | 0.966 | 0.954 | 0.961 | 0.964 | 0.960 | 0.958 |
For chains exceeding 10 turns, all 6 LLMs and the 3 sandbox reruns completed the same 11-, 15-, and 19-turn chains. Following Table 9, we report first-turn → last-turn mean scores:
| Model | Setting | IoU ↑: first → last | CC₀ ↑: first → last |
|---|---|---|---|
| Astra | Original | 0.97 → 0.66 | 0.99 → 0.63 |
| Astra | Sandbox | 0.97 → 0.71 | 0.99 → 0.65 |
| Sol 6 | Original | 0.96 → 0.60 | 0.99 → 0.60 |
| Sol 6 | Sandbox | 0.97 → 0.67 | 0.99 → 0.64 |
| Opus 5.5 | Original | 0.97 → 0.70 | 0.99 → 0.65 |
| Opus 5.5 | Sandbox | 0.96 → 0.77 | 0.98 → 0.97 |
| Fable 5.1 | Original | 0.97 → 0.61 | 0.98 → 0.62 |
| GLM 5.3 | Original | 0.93 → 0.57 | 0.95 → 0.60 |
| DeepSeek 4.1 Flash | Original | 0.97 → 0.57 | 0.99 → 0.58 |
On the 3 chains longer than 10 turns, Opus 5.5 achieves the highest final IoU (0.77) in the sandbox.
Q5. Why 55 chains? Can the evaluation be expanded?
Is there a particular reason why only 55 chains are evaluated on LLMs? Can you report on a larger subset, as now it’s less than 15% of the set?
We used 55 chains for two main reasons: the sample size already produces stable comparative conclusions, and evaluating all 6 LLMs on additional chains is computationally expensive.
Notably, each LLM performs 416 edits across 55 chains (2,496 edits total across the 6 LLMs), surpassing the per-model edit scale of benchmarks, such as BlenderGym (245 editing tasks) and EditBench3D (240 scene–edit pairs). Table 12 details the token and runtime costs.
To verify that our conclusions are robust to subset size and selection, we performed two additional checks:
1. Selection stability: Randomly sampling 20 of the 55 chains 1,000 times (evaluating all 6 LLMs on the identical sample per draw) yields high rank stability:
| Metric | Mean Spearman rank correlation with full 55 | Mean absolute score change |
|---|---|---|
| IF | 0.946 | 0.0360 |
| CCprev | 0.921 | 0.0056 |
| CC₀ | 0.915 | 0.0148 |
2. Out-of-sample evaluation: Evaluating Astra and Sol 6 on 20 additional unseen chains under the turn-by-turn sandbox protocol preserves the same Astra–Sol 6 ordering on IF, CCprev, and F_W:
| Model | IF: original 55 (sandbox) → additional 20 (sandbox) | CCprev | CC₀ | F_W |
|---|---|---|---|---|
| Astra | 0.5774 → 0.5972 | 0.9847 → 0.9832 | 0.9627 → 0.9374 | 0.8246 → 0.8484 |
| Sol 6 | 0.5106 → 0.5308 | 0.9799 → 0.9812 | 0.9380 → 0.9390 | 0.8057 → 0.8387 |
W1. Backbone diversity
All the conventional methods are built on top of TRELLIS latent. This represents a significant limitation in diversity when compared with the benchmark.
Using a shared TRELLIS backbone allows us to isolate the editing mechanism from differences in base generator quality; moreover, TRELLIS is a widely adopted state-of-the-art 3D backbone, and few alternative backbones natively support both single-view conditioning and textured mesh editing.
To further broaden backbone diversity, we additionally evaluated ShapeUP, a recent method built on Step1X-3D (a non-TRELLIS generator) with SDXL-based texture synthesis, across all 55 chains and 416 self-rollout turns (conditioned on the current target image and previous exported mesh):
| Method | Delivered / Tried | IF ↑ | CCprev ↑ | F_W ↑ | LPIPS_E ↓ | LPIPS_W ↓ | PSNR_W ↑ |
|---|---|---|---|---|---|---|---|
| ShapeUP | 416/416 | 0.15 | 0.48 | 0.19 | 0.673 | 0.392 | 11.0 |
ShapeUP achieves an IF below VoxHammer, Nano3D, and 3DEditFormer (and above PartFlow), confirming that sequential instruction following and content preservation remain challenging across distinct 3D generative backbones.
W2. Representation and inference budget
Code-editing agents operate directly on part-structured meshes with iterative render-and-compare refinement, while conventional editors do not have equivalent capabilities or inference budgets.
Our benchmark evaluates each method under its standard operating setup, with full token and runtime cost breakdowns detailed in Table 12.
We completely agree that combining the complementary strengths of LLMs and generative models is a promising path forward: LLMs excel at high-level reasoning, spatial planning, and instruction following, whereas generative models excel at synthesizing fine-grained geometry and rich textures. As our benchmark shows, LLM-based editors already lead in instruction following and content preservation while struggling primarily with detailed part synthesis—suggesting that a hybrid architecture combining both paradigms is a natural and exciting next step.
Thank you for the helpful and thorough review. We address each of your questions and comments below.
Q1. Reconstruction loss
The 4 TRELLIS-based editors encode the source asset once, propagate their latent state across turns, and decode the output at each turn. We evaluated both single encode–decode and repeated encode–decode reconstruction on the 55-chain set:
Single-pass reconstruction maintains a CCprev of 0.976, well above all 4 conventional editors (0.732–0.900), indicating that most preservation loss arises from the editing updates rather than a single encoder–decoder pass.
For reference, repeated encode–decode reconstruction yields a CCprev of 0.864, illustrating error accumulation when meshes are repeatedly re-encoded.
Q2. Semantic part names and merged meshes
Yes, the original source meshes include semantic node labels (e.g.,
slot_helmetandslot_boot_left).To examine how strongly agents depend on semantic labels and part decomposition, we evaluated Opus 5.5 and Sol 6 on 10 chains (82 turns) after stripping semantic names (retaining geometry parts). We also tried merging the initial parts into a single mesh for agents. All runs used medium reasoning under the turn-by-turn sandbox protocol. While scores decrease slightly without semantic names or part separation, LLM agents still substantially outperform conventional editors: