acceptodds
Withdrawn from ICLR 2027

ARGUS: An Evolving, Auditable Evaluator for 3D Scene Editing Agents via Evidence-Driven Skill Induction

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

A 3D scene editing agent can produce a plausible scene without completing the requested edit, so evaluators that accurately check the executed result are increasingly important. Image-based judgment often fails when the target is occluded or the change is small, and an aggregate score or binary verdict discards which conditions were checked, whether a checker existed, and whether the evidence sufficed, which is the information repair needs. We propose ARGUS, an experience-driven framework whose primary objective is accurate evaluation and whose downstream objective is scene improvement. It compiles a task specification, binds each requirement to compatible checkers, and combines state, visual, physical, and preservation evidence; confirmed violations go to repair, missing evidence to additional observation, and missing capability to skill learning. A separate pipeline has an LLM propose executable evaluation and repair programs from reviewed evidence. Admission tests each candidate on labeled counterexamples and on a label-free invariance test that replays it under meaning-preserving transformations such as unit conversion and world-frame rotation, which in a controlled test rejects faulty measurements that single-axis labels admit; only programs that pass both become reusable skills. On 48 edits where every evaluator sees only the instruction and the scenes, ARGUS has the lowest selective risk (error rate among decided conditions), 7.0%, and the highest scene AUROC, 81.9, and it holds conditions it cannot settle with the missing evidence named. Registered skills, together with a fixed relation frame in the binder, raise coverage (the fraction of conditions decided) from 44.2% to 60.0% at the same error rate; the skills alone decide 11 held conditions, 10 correctly. Accurate diagnosis carries over to repair, raising the holistic success rate from 56.8% to 65.1%, and admitted skills are reused without LLM calls and revised when new counterexamples appear.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
3D scene editing agents, Scene evaluation, Skill induction, 3d-vision

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_9386OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a self-evolving evaluator for 3D scene editing. The system first collects historical editing failures and repair trajectories, then uses an LLM-based agent to propose new evaluation or repair programs. These programs are subsequently tested on historical cases, and those that pass the validation process are integrated into a reusable skill library.

Critical strengths
  1. The proposed idea is straightforward and intuitive, and the overall methodology is clearly presented and easy to follow.

  2. The proposed method achieves better experimental results than the baselines.

Critical weaknesses
  1. SceneCraft (ICML 2024) has already introduced the idea of Library Learning, which updates the verification functions during 3D scene generation and integrates them into the libarary. I believe the core idea of this paper is essentially the same as SceneCraft.

  2. According to Table 1, ARGUS with learned skills makes explicit decisions on only 57 out of 95 labeled conditions, significantly fewer than the baselines. Therefore, I think ARGUS adopts a more conservative evaluation strategy, sacrificing coverage and recall to achieve lower selective risk.

  3. The authors only evaluate their method on indoor scene editing tasks, without conducting experiments on more general or open-world 3D scenes.

  4. The evolving process is not fully automated. For example, the authors still rely on human-designed mechanisms, such as Label-Free Invariance Testing, to validate newly generated skills, rather than allowing the agent to automaticly verify them.

  5. Based on the learned skills presented in Supplementary Table 8, most of them involve relatively simple and deterministic computational tasks. Is LLM-based program synthesis really necessary for these operations? I believe a general geometric checker library could be implemented manually instead.

  6. In the main paper, the HSR improves from 56.8% without repair to 65.1% with ARGUS repair skills. However, the learned repair skills successfully fix only 4 out of 19 failed scenes, which I think a rather limited improvement.

[1] Hu Z, Iscen A, Jain A, et al. SceneCraft: An LLM Agent for Synthesizing 3D Scenes as Blender Code[C]//ICML. 2024: 19252-19282.

Questions for authors

Please see the "Critical Weaknesses" section

Reviewer_e43YOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes ARGUS, an evaluator for 3D scene editing agents. Given the instruction, ARGUS checks the scene via 4 paths (object state, VLM visual, physics, preservation), and automatically repair failed checks. The checking and repairing trajectory are then used in a proposing and revising process to obtain new evaluation and repairing skills. ARGUS gets the lowest selective risk (7.0%) and highest scene AUROC (81.9).

Critical strengths

ARGUS separates "cannot decide" from "fail" with explicit hold reasons, and converts this into new skill learning opportunity. This is a good design for evaluator, which broadens the decision space of evaluator from true/false in the three-way pass/fail/hold, and naturally improves the evaluator.

Critical weaknesses
  1. The evaluation set is very small, with only 48 edits and 95 labeled conditions. And the paper also acknowledges than "With 4/57 errors, the risk interval of LE [1.9, 17.0] overlaps that of LEGO-Eval".
  2. Invariance test never refused any LLM proposal. Appendix C.4 "Every source gate and admission gate passed, including the invariance test". So the claim of "label-free invariance rejects faulty measurements" is not convincible.
  3. The role of skills is not sure. The frozen library of Table 8 consists of 9 basic geometric measurements. On EditRoom, only 1 of 5 newly correct decisions comes from a learned skill, while the other 4 come from the hand designed relation frame.
  4. The success rate of repairing is relatively low (4 of 19, all EditRoom).
Questions for authors
  1. Does the invariance test ever rejected any real LLM proposed evaluation program?
  2. Is there any skill learnt during ARGUS process, that goes beyond basic geometric condition to high level 3d scene design?
  3. How much do the gold labels (state differences) overlap with ARGUS's state checker on EditRoom?
Reviewer_mfp1OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces ARGUS, an evaluator and repair framework for 3D scene editing agents designed to verify instruction fulfilment through systematic diagnostics. ARGUS operates via two core pipelines. First, Scene Execution compiles editing instructions and pre-edit scenes into fixed specifications consisting of predicates, object bindings, coordinate systems, and tolerances. It evaluates these conditions across four specialized checking paths, including object state, VLM visual judgment, physical constraints, and non-target invariance, issuing a pass or fail only when evidence is compatible, fresh, observed, and sufficient, or otherwise recording a "hold" status. Unverified violations trigger a repair phase executed on scene replicas, accepting updates only upon strict improvement. Second, Experience Learning prompts an LLM to propose evaluation or repair programs from training trajectories, iteratively refining them with counterexamples before validating them on an independent admission set for execution scope, compatibility, class discrimination, lack of regression, and utility gains, alongside an unsupervised invariance test for robustness against renaming, unit conversion, translation, and rotation. Passing skills are then frozen for reuse without further LLM intervention. Evaluated on 48 single-room editing tasks (24 from E2A and 24 from EditRoom) using GPT-5, ARGUS achieves a condition-level selective risk of 7.0% at 60.0% coverage, which outperforming LEGO-Eval's 17.9% under full coverage, and a scene-level AUROC of 81.9. Moreover, incorporating learned skills and a fixed relation frame increases coverage from 44.2% to 60.0%, while feedback from ARGUS improves the task success rate (HSR) from 56.8% to 65.1% with an average of only 0.17 LLM calls per edit.

Critical strengths

Clear problem definition and practical significance: The paper distinguishes between whether an edit is genuinely executed and whether the scene looks plausible, replacing a single score with a ternary evaluation (pass, fail, hold) where the reason for a "hold" dictates the next action, such as rebinding, supplementary observation, learning new verification capabilities, or conducting repairs. Design choices such as specification hash freezing, read-only evaluation skills, and restricting repair skills to write only to permitted fields on scene replicas prevent the evaluator from shifting goalposts or grading itself during the self-improvement loop, offering insights for mitigating reward hacking in agent self-improvement.

Good reporting transparency: The authors proactively disclose that the amount guard was introduced post-hoc, that the risk confidence interval of experience learning overlaps with that of LEGO-Eval, and that EVC is a self-reported metric rather than an independently audited one. Additionally, the paper reports metrics capable of exposing over-editing, such as HSR48, NET, and OVR, and establishes tie-breaking rules for judge scoring based on the standard deviation across multiple runs.

Critical weaknesses

Omitted literature and the need to reposition novelty: (1) Generating tools via LLMs, verifying them on validation samples, caching them for reuse, and bypassing frontier models at inference time mirrors LATM (ICLR 2024), CRAFT (ICLR 2024), and TroVE (ICML 2024), none of which are cited, with the paper referencing only Voyager, ASI, and AWM; (2) LLM evaluators equipped with abstention mechanisms appear in Trust or Escalate (ICLR 2025), which performs selective evaluation under provable human-alignment guarantees, directly aligning with the selective risk framework; (3) Decomposing instructions into atomic conditions and verifying them via procedural checks has been explored in 2D generation evaluation via VPEval (NeurIPS 2023) and Davidsonian Scene Graphs (ICLR 2024). The paper's incremental contribution lies in combining these components for 3D editing evaluation while adding permission separation and specification freezing.

Small evaluation scale and lack of statistical support for core claims: The testbed comprises only 48 edits and 95 conditions. Based on the False Acceptance Rate (FAR) and False Rejection Rate (FRR) in Table 1, the gold standard contains a total of only 22 failed conditions (including just 3 in E2A) and 19 failed scenes (including only 3 in E2A). Consequently, all methods on E2A can only take discrete FAR values of 0, 33.3, 66.7, or 100, and the scene-level AUROC of 90.0 on E2A is anchored on merely 3 failed scenes. As the authors themselves concede, the risk interval for Experience Learning (LE) of [1.9,17.0][1.9, 17.0] overlaps with that of LEGO-Eval. With 23 passing and 19 failing samples, the 95% confidence interval for a scene-level AUROC of 81.9 (estimated via the Hanley–McNeil method) spans roughly 69 to 95, fully encompassing SceneCritic's 75.1. Regarding repair, the Task Success Rate (HSR48) rises from 25/4825/48 to 28/4828/48 (recovering 4 and losing 1), yielding an exact McNemar test pp-value of approximately 0.380.38. Therefore, claims in the abstract and conclusion regarding "making the most reliable decisions," achieving the "highest AUROC," and delivering statistically significant repair improvements are currently indistinguishable from noise. Furthermore, the evaluation is restricted to only four edit types (removal, movement, rotation, scaling) within single-room, single-instruction settings. I recommend scaling the evaluation to hundreds of edits and incorporating addition, replacement, appearance modification, and multi-step instructions.

Low selective risk stems primarily from selective abstention concentrated on failing samples, revealing weak failure-detection capability: Back-calculating from precision and recall in Table 1 reveals that out of 22 failed conditions, LE evaluated only 7 (correctly rejecting 5 and falsely accepting 2), while holding back the remaining 15 (68%); conversely, out of 73 passing conditions, LE held back 23 (32%). A similar pattern appears at the scene level: out of 19 failed scenes, LE predicted fail on only 6, held 12, and falsely predicted pass on 1. Table 3 further indicates that without ternary aggregation (V2), ARGUS's check-level FAR reaches 59.1% (13/2213/22), and V0 reaches 81.8%, both substantially worse than LEGO-Eval (22.7%) and Direct State LLM (45.5%). In short, ARGUS's checking mechanism is not inherently skilled at uncovering failures; the reduction in risk is driven by converting those failures into "holds." Moreover, selective risk is incomparable across differing coverage rates: a trivial evaluator that always predicts "pass" achieves a full-coverage risk of only 22/95=23.2%22/95 = 23.2\%, which already undercuts Direct VLM's 24.2%. While baseline models were compared at matching coverage rates for scene-level scoring, all condition-level baselines were compared at full coverage or their respective native coverages. Over the exact same 57 conditions judged by LE, back-calculating from the authors' reported LEGO-Eval P/R (71.9/85.9) yields 1 false acceptance and 7 false rejections for LEGO-Eval, compared to 2 false acceptances and 2 false rejections for LE. This shows that LE's advantage derives entirely from fewer false rejections; on the specific failure mode emphasized in the introduction that uncompleted edits being falsely accepted, LE performs no better. Additionally, because "holds" are concentrated on failing samples while scene-level AUROC assigns a score of 0.5 to holds, holds are effectively treated as failure signals.

Learned skills contribute marginally and rely on a post-hoc rule: Coverage rises from 44.2% to 60.0% (a 15-condition increment), of which 4 conditions originate from the "relation frame." Because the relation frame is a manually engineered binding component rather than an LLM-learned skill, grouping it under LE (+ registered skills) conflates results. The actual number of judgments newly contributed by learned skills is 11 (10 of which are correct). More critically, the "amount guard" was introduced post-hoc after error analysis and successfully blocks 3 false acceptances. If these 3 false acceptances occurred on the testbed (as the framing of the main text implies), removing this rule means that among the 14 skill-added judgments, 4 are incorrect, driving LE's FAR to 5/22=22.7%5/22 = 22.7\% (identical to LEGO-Eval) and its risk to 7/60=11.7%7/60 = 11.7\% (undermining claims of equal-error-rate coverage gains). This also creates a contradiction with the protocol stating that "libraries are frozen before E"; please clarify whether this error analysis was conducted on the development split (H), the validation split (G), or the testbed. Furthermore, Table 4 shows that the base checker completely fails to evaluate rotation, meaning the learned rotation skill essentially computes the relative SO(3)SO(3) angle, an engineering task trivially solved in a few lines of code. In addition, with V-on/R-offV\text{-on} / R\text{-off} already achieving 18/18, the repair skill merely reduces tool-call counts from 12 to 0.

Questions for authors

Q1. When grouped by gold-standard labels and condition types (C0, C1, C2), what are the distributions of hold rates and hold reason codes? Furthermore, under the exact same coverage rate as LE, what are the selective risks of Direct VLM, Direct State LLM, and LEGO-Eval when augmented with confidence score thresholds?

Q2. Compared to a baseline where GPT-5 writes a bespoke checking program from scratch for each instance without admission or caching, how does ARGUS's cached skill library perform in terms of both accuracy and overall invocation cost?

Q3. In the repair comparison, were the linked baselines granted access to the constraint solver? Additionally, when accounting for compilation, VLM checks, and skill learning, what are the total end-to-end LLM call counts and latency per edit? Finally, which model was used to score IF, SC, and PP?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.