Omitted literature and the need to reposition novelty: (1) Generating tools via LLMs, verifying them on validation samples, caching them for reuse, and bypassing frontier models at inference time mirrors LATM (ICLR 2024), CRAFT (ICLR 2024), and TroVE (ICML 2024), none of which are cited, with the paper referencing only Voyager, ASI, and AWM; (2) LLM evaluators equipped with abstention mechanisms appear in Trust or Escalate (ICLR 2025), which performs selective evaluation under provable human-alignment guarantees, directly aligning with the selective risk framework; (3) Decomposing instructions into atomic conditions and verifying them via procedural checks has been explored in 2D generation evaluation via VPEval (NeurIPS 2023) and Davidsonian Scene Graphs (ICLR 2024). The paper's incremental contribution lies in combining these components for 3D editing evaluation while adding permission separation and specification freezing.
Small evaluation scale and lack of statistical support for core claims: The testbed comprises only 48 edits and 95 conditions. Based on the False Acceptance Rate (FAR) and False Rejection Rate (FRR) in Table 1, the gold standard contains a total of only 22 failed conditions (including just 3 in E2A) and 19 failed scenes (including only 3 in E2A). Consequently, all methods on E2A can only take discrete FAR values of 0, 33.3, 66.7, or 100, and the scene-level AUROC of 90.0 on E2A is anchored on merely 3 failed scenes. As the authors themselves concede, the risk interval for Experience Learning (LE) of [1.9,17.0] overlaps with that of LEGO-Eval. With 23 passing and 19 failing samples, the 95% confidence interval for a scene-level AUROC of 81.9 (estimated via the Hanley–McNeil method) spans roughly 69 to 95, fully encompassing SceneCritic's 75.1. Regarding repair, the Task Success Rate (HSR48) rises from 25/48 to 28/48 (recovering 4 and losing 1), yielding an exact McNemar test p-value of approximately 0.38. Therefore, claims in the abstract and conclusion regarding "making the most reliable decisions," achieving the "highest AUROC," and delivering statistically significant repair improvements are currently indistinguishable from noise. Furthermore, the evaluation is restricted to only four edit types (removal, movement, rotation, scaling) within single-room, single-instruction settings. I recommend scaling the evaluation to hundreds of edits and incorporating addition, replacement, appearance modification, and multi-step instructions.
Low selective risk stems primarily from selective abstention concentrated on failing samples, revealing weak failure-detection capability: Back-calculating from precision and recall in Table 1 reveals that out of 22 failed conditions, LE evaluated only 7 (correctly rejecting 5 and falsely accepting 2), while holding back the remaining 15 (68%); conversely, out of 73 passing conditions, LE held back 23 (32%). A similar pattern appears at the scene level: out of 19 failed scenes, LE predicted fail on only 6, held 12, and falsely predicted pass on 1. Table 3 further indicates that without ternary aggregation (V2), ARGUS's check-level FAR reaches 59.1% (13/22), and V0 reaches 81.8%, both substantially worse than LEGO-Eval (22.7%) and Direct State LLM (45.5%). In short, ARGUS's checking mechanism is not inherently skilled at uncovering failures; the reduction in risk is driven by converting those failures into "holds." Moreover, selective risk is incomparable across differing coverage rates: a trivial evaluator that always predicts "pass" achieves a full-coverage risk of only 22/95=23.2%, which already undercuts Direct VLM's 24.2%. While baseline models were compared at matching coverage rates for scene-level scoring, all condition-level baselines were compared at full coverage or their respective native coverages. Over the exact same 57 conditions judged by LE, back-calculating from the authors' reported LEGO-Eval P/R (71.9/85.9) yields 1 false acceptance and 7 false rejections for LEGO-Eval, compared to 2 false acceptances and 2 false rejections for LE. This shows that LE's advantage derives entirely from fewer false rejections; on the specific failure mode emphasized in the introduction that uncompleted edits being falsely accepted, LE performs no better. Additionally, because "holds" are concentrated on failing samples while scene-level AUROC assigns a score of 0.5 to holds, holds are effectively treated as failure signals.
Learned skills contribute marginally and rely on a post-hoc rule: Coverage rises from 44.2% to 60.0% (a 15-condition increment), of which 4 conditions originate from the "relation frame." Because the relation frame is a manually engineered binding component rather than an LLM-learned skill, grouping it under LE (+ registered skills) conflates results. The actual number of judgments newly contributed by learned skills is 11 (10 of which are correct). More critically, the "amount guard" was introduced post-hoc after error analysis and successfully blocks 3 false acceptances. If these 3 false acceptances occurred on the testbed (as the framing of the main text implies), removing this rule means that among the 14 skill-added judgments, 4 are incorrect, driving LE's FAR to 5/22=22.7% (identical to LEGO-Eval) and its risk to 7/60=11.7% (undermining claims of equal-error-rate coverage gains). This also creates a contradiction with the protocol stating that "libraries are frozen before E"; please clarify whether this error analysis was conducted on the development split (H), the validation split (G), or the testbed. Furthermore, Table 4 shows that the base checker completely fails to evaluate rotation, meaning the learned rotation skill essentially computes the relative SO(3) angle, an engineering task trivially solved in a few lines of code. In addition, with V-on/R-off already achieving 18/18, the repair skill merely reduces tool-call counts from 12 to 0.