- Rating
- 1
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduces a framework for improving long-horizon artifact editing by organizing previous editing attempts, validation outcomes and conflicting repairs into a persistent memory structure. The method combines an edit graph that preserves recoverable artifact versions, relevance-based retrieval of previous editing experiences and conflict analysis to identify repairs that repeatedly undo earlier progress. It is evaluated on 3 artifact-editing domains. 1. HTML/CSS poster editing; 2. SVG editing; 3. Blender scene editing. They evaluate this using Claude Opus 4.8 and GPT-5.6 Terra. Experiments show good improvements in constraint-satisfaction success.
Critical strengths
- The paper addresses a nice important problem in the space of long-horizon artifact editing.
- The memory framework combines edit graphs, relevance-based retrieval and conflict analysis to preserve previous successes and failures and guide subsequent editing decisions. I feel this is a nice approach.
- The experiments also show good improvements across three editing domains and two models.
- The trajectory-level analyses also provide some useful insights.
Critical weaknesses
- The contribution of individual components like edit graphs or relevance retrieval, etc are not isolated properly through ablations and now this makes it difficult to determine which mechanisms actually contribute to the improvements and their weightage as well.
- The eval relies heavily on structured artifacts and predefined validators. This limits the evidence of generalization to less structured editing tasks.
- The repair-cycle eval is a bit unclear to me. I feel the main text and appendix describe different evaluation setups. The reported episode counts also differ between methods.
- The evals cover only 102 tasks and 2 models. The efficiency gains are also reported on jointly solved tasks without accounting for the full computational overhead of memory maintenance and retrieval.
Questions for authors
- Can you please provide component-wise ablations to isolate the contributions of edit graphs, relevance retrieval, conflict analysis and search instructions? This is a really important requirement.
- Please clarify whether repair-cycle recovery uses matched checkpoints or independently observed cycles. Also, explain why the episode counts differ between methods?
- How well does Artifact Memory generalize to tasks with subjective visual requirements or incomplete validators where constraint violations cannot be reliably measured?
- Can the authors report end-to-end computational costs. This includes memory maintenance and retrieval. Also pls evaluate efficiency across all tasks rather than only jointly solved ones.
We thank the reviewers and area chair for their feedback. The reported results show higher constraint satisfaction across all three domains and both models. With limited additional computational resources that we have access to during this rebuttal period, we clarify the existing evidence below.
Baseline comparisons. Both +Val and AM retain chronological editing history. AM adds context linking recoverable versions, validation outcomes, and previous repairs. Its advantage over simpler supplementary summaries or retrieval remains unresolved.
Component attribution and instructions. The results support the complete system. Isolating the graph, retrieval, conflict analysis, and repair advice requires controls with matched instructions.
Recovery protocol and reporting. The scripts analyze each run's first encountered cycle within matched task cohorts, explaining unequal episode denominators. This analysis is observational. The appendix's shared-checkpoint description conflicts with that procedure. Cohorts contain 20 Poster, 20 SVG, and 22 Blender pairs. The available Blender audit assesses repetition for one AM return, with five additional cases unassessed. We withdraw the unsupported 0/6 repetition figure and the productivity comparison, whose eligibility rules differed across methods.
Visual quality. Our strongest evidence concerns specified constraints. Visual gains are modest, and Claude's Blender judge score decreases from 3.91 to 3.86. Systematic output comparisons and fuller judge documentation remain gaps.
Generalization. The evaluation covers structured artifacts with predefined validators. Applicability to subjective goals and incomplete feedback remains unestablished.
Statistical strength. Small samples limit precision. Paired uncertainty and stability across independent repetitions require further evidence.
Efficiency accounting. The reported 26–39% savings concern inference cost to first PASS on jointly solved tasks. Costs across all attempted tasks and memory overhead require further accounting.
Verification and presentation. We acknowledge unresolved references, editorial remnants, incomplete evaluation documentation, and limitations of model-assisted, unblinded trajectory annotation. The extent of human verification needs clearer documentation.