acceptodds
Withdrawn from ICLR 2027

Artifact Memory: Guiding Agentic Artifact Editing with Edit Graphs and Conflict Retrieval

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Long-horizon artifact editing requires agents to manage interactions among components and dependencies across editing decisions. Many edit consequences become apparent only after rendering, yet retaining every intermediate version and observation is costly and does not explain recurring repair failures. We propose Artifact Memory to address both the representational and algorithmic needs of artifact editing. Its edit graph shares unchanged components across recoverable versions and links edits to component-level validation outcomes. Relevance-based retrieval selects prior attempts and partial successes, while conflict analysis connects outcomes across versions to identify repairs that undo earlier progress. This context supports coordinated repairs, backtracking, and alternative editing sequences while retaining lessons from abandoned branches. Across HTML/CSS poster editing, SVG editing, and Blender scene editing with Claude Opus 4.8 and GPT-5.6 Terra, Artifact Memory improves constraint-satisfaction success rates over validator-feedback baselines by 10.0–31.8 percentage points. and reduces mean inference cost by 26–39% on tasks solved by both methods. Further trajectory analyses show lower capped repair-cycle recovery costs across domains, more productive backtracking and less repetition of failed continuations.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Memory, Artifact, Agentic Editing, Agents

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_vSLNOpenReview ↗
Rating
1
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces a framework for improving long-horizon artifact editing by organizing previous editing attempts, validation outcomes and conflicting repairs into a persistent memory structure. The method combines an edit graph that preserves recoverable artifact versions, relevance-based retrieval of previous editing experiences and conflict analysis to identify repairs that repeatedly undo earlier progress. It is evaluated on 3 artifact-editing domains. 1. HTML/CSS poster editing; 2. SVG editing; 3. Blender scene editing. They evaluate this using Claude Opus 4.8 and GPT-5.6 Terra. Experiments show good improvements in constraint-satisfaction success.

Critical strengths
  • The paper addresses a nice important problem in the space of long-horizon artifact editing.
  • The memory framework combines edit graphs, relevance-based retrieval and conflict analysis to preserve previous successes and failures and guide subsequent editing decisions. I feel this is a nice approach.
  • The experiments also show good improvements across three editing domains and two models.
  • The trajectory-level analyses also provide some useful insights.
Critical weaknesses
  • The contribution of individual components like edit graphs or relevance retrieval, etc are not isolated properly through ablations and now this makes it difficult to determine which mechanisms actually contribute to the improvements and their weightage as well.
  • The eval relies heavily on structured artifacts and predefined validators. This limits the evidence of generalization to less structured editing tasks.
  • The repair-cycle eval is a bit unclear to me. I feel the main text and appendix describe different evaluation setups. The reported episode counts also differ between methods.
  • The evals cover only 102 tasks and 2 models. The efficiency gains are also reported on jointly solved tasks without accounting for the full computational overhead of memory maintenance and retrieval.
Questions for authors
  • Can you please provide component-wise ablations to isolate the contributions of edit graphs, relevance retrieval, conflict analysis and search instructions? This is a really important requirement.
  • Please clarify whether repair-cycle recovery uses matched checkpoints or independently observed cycles. Also, explain why the episode counts differ between methods?
  • How well does Artifact Memory generalize to tasks with subjective visual requirements or incomplete validators where constraint violations cannot be reliably measured?
  • Can the authors report end-to-end computational costs. This includes memory maintenance and retrieval. Also pls evaluate efficiency across all tasks rather than only jointly solved ones.
Reviewer_oKjFOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Artifact Memory helps an agent keep track of previous edits and their results. It stores artifact versions in an edit graph and retrieves relevant attempts when the agent faces a new repair problem. It also records conflicts, such as an edit fixing one constraint but breaking another. The method is tested on poster, SVG, and Blender editing, with improvements in constraint satisfaction and inference cost.

Critical strengths

The problem is practical. An editing agent can spend many turns fixing something and then undo its own progress. Keeping earlier versions together with the reasons why an edit failed seems useful.

The reported gains are also worth attention. Compared with the validator-feedback baseline, success improves across all three domains and both models. For Claude, poster success goes from 56% to 82%, and Blender success goes from 59.1% to 90.9%. These are substantial differences. Looking at recovery and backtracking behavior also gives more information than only reporting the final success rate.

Critical weaknesses

What I cannot tell from the experiments is how much of this improvement needs the proposed memory structure. The main comparison adds Artifact Memory to an agent with a fixed-length history window. A simpler baseline could keep a short summary of failed attempts, retrieve relevant past edits, or save useful checkpoints. Without these comparisons, the results show that the whole system helps, but do not clearly establish the benefit of the edit graph and conflict retrieval.

The method also includes instructions about how to inspect history, choose repairs, and backtrack. Appendix A explicitly says the memory and these instructions jointly define the method. This makes a controlled comparison especially important. Giving the baseline the same repair instructions would help separate better instructions from better memory.

There is a reporting issue in the recovery experiment that needs clarification. The main text describes recovery from cycles observed in each method’s own runs. Appendix F instead says both methods resume identical repair-cycle checkpoints. Tables 2 and 3 report the same numbers, but the episode counts differ between methods, for example 15 versus 12 on Poster. These descriptions suggest different experimental designs. It is difficult to judge the recovery result without knowing which protocol was actually used.

The strongest evidence is for satisfying the specified constraints. The visual-quality changes are smaller, and are not positive in every setting: Claude’s Blender judge score changes from 3.91 to 3.86. The tasks also rely on validators that can identify affected components. This is a useful setting, but the results leave open how much the method helps when feedback is incomplete or the editing goal is more subjective.

Questions for authors

How does Artifact Memory compare with a history-summary or retrieval baseline under the same context and inference budgets?

Reviewer_vSqwOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes Artifact Memory to improve the use of editing history and error recovery in long-horizon agentic artifact editing. It maintains recoverable versions in an edit graph, associates edits with component-level validation outcomes, and retrieves previous failures and partial successes to support coordinated repairs and backtracking. Experiments cover HTML/CSS posters, SVG graphics, and Blender scenes using two backbone models. The paper reports improvements of 10.0–31.8 percentage points in constraint-satisfaction success rate over validator-feedback baselines, together with 26–39% lower inference costs on tasks solved by both methods.

Critical strengths
  1. The problem is practically relevant. Local edits frequently invalidate previously satisfied requirements. Linking historical versions, editing actions, and validation feedback is a well-motivated approach.
  2. The evaluation has some breadth. Improvements across three editing domains and two backbone models provide preliminary evidence of effectiveness.
  3. The paper considers both effectiveness and cost. It reports visual scores, editing turns, and inference costs alongside constraint satisfaction, and provides a Blender example illustrating memory-guided backtracking.
Critical weaknesses
  1. The paper lacks systematic comparisons of final visual outputs, leaving its visual-quality claims insufficiently supported. Figure 2 illustrates tasks and constraints, while Figure 4 presents a single Blender editing trajectory. Neither provides systematic, side-by-side comparisons of the final outputs produced by the baselines and AM across all three domains. Representative failure cases are also missing. Validator success cannot substitute for direct assessment of artifact quality. The visual-score gains in Table 1 are modest; for Claude on Blender, the score actually decreases from 3.91 to 3.86 despite success increasing from 59.1% to 90.9%. Constraint satisfaction and visual quality therefore require separate empirical justification. (pp. 6–8)

  2. The baseline coverage is insufficient to establish an advantage over existing memory approaches or simple alternatives. The experiments primarily compare Base, the validator-feedback baseline (+Val), and the complete AM system. Although the related-work discussion includes A-MEM, AgentFold, Git Context Controller, and BlenderAlchemy, no experimental comparisons or adapted baselines are provided. At a minimum, the evaluation should include token-matched history summarization, recent-edit retrieval, and a simple memory system with version rollback. The current results establish an advantage over the selected bounded-history baseline, but do not establish the necessity of the proposed memory design. (pp. 2–3, 6–7, 22–23)

  3. There are no convincing component ablations, and memory mechanisms are confounded with additional search instructions. Despite its “Analysis and Ablation” title, Section 5.3 primarily presents examples and trajectory statistics rather than experiments isolating the edit graph, conflict retrieval, or backtracking information. Appendix D also supplies detailed search instructions for AM. On page 19, the authors explicitly acknowledge that memory and these instructions jointly define the method and that behavior cannot be attributed to the representation without controlling the instructions. Controls such as identical search instructions without AM, and retrieval without conflict analysis, are therefore necessary. (pp. 8, 19, 21)

  4. The repair-cycle experiment contains a substantive inconsistency in its stated design. The main text describes Table 2 as evaluating the first qualifying repair cycle independently within each run. The appendix instead describes restarting both methods from identical repair-cycle checkpoints to control difficulty. These are different experimental designs, yet they report identical numerical results. Under a shared-checkpoint design, why do the denominators differ between methods—15 versus 12 for Poster and 17 versus 13 for Blender? Furthermore, the analysis states that it uses 20 tasks per domain, but the main text reports Blender cycle incidence as 13/22 versus 17/22. These inconsistencies prevent a reliable interpretation of the recovery results. (pp. 8–9, 28)

  5. The evaluation protocol and statistical evidence remain insufficient for robust conclusions. SVG and Blender contain only 30 and 22 tasks, respectively; the GPT success-rate gains in both domains correspond to a net increase of approximately three successful tasks. Although the appendix provides some confidence intervals, repeated independent runs and uncertainty estimates for paired method differences are still needed. The visual evaluation also lacks sufficiently clear documentation of the evaluator configuration, scoring procedure, and human validation. Finally, the reported cost reductions are conditional on jointly solved tasks, whose actual subset sizes are absent from the relevant tables. These reductions should not be generalized to average costs over the full task distribution. (pp. 7, 29–34)

  6. The manuscript contains conspicuous remnants of an incompletely consolidated automated research/writing workflow. Page 10 discloses AI assistance in implementation, benchmark construction, and manuscript polishing, while trajectory analysis uses model-assisted annotation. AI assistance itself is not grounds for rejection. However, the manuscript retains unresolved references such as “Appendix ??,” “Figure ??,” and “Table ??,” together with insufficiently integrated statements including “The final experimental report should release these resources” (p. 10), “Compare B and B+V” (p. 26), and “We enable backtracking in SKILL” (p. 28). Combined with duplicated experimental tables and conflicting protocol descriptions, these issues raise concerns about the adequacy of human verification. They do not establish the use of any particular auto-research system, but they materially weaken the manuscript’s completeness and reproducibility.

Questions for authors
  1. Can you provide matched input, +Val, and AM renderings across all three domains, including detailed views, failures, and cases that pass validation despite poor visual quality?
  2. Under matched model, context, and inference budgets, how does AM compare with history summarization, recent-edit retrieval, simple version rollback, and relevant memory methods?
  3. With search instructions held fixed, what are the individual contributions of the edit graph, conflict retrieval, and backtracking information?
  4. Were Tables 2 and 3 obtained from independently observed repair cycles or controlled restarts from shared checkpoints? Please reconcile the unequal denominators and the inconsistent Blender sample sizes.
  5. Please specify the visual evaluator, scoring prompts, evaluated-subset sizes, jointly solved-subset sizes, and number of independent runs. Are the paired improvements stable across runs?
  6. What human verification was performed on AI-assisted benchmark construction and model-annotated trajectories? Can you provide verification records, reconcile the experimental protocols, and complete the references and run configurations?

Comments on the paper

1 comment

We thank the reviewers and area chair for their feedback. The reported results show higher constraint satisfaction across all three domains and both models. With limited additional computational resources that we have access to during this rebuttal period, we clarify the existing evidence below.

  • Baseline comparisons. Both +Val and AM retain chronological editing history. AM adds context linking recoverable versions, validation outcomes, and previous repairs. Its advantage over simpler supplementary summaries or retrieval remains unresolved.

  • Component attribution and instructions. The results support the complete system. Isolating the graph, retrieval, conflict analysis, and repair advice requires controls with matched instructions.

  • Recovery protocol and reporting. The scripts analyze each run's first encountered cycle within matched task cohorts, explaining unequal episode denominators. This analysis is observational. The appendix's shared-checkpoint description conflicts with that procedure. Cohorts contain 20 Poster, 20 SVG, and 22 Blender pairs. The available Blender audit assesses repetition for one AM return, with five additional cases unassessed. We withdraw the unsupported 0/6 repetition figure and the productivity comparison, whose eligibility rules differed across methods.

  • Visual quality. Our strongest evidence concerns specified constraints. Visual gains are modest, and Claude's Blender judge score decreases from 3.91 to 3.86. Systematic output comparisons and fuller judge documentation remain gaps.

  • Generalization. The evaluation covers structured artifacts with predefined validators. Applicability to subjective goals and incomplete feedback remains unestablished.

  • Statistical strength. Small samples limit precision. Paired uncertainty and stability across independent repetitions require further evidence.

  • Efficiency accounting. The reported 26–39% savings concern inference cost to first PASS on jointly solved tasks. Costs across all attempted tasks and memory overhead require further accounting.

  • Verification and presentation. We acknowledge unresolved references, editorial remnants, incomplete evaluation documentation, and limitations of model-assisted, unblinded trajectory annotation. The extent of human verification needs clearer documentation.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.