acceptodds
Under review as a conference paper at ICLR 2027

World Editing: Intervening on Executable Worlds at Increasing Depth

Abstract

Interactive world models are increasingly capable of generating environments and acting within them, yet deliberately editing an existing executable world remains underexplored. We formulate world editing as intervening on an existing world while preserving properties that should remain unchanged, and introduce intervention depth as an axis describing how strongly an edit couples world entities, dynamics, and systems. We instantiate this capability through industry-grade game modding and introduce IGMWorld, together with IGMBench, a benchmark of 110 tasks and over 1.1K executable state and behavioral criteria across Minecraft and Terraria. The tasks span property, entity, dynamics, and system interventions and are evaluated through deterministic executability, behavioral, preservation, and visual checks. Frontier coding agents already exhibit substantial world-editing capability: the strongest configuration solves 78.2% of tasks under a strict task-level criterion, while criterion-level performance reaches 94.8%. Reliability generally decreases with intervention depth, and this pattern persists even among tasks with similar numbers of evaluation criteria. Most failed edits still build and load successfully, suggesting that the main difficulty is making the edited world behave as requested. Visual consistency remains a separate weakness, with all evaluated configurations below 50% joint visual pass rate. These results show that world editing is a distinct capability from world generation and interaction, and that executable games provide a practical testbed for studying it.

Primary area:
datasets and benchmarks
Keywords:
World Editing, Coding Agents, World Models, Game Modding, Executable Worlds

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_yJZEOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper formulates world editing as modifying an existing executable world while preserving unrelated properties, and organizes edits into four intervention levels: property, entity, dynamics, and system. It introduces IGMWorld and IGMBench, comprising 110 game-modding tasks across Minecraft and Terraria, with executable behavioral checks and separate visual-consistency evaluation. The experiments report decreasing reliability for deeper interventions, predominantly behavioral rather than compilation failures, and a gap between functional correctness and visual integration. The main contribution is a benchmark and an analysis of coding-agent capabilities in game modification.

Critical strengths
  1. Visual consistency is the most interesting aspect of the evaluation. Separating semantic correctness from compatibility with the host game's art style captures a meaningful requirement beyond functional implementation. The reported differences between functional and visual rankings are worth investigating further.
  2. The benchmark has practical value for game-development agents. Evaluating modifications through actual game states and interactions, rather than compilation alone, provides a useful test of whether agents implement the requested behavior.
Critical weaknesses
  1. The connection to world modeling and the novelty beyond agentic game development remain insufficiently established. The current setting primarily evaluates coding agents’ ability to modify game properties, content, and systems through existing code, assets, and modding interfaces, rather than learning or editing an environment model. While the benchmark is valuable for game-development and software-engineering agents, its direct contribution to interactive world models remains motivational rather than empirically demonstrated, even though Section 5 appropriately acknowledges the complementary relationship. Moreover, GameDevBench already evaluates modifications to existing game projects using engine-based tests [1]. The authors therefore need to explain what new capabilities or challenges intervention depth reveals beyond reorganizing familiar game-modification tasks. The distinctive contribution should be established through insights into cross-system coupling, preservation of non-target properties, or visual integration—not through the generation-versus-editing framing alone.

  2. The experiments do not yet convincingly establish intervention depth as an explanatory axis beyond ordinary task complexity. Figure 2 stratifies tasks by criterion count, but tasks within a bin can still differ in game, task type, API difficulty, and the complexity of individual requirements. The results support an association between the assigned levels and success, but do not clearly identify cross-component coupling as the relevant source of difficulty. A small within-game comparison of tasks with similar requirements but different dependencies would be more informative than simply expanding the benchmark.

[1] Wayne Chi et al. GameDevBench: Evaluating Agentic Capabilities Through Game Development. arXiv:2602.11103, 2026.

Questions for authors
  1. Is this editing in dynamic? In the static case, what distinguishes the evaluated capability from generating a new world variant from a revised prompt?
  2. Could you clarify the counting units in Table 2, Figure 2b, and CPR? Are the 1,161 criteria in Table 2 unique criteria that can be reused or combined across tasks, or the sum of all per-task criteria? Under the latter interpretation, the average is approximately 10.55 criteria per task, which seems difficult to reconcile with Figure 2b's quartile bins of ≤13, 14–19, 20–28, and >28.
  3. How well is preservation tested at L3/L4? Since regression checks are concentrated at L1, please report how many deeper tasks include them and which non-target behaviors they protect.
  4. Could strict thresholds partly explain the low visual pass rates? Table 18 reports mean balanced accuracy of 0.61. Could a small held-out human evaluation verify the main visual conclusion and model rankings?
  5. One more thing I'm interested in: Did the authors analyze what drives the visual failures? Could you break down visual performance by asset-creation method, image-generation backend, and whether agents inspected native reference assets?
4 comments
AuthorsOpenReview ↗

Response to Reviewer yJZE

We thank the reviewer for the detailed and constructive review.

W1: Connection to World Modeling and Distinction from Game Development

We agree that our current experiments evaluate coding agents editing executable worlds. Applications to learned world models remain future work. IGMWorld provides a framework for constructing controlled world variants with changes to entities, dynamics, and interacting systems. These variants could provide intervention data for future world-model training and evaluation (§5). We also acknowledge that GameDevBench also evaluates modifications to existing game projects using engine-based tests. We will clarify this distinction in the related work.

AspectGameDevBenchIGMBench
Editable scopeModify game-project source codeExtend shipped commercial games without modifying host source
Intervention depthNo world-system coupling taxonomyL1–L4, from properties to interacting systems
PreservationNot testedPaired runtime checks against the original game
Visual assetsProvides visual assetsAgents create assets evaluated against native game art

Evidence of distinctive challenges:

  • Cross-system integration: Interaction/progression failures become more prominent at deeper intervention levels, reaching 43% of behavioral failures at L4 (§4.3, Figure 3b). Additional failures involve registration, initialization, and resource conventions (Appendix Figure 9).
  • Preservation: Targeted interventions require non-target properties to remain unchanged. We evaluate this through paired regression checks and have expanded preservation coverage to all 110 tasks (Q3).
  • Visual integration: Functional success does not guarantee visual consistency, and model rankings differ between functional and visual correctness (§4.5, Table 4).

W2: Intervention Depth and Cross-System Coupling

We agree that criterion-count stratification alone cannot establish why deeper edits are harder. We conducted additional analyses. Across 110 tasks, adding intervention level to a regression controlling for game, task genre, and complexity measures improves explained variance (R²: 0.455 → 0.518, p = 0.007). The adjusted L1–L4 success gap remains 29 percentage points (95% CI: 7–56), compared with 48 points before adjustment. Following the reviewer's suggestion, we compared 10 pairs of tasks from the same game and genre, with similar executable-check counts but different runtime dependencies.

MeasureFewer dependenciesMore dependencies
WSR51%43%
Integration-related failures38%55%

For example, in Terraria, a silver-coin sword (23 checks, one runtime dependency) succeeds in 5/7 runs, while a reflecting buckler (24 checks, four dependencies) succeeds in 1/7. The buckler frequently fails projectile ownership and reflection requirements.

We found that intervention depth helps predict task difficulty beyond measured complexity factors, especially between L1 and deeper levels. Cross-system coupling may contribute to this difficulty, but our current evidence cannot confirm its independent effect. The matched-pair success difference is inconclusive (p = 0.49).

AuthorsOpenReview ↗

Response to Reviewer yJZE (part 2)

Q1: Is Editing Static or Dynamic?

Edits are static. We were aware that dynamic edits were possible in minecraft and terraria, but deeper changes to world mechanics often require underlying implementation.

Distinction from World Generation. World editing starts from an existing world and applies a targeted intervention. The original world provides a fixed reference for verifying intended changes and preserving non-target properties through paired runtime comparisons (§3.1). This introduces explicit consistency requirements across existing mechanics and systems, which are not necessarily enforced when generating a world variant from a revised prompt.

Q2: Clarification of Criterion Counting

Thank you for pointing out this ambiguity. Table 2 counts 1,161 behavioral criteria (10.55 per task), each with at least one executable check. Each criterion may involve multiple checks, yielding 2,771 executable checks (25.19 per task). Figure 2b uses executable check counts for its quartile bins (≤13, 14–19, 20–28, >28), explaining the apparent discrepancy. CPR is calculated over individual executable checks. We will clarify these counting units in the paper.

DepthTasksBehavioral criteriaExecutable checks
L127174459
L229306693
L327323882
L427358737
Total1101,1612,771

Q3: Preservation Coverage at L3/L4

The original 225 regression checks (Appendix Table 7) were derived from task-specific requirements. L1 tasks often specify properties to preserve due to parameter change, while deeper tasks L2-L4 primarily describe new content and mechanics, resulting in fewer explicit preservation requirements and sparse regression coverage.

To improve this, we have expanded preservation testing to all 110 tasks, with 711 paired regression checks across four intervention depth levels. Each check compares properties of existing vanilla content against a mod-free baseline under the same environment, with task-specific exemptions for intended changes. The checks cover items, entities, recipes, loot tables, player statistics, and other game-specific categories.

DepthMinecraftTerrariaTotal checks
L18291173
L29098188
L38491175
L48491175
Total340371711

Across runs, only one preservation check failed, in a run that already failed its authored requirements. Thus, the expanded checks introduced no additional WSR failures, leaving our reported WSR results unchanged.

Q4: Visual Metric Validation

Following the reviewer's suggestion, we conducted a held-out human study of 112 unseen agent-created textures across L1–L4 and 7 model configurations. Annotators familiar with Minecraft and Terraria evaluated each texture in its game context, spending approximately 30 minutes each. We also collected reasons for negative judgments (Q5). TPIPS rejects more textures than human majority judgments, and adjusting α barely improves agreement. Human–TPIPS model-ranking correlation equals to ρ = 0.36. We will revise the paper to state that visual integration is weak under both human and automatic evaluation.

MeasurePreliminary (40 textures, 9 annotators)Held-out (112 textures, 3 annotators)
Style balanced accuracy (α = 0.15)0.600.49
Semantic balanced accuracy (α = 0.15)0.620.59
Mean balanced accuracy (α = 0.15)0.610.54
Best style accuracy (α)0.61 (0.10)0.56 (0.05)
Best semantic accuracy (α)0.65 (0.25)0.62 (0.20)
Human majority joint pass55%47%
Automated joint TPIPS pass (α = 0.15)23%26%

Q5: What Drives Visual Failures?

In our held-out human study (Q4), annotators provided reasons for negative judgments (Annotators could select multiple reasons). Among 181 negative responses, the most common reasons were:

Human-reported failure reason% of negative responses
Mismatched pixels, shading, or outlines65%
Unrecognizable content43%
Inconsistent colors39%
Broken or unfinished appearance22%

We further analyzed our 1,968 scored custom textures using the paper's visual thresholds (α = 0.15, k = 5).

FactorTexturesJoint visual pass rate
Image-generated54118%
Code-drawn64829%
Native assets inspected60936%
Native assets not inspected78521%

After controlling for model, game, and asset category, neither creation method (p = 0.70) nor reference inspection (p = 0.98) significantly predicts success.

We also identify a concrete stylistic mismatch: image-generated Minecraft item sprites contain a median of approximately 115 colors, compared with 10 in native assets and 9 in code-drawn assets. Generated sprites frequently exhibit smooth gradients and anti-aliased edges that differ from native pixel art (Appendix Figure 7).

Thank you for the detailed responses and additional analyses, particularly the expanded preservation checks and held-out human evaluation. These additions address several of my concerns, and I will raise my overall assessment to 3.

I encourage the authors to incorporate these findings and clarifications into the revised paper. Regarding W2, the additional experiment already provides useful insights. I encourage the authors to build on these encouraging results and explore the question in greater depth, which would further strengthen the paper.

For future work, I would be interested in dynamic world editing and in demonstrating how controlled world variants can support agent adaptation and model training or evaluation.

Dear Reviewer yJZE,

Thank you for your feedback and for increasing the rating. We have revised the manuscript. Thank you again for your time!

Reviewer_f9fuOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces world editing as modifying existing executable worlds while preserving unrelated properties. It proposes a four-level intervention taxonomy and presents IGMWorld and IGMBench, covering 110 Minecraft and Terraria tasks. Using existing coding-agent configurations, the authors evaluate executability, functional correctness, and visual consistency. The results characterize performance across intervention levels, identify behavioral validation as the dominant failure stage, and report low visual pass rates under the selected metric.

Critical strengths

Useful executable evaluation infrastructure. The benchmark evaluates actual game states and behavior rather than relying solely on compilation or screenshots. This provides a concrete setting for assessing agents on modifications involving code, assets, and gameplay mechanics.

Broad empirical coverage within the setting. Experiments across two game ecosystems and seven agent configurations, together with failure-stage analysis, provide useful baseline measurements.

Critical weaknesses

Limited differentiation and scientific insight. The contribution primarily consists of benchmark construction and descriptive evaluation. While such contributions can be valuable, the paper does not sufficiently establish what distinct capability is measured beyond complex coding and game-development tasks.

The defining preservation requirement is incompletely validated. Preserving unrelated world properties is central to the formulation, yet regression checks cover only 30 of 110 tasks and are concentrated at L1. Consequently, the reported functional success rates provide limited evidence that deeper interventions satisfy this defining requirement, weakening the alignment between the proposed capability and its measurement.

Questions for authors

Which specific capabilities or failure modes does IGMBench capture that existing coding and game-development benchmarks do not, and what evidence demonstrates this distinction?

How are intervention levels assigned consistently, particularly at the L2/L3 boundary? Beyond criterion-count stratification, what evidence supports intervention depth as an explanatory factor rather than differences in task composition or ecosystem integration?

What is the regression-test coverage at each intervention level, and do regression failures affect WSR? How should successful deeper edits be interpreted when preservation is not tested?

3 comments
AuthorsOpenReview ↗

Response to Reviewer f9fu

We thank reviewer for the precise questions

W1/Q1: Differentiation and scientific insight. World editing studies whether agents can modify existing world rules and dynamics while preserving unrelated properties (§1–2). IGMBench evaluates this capability through executable Minecraft and Terraria games, where agents extend shipped games without modifying host source code. Such executable interventions could also provide data for future world-model research (§5). Difference in below table:

AspectGameDevBenchIGMBench
Editable scopeModify game-project source codeExtend shipped commercial games without modifying host source
Intervention depthNo world-system coupling taxonomyL1–L4, from properties to interacting systems
PreservationNot testedPaired runtime checks against the original game
Visual assetsProvides visual assetsAgents create assets evaluated against native game art

Our setting reveals challenges and findings:

  • Behavioral correctness: Most failed edits successfully build and load, yet fail to realize the requested behavior (§4.3, Figure 3a).
  • Cross-system integration: Interaction/progression failures become more prominent at deeper intervention levels, reaching 43% of behavioral failures at L4 (Figure 3b). Additional failures involve registration, initialization, and resource conventions (Appendix Figure 9).
  • Visual consistency: Functional success does not guarantee visual integration, and model rankings differ between functional and visual correctness (§4.5, Table 4).

We hypothesize that behavioral integration, cross-system consistency, and visual integration will remain important challenges in world editing across different world representations, including learned world models. Our current empirical evidence comes from Minecraft and Terraria; validating these hypotheses in other settings remains future work (§5).

W2/Q3: Preservation Test

Do regression failures affect WSR? Yes, regression failures count into WSR. The original 225 regression checks (Appendix Table 7) were derived from task-specific requirements. L1 tasks often specify properties to preserve due to parameter change, while deeper tasks L2-L4 primarily describe new content and mechanics, resulting in sparse regression coverage.

To improve this, we have expanded preservation testing to all 110 tasks, with 711 paired regression checks across four intervention depth levels. Each check compares properties of existing vanilla content against a mod-free baseline under the same environment, with task-specific exemptions for intended changes. The checks cover items, entities, recipes, loot tables, player statistics, and other game-specific categories.

DepthMinecraftTerrariaTotal checks
L18291173
L29098188
L38491175
L48491175
Total340371711

Across 7 model runs from 110 tasks (total 770 runs), only one preservation check failed, in a run that already failed its authored requirements. Thus, the expanded checks introduced no additional WSR failures, leaving our reported WSR results unchanged.

AuthorsOpenReview ↗

Response to Reviewer f9fu (part 2)

Q2: Assignment on Intervention Levels

2a) Boundary on L2/L3 tasks. Levels were assigned using the level rubric (Table 1). When a task introduces both an entity and a new mechanic, we assign the level based on the highest-depth required intervention. To assess the consistency of this distinction, we conducted an independent human annotation study covering all 56 L2/L3 tasks (29 L2, 27 L3). 3 human with minecraft and terraria gaming experience participated and classified each task using only its description and the Table 1 definitions, without seeing the original level assignments or evaluation criteria. Overall human study mostly picked same assignment as assigned ones. Most disagreements involve adding entities with new mechanics. We will clarify this boundary in Table 1.

Agreement measureResult
Individual agreement with assigned levels82–89% (κ = 0.64–0.79)
Majority agreement with assigned levels49/56 (88%, κ = 0.75)
Inter-annotator agreementKrippendorff's α = 0.62
Tasks with unanimous agreement40/56 (71%)

2b) Intervention Depth Beyond Task Complexity. We examined whether intervention depth remains associated with success after accounting for task complexity. We performed a within-game comparison, to group all 110 tasks by game and executable-check-count quartile. Shallower levels outperform deeper levels in 29/32 within-group comparisons. L1 achieves the highest success rate in every group containing L1.

GameExecutable checksL1L2L3L4
Minecraft≤1391% (11)60% (6)38% (3)–
Minecraft14–1971% (1)51% (7)50% (2)36% (2)
Minecraft20–2893% (2)29% (2)34% (5)31% (6)
Minecraft>28––36% (4)19% (6)
Terraria≤13––67% (3)65% (7)
Terraria14–1998% (6)78% (7)57% (2)62% (3)
Terraria20–2896% (4)71% (2)57% (4)–
Terraria>2871% (3)54% (5)46% (4)43% (3)

Cells report WSR (number of tasks). Bold indicates the highest WSR within each group.

We further controlled for task genre and other complexity measures. Adding intervention level improves the regression fit (R²: 0.455 → 0.518, p = 0.007). The L1–L4 success gap remains 29 percentage points after adjustment, compared with 48 points before adjustment.

Our failure analysis provides complementary evidence: interaction/progression failures become more prominent at deeper levels, reaching 43% of behavioral failures at L4 (§4.3, Figure 3b). The difference is strongest between L1 and deeper levels, while L2–L4 are less consistently ordered. So we cannot yet determine whether cross-system coupling causes these differences.

Dear Reviewer f9fu,

Thank you again for your thoughtful feedback. We have addressed your concerns in our rebuttal and revised manuscript.

TL;DR: We clarified the distinction from GameDevBench, expanded preservation checks to cover all 110 tasks, and independently validated L2/L3 assignments (88% agreement). We also added controlled analyses showing that intervention depth predicts task success beyond measured task complexity.

As the discussion period is nearing its end, we would appreciate hearing whether our responses have adequately addressed your concerns and if you could consider increasing aspect scores and overall rating.

Reviewer_CsTEOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper evaluates AI agent’s ability to perform world editing. The task involves deliberately intervening on an existing executable world while preserving unrelated properties. The authors propose a four-level taxonomy of intervention depth and instantiate it through IGMWorld/IGMBench, a benchmark of 110 tasks.

Critical strengths
  1. The distinction between world generation, world interaction, and world editing is a useful conceptual note.
  2. Substantial engineering effort. Building a benchmark that requires significant engineering effort.
  3. The empirical findings are interesting.
Critical weaknesses
  1. It is unclear why such a benchmark is being built. IGMbench is a well-structured game editing task but what makes this task of interest for testing? What existing SWE and game dev development falls short? The authors need to elaborate on what this benchmark is aim to measure, why it is important, and what findings are aiming to generalize.

  2. The criteria-based evaluation requires more detail. As the primary contribution of this paper is the benchmark itself, I would suggest that the authors to discuss how IGMBench is created and validated. For example, the size of the tasks are small and human judge agreement is low. Can such a benchmark offer any reliable and generalizable evaluations? In my opinion, the evaluation results are secondary, the focus of the paper instead should be validating the proposed benchmark.

Questions for authors
  1. How do you determine T=3600?
  2. Why is regression/preservation testing so sparse at L3/L4?
6 comments
AuthorsOpenReview ↗

Response to reviewer CsTE

We thank the reviewer for the careful review. We agree that benchmark validation deserves greater emphasis and we will move key construction and validation details into the main text.

W1: What this work measures and why it matters

World editing asks whether AI systems can intentionally modify the rules and dynamics of an existing world while preserving unrelated properties (§1–2). This capability could enable controlled variations of world dynamics for studying agent adaptation and generalization (§5), and such executable world variants could serve as a source of intervention data for training and evaluating future world models, although these downstream applications remain future work. IGMBench provides a concrete evaluation of this capability through executable games, where requested interventions and their consequences can be verified through in-game states and interactions (§3). Minecraft and Terraria offer complex, established worlds with interconnected mechanics and systems, making them suitable testbeds for evaluating interventions at increasing depth, and agent does not modify the game's (host's) source code like prior SWE and GameDev works.

AspectSWE-bench / GameDevBenchIGMBench
What the agent editsSource code of an editable projectA shipped game extended through modding interfaces; the host source cannot be modified
Intervention depthNot explicitly organized by world-system couplingDepth L1–L4, from properties to interacting systems
How correctness is checkedRepository or engine-based testsActions performed in the running game, checking the resulting state
PreservationExisting project tests (SWE-bench); Not tested (GameDevBench)Paired runtime comparisons with the unmodified game
Visual assetsGameDevBench provides assetsCreated by the agent; must match the host game's art style

Our setting reveals challenges and findings:

  • Behavioral correctness: Most failed edits successfully build and load, yet fail to realize the requested behavior (§4.3, Figure 3a).
  • Cross-system integration: Interaction/progression failures become more prominent at deeper intervention levels, reaching 43% of behavioral failures at L4 (Figure 3b). Additional failures involve registration, initialization, and resource conventions (Appendix Figure 9).
  • Visual consistency: Functional success does not guarantee visual integration, and model rankings differ between functional and visual correctness (§4.5, Table 4).
  • We hypothesize that behavioral integration, cross-system consistency, and visual integration will remain important challenges in world editing across different world representations, including learned world models. Our current empirical evidence comes from Minecraft and Terraria; validating these hypotheses in other settings remains future work (§5).

W2: How the benchmark is created and validated

Construction (§3). 11 annotators with modding experience authored 110 tasks with instructions and expected behaviors (example at Appendix Figure 12). Agents receive prompt requirements (Appendix Figure 11), while grading uses hidden executable checks. Each behavioral check sets up the game, performs an action, and measures the result (Appendix Figure 13). For example, a Fire Sword check spawns a target, attacks it with the sword, and verifies that the target is set on fire in the running game.

Validation. We conducted control tests in the table below:

TestExpectedResult
(1) Blank mod graded on every taskNo task succeeds0/110 succeed
(2) Task solvabilityAll tasks are feasible104/110 (95%) solved by at least 1 agent; remaining 6 manually verified as feasible
(3) Preservation checks on the unmodified gameAll pass705/705 pass
(4) Preservation checks catch deliberate vanilla editsonly editted area's check fails13/13 caught with no false alarm
(5) Resampling tasks (bootstrap)Stable results95% CI within 8 points per model; top and bottom ranks stable

Human agreement. The reported Krippendorff's α values (style and object) come only from texture study used to calibrate the visual metric (Appendix Table 18). In this study, TPIPS showed better agreement with human judgments than CSD (AUC 0.66 vs. 0.54; Appendix Table 14). Functional scores (CPR, WSR) come only from automatic checks with no human scoring involved.

We are additionally conducting an independent human audit of sampled functional evaluation. We will follow up and report statistical results during the author-reviewer discussion. (independent human audit of sampled functional evaluation now in Part 3 Response)

AuthorsOpenReview ↗

Response to reviewer CsTE (part 2)

Q1: Why a T=3600 budget?

We chose a T=3600 budget to give agents time for several rounds of editing, building, and testing, while keeping the cost manageable. If time runs out, we grade whatever the agent left.

Empirical justification. Median agent runtimes range from 1 to 47 minutes (Appendix Table 8), and only 4.3% of runs reached the timeout.

DepthRuns reaching the limit
L16/189 (3.2%)
L210/203 (4.9%)
L312/189 (6.3%)
L45/189 (2.6%)
All33/770 (4.3%)

Timeout rates show no monotonic increase with intervention depth; L4 has the lowest rate. So deeper tasks don’t fail more often because agents run out of time.

Budget sensitivity. We further estimate bounds under cases in a 30-minute limit and for no limit.

ModelWSR @1800s (floor)WSR @3600sNo-limit ceiling
GPT-5.6 Sol76.4%78.2%78.2%
GPT-5.6 Luna64.5%64.5%64.5%
Claude Opus 4.869.1%71.8%74.5%
Gemini 3.5 Flash48.2%57.3%67.3%
Kimi-K338.2%60.0%67.3%
GLM-5.336.4%48.2%50.9%
DeepSeek-V4-Pro31.8%31.8%31.8%

The 1800s floor assumes that runs finishing successfully after 30 minutes fail under the shorter budget. The no-limit ceiling assumes every unsuccessful timed-out run eventually succeeds. These are bounds derived from recorded runs. Under both bounds, the top two and bottom-ranked configurations retain their positions, while the middle rankings are budget-sensitive. Only 25/770 runs could gain success by removing the one-hour limit (if we assume it eventually does the job).

So we are using T=3600 as a practical trade-off between evaluation cost and task completion.

Q2: Why was preservation testing sparse at L3/L4?

The original 225 regression checks (Appendix Table 7) were derived from task-specific requirements. L1 tasks change existing values, so they naturally list what must stay the same. L2–L4 tasks mostly add new content, so they rarely do.

To improve this, we have expanded preservation testing to all 110 tasks, with 711 paired regression checks across four intervention depth levels. Each check compares the game’s original content with and without the mod, ignoring anything the task asked to change. The checks cover items, entities, recipes, loot tables, player statistics, and other game-specific categories.

DepthMinecraftTerrariaTotal checks
L18291173
L29098188
L38491175
L48491175
Total340371711

Across 7 model runs from 110 tasks (total 770 runs), only one preservation check failed, in a run that had already failed the task. Thus, the expanded checks introduced no additional WSR failures, leaving our reported WSR results unchanged. These checks verify non-target vanilla-state properties; dynamic behavioral side effects outside the checked properties remain a limitation.

AuthorsOpenReview ↗

Response to reviewer CsTE (part 3)

Human Validation of Functional Evaluation. We conducted independent human playtesting on 39 agent-generated mods across 6 tasks from L2 and L3. Playtesters followed step-by-step guides derived from the executable checks, spending approximately 20 minutes per mod. We provided cheat tools to help testers set up game states, obtain items, and access most conditions needed for testing. We found strong agreement between automated checks and human playtesting. Some behaviors remain difficult to trigger manually, such as a parry mechanic with a 0.5-second blocking window. Human testers may miss these conditions even when the implementation is correct. We provide confusion matrix to accommodate our agreement results.

Agreement Results

Evaluation unitHuman–automated agreementCohen's κ
Individual behaviors298/306 (97.4%)0.80
Complete runs36/39 (92.3%)0.82

Confusion Matrix (Individual Behaviors)

Human \ automatedPassFail
Pass2815
Fail317

Comments on the paper

1 comment
AuthorsOpenReview ↗

Revision Change Log

We thank all the reviewers for their thoughtful comments. We have revised our manuscript accordingly. Here is the change log:

  • Evaluation validity: As requested by reviewer CsTE, we added human playtesting, task feasibility validation, and negative controls (Appendix G.1). Human and automated evaluations agree on 97.4% of tested behaviors.

  • Behavioral preservation: To address concerns raised by reviewers CsTE and f9fu, we expanded regression checks (total 711) to cover all 110 tasks, with per-level coverage and additional controls (Appendix G.2; Table 7). Preservation checks caused no additional task-level failures.

  • Intervention-level consistency: In response to questions raised by reviewer f9fu, we independently validated L2/L3 task assignments (Appendix G.3; Table 21). Human majority labels agree with 88% of the original assignments.

  • Intervention depth and task complexity: As suggested by reviewers f9fu and yJZE, we added within-game check-count stratification, adjusted regression analysis, and matched runtime-dependency comparisons (Appendix G.3; Tables 22–24). Intervention depth remains predictive of task success after controlling for measured task complexity.

  • Visual evaluation: To address concerns raised by reviewer yJZE, we added held-out human validation, visual failure analysis, and controlled comparisons of asset-creation strategies (Appendix G.4; Tables 25–29). TPIPS is more restrictive than human judgments and shows limited agreement in model rankings.

  • Agent time budget: In response to reviewer CsTE's question, we added timeout analysis and budget-sensitivity bounds for the 3,600-second limit (Appendix G.5; Tables 30–32). Only 4.3% of runs reached the time limit.

  • Motivation and related work: To address concerns raised by reviewers CsTE, f9fu, and yJZE, we clarified the motivation for world editing and its distinction from existing game-development benchmarks (Sections 1, 2, and 6).

  • Clarification: We revised the description of IGMBench to avoid confusion. In the 110 tasks there are 1.1K state and behavioral criteria, implemented through 2.7K executable checks across Minecraft and Terraria.

Some statistics in the revised appendix may differ from those in our earlier rebuttal responses following updates to the evaluation and regression checks. The revised manuscript reports the latest available analyses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.