- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies how two AI agents: one that edits game code and another that plays the game can learn from failures. Its main idea is to determine whether a failure comes from the game, the player, or both before deciding what to change. The system saves useful experience only after checking that the change works and does not damage previous abilities. The paper introduces a benchmark and reports improvements in task completion and fewer updates to the wrong component.
Critical strengths
-
The paper addresses a meaningful problem. A failed game attempt does not necessarily mean the game is broken. Distinguishing a software problem from a player mistake helps avoid unnecessary or harmful changes.
-
The system checks more than whether a fix solves the current task. It also checks whether the improvement works on other tasks and preserves previously successful behavior. This is a sensible way to prevent learning an unreliable shortcut.
-
Testing agents with partners from different generations is a thoughtful experiment. It helps determine whether each agent becomes more capable or simply gets better at working with one particular teammate.
-
The reported reduction in wrong-target updates supports the main idea. In Table 5, this rate drops from 25.0% without CARE to 8.3% with CARE, while more previously successful behavior is retained.
Critical weaknesses
-
The paper does not explain the central decision clearly enough. It lists the evidence CARE examines, but gives limited detail about how that evidence leads to a particular decision. A worked example showing how the system rules out competing explanations would make the method much easier to understand and reproduce.
-
The training comparison does not clearly separate better training content from more training data. The full model receives additional joint-task examples, but dataset sizes and an equal-data comparison are not provided. Some improvement could therefore come from receiving more relevant practice.
-
Several important performance differences represent only two additional successful tasks. For example, Table 4 reports 23.3% versus 20.0% on 60 joint tasks: 14 versus 12 successes. Table 5’s comparison with and without CARE similarly gives 19 versus 17 successes. Repeated runs are needed to show whether these differences are reliable.
-
The separation between training, update verification, and final testing is insufficiently explained. The system uses transfer tasks to decide which experiences to keep, then reports performance on held-out tasks. Readers need a clearer description of how these groups are separated to assess whether the improvements extend to genuinely unseen problems.
Questions for authors
- Can the authors show one complete example of how CARE distinguishes a game bug from a player mistake?
- How many training examples does each variant receive, and does the advantage remain with equal training budgets?
- Do the main improvements hold across repeated runs?
- Are the tasks used to approve updates separate from the final evaluation tasks, including their underlying game families?