GameMod: Evaluating Software Compliance through Game Modding

Rapid advances have brought coding agents into everyday development workflows, yet they sometimes reinvent existing components and overlook compatibility with established code, undermining maintainability and robustness. Game modding is well suited to evaluating this gap: the game runtime provides comprehensive and strict compliance verification of the implemented mod, and the compliance complexity can be further increased by requiring the mod to remain compatible with other independently developed mods. To this end, we introduce GAMEMOD, a benchmark of 150 game-mod construction and repair tasks spanning 12 games for evaluating coding agents under these constraints. Our analysis shows that agents can break existing functionality when rewriting components, and that their fixes for reported bugs can cause previously compatible mods to stop working together. Moreover, even when an implemented mod passes the agent’s own tests, it can still fail in the actual game, showing that the real runtime provides more rigorous compliance verification. By exposing these limitations, GAMEMOD provides concrete targets for improving generalist coding agents’ ability to extend existing software while preserving compatibility, maintainability, and robustness.