OmniGameArena: A Unified UE5 Benchmark for VLM Game Agents with Improvement Dynamics

Most game benchmarks score a foundation model the first time it meets a game, and rarely measure what the model can make of the game once it has played it and reflected on its play. We measure a model at three levels, a cold-start score at its first contact with a game, an Improvement Dynamics Curve (IDC) of its score after every round of reflection, and a transfer test that re-runs the skill it learned through reflection on variants of the game. These levels need games that no model has seen and that ship with variants of themselves. We built OmniGameArena, a single Unreal Engine 5 build of twelve such games played from the rendered frame, covering solo, adversarial and cooperative play, each with four variants that alter it in one respect. Across 72 curves of six VLMs, reflection raises the best round more than 0.1 above the cold-start score in 42, yet in half of these the first round shows at most 0.05 of the gain and in a quarter the last round ends more than 0.1 below the best. The learned skills gain 0.19 on average on the game they were written from and 0.03 on its variants, and the size of the gain on the practiced game explains about a tenth of what carries over. Neither pattern is visible in a single score, whether it is taken at first contact or after one round of reflection.