- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper identifies the problem of "scoreboard traps" in long-horizon strategy games, a phenomenon where the currently higher-scoring position actually leads to a worse final outcome. The authors build CivHarness, a system that saves game states and replays branched futures to obtain ground-truth long-horizon outcomes, then train CivTelescope, a pairwise LLM judge fine-tuned on replay-labeled data that correctly identifies trap positions with 77.5% accuracy. The authors further use this judge as a dense reward signal to train a FreeCiv gameplay agent via a reverse-curriculum phased schedule, achieving a 77% mid-game improvement over the best sparse-reward baseline.
Critical strengths
- The paper thoroughly investigates the "scoreboard trap" phenomenon, demonstrating its prevalence across eight strategy games and showing that even frontier LLMs perform below chance on these trap pairs
- As promised by the authros, the open-source project CivHarness will potentially provide valuable support for future research on long-horizon agent training
Critical weaknesses
The primary concern is the limited technical novelty. While the engineering is solid, the methodological contribution feels incremental. The core concepts of using future rollouts (Monte Carlo evaluation) to estimate state value, distilling them into a learned reward model, and using reverse curricula are well-established in reinforcement learning and game AI. In the meanwhile, The approach is highly tailored to Freeciv's specific mechanics, it's generalization to other domains is unknown.
Questions for authors
- Compared to distilling replay outcomes into a separate model, how does performance compare if you simply run a small number of actual replay rollouts via CivHarness directly during training to compute a Monte Carlo value estimate as the dense reward? What are the computational trade-offs?
- How well does your method generalize to other domains?
5 comments
Update on W2 + Q2: generalization beyond Freeciv
The Othello result that we promised is now complete. With the same judge-reward pipeline as in Freeciv, the judge reward beats Scoreboard on all three seeds in both Settlers of Catan and Othello.
Terminal score of the trained policy (higher is better):
| Game | Reward | Seed 1 | Seed 2 | Seed 3 | Mean |
|---|---|---|---|---|---|
| Settlers of Catan | Scoreboard | 6.53 | 8.01 | 7.90 | 7.48 |
| Settlers of Catan | CivTelescope (Ours) | 8.34 | 8.23 | 8.26 | 8.27 |
| Settlers of Catan | Difference | +1.82 | +0.21 | +0.36 | +0.80 |
| Othello | Scoreboard | -9.24 | -8.76 | -6.43 | -8.14 |
| Othello | CivTelescope (Ours) | -3.74 | +1.81 | -3.74 | -1.89 |
| Othello | Difference | +5.50 | +10.57 | +2.68 | +6.25 |
Catan is scored in victory points and Othello in final disc difference, each averaged over 45 held-out starting positions with eight games each. Each reward trains the same base model with three seeds and the same training budget. The Othello base model scores -8.38, so the Scoreboard reward barely moves it (-8.14), while the judge reward lifts it to -1.89. For the judge reward we chose the learning rate on separate validation positions; Scoreboard keeps its default configuration.
[update] Released checkpoints
We release the checkpoints behind these results through an anonymous Hugging Face account: https://huggingface.co/Edward11451/CivMarshmallow-ckpts
| Folder | Policy |
|---|---|
| sparse-to-civtelescope | Sparse→CivTelescope hybrid, full game (Table 3) |
| civtelescope | CivTelescope reward alone, full game (Table 3) |
| scoreboard | Scoreboard, full game (Table 3) |
| sparse | Sparse, full game (Table 3) |
| scoreboard-to-civtelescope_from-scoreboard-rem80 | Hybrid from a Scoreboard rem80 checkpoint |
Thank you for your review. We appreciate your questions on novelty and generality. However, we believe that these concerns come from misunderstandings of what the paper claims and already shows, so please allow us to clarify them first. With limited compute, the experiment described below is still running, and we will report back in a few days. The code, including the trap pipeline for the other games, is available in anonymized form at https://anonymous.4open.science/r/CivMarshmallow/.
W1: limited technical novelty
The paper does not claim Monte Carlo evaluation, value distillation or reverse curricula as its contribution. It uses them as tools and cites them (Section 5.3). Its contribution is to show that the standard distillation fails, and what works instead.
A regression value head trained on the same replay outcomes “is at chance on held-out trap pairs” (0.520, against 0.688 for CivTelescope; Section 6.2). A judge distilled from a frontier model's preferences reaches only 0.391, “whereas the replay-labeled judge reaches 0.610” (Section 6.5). The paper also identifies the problem itself: scoreboard traps occur “in 8–68% of position pairs across eight strategy games” (Abstract), and five frontier models are below chance on them.
W2 + Q2: tailored to Freeciv; generalization to other domains unknown
No part of the method uses Freeciv's rules. It needs only a simulator that can save a position and play it to the end, and a text description of the position. The paper already applies the trap pipeline to seven other games (Appendix A.2), and the judge transfers to an unseen ruleset: 0.592 on trap pairs, “against 0.241 for its backbone” (Section 6.2).
We have now run the full method on Settlers of Catan and Othello. Preliminary results: the trained judge escapes scoreboard traps in both games, and in Catan policy training the judge reward is ahead of Scoreboard on all three seeds. We are running the same policy training on Othello.
Trap accuracy of the judge:
Catan policy training, terminal score in victory points:
Q1: Monte Carlo rollouts directly as the reward; compute trade-offs
Direct Monte Carlo values are too slow for a dense reward, because every decision needs its own rollouts. A full-game iteration has about 7,600 decisions. With K = 8 branches each, as in our replay labels, this adds about 61,000 branches, or about 2.3 hours to an iteration that now takes 23 minutes. The judge pays this cost once, offline.
Fewer branches are cheaper but too noisy. Even at K = 8, the paper keeps only pairs whose scores “differ more than twice their pooled error” (Section 5.1).