acceptodds
Under review as a conference paper at ICLR 2027

Rome Wasn’t Built in One Turn: Replay-Grounded Reward Signals for Long-Horizon Freeciv Agents

Abstract

Training language-model agents to play long-horizon strategy games is hard because deterministic feedback is sparse and often arrives only at the end of a game. In-game scores can offer dense signals, but they do not reliably predict future outcomes: in 8–68% of position pairs across eight strategy games, the higher-scoring position leads to the worse outcome. We term this phenomenon scoreboard traps. In Freeciv, an open-source turn-based empire-building game where several players manage units, cities, and diplomacy over hundreds of turns, we look into better reward signals for gameplay agents. To facilitate this, we build CivHarness, an open system that saves current game states, branches on different decisions, and replays the game to future turns to provide measurable long-horizon feedback from saved positions. We use CivHarness to train CivTelescope, an LLM Judge that predicts which one of two game positions will yield the better outcome. On scoreboard-trap pairs, CivTelescope is 1.3–2.0 times more accurate than off-the-shelf language models, and it generalizes to new game starts, later turns, and unseen rules. We then train a Freeciv gameplay agent with CivTelescope as a dense reward, and call this recipe CivMarshmallow. When used in the middle stage of the game where scoreboard traps are common, CivMarshmallow reaches an average training phase gain of 8.94 points, 77% more than the best baseline, the end-of-game reward. Over the full 120-turn game, CivMarshmallow uses a hybrid reward that starts from the scoreboard or the end-of-game reward and shifts to CivTelescope as the game advances; this hybrid reward scores highest among the rewards we compare.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Model Agents, Long-Horizon Reinforcement Learning, Reward Shaping, LLM-as-a-Judge, Replay-Grounded Supervision, Strategic Games

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_KQ2YOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper identifies the problem of "scoreboard traps" in long-horizon strategy games, a phenomenon where the currently higher-scoring position actually leads to a worse final outcome. The authors build CivHarness, a system that saves game states and replays branched futures to obtain ground-truth long-horizon outcomes, then train CivTelescope, a pairwise LLM judge fine-tuned on replay-labeled data that correctly identifies trap positions with 77.5% accuracy. The authors further use this judge as a dense reward signal to train a FreeCiv gameplay agent via a reverse-curriculum phased schedule, achieving a 77% mid-game improvement over the best sparse-reward baseline.

Critical strengths
  • The paper thoroughly investigates the "scoreboard trap" phenomenon, demonstrating its prevalence across eight strategy games and showing that even frontier LLMs perform below chance on these trap pairs
  • As promised by the authros, the open-source project CivHarness will potentially provide valuable support for future research on long-horizon agent training
Critical weaknesses

The primary concern is the limited technical novelty. While the engineering is solid, the methodological contribution feels incremental. The core concepts of using future rollouts (Monte Carlo evaluation) to estimate state value, distilling them into a learned reward model, and using reverse curricula are well-established in reinforcement learning and game AI. In the meanwhile, The approach is highly tailored to Freeciv's specific mechanics, it's generalization to other domains is unknown.

Questions for authors
  • Compared to distilling replay outcomes into a separate model, how does performance compare if you simply run a small number of actual replay rollouts via CivHarness directly during training to compute a Monte Carlo value estimate as the dense reward? What are the computational trade-offs?
  • How well does your method generalize to other domains?
5 comments

Thank you for your review. We appreciate your questions on novelty and generality. However, we believe that these concerns come from misunderstandings of what the paper claims and already shows, so please allow us to clarify them first. With limited compute, the experiment described below is still running, and we will report back in a few days. The code, including the trap pipeline for the other games, is available in anonymized form at https://anonymous.4open.science/r/CivMarshmallow/.


W1: limited technical novelty

The paper does not claim Monte Carlo evaluation, value distillation or reverse curricula as its contribution. It uses them as tools and cites them (Section 5.3). Its contribution is to show that the standard distillation fails, and what works instead.

A regression value head trained on the same replay outcomes “is at chance on held-out trap pairs” (0.520, against 0.688 for CivTelescope; Section 6.2). A judge distilled from a frontier model's preferences reaches only 0.391, “whereas the replay-labeled judge reaches 0.610” (Section 6.5). The paper also identifies the problem itself: scoreboard traps occur “in 8–68% of position pairs across eight strategy games” (Abstract), and five frontier models are below chance on them.


W2 + Q2: tailored to Freeciv; generalization to other domains unknown

No part of the method uses Freeciv's rules. It needs only a simulator that can save a position and play it to the end, and a text description of the position. The paper already applies the trap pipeline to seven other games (Appendix A.2), and the judge transfers to an unseen ruleset: 0.592 on trap pairs, “against 0.241 for its backbone” (Section 6.2).

We have now run the full method on Settlers of Catan and Othello. Preliminary results: the trained judge escapes scoreboard traps in both games, and in Catan policy training the judge reward is ahead of Scoreboard on all three seeds. We are running the same policy training on Othello.

Trap accuracy of the judge:

GameUntrained backboneTrained judge
Settlers of Catan22.5%87.1%
Othello36.5%81.8%

Catan policy training, terminal score in victory points:

RewardSeed 43Seed 44Seed 45Mean
Scoreboard6.538.017.907.48
CivTelescope (Ours)8.348.238.268.27
Difference+1.82+0.21+0.36+0.80

Q1: Monte Carlo rollouts directly as the reward; compute trade-offs

Direct Monte Carlo values are too slow for a dense reward, because every decision needs its own rollouts. A full-game iteration has about 7,600 decisions. With K = 8 branches each, as in our replay labels, this adds about 61,000 branches, or about 2.3 hours to an iteration that now takes 23 minutes. The judge pays this cost once, offline.

Fewer branches are cheaper but too noisy. Even at K = 8, the paper keeps only pairs whose scores “differ more than twice their pooled error” (Section 5.1).

Update on W2 + Q2: generalization beyond Freeciv

The Othello result that we promised is now complete. With the same judge-reward pipeline as in Freeciv, the judge reward beats Scoreboard on all three seeds in both Settlers of Catan and Othello.

Terminal score of the trained policy (higher is better):

GameRewardSeed 1Seed 2Seed 3Mean
Settlers of CatanScoreboard6.538.017.907.48
Settlers of CatanCivTelescope (Ours)8.348.238.268.27
Settlers of CatanDifference+1.82+0.21+0.36+0.80
OthelloScoreboard-9.24-8.76-6.43-8.14
OthelloCivTelescope (Ours)-3.74+1.81-3.74-1.89
OthelloDifference+5.50+10.57+2.68+6.25

Catan is scored in victory points and Othello in final disc difference, each averaged over 45 held-out starting positions with eight games each. Each reward trains the same base model with three seeds and the same training budget. The Othello base model scores -8.38, so the Scoreboard reward barely moves it (-8.14), while the judge reward lifts it to -1.89. For the judge reward we chose the learning rate on separate validation positions; Scoreboard keeps its default configuration.

[update] Released checkpoints

We release the checkpoints behind these results through an anonymous Hugging Face account: https://huggingface.co/Edward11451/CivMarshmallow-ckpts

FolderPolicy
sparse-to-civtelescopeSparse→CivTelescope hybrid, full game (Table 3)
civtelescopeCivTelescope reward alone, full game (Table 3)
scoreboardScoreboard, full game (Table 3)
sparseSparse, full game (Table 3)
scoreboard-to-civtelescope_from-scoreboard-rem80Hybrid from a Scoreboard rem80 checkpoint
Reviewer_sCHtOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the mismatch between current game scores and future outcomes in long-horizon games. It introduces a replay harness for Freeciv, trains a pairwise judge from replay outcomes, and uses the judge as a dense signal for policy training. The experiments show improved recognition of scoreboard traps and study when judge-based rewards help during a reverse curriculum.

Critical strengths
  1. The replay harness and the analysis of scoreboard traps are useful. The paper checks both cross-game comparisons and rollouts from the same starting position, and provides detailed replay and evaluation protocols.
  2. The judge evaluation includes meaningful controls, including masked scores, alternative label sources, a regression value head, and transfer to another ruleset. These help establish that the judge learns more than the visible score heuristic.
Critical weaknesses
  1. My concern: How much of the policy-training gain comes from the reward signal itself. Phase gain depends on the checkpoint entering a phase. At rem120, hybrid and pure-judge methods share the judge-rem80 checkpoint, while Sparse and Scoreboard start from their own checkpoints. Comparisons across these reward families do not fully isolate the reward effect. The best full-game score is 23.35, versus 23.29 for Scoreboard and 23.21 for Sparse (Table 3). The practical magnitude and robustness of this final-score improvement remain uncertain.
  2. The regression value head is evaluated as a judge, not as a reward in the same policy-training pipeline. The terminal-only control also changes initialization and reference-set updates. Thus, the extra training benefit of pairwise calibration and dense intermediate feedback is not yet isolated.
Questions for authors
  1. Can you compare reward families from matched checkpoints at rem80 and rem120, and report final scores with uncertainty across training seeds?
  2. How does a regression value model trained on the same replay data perform in the same policy-training pipeline under comparable rollout and compute budgets?
  3. Can you repeat the terminal-only comparison with matched initialization and reference-set updates, including λ=1 in Eq. 4 while retaining the true terminal reward?
5 comments

Thank you for your review and for recognizing the value of this work. Your three questions are clear, and we are running all three experiments now. Our compute is limited, so they need a few more days, and we will report back as soon as they finish. We have also made our code available, anonymized, at https://anonymous.4open.science/r/CivMarshmallow/.


W1 + Q1: phase gains depend on the entry checkpoint; the full-game margin is small

The paper already has one comparison from the same checkpoint: “Only pure CivTelescope starts from the same checkpoint as the hybrids, and both hybrids beat it in all three seeds, by 2.4 points on average” (Section 6.4). Each full-game score in Table 3 is the mean over 45 held-out starting positions with eight games each.

We are running the matched comparisons that you ask for: our method from the same checkpoint as Sparse at rem80 and as Scoreboard at rem120. We will report the final score of every training seed with its standard error.


W2 (part 1) + Q2: the regression value head is not used as a reward

We have now run this experiment. Preliminary result (phase gain at rem80): with the regression value head as the reward in the same pipeline (same starting checkpoint and rollout budget), CivTelescope is ahead on all three seeds.

RewardSeed 43Seed 44Seed 45Mean
Regression value head4.075.6110.546.74
CivTelescope (Ours)4.4810.3814.259.70
Difference+0.40+4.77+3.70+2.96

This agrees with the paper's offline result, where the value head “is at chance on held-out trap pairs” (0.520, against 0.688 for CivTelescope; Section 6.2). On compute: both scorers are 7B models; the value head is queried once per position and the judge four times (M = 4, one order).


W2 (part 2) + Q3: the terminal-only control is not matched

We are running exactly this control: λ = 1 in Equation 4 with the true terminal reward, with matched initialization and reference-set updates. With λ = 1, “the potential serves only as a baseline” (Section 5.3), so the control removes only the intermediate feedback.

Thanks for the rebuttal.

Two follow-up questions on the value head result: fq1. The value head regresses directly onto final returns, which is exactly what a value function is trained to do, yet it scores 0.520 on held-out trap pairs, which is essentially random. Can the authors walk through why this happens? For example, does the model mostly latch onto the current scoreboard signal rather than learning anything about future trajectory quality?

fq2. Reinforcement learning has a lot of mature machinery for credit assignment and value learning, like GAE, prioritized replay, better sampling of critical states, PPO-style objectives. Given that, why does the value head remain this weak? Have you tried any of these, or a ranking/pairwise loss within the same turn (similar in spirit to CivTelescope's training target) to see if the value head can close the gap?

I'm mainly trying to understand whether the gap is fundamental to absolute-score regression in this setting, or just a matter of training choices that haven't been explored yet.

AuthorsOpenReview ↗

Re: Follow-up quetsions 1 2 from Reviewer sCHt

Thank you for your reply, your follow-up questions, and for recognizing our work. Based on your questions, we have run an additional experiment (Follow-up 2), and we are happy to answer any further questions.


Follow-up 1: why the value head is at chance on held-out trap pairs

The value head does not follow the scoreboard; it memorizes its training games. A scoreboard reader always picks the position with the higher current score; the head picks it only about half the time and is at chance on both trap and non-trap pairs (0.520 and 0.548; Appendix D.3). It fits its training positions but not new games (r = 0.77 against 0.16; 0.93 against 0.01 with longer training), because 90% of the target's variance lies between games: recognizing the game is enough to fit the training set.

Both replay training and reading the two positions together are needed. The head shares “CivTelescope’s base, adapter, and recipe” (Appendix D.3). On the same 200 held-out trap pairs:

ModelTrained on replay outcomesReads both positions togetherTrap accuracy
Zero-shot backboneNoYes0.219
Regression value headYesNo0.520
CivTelescope (Ours)YesYes0.688

Follow-up 2: a ranking loss within the same turn does not close the gap

We ran the within-turn ranking loss you suggest; it does not close the gap. Same head, data and recipe; each batch holds four positions of the same turn from different games, and each pair with clearly different replay outcomes (the rule for CivTelescope's training pairs, Section 5.2) adds a pairwise logistic loss. On the same held-out pairs:

ModelLossReads both positions togetherTrap pairsNon-trap pairs
Regression value headRegressionNo0.5200.548
Ranking value headWithin-turn rankingNo0.3800.650
CivTelescope (Ours)Pairwise choiceYes0.6880.777

The ranking head learns its training pairs (83% right) but falls back on the current score in new games: on held-out trap pairs it picks the higher current score 63% of the time. So the gap does not come from the regression loss: with either loss, a head that scores one position at a time stays well below CivTelescope (the 95% confidence intervals do not overlap).

The other tools you list do not address this. GAE and PPO-style objectives reduce the noise and instability of online RL targets, and prioritized replay and critical-state sampling reweight the training positions. Our head is trained offline on fixed targets, each the mean of eight replays, and it already fits its training positions; it fails only on new games. GAE is also already part of our pipeline (Equation 4).

Reviewer_ct2DOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes to train an LLM to act as a preference/value function in a long-horizon strategy game (Freeciv), and then train an agent with this as dense reward, instead of binary outcome or game-provided scoreboards per turn. This targets a common trap in long-horizon games, where a myopic short-term reward might look good but turns out bad in the long run.

Critical strengths
  1. The analysis of leaderboard traps, and showing that they exist in several games (and how often and where) is a very welcome motivation to the problem
  2. Advantage estimation in long-horizon games is an important topic.
Critical weaknesses

In the order from strongest to least important:

  1. The primary weakness is scope. The paper considers a single benchmark, with a single model. This makes it seem more like a workshop than a full conference paper.
  2. The method does not work better than standard RL. As per Table 3, using the method as reward leads to less score than using either terminal reward or the scoreboard as reward, and only when mixing with either of them it matches (but still does not outperform by a larger margin) in all reported metrics. The results in Table 2 do not look like a significant difference either, given the heavy spread between runs.
  3. It looks like many RL runs crashed below random performance. Though training curves are not given (which is another weakness), the seeds reported in Table 2 look very instable.
  4. The paper does not compare to any RL-training methods from literature. It does compare to prompting baselines on various models, but the only RL train runs are basic RL with terminal or scoreboard rewards. The literature on GAE in long-horizon games is left unaccounted for.
  5. I cannot follow the math on how the rewards are built in Section 5.3. Partly because the paper does not stick to standard RL terminology (value, q value, advantage, ...), partly because things are not defined (what are “positions rj from nearby game turns whose terminal scores Y(rj) are known”, what is a nearby state and why should terminal scores ever not be known? Because the game did not finish within the step budget?), and party because the math just does not make sense to me, like Formula (2). I would really have to guess together what is happening here, which reduces reproducibility, readability, and defeats the purpose behind mathematical formalization.
  6. The environment / POMDP setup is not described. For that, consider repurposing Section 4. Currently, it is just trying to sell the existence of an RL env around FreeCiv as a contribution, without really giving RL-relevant details (details like that you are batching seem very standard). I would argue it is implicit that you must have built an RL environment to train in, and it is nice that you plan to open-source it, but the actual scientific question is how exactly it looks to the agent. I would encourage giving a brief technical overview of how this wrapper turns FreeCiv into something Qwen can digest: This is not described anywhere in the paper (neither in the appendix), so I’m unsure what the task exactly is. How are prompts built? What is the observation space / what does the agent see and in which format? What is the action space? How many actions are there to choose from? What is the intended output?
  7. Some of the writing, though clearly not all of it, feels like AI slop. For example, the last paragraph in the introduction slowly starts creeping in undefined terms “phase gain”, or unexplained sentences “which keeps the early-game signal stable“. Another example is Appendix B, none of the terms there are defined. Undefined terms, missing definitions, etc, lower both readability and reproducibility of the paper.

Smaller weaknesses that don’t influence my score:

  • Consider renaming “CivTelescope” and “CivMarshmallow”. If you’re presenting at the conference and people see this at the poster, they will lack the context to know what they mean. Maybe stick to more common terms like “CivValuejudge” and “CivAgent”.
  • Cite slime.
Questions for authors

What do you mean by “to keep the early-game signal stable”?

In Section 5.2, you say that you execute the value head twice, in either order of the two game states, because the model is biased by the order. Shouldn’t this go away with training? It effectively increases the compute by 2x.

8 comments

Thank you for your constructive comments. However, we sincerely believe that our work deserves a better evaluation: several of the main concerns rest on misunderstandings of the paper, which we correct below. Because our compute is limited, the experiments described below are still running, and we will report the results in a few days. We kindly ask you to reconsider your score. An anonymized copy of our code, including CivHarness, is at https://anonymous.4open.science/r/CivMarshmallow/.


W1: one benchmark, one model

We respectfully disagree that this scope is "more like a workshop". A single complex game is a standard scope at top venues: CivRealm [1] (ICLR 2024), CivBench [2] (COLM 2026) and CivBench [3] (NeurIPS 2026) each study one Civilization game, and Motif [4] (ICLR 2024) studies NetHack alone. Our paper cites all four. Civilization is not a simple game: CivRealm reports that both RL and LLM agents struggle to make substantial progress in the full game. Unlike Minecraft-style tasks, where achievements mark the way, Freeciv gives the agent no sub-goals, and its only running signal, the scoreboard, misleads in 26% of position pairs (Section 3).

We acknowledge that our main policy training focuses on Freeciv, so we are extending it to two more games. Preliminary results: the trained judge escapes scoreboard traps in both games, and in Catan policy training the judge reward is ahead of Scoreboard on all three seeds. Running: the same policy training on Othello.

Trap accuracy of the judge:

GameUntrained backboneTrained judge
Settlers of Catan22.5%87.1%
Othello36.5%81.8%

Catan policy training, terminal score in victory points:

RewardSeed 43Seed 44Seed 45Mean
Scoreboard6.538.017.907.48
CivTelescope (Ours)8.348.238.268.27
Difference+1.82+0.21+0.36+0.80

Beyond policy training, the paper already measures scoreboard traps “across eight games” (Section 3) and compares the judge with five frontier models (Table 1 of the paper).


W2: the method does not beat standard RL; Table 2 has a large spread

This reading of Table 3 is incorrect: the 21.30 row is the judge reward alone, not our full-game method. The paper states that for full games “CivMarshmallow combines the CivTelescope reward with a baseline reward” (Section 5.3). The judge alone is expected to score lower, because early in the game “CivTelescope cannot yet tell positions apart” (Section 6.4). Our method, the hybrid, has the highest score in Table 3 (23.35), measured on 45 held-out starting positions with eight games each. In order to clarify which rows of Table 3 are our full-game method, we have added this to its caption with blue text: “[For full games CivMarshmallow uses the hybrid rewards (Section 5.3). The CivTelescope row is the judge reward alone.]” Running: more training seeds from the same checkpoint as the baselines, with every seed and its standard error reported.

On Table 2: at rem80, where scoreboard traps are common, the judge reward has the highest phase gain (8.94, against 5.04 for Sparse and 0.83 for Scoreboard). The spread between seeds does not hide this: the paired difference over the 45 evaluation positions (mean ± standard error) is positive on every seed.

CivTelescope minusSeed 43Seed 44Seed 45
Sparse+0.69 ± 1.03+6.89 ± 0.61+4.14 ± 0.85
Scoreboard+5.41 ± 0.70+8.45 ± 0.67+10.49 ± 0.63

W3: RL runs seem to crash below random; no training curves

This statement is factually incorrect: no run crashed, and every trained policy is far above the untrained model. The negative numbers in Table 2 are phase gains, not scores: the phase gain “is the change in S over a training phase” (Section 6.1), so −0.4 means that a phase ended 0.4 points below where it started. The scores are in Table 3: every trained policy scores 21.30–23.35, about three times the untrained model (7.38), and “Every trained policy outscores every prompting agent” (Section 6.4). In order to clarify that the numbers in Table 2 are changes in score, not scores, we have added this to its caption with blue text: “[G is a change in terminal score within one phase, not a score. A negative value means that the policy ended the phase slightly below where it started.]”


References

[1] Qi et al. CivRealm: A Learning and Reasoning Odyssey in Civilization for Decision-Making Agents. ICLR 2024.

[2] Chen et al. CivBench: Progress-Based Evaluation for LLMs' Strategic Decision-Making in Civilization V. COLM 2026.

[3] Andrews et al. CivBench: A Long-Horizon Benchmark for Tool-Mediated Agents in Civilization VI. NeurIPS 2026.

[4] Klissarov et al. Motif: Intrinsic Motivation from Artificial Intelligence Feedback. ICLR 2024.

W4: no RL methods from the literature; GAE work not accounted for

This statement is factually incorrect: both baselines are RL methods from the literature, and our reward is itself GAE. Sparse is GRPO [5]: its advantage “normalizes the terminal score across the rollouts from the same starting position, as in GRPO” (Section 5.3). The CivTelescope reward uses “generalized advantage estimation” [6, 7] (Section 5.3). In order to clarify that Scoreboard uses the same estimator, we have added this to Section 5.3 with blue text: “[It uses the same advantage estimator as the CivTelescope reward (Equation 4), with the current score as the potential.]”

We have also tested three dense rewards from the literature in the same pipeline. Preliminary results: CivTelescope has the highest mean phase gain of the four, 9.70, against 6.74 for a regression value head, 7.59 for a reward model distilled from LLM preferences (as in Motif [4]) and 9.05 for a zero-shot LLM judge. We will also extend the related work on GAE in long-horizon games.


W5: the math in Section 5.3 is hard to follow

Section 5.3 is standard RL: GAE with a learned state value. Φ is the state value and AtA_t is the advantage. In order to clarify the terms, we have added these to Section 5.3 with blue text: “[The potential ΦB(s)\Phi_{\mathcal{B}}(s) is a state value: it estimates the terminal score expected from s. The reference positions come from finished games, at the sampled game turn nearest to s, so the terminal score of each is known.]” and “[Equation 4 is generalized advantage estimation with γ = 1, state value Φ, and the terminal score R as the only reward. AtA_t is the advantage of the decision taken at sts_t.]” The terminal scores of the references are known because their games are finished. No step budget is involved.

Equation 2 turns the judge's comparisons into a score. The judge compares s with each of the M reference positions. The mean of its M answers is the share of the references that s is expected to beat. Φ(s) is the terminal score found at that share in the sorted reference scores. In order to clarify this, we have added a worked example to Section 5.3 with blue text: “[For example, if four references end at 10, 20, 30, and 40 points and the judge prefers s to them with probabilities 0.9, 0.8, 0.3, and 0.1, the mean is w = 0.525 and ΦB(s)\Phi_{\mathcal{B}}(s) = 25.75.]”


W6: the environment / POMDP is not described

In order to clarify what the task looks like to the agent, we have added the full description as the new Appendix B.2 (Agent interface) with blue text. We have also added a pointer to it in Section 4: “[Appendix B.2 describes the task as the agent sees it: the observation, the action menu, and the output format.]” In short, the observation is one text prompt per decision, with the task description, the last five decisions, the game state as compact JSON and the action menu (about 1,560 tokens). The action space is a menu of at most 96 legal actions in six domains: technology, government, diplomacy, cities, civilian units and military units. The output is one JSON object that selects at most four actions. The agent makes one decision per game turn, 119 in a full game, and “sees only its own player's view of the game, not the full game state” (Section 4).

CivHarness is also more than an RL wrapper. Unlike earlier Freeciv environments, it “branches and replays any saved position” (Section 2), and every replay label in the paper rests on this. An anonymized copy of the code, including CivHarness, is available at https://anonymous.4open.science/r/CivMarshmallow/.


W7: writing reads machine-generated; terms used before definition

We disagree with this description of the writing. Both terms are defined in the paper: the phase gain “is the change in S over a training phase” (Section 6.1), and the early-game sentence refers to Section 6.4, where early in the game “CivTelescope cannot yet tell positions apart”. To make both terms clear where they first appear, we have added these to the introduction with blue text: “[The phase gain is the change in the agent's held-out terminal score over one training phase (Section 6.1).]” and “[The hybrid starts from a baseline reward because CivTelescope cannot yet tell positions apart early in the game (Section 6.4).]” We will also define each acceptance test of Appendix B.1 in one sentence.


References

[4] Klissarov et al. Motif: Intrinsic Motivation from Artificial Intelligence Feedback. ICLR 2024.

[5] Shao et al. DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models. arXiv:2402.03300, 2024.

[6] Schulman et al. High-Dimensional Continuous Control Using Generalized Advantage Estimation. ICLR 2016.

[7] Ng et al. Policy Invariance Under Reward Transformations: Theory and Application to Reward Shaping. ICML 1999.

Minor 1: rename CivTelescope and CivMarshmallow

Thank you for the suggestion. We keep the names: the code, checkpoints and dataset use them, and the paper defines both where they first appear. CivTelescope is “a pairwise language-model judge” (Section 1), and the training recipe that uses it as a reward is CivMarshmallow, named after “the marshmallow test of delayed gratification” (Section 1, footnote). We think this metaphor is apt: CivTelescope “rewards a position for an outcome that arrives many turns later, even when the position gives up score now” (Section 1, footnote).


Minor 2: cite slime

Done. We have added the citation of slime [8] to Section 4 with blue text.


Q1: what does "keeps the early-game signal stable" mean?

It means that the hybrid uses the game score, not the judge, early in the game. The paper gives the reason in Section 6.4: “CivTelescope cannot yet tell positions apart: in turns 10–29 its potential has a rank correlation of 0.09–0.10 with the terminal score, against 0.34–0.36 for the scoreboard”. To explain this phrase where it appears, we have added this to the introduction with blue text: “[The hybrid starts from a baseline reward because CivTelescope cannot yet tell positions apart early in the game (Section 6.4).]”


Q2: the judge is run in both orders; shouldn't position bias vanish with training? 2× compute

Training does not pay the 2× cost. During training “the judge is queried in one presentation order” (Appendix C.1). Both orders are used only when we measure the judge's accuracy (Section 6.1). To make this clear, we have added this to Section 5.2 with blue text: “[The two-order query applies when we measure the judge's accuracy (Section 6.1). During policy training the judge is queried in one order only (Appendix C.1), so the reward adds no second query.]”

Yes: after training, the judge no longer prefers a position. The judge is trained on both orders of every pair. In all three games, the trained judge picks the first-listed position in about half of its answers. Its answer still changes with the order on some pairs (17.5% in Freeciv), which is why our accuracy numbers average both orders.

GameJudgeFirst-position picksSame answer in both orders
FreecivUntrained backbone46.1%80.0%
FreecivCivTelescope (Ours)51.1%82.5%
Settlers of CatanUntrained backbone37.4%66.0%
Settlers of CatanTrained judge49.1%92.5%
OthelloUntrained backbone19.7%39.3%
OthelloTrained judge51.5%92.3%

References

[8] Zhu et al. slime: An LLM Post-Training Framework for RL Scaling. GitHub repository, 2025.

Reviewer_fnW5OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces "scoreboard traps". In scoreboard traps, positions with higher current scores lead to worse future outcomes.

It has three parts. CivHarness is used to save, branch and replay Freeciv states. CivTelesope is a pairwise judge trained on replay outcomes. CivMarshmallow is a training recipe that use the judge's calibrated state values for dense feedback.

The experiments include trap measurement across eight games, judge evaluation, transfer to an unseen Freeciv ruleset, and policy training across three seeds.

Critical strengths
  1. The replay supervision results are convincing. CivTelescope gets 77.5% accuracy on held-out trap pairs and 87.8% on non-trap pairs, suggesting that it does more than reverse the scoreboard ranking. The label-source ablation gives a more direct comparison that with the same training setup, replay labels work substantially better than the tested model’s preference labels.

  2. The paper includes useful checks on where scoreboard traps come from and whether the judge helps training. Traps persist when comparing positions from the same start at the same turn, so differences between maps cannot fully explain them. At rem80, CivTelescope also outperforms both sparse and scoreboard rewards across all three seeds. This supports its usefulness for policy training in the middle game, though the gains are less consistent in other phases.

Critical weaknesses
  1. The evidence for a meaningful full-game advantage is limited. In Table 3, the best hybrid scores 23.35, compared with 23.29 for SCOREBOARD and 23.21 for SPARSE. The result has no uncertainty estimates for these differences. The pure judge scores only 21.30.

  2. The rem120 hybrids and baseline rewards inherit different checkpoints, so their phase gains do not cleanly isolate the effect of changing the reward. The middle-game result is encouraging, but the broader practical benefit remains unclear.

  3. The mechanism behind the improvement is not sufficiently isolated. The regression value head is evaluated offline but is not tested as a critic in the same policy-training setup. This leaves open whether the pairwise judge and reference calibration improve on a simpler replay-trained value estimator.

  4. The terminal-only control changes more than the intermdiate feedback. The main method retains the true terminal score (R) in Equation 4, whereas the control replaces it with the judge’s prediction. Its weaker performance could therefore reflect errors in the predicted terminal reward, rather than the absence of dense intermediate feedback.

Questions for authors
  1. Do the rem120 hybrid gains persist when SPARSE, SCOREBOARD, and the hybrids all start from the same checkpoint?

  2. For the dense-feedback claim, what happens in a matched comparison while preserving the true terminal reward and reference-update procedure?

  3. How does a replay-trained scalar value estimator perform when used within the same policy-training pipeline? The offline ranking comparison alone does not resolve whether the proposed pairwise construction is necessary.

If the authors can fully respond to the above questions and address these weakness through necessary experiments, I am willing to consider raising the score.

3 comments

Thank you for your careful review. We appreciate the weaknesses and questions that you listed. Due to limited compute, we are still running the experiments detailed below and will report back in a few days. Our code is available in anonymized form at https://anonymous.4open.science/r/CivMarshmallow/.


W1: the full-game advantage is small and has no uncertainty

Each score in Table 3 (full-game result) is the mean over 45 held-out starting positions with eight games each, so the hybrid's lead, though small, is measured on 360 games. To give the uncertainty, we are running more seeds and will report every seed with its standard error.

The pure judge (21.30) is expected to score lower over the full game, because early in the game “CivTelescope cannot yet tell positions apart” (Section 6.4). This is why our full-game method is the hybrid, which beats the pure judge “in all three seeds, by 2.4 points on average” (Section 6.4). In order to clarify which rows of Table 3 are our full-game method, we have added this to its caption with blue text: “[For full games CivMarshmallow uses the hybrid rewards (Section 5.3). The CivTelescope row is the judge reward alone.]”


W2 + Q1: at rem120 the hybrids and the baselines enter from different checkpoints

We are running the comparison that you ask for: the hybrid from the same checkpoint as Scoreboard, the strongest baseline at rem120. We will report every seed with its standard error.


W3 + Q3: the regression value head is only tested offline

The paper already shows that the simpler value estimator fails offline: a regression value head trained on the same replay outcomes “is at chance on held-out trap pairs” (0.520, against 0.688 for CivTelescope; Section 6.2).

We have now also tested it as the reward in the same policy-training pipeline, with the same starting checkpoint and rollout budget. Preliminary result (phase gain at rem80): CivTelescope is ahead on all three seeds.

RewardSeed 43Seed 44Seed 45Mean
Regression value head4.075.6110.546.74
CivTelescope (Ours)4.4810.3814.259.70
Difference+0.40+4.77+3.70+2.96

W4 + Q2: the terminal-only control changes more than the intermediate feedback

The matched control that you describe is Equation 4 with λ = 1. It keeps the true terminal score R, and “the potential serves only as a baseline” (Section 5.3), so it removes only the intermediate feedback.

We are running it with the same starting checkpoint and reference-set updates as our method, and will report every seed.

[update] W1: full-game scores with standard errors

Every score is measured on 360 full games per policy: 45 held-out starting positions, eight games of 120 turns from each. ± is the standard error over the 45 starts.

PolicyTerminal score
Sparse→CivTelescope hybrid (Ours)23.35 ± 0.91
Scoreboard23.29 ± 0.95
Sparse23.21 ± 1.05
CivTelescope alone21.30 ± 1.13

We therefore believe that CivMarshmallow has the advantage. It reaches the highest full-game score of all methods in Table 3, and, unlike Scoreboard, its reward is not misled by scoreboard traps (Section 6.2). The hybrid is also clearly above the judge reward alone (paired over the 45 starts, the 95% confidence interval excludes zero), which is exactly the early-game weakness that our hybrid is built to remove.

We have added these standard errors to Table 3 with blue text.


[update] W2 + Q1: hybrid started from a Scoreboard checkpoint

We additionally trained the model with our method from a Scoreboard rem80 checkpoint. Same evaluation as above: 360 full games per policy (45 held-out starts × 8 games); ± is the standard error over the starts.

PolicyTerminal score
Hybrid from a Scoreboard rem80 checkpoint23.35 ± 0.97
Scoreboard (Table 3)23.29 ± 0.95

It shows that CivMarshmallow can take over from a Scoreboard-trained policy and still reach the top score of Table 3. In this run, switching the reward to CivTelescope for the final phase costs no score, while it replaces a signal that traps can mislead with one that they do not.


[update] W4 + Q2: terminal-only control

We trained the control you asked for: the hybrid with λ = 1 in Equation 4. The control and a hybrid run start from the same Scoreboard rem80 checkpoint, with the same initialization and reference-set updates. We conduct the same evaluation as above: 360 full games per policy (45 held-out starts × 8 games); ± is the standard error over the starts.

PolicyTerminal score
Hybrid trained from this checkpoint22.05 ± 1.00
Starting checkpoint17.20 ± 0.71
Terminal-only control (λ = 1)15.88 ± 0.66

Therefore, we believe that the intermediate feedback from CivTelescope is what drives the gain. With λ = 1, “the potential serves only as a baseline” (Section 5.3), so every decision is credited only through the final score; over the 120 turns of a full game this signal does not improve the policy, and the control even ends below the checkpoint it started from.


[update] Released checkpoints

To prove the reproducibility of our work, we release the checkpoints behind these results through an anonymous Hugging Face account: https://huggingface.co/Edward11451/CivMarshmallow-ckpts

FolderPolicy
sparse-to-civtelescopeSparse→CivTelescope hybrid, full game (Table 3)
civtelescopeCivTelescope reward alone, full game (Table 3)
scoreboardScoreboard, full game (Table 3)
sparseSparse, full game (Table 3)
scoreboard-to-civtelescope_from-scoreboard-rem80Hybrid from a Scoreboard rem80 checkpoint
AuthorsOpenReview ↗

Reminder before the discussion ends: all three experiments you asked for are now reported

Dear Reviewer fnW5,

Thank you again for your review, and for finding our replay-supervision results convincing. You wrote that you would consider raising your score if we answered your three questions with the necessary experiments. We have now run all three, and the results are in our response and our latest update:

  • Same checkpoint at rem120 (Q1): started from a Scoreboard checkpoint, our best run of the hybrid reaches 23.35 ± 0.97, the top score of Table 3.
  • Dense feedback with the true terminal reward (Q2): the terminal-only control (λ = 1, same checkpoint, initialization and reference-set updates) ends at 15.88 ± 0.66, against 22.05 ± 1.00 for the hybrid.
  • Scalar value estimator in the same pipeline (Q3): used as the reward, the regression value head reaches a mean phase gain of 6.74, against 9.70 for CivTelescope.
  • Code and checkpoints: our code is available in anonymized form at https://anonymous.4open.science/r/CivMarshmallow/, and the trained checkpoints at https://huggingface.co/Edward11451/CivMarshmallow-ckpts.

Every score in Table 3 now also has its standard error (W1).

The discussion period ends on November 18. We would be grateful if you could let us know whether these results address your concerns, and if they do, we hope you will consider raising your score.

Sincerely,
Paper22433 Authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of our rebuttal: new experiments and paper revisions

Dear ICLR Chairs,

Thank you for organizing this review process. The reviews of our paper were helpful. All four reviewers appreciated our analysis of scoreboard traps (positions with a higher current score that lead to a worse final outcome), Reviewer fnW5 found our replay-supervision results convincing, and Reviewer sCHt noted the meaningful controls in our judge evaluation.

Our rebuttal addressed the main concerns with new experiments and a revised PDF (changes in blue):

  • Isolating the reward effect (fnW5, sCHt). We ran the three controls both reviewers asked for. (1) A regression value head used as the reward in the same pipeline reaches a mean phase gain of 6.74, against 9.70 for our judge, which is ahead on all three seeds. (2) The terminal-only control (λ = 1, true terminal reward, matched initialization and reference-set updates) ends at 15.88, against 22.05 for our hybrid: the intermediate feedback drives the gain. (3) Started from a Scoreboard checkpoint, our best run reaches 23.35, the top score of Table 3. Every Table 3 score now has its standard error over 360 games.
  • Scope and generalization (ct2D, KQ2Y). We ran the full method on Settlers of Catan and Othello. Our reward beats Scoreboard on every seed in both games (new Appendix D.8).
  • Novelty (KQ2Y). The paper shows that the standard distillation of replay outcomes fails, and what works instead: a regression value head is at chance on held-out trap pairs (0.520, against 0.688 for our judge), and, as we showed in reply to Reviewer sCHt, a ranking-loss head does no better (0.380).
  • Clarity (ct2D). We added the agent's observation and action space (new Appendix B.2), standard RL terms and a worked example in Section 5.3, definitions of every term the review raised, related work on GAE, and the slime citation.
  • Code and checkpoints. We released our code in anonymized form at https://anonymous.4open.science/r/CivMarshmallow/, and the trained checkpoints at https://huggingface.co/Edward11451/CivMarshmallow-ckpts.

We would be grateful if the chairs could encourage our reviewers to read our rebuttal and update their reviews. Thank you for your time.

Sincerely,
Paper22433 Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.