acceptodds
Under review as a conference paper at ICLR 2027

FM-Bench: A Benchmark for Long-Horizon Management with Competing Agents

Abstract

Language model agents now execute bounded tasks reliably, but whether they can sustain effective decision-making over long horizons, where actions have cumulative consequences and the environment responds to their choices, is less well studied. We introduce FM-Bench (Football Management Benchmark) to study this setting. An LLM agent runs a football club for 20 in-game years through 26 tools and roughly 340 to 400 decision stops, drafting on a fixed budget, trading, negotiating contracts, investing in facilities and youth, and answering to a board that can fire it. A deterministic engine accumulates every season into one final score with no LLM judge. A solo track plays 15 frontier models against a frozen scripted world; an Arena places them in one shared 20-year world. Across three solo seeds, all models complete the horizon, whereas the blind scripted baselines are eliminated in most runs; claude-fable-5 has the highest mean solo score and finishes first in the Arena, although the season title rotates among ten models. In our sample, neither model scale, price, nor vendor clearly predicts the ranking, which stabilizes only late in the horizon, and the best of six first-time human players scores near the bottom of the model board. Decomposing the score into six behavioral capabilities suggests that differences in managerial behavior, rather than computation, account for much of the gap: stronger models tend to reduce slow-payoff investment near the end, keep cash invested, and renew contracts early. Token spend is not correlated with score, no model reliably infers the market's hidden prices, and self-managed memory degrades into an ever-growing archive or a plan rewritten every season. The code is available at https://anonymous.4open.science/r/FM-Bench-B644.

Primary area:
datasets and benchmarks
Keywords:
LM agents, benchmark, long-horizon decision making, multi-agent evaluation, management simulation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_PqQjOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces FM-Bench, a purpose-built football-management simulator for evaluating long-horizon decision-making under hidden information, delayed consequences, a reactive market and competing objectives. The benchmark has 2 modes: Solo, where agents manage a club against scripted opponents, and Arena, where agents compete directly against each other. The benchmark evaluates a wide selection of LLMs and measures their performance in terms of a composite score as well as various capability/behavioral metrics. The benchmark is well defined and the code is well structured and extendable.

Critical strengths
  • The benchmark follows established principles for reproducibility and includes seeded deterministic simulation, engine-computed scoring, reference policies and thorough logging. The analysis goes beyond just a leader board and includes a wide range of behavioural metrics that are relevant to the benchmark’s objectives.
  • The paper is well written and structured. Introduced concepts are clearly defined in the main body and the appendix provides thorough documentation of most additional aspects of the benchmark and evaluation.
Critical weaknesses
  • The behavioural metrics are not yet sufficiently validated as measures of the capabilities they name. Offers per completed signing does not distinguish accurate pricing from overpayment, while notebook similarity does not directly measure retention of useful information. The memory score additionally rewards proximity to an empirically selected similarity value, and other metrics were discarded based on their observed correlations. This makes it difficult to distinguish independently validated capability measures from exploratory descriptions chosen after examining outcomes. Ideally I would like analysis of the correlation and contribution of each of the capabilities to the overall score, as well as a more thorough statistical analysis of the hypotheses and documentation of experimental hypotheses setup.
  • One 1 seed is used in the arena case which limits the conclusions that can be drawn. The paper acknowledges this and reports descriptive statistics and correlations, but the ability to do a more thorough statistical analysis is limited.
  • Many figures contains text that is significantly smaller than latex "\small", and some figures in the appendix have formatting issues.
Questions for authors
  • Why was consistency of correlation with the final score used to retain or reject capability metrics, and how does this selection affect the interpretation of the reported capability–score associations?
  • Would it be possible to investigate the contribution of each of the capabilities to the overall score?
  • A mix of sd, mean and mean (with min/max) is used across tables and figures. I think the min/max makes sense, but in some cases a 95CI interval might be more useful. Can the authors clarify their choice of reporting metrics and consider using a more consistent approach across tables?
  • For the price discovery attribute, can this not be hacked by always overpaying for players? Does the game engine have the true (mean) market value of players? Would it not make more sense to define the price discovery attribute as how close the agent is to the true market value of players or some compound of this and transfer success rate?
  • Given that models are now trained in persistent harness settings, was using their own harness and allowing them to manage context and how they want to memorise things themselves considered? This may improve results and be more realistic to how models are trained and used in practice.
  • How was the memory-similarity target of 0.35 chosen and validated as measuring useful memory curation?
  • Can you make it more clear what attributes FM-Bench tests that the other long horizon benchmarks do not? How does it compare to other benchmarks in terms of difficulty and what capabilities are being tested?
  • Can it be made more clear that the benchmark is not based on the football manager game but an environment that covers a similar setting? I did not realise this until the end of my first reading.
  • Does the game constitute a POMDP? If so maybe some formalization of the problem in terms of POMDP would be useful. Also a RL baseline would be useful to compare against. If not, what is the formalization of the problem?
8 comments
AuthorsOpenReview ↗

Response to Reviewer PqQj (1/5)

Thank you for a thorough review and for the many concrete suggestions. You found the paper well written and the benchmark built on established principles for reproducibility; your concerns are about validation and statistics, and we ran new analyses for each of them. Because several answers include full tables, this reply is split over a few comments.

W1 / Q1. Validation of the behavioral metrics; selection by observed correlation

We agree that choosing metrics by how consistently they correlate with the score makes the results exploratory, and we will say so. Our reason for the rule was that a metric whose sign flips between seeds cannot describe a stable capability, but applying it after looking at the outcome selects for metrics that happen to agree on three seeds. Below we (a) re-test all nine candidates under one procedure, so the dropped metrics are judged exactly like the kept ones, (b) report what each metric does and does not measure, and (c) describe the hypothesis documentation for the new experiments.

(a) All nine candidates, one procedure. For each metric we computed the Spearman correlation with the final score on each seed separately (15 models per seed), a permutation p-value that shuffles model labels within each seed (20,000 permutations), and corrected for multiple comparisons over all nine with Holm (family-wise) and Benjamini–Hochberg (false discovery rate). "Within model" is the correlation after removing each model's mean, that is, whether a model's better seeds are the ones where it shows more of the behavior.

metricstatus in submissionper-seed ρ\rho (1, 1024, 2026)permutation ppHolm ppBH ppwithin model
idle cashkept−0.74, −0.60, −0.34<0.0010.0020.002−0.28
renewal lead timekept+0.68, +0.18, +0.470.0020.0190.011+0.17
endgame investment cutkept−0.31, −0.34, −0.280.0430.260.096−0.18
tactic reversalsdropped−0.46, 0.00, −0.590.0210.150.063−0.12
youth harvestdropped−0.52, −0.18, +0.040.160.820.29+0.19
tokenskept (compute axis)−0.14, −0.23, −0.150.261.00.39−0.22
warning exposuredropped+0.45, +0.07, −0.780.581.00.74−0.46
offers per signingkept−0.25, +0.02, +0.020.661.00.74−0.03
distance from 0.35kept−0.28, −0.06, +0.240.831.00.83−0.31

Two metrics survive correction: idle cash and renewal lead time. The endgame cut keeps its sign on every seed but does not survive correction. Of the three dropped metrics, tactic reversals is as strong as the endgame cut before correction, so the earlier selection was not conservative in a useful way: it kept two metrics (offers per signing, distance from 0.35) that are weaker than one it dropped. Idle cash needs a caveat: much of its correlation comes from the score's own discount on hoarded cash (without the discount the per-seed correlations are −0.24, −0.12, −0.17). Renewal lead time is therefore the one behavior with a clean, corrected association with score.

As a result, we will present the capability profile as exploratory description, report this full table (including the dropped metrics) in Appendix J, and limit association claims to renewal timing, with idle cash flagged as coupled to the score.

(b) What the two questioned metrics measure. Offers per completed signing does not separate accurate pricing from overpayment, as you say. We now measure pricing directly against the engine's exact acceptance threshold; see Q4 and our reply to Reviewer 2mbG (W6). Notebook similarity does not measure retention of useful information directly. To test what it does capture, two blinded LLM judges scored 225 season-end notebooks on whether they held the information the agent needed for the next season; distance from 0.35 tracks lower judged quality (−0.30-0.30 and −0.32-0.32), mainly by flagging append-only notebooks. We will keep it only as a description of that pattern. The notebook intervention (removing or capping it) shows that whether memory matters depends on the model; see Q6.

(c) Hypothesis documentation. For every new experiment in this rebuttal, the hypotheses, the treated and control runs, the decision rule (an effect must exceed twice the same-seed run-to-run SD and replicate in a repeat), and the manipulation checks were written down before launch (in the supplementary archive, rebuttal/PREREGISTRATION.md); amendments made after launch are dated and listed in the same file. We will add this protocol to the appendix.

AuthorsOpenReview ↗

Response to Reviewer PqQj (2/5)

Q2. Contribution of each capability

Yes, and we did so in three ways. The short answer is that the capabilities overlap, and together they explain only a small part of the differences in score.

1. The capabilities are correlated with each other. Spearman correlations between the six capability metrics (15 models, each metric averaged over the three seeds):

endgame cutidle cashrenewal leadoffers per signingdistance from 0.35tokens
endgame cut10.29−0.700.57−0.200.30
idle cash1−0.350.030.010.00
renewal lead1−0.21−0.020.26
offers per signing1−0.270.46
distance from 0.3510.03
tokens1

The endgame cut and renewal timing correlate at −0.70: models that renew early are largely the same models that wind investment down near the end. Attributing score to one of them separately is therefore not well defined with 15 models.

2. Predictive contribution. In leave-one-model-out prediction of the score (fit on 14 models, predict the 15th), the best single metric (endgame cut) explains 13% of the variance, the three headline metrics together 4%, and all six together do worse than predicting the mean (R2=−0.69R^2 = -0.69, overfitting). Behavioral metrics describe how models differ, but they do not add up to a decomposition of the score.

3. Interventional contribution. We tried two ways to change one capability and observe the score. (i) Prompting a model to renew early or to keep less idle cash did not change its behavior enough to pass our pre-registered manipulation check, so we cannot read off an effect. (ii) In a scripted policy we can switch each behavior on and off exactly. In heuristic_v2 (our stronger blind baseline), switching off any single rule, including the survival controller, changes the score over 13 seeds by an amount indistinguishable from zero (every paired 95% interval contains 0). In a script, at least, none of these behaviors has a measurable effect on its own.

Finally, the composite score itself has three channels, and across the 15 models the honors channel accounts for 74% of the variance in score, value added for 13% and squad value for 13%. Most of the separation between models therefore comes from on-pitch results, which the capability metrics reach only indirectly.

We will add these three analyses to Appendix J and state in §4.2.1 that the capabilities are a descriptive profile, not a decomposition of the score.

Q3. Reporting (sd, min/max, CI)

We agree that the mix is confusing. We will use one convention in every results table: the mean, the three seed values and a t-based 95% interval, which is wide with three seeds (for example [78.0, 103.9] for the top model). With three seeds and run-to-run variation of up to 16 points, only 11 of the 105 model pairs are separated by a seed-paired 95% interval, and we will say so next to Table 1. Model comparisons will use seed-paired differences, and Arena results will be mean ranks over the three matched-seed worlds.

AuthorsOpenReview ↗

Response to Reviewer PqQj (3/5)

Q4. Can price discovery be gamed; does the engine know true values?

We thank the reviewer for this sharp observation, which raises three questions that we answer in turn.

Does the engine have the true market value of players? Yes. Every player has a deterministic market value, and every seller has a deterministic accept threshold, which is the asking price adjusted by the seller's style and financial distress. Both are pure functions of the seed and are hidden from agents, who see only noisy scout bands. The oracle reads the threshold directly from the engine and bids exactly that, so each of its purchases closes on the first offer at the lowest price the seller would take (Appendix H). This is what fixes the reference at 1.0 offers per completed signing.

Can the metric be hacked by always overpaying? Taken in isolation, yes: a policy that bids far above the threshold would also close on the first offer. In the benchmark this cannot be exploited, for two reasons. First, the six capability metrics are post-hoc diagnostics. They do not enter the composite score and are never shown to the agent, so there is nothing for an agent to optimize against. Second, overpaying is penalized by the score that does rank the models. By Proposition A.3, buying a player of market value vv at price p≥vp \ge v changes net worth by −p+λv<0-p + \lambda v < 0, so every credit paid above fair value lowers the value-added channel. The fee is also not burned but transferred to the selling club, a direct rival in the same league, which weakens the buyer's relative position and hence the honors channel. An always-overpay policy would look good on this one diagnostic and lose on the ranking.

Should the attribute be defined by closeness to true market value instead? We agree with the underlying point: offers per signing measures search cost, that is, how many attempts an agent needs to close a deal, and it does not by itself measure price efficiency. A low value is not evidence of accurate pricing. We will revise §4.2.1 and Appendix J to state this explicitly and remove the wording that read the statistic as a statement about the prices models bid. We keep the diagnostic as a measure of search cost because price efficiency is already what the value-added channel of the composite rewards, and we prefer each diagnostic to capture a behavior the score does not directly measure.

Q5. Letting models use their own harness

We considered it and kept one harness so that the score compares models rather than harnesses. Our new experiment shows the choice matters: removing the notebook lowered gpt-5.6-terra's score in both repeats, while other models were unaffected, and the models manage the same notebook in very different ways (see Q6). A model's own harness would mix these differences with harness engineering. In the Limitations section we will add: "We fix one harness so that scores compare models rather than harnesses; a track in which each model brings its own context and memory management is future work."

Q6. The memory target of 0.35

0.35 was chosen after seeing the seed-1 results, and the submission did not validate it. We have now checked what it does and does not measure.

It does track notebook quality. Two LLM judges (gpt-5.6-terra and claude-fable-5) scored 225 season-end notebooks against what the agent did the next season, without seeing the model name or the similarity value. Notebooks further from 0.35 received lower memory-quality scores from both judges (Spearman −0.30-0.30 and −0.32-0.32, p<0.01p < 0.01 within models; the two judges agree with each other at 0.67). The effect comes mainly from append-only notebooks (similarity above 0.5), which scored worst.

It does not predict score. The distance does not predict the final score on held-out seeds, and re-fitting the target with each model held out in turn gives unstable targets (0.34 to 0.90) that do not predict the held-out models' scores.

And memory matters for some models but not others. Removing the notebook lowered gpt-5.6-terra's score in both repeats (89.7 to 85.8, and 91.3 to 85.9). gpt-5.6-terra writes at nearly every stop and almost never rewrites, so its notebook is a running record it depends on. claude-fable-5 keeps a short notebook it rewrites often and plays about as well without it; gpt-5.6-sol keeps an append-only archive, and capping it changed nothing; glm-5.2 writes far less often than the other models.

We will therefore keep the distance from 0.35 only as a descriptive measure of curation style, no longer call 0.35 the best value, and present memory as a per-model profile.

Reviewer_s6yJOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

FM-Bench evaluates LLM agents managing a synthetic football club for 20 in-game years through 26 tools and approximately 340–400 decision stops. The environment combines hidden player quality, delayed investments, financial and sporting objectives, competing clubs, and a notebook that carries the agent's self-authored state between fresh conversations. A deterministic composite score combines accumulated honors, value added, and squad value. The study evaluates 15 models over three Solo seeds and one shared-world Arena, includes scripted and privileged-information anchors and six first-time human runs, and analyzes behavioral metrics such as renewal timing, idle cash, bidding, notebook changes, and endgame investment.

Critical strengths
  1. The benchmark offers a concrete and reusable setting for sustained decision-making with cumulative consequences and competition. The Solo and Arena tracks share an engine and tool interface, while the explicit notebook makes cross-stop state management observable. Mechanism-computed scoring avoids an LLM judge. I consider this environment contribution sufficient to make the paper potentially valuable even if the initial leaderboard is not definitive.

  2. The study goes beyond reporting final scores. The season trajectories show why shorter evaluations can produce a different ordering, and the action-level analyses identify specific behaviors that future work can test. The paper appropriately acknowledges the small seed count, the privileged oracle's non-optimality, and the limitations of the human comparison rather than treating them as general capability rankings.

Critical weaknesses
  1. Some capability interpretations are stronger than the measurements establish. Appendix J scores notebook curation by distance from an empirically selected cosine-similarity target of 0.35; this makes the preferred memory style partly fitted to these runs. Idle cash is also directly affected by the score's anti-hoarding discount, and score-per-token includes the outcome in the proposed compute-efficiency axis. These can be useful descriptive metrics, but they do not independently validate memory curation, credit assignment, or a claim that behavior rather than computation explains the performance gap. The acknowledged absence of causal probes and the single Arena seed further limit the explanations. This matters because behavioral diagnosis is a central contribution, not merely an incidental plot.

  2. The reproducibility boundary is unclear in the accessible release. The PDF's reproducibility statement says that transcripts, action logs, and a self-contained analysis folder are released, whereas the linked anonymous repository README explicitly excludes official trajectories/result logs and says Arena is not self-run. The separate supplementary archive may resolve this, but I could not verify its contents. Appendix J also describes two passes over 15 Solo runs and 30 matching replays, while its metrics are stated to average three seeds for each of 15 models. An explicit mapping from the reported tables to the released runs and analysis scripts is needed to distinguish reproducibility of the engine from auditability of the reported conclusions.

Questions for authors
  1. Was the notebook target of 0.35 fixed before the scored runs, or selected from the observed score–similarity relationship? Can you show whether the memory-profile interpretation survives alternative targets or, preferably, a notebook intervention? A clarification that the axis is descriptive would also narrow the claim appropriately.

  2. Where can reviewers access the exact reported trajectories, manifests, and behavioral-analysis inputs? Please reconcile the PDF's release statement with the anonymous README's exclusions, and clarify whether the replay gate covers all 45 model-by-seed Solo runs or only 15. If the supplementary archive supplies them, please identify the relevant paths and engine/scoring version.

  3. Which conclusions about competition survive a matched-seed Solo/Arena comparison or additional Arena worlds? Section 4.3 already acknowledges the seed confound; I would like the conclusion to clearly separate the demonstrated within-world observations from an effect attributed to adaptive rivals. I lean toward acceptance based on the benchmark contribution, with the explanatory and release claims tightened.

4 comments
AuthorsOpenReview ↗

Response to Reviewer s6yJ (1/2)

Thank you for a careful review and for leaning toward acceptance on the benchmark contribution. We agree with both conditions you attach: the explanations should be no stronger than the measurements, and the release should let a reader check the conclusions, not just rerun the engine. Below is what we did about each.

W1 / Q1. Interpretations stronger than the measurements

We agree with all four points.

The 0.35 target was read off seed 1, not fixed in advance, and we will say so and call the axis descriptive. On alternative targets: if the target is re-fitted with each model held out in turn, the fitted value ranges from 0.34 to 0.90, and closeness to it does not predict the held-out models' scores (Spearman −0.60-0.60, −0.51-0.51 and +0.07+0.07 on the three seeds, the first two in the wrong direction). No single target predicts score. What the distance does capture is notebook quality: two blinded LLM judges scored 225 season-end notebooks against what the agent did next season, and notebooks further from 0.35 scored lower with both judges (Spearman −0.30-0.30 and −0.32-0.32, p<0.01p < 0.01 within models), mainly because append-only notebooks score worst.

We also ran the notebook intervention you suggest. Removing the notebook lowered gpt-5.6-terra's score in both repeats (89.7 to 85.8, and 91.3 to 85.9); for glm-5.2 and claude-fable-5 the effect was within run-to-run noise, and capping gpt-5.6-sol's archive-like notebook changed nothing. The models use the notebook very differently (terra keeps a growing running record, fable a short notebook it rewrites often, sol an append-only archive, glm writes far less often than the rest); details are in our answer to Reviewer 2mbG (W5). The intervention therefore supports a per-model memory profile, which is how we will present the axis.

The coupling between idle cash and the discount is real. With the discount removed from both the score and the metric, the per-seed correlations fall from −0.74-0.74, −0.60-0.60, −0.34-0.34 to −0.24-0.24, −0.12-0.12 and −0.17-0.17. We still think the behavior is informative: the rules every agent receives say that cash hoarded far beyond the wage bill is discounted, so a model that keeps hoarding is ignoring an incentive it was told about. As in a real club, idle cash has a cost. We will present it that way, not as evidence of credit assignment.

Score per token was a mistake. The score sits in the numerator, so high-scoring models look efficient by construction. We will remove that axis from the capability figure. Token spend alone does not predict the ranking (per seed −0.14-0.14, −0.23-0.23, −0.15-0.15).

We will restate "behavior rather than computation" as an association. We tried to test causation by prompting models to change one behavior, but the prompts did not change the behavior enough to tell us anything. On the single Arena seed, we now have three more worlds; see Q3.

W2 / Q2. Reproducibility boundary and replay coverage

Both gaps were real, and we thank the reviewer for checking. The replay check now covers all 45 solo runs (90 of 90 replays exact) and every Arena world, including the new ones; the submission's extraction had used seed 1 only. Recomputing Appendix J found 15 errors, which we will list and fix. The updated supplementary archive contains, under results/runs/<run_id>/, each solo run's manifest, action log and per-turn transcripts, and each Arena world's manifest and action log (Arena transcripts were not kept, which the reproducibility statement in the final version will say); the replay and analysis scripts under rebuttal/ and scripts/; and MAPPING.md, which maps every table and figure to its run ids and script. Every run was scored with score version score_v0.2 (score/composite.py) and parameter hash 06ea9d609104, which is recorded in each run's manifest. The anonymous repository holds the engine code only, which is why its README excludes run logs; we will make both statements consistent.

AuthorsOpenReview ↗

Response to Reviewer s6yJ (2/2)

Q3. Which competition conclusions survive?

We ran three more 20-year Arena worlds on exactly the three solo seeds (1, 1024, 2026), with seats rotated so that no model keeps its club; all three replay to their recorded state hashes. This removes the seed confound: each Arena world now has a solo counterpart on the same seed. We separate what we observe from what we had attributed.

Observations that hold in all three worlds.

  1. The title rotates. There are 8, 11 and 8 different champions in 20 seasons, and a champion retains its title in only 6 of 57 opportunities. On the same seeds, solo runs hold dynasties (median 10 titles per run).
  2. The top and bottom of the ranking are stable. claude-fable-5 has the best mean rank (2.33; ranks 4, 2, 1), followed by qwen3.7-max (3.33) and gpt-5.6-sol (4.00); these three are in the top five in every world. claude-haiku-4.5 is in the bottom three of the 11 shared models in every world (9th, 11th, 11th).
  3. The middle is not stable. Agreement between worlds is moderate (Kendall's W = 0.49 for the 11 shared models, 0.42 for all 15 seats). For example, glm-5.2 is 10th, 1st and 7th, and gemini-3-flash 1st, 8th and 8th.
  4. Solo and Arena agree at the ends. On matched seeds, solo mean rank and Arena mean rank correlate at Spearman 0.60 (p=0.05p = 0.05) for the 11 models that played both tracks. Without claude-haiku-4.5 the correlation falls to 0.47, so the agreement is carried by the extremes; mid-table models move by up to three places (muse-spark-1.1 and gpt-5.6-terra do worse in the Arena, qwen3.7-max and claude-opus-4.8 better).
  5. Winning is not the same as finishing first on the composite. claude-fable-5 has the best mean rank but wins only one world, so we will report mean rank over worlds rather than a winner, and we will change "finishes first in the Arena" in the abstract accordingly.

Attributions that do not hold.

  1. "Adaptive rivals cause rotation." Sixteen identical scripted managers, which cannot adapt, also rotate the title on the same three seeds (8–12 champions in 20 seasons). Rotation follows from rivals of comparable strength, not from adaptation.
  2. "Rivals bid talent away from whoever leads." In the three worlds the reigning champion is the seller in 9.0% of completed transfers (6.7% by chance), but it receives only 6.0% of rivals' bids. Rivals do not single out the leader.
  3. "The contrast isolates the counter-adaptive market." The same market response operates in the solo track, where dynasties form, so it is not sufficient for rotation. Whether it changes how often titles change hands would require an ablation with the market response switched off, which we could not run during the discussion period (see our answer to Reviewer 2mbG, Q3).

We will rewrite the conclusion of §4.3 to state observations 1–5 as the competition findings, and the "two tracks contrasting fixed and adaptive rivals" contribution as a contrast in opponent strength and type, without attributing the difference to adaptation.

Reviewer_s6yJOpenReview ↗

Response to authors

Thank you for the thorough response and the new experiments.

W1/Q1: I appreciate that the 0.35 target will be described as descriptive, and that score-per-token will be removed. The notebook-removal experiment supports presenting memory as a per-model profile rather than a general capability.

W2/Q2: Extending the replay check to all 45 solo runs and adding MAPPING.md resolves my reproducibility concern.

Q3: The three new Arena worlds on matched seeds remove the seed confound, and the scripted-manager result shows that title rotation comes from comparable rival strength, not adaptation. Restating §4.3 this way makes the claims match the evidence.

My concerns are addressed. I will keep my score of 3.

Thank you for following up, and for the time you spent on our paper. Your comments on the target framing, reproducibility, and the Arena results led to experiments that improved the paper. We'll include all the agreed changes in the final version, including the rewritten §4.3.

Reviewer_2mbGOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

FM-Bench evaluates long-horizon management through a football club simulation. An agent manages a club for 20 in-game years, including transfers, contracts, squad selection, and facility investment. Each decision stop starts a new conversation, and the agent uses a notebook to keep its plans. The benchmark includes a Solo track against scripted opponents and an Arena track where different models compete in the same world. The paper evaluates 15 language models, several scripted policies, a privileged oracle, and six first-time human players. It also studies investment timing, contract renewal, bidding, and notebook use.

Critical strengths
  1. The combination of long-term investment, hidden player information, and competition is interesting. Decisions change both the club and the opportunities available later. This gives the benchmark more substance than simply extending the number of tool calls.
  2. The environment has considerable engineering work. Deterministic scoring, action-log replay, scripted reference policies, and the detailed interface make the results easier to inspect. The season-level figures also show how early success can change over a longer run.
Critical weaknesses
  1. The non-LLM baselines leave a large gap. Random and idle are basic references, while the heuristic performs much worse than most models. The oracle uses hidden information. A stronger policy using the same observations is needed to judge whether the task requires substantial planning or can be handled well by a small set of domain rules.
  2. The final ranking depends on several designed preferences: score weights, cash discounts, asset valuation, and the revival penalty. There is no sensitivity analysis showing whether the main conclusions remain under other reasonable settings.
  3. The memory capability score is not sufficiently justified. Appendix J rewards notebook similarity close to an “empirically best” value of 0.35. This uses an observed pattern to define what good memory should look like.
  4. The Arena has only one world, and its seed differs from the Solo seeds. The change from repeated champions to rotating champions is interesting, but opponent type and world variation change together.
  5. The main mechanisms are discussed more than tested. There are no controlled comparisons removing the market response, changing the memory setup, or varying the planning horizon.
  6. The price-discovery conclusion needs a more direct measurement. A high ratio alone does not show that the model failed to learn prices over time.

After rebuttal: I raise my rating from 1 to 2. The stronger baseline and the new Arena runs on matched seeds are useful. The paper shows that different clubs win the league across seasons, but the text and figures do not explain this result well. In particular, it is still unclear whether the market response contributes to it. I remain at weak rejection.

Questions for authors
  1. How do the rankings and behavioral conclusions change under reasonable changes to score weights, the excess-cash discount, and revival penalties?
  2. How was 0.35 selected for the memory metric? Does this criterion predict performance on held-out models or seeds, and does changing notebook retention actually change task outcomes?
  3. Can Solo and Arena be evaluated on matched seeds across several worlds? Does the title-rotation pattern remain when the market-response mechanism is disabled?
4 comments
AuthorsOpenReview ↗

Response to Reviewer 2mbG (1/4)

Thank you for a detailed review. You found the combination of long-term investment, hidden information and competition interesting and noted the considerable engineering work, and your point that the main mechanisms are discussed more than tested is fair. For each point below we say what we ran or measured and what will change in the final version.

W1. Stronger blind baseline

We built heuristic_v2, which sees only what the agents see (a static check confirms it never reads hidden state). It adds a survival controller and hand-coded versions of the behaviors our analysis links to high scores: early contract renewal, deploying idle cash, an endgame investment cut and disciplined bidding. It scores 50.1 on the official seeds (60.3, 44.9, 45.2), against 17.1 for the old heuristic, but it still ranks below 14 of the 15 models, and on ten additional seeds it averages about 28 and is eliminated early in 2 of 13 runs, so the official-seed figure flatters it. A 61-candidate random search over its parameters on five disjoint training seeds did not help (45.3 on the official seeds). Over all 13 seeds, switching off any single rule, including the behaviors our analysis links to high scores, changes the score by an amount indistinguishable from zero (every paired 95% interval contains 0). A small rule set does not solve the task. The oracle remains in the paper only as a ceiling that reads hidden state.

W2 / Q1. Sensitivity to the score's design

We re-scored all 45 solo runs and the anchor runs with behavior held fixed. One knob at a time (each weight, divisor and discount at 0.5x to 2x), the median Kendall τ\tau with the official ranking is 0.98, and only the extreme "honors only" setting changes the top model. Across 2,000 random joint settings of the weights, the excess-cash discount, asset valuation and the revival penalty, claude-fable-5 is first in 97.6% of them, the median Kendall τ\tau with the official ranking is 0.79 (5th percentile 0.50), and the order of the scripted anchors never changes. The behavioral correlations keep their sign on every seed in 91–100% of settings. The top and bottom of the board are stable; the middle reorders. We will also correct one sentence: the weights were calibrated on the scripted anchors with exploit checks on pilot LLM runs, then frozen before the first official run.

W3 / Q2. The memory target of 0.35

0.35 was chosen after looking at the seed-1 results, and we will say so. We still think it is a reasonable analysis tool, and we can now show why. Two LLM judges (gpt-5.6-terra and claude-fable-5) scored 225 season-end notebooks against what the agent actually did the next season, without seeing the model name or the similarity value. Notebooks further from 0.35 got lower memory-quality scores from both judges (Spearman −0.30-0.30 and −0.32-0.32, p<0.01p < 0.01 within models), mostly because append-only notebooks scored worst. So distance from 0.35 does pick out poor curation. It does not predict the final score on the held-out seeds (+0.06+0.06 and −0.24-0.24), so we will keep it as a descriptive memory measure and will no longer call 0.35 the best value. Note that 0.35 enters only the memory axis of the capability profile, never the composite score.

Removing the notebook lowered gpt-5.6-terra's score in both repeats (89.7 to 85.8, and 91.3 to 85.9).

AuthorsOpenReview ↗

Response to Reviewer 2mbG (2/4)

W4 / Q3 (first part). One Arena world, on a different seed

Yes. We ran three more 20-year Arena worlds on exactly the solo seeds (1, 1024, 2026), rotating seats so that no model keeps its club, and every world replays to its recorded state hash. Ranks of the 11 models that played both tracks (four providers retired their endpoints, so those Arena seats used successor models, which are included in the 15-seat statistics but left out of this table):

modelseed 1seed 1024seed 2026mean rank
claude-fable-54212.33
qwen3.7-max2353.33
gpt-5.6-sol3544.00
gemini-3-flash1885.67
grok-4.55935.67
glm-5.210176.00
gpt-5.6-terra7666.33
claude-opus-4.864106.67
muse-spark-1.181026.67
claude-sonnet-511799.00
claude-haiku-4.59111110.33

Agreement between worlds is moderate (Kendall's W = 0.49 for these 11 models, 0.42 for all 15 seats), so a single world is not a ranking. claude-fable-5 has the best mean rank but wins only one of the three worlds, so we will report mean rank over worlds rather than a winner. On the matched seeds, solo and Arena mean ranks correlate at Spearman 0.60 (p=0.05p = 0.05): the strongest and weakest models agree, while mid-table models move by up to three places (for example, muse-spark-1.1 and gpt-5.6-terra do worse in the Arena than in solo, qwen3.7-max better). The title rotates in all three worlds (8, 11 and 8 different champions in 20 seasons; a champion kept its title in only 6 of 57 opportunities), while solo runs on the same seeds hold dynasties.

Q3 (second part). Rotation with the market response disabled

We can answer two parts of this question now; the third needs an experiment we have not run.

First, the market response alone does not produce rotation. The same market response (asks that rise after rejected bids, repeat-pair markups, cooldowns) is active in the solo track, and there strong models still build dynasties against the 15 scripted clubs (median 10 titles in 20 seasons). So the mechanism alone does not make the title rotate.

Second, rotation does not need adaptive rivals. Sixteen identical scripted managers, which cannot adapt to anything, produce 8–12 different champions in 20 seasons on the same three seeds, about as many as the LLM worlds (8–11). Rotation appears whenever the clubs are evenly matched. We also checked the "rivals bid talent away from the leader" explanation directly in the three LLM worlds: the reigning champion is the seller in 9.0% of completed transfers (6.7% would be chance), but it receives only 6.0% of rivals' bids, so rivals do not single out the leader. We withdraw that explanation and will rewrite the §4.3 passage to say that the contrast reflects field parity rather than adaptation. The last part of the §4.3 passage still holds: in the Arena the composite winner is set by 20-year asset accumulation rather than by title count: in the seed-2026 world, for example, muse-spark-1.1 won 8 of the 20 titles and claude-fable-5 won 4, yet claude-fable-5 finished first on the composite.

Third, we do not yet know whether the market response changes how often titles change hands. The LLM worlds keep the title less often (6 of 57 defences) than the identical scripted leagues on the same seeds (14 of 57), but the fields differ in more than the market response. A clean answer requires an ablation: the same seeds and seats with the ask increases, markups and their decay set to zero, comparing how often the champion retains the title. If retention rises clearly, the market response is a factor; if it does not change, it is not. Each 20-year, 15-LLM world costs about USD 1,000 to run, so we could not complete it during the discussion period; we will state the question as open in the paper rather than assert an answer.

AuthorsOpenReview ↗

Response to Reviewer 2mbG (3/4)

W5. Mechanisms discussed more than tested

We ran controlled comparisons for memory and horizon, and one more for behavior prompting. Each treated run was compared with a fresh control of the same model and seed, the predictions and decision thresholds (twice the same-seed run-to-run SD) were registered before launch (in the supplementary archive, rebuttal/PREREGISTRATION.md), and every run passed the replay gate. The market response is discussed in our answer to Q3.

Memory: whether it matters depends on the model. We removed the notebook for gpt-5.6-terra, glm-5.2 and claude-fable-5, and capped gpt-5.6-sol's notebook; each pair was repeated except claude-fable-5's. Only gpt-5.6-terra lost score reliably (89.7 to 85.8 and 91.3 to 85.9, a mean drop of 4.7 against a threshold of 1.7). For glm-5.2 the two repeats went in opposite directions (+3.0 and −6.8) and for claude-fable-5 and gpt-5.6-sol the change stayed within run-to-run noise. To understand why, we measured how each model uses its notebook across the three new Arena worlds (seeds 1, 1024, 2026), where all 15 models play under identical conditions:

modelnotebook writes per seasonshare of writes that rewrite the whole notebooknotebook length at season 20 (chars)season-to-season similarity
gpt-5.6-terra19.33%28,1910.66
claude-fable-518.728%7,0860.26
glm-5.28.813%7,9070.52
gpt-5.6-sol18.21%212,4190.93
qwen3.7-max16.594%1,7640.18
grok-4.519.182%3,9740.29
claude-opus-4.818.04%67,0450.45

The models fall into clearly different styles, and the same styles appear in the solo runs (season-to-season similarity on the three solo seeds: gpt-5.6-sol 0.91, gpt-5.6-terra 0.66, claude-fable-5 0.39). gpt-5.6-terra writes at almost every stop and almost never rewrites, so its notebook is a running record of its own state and plans; it is the model that loses score without it. glm-5.2 writes far less often than any other model (8.8 times per season, against 14 to 24 for the rest), so removing the notebook takes away little. gpt-5.6-sol only appends, ending with a 200,000-character archive that barely changes between seasons; capping it forced compression without changing the score, which suggests that the archive was not working memory to begin with. claude-fable-5 keeps a short notebook that it rewrites often; the blinded judges rated its notebooks among the best (8.9 out of 10 from one judge, the highest of all models, and 8.3 from the other), yet it plays about as well without them. Its notes are good, but its decisions do not appear to depend on them. Memory therefore matters for models that rely on accumulated notes and not for models that rebuild their picture of the club at each stop. This is a property of the model rather than of the benchmark, and it is why we keep memory as a per-model profile rather than a single "memory helps" claim.

Horizon. With a 10-year horizon, claude-fable-5 and gpt-5.6-terra stopped long-term investment in the last seasons of the new horizon instead of at years 19 and 14 of the 20-year runs, so they follow the announced horizon. claude-sonnet-5 and muse-spark-1.1 did the same, so this behavior does not separate strong from weak models, and we will drop that claim.

Behavior prompting. Prompting claude-opus-4.8 to renew contracts early and glm-5.2 to keep less idle cash did not change their behavior enough to pass our pre-registered manipulation checks, so those runs say nothing about causation.

Run-to-run variation. Rerunning the same model on the same seed can move its score by up to 16 points. We will report this, and we will describe the links between behavior and score as associations, not causes.

AuthorsOpenReview ↗

Response to Reviewer 2mbG (4/4)

W6. Price discovery needs a direct measurement

We agree, and we have now measured it directly. The engine knows, for every bid, the exact lowest price the seller would accept (the asking price for that buyer, including any markups, times the seller's acceptance ratio). We replayed the three new Arena worlds (seeds 1, 1024, 2026) and recorded this threshold at the moment of each of the 3,335 bids that LLM managers made on clubs able to sell. The recorded threshold reproduces the engine's accept or reject decision for 99.9% of the bids (the rest are rounding at the boundary), so it is the true price.

Bids do not get closer to the true price over time. The typical distance between a bid and the threshold is about the same in the last five seasons as in the first five:

seasonsbidsmedian bid / thresholdmedian distance, as a factorbids within 10% of the thresholdcompleted purchases: median price paid / threshold
1–55001.621.9111%2.24
6–108361.812.188%2.56
11–159491.712.087%2.50
16–201,0501.111.8010%1.97

Across all bids, the rank correlation between season and distance from the threshold is −0.05-0.05. Per model and world (44 cases with enough bids), the median correlation is +0.01+0.01; bids get significantly closer in 10 cases and significantly further in 4. The error is mostly overpayment: completed purchases paid a median 2.3 times the lowest price the seller would have taken, and 87% paid more than 20% above it. Even the most careful buyer, claude-fable-5, paid a median 1.6 times the threshold.

Models do learn within a single negotiation: after a counter-offer, the next bid moves closer to the threshold 85% of the time. Models use the information in a counter-offer, but they do not carry what they learned about prices from one negotiation or season to the next.

So the claim that models do not learn where the acceptance boundary lies holds, now on a direct measurement. We will replace the offers-per-signing ratio in §4.2.1 with this measurement, and we answer the related question of whether the old ratio rewards overpaying in our reply to PqQj (Q4).

We will also shorten Section 4 and move the score and metric definitions before their first use, which we hope will make the paper easier to follow.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response to All Reviewers

Thank you to all three reviewers for the time and care in these reviews. All three valued the environment itself and its engine-computed score with no LLM judge, and s6yJ leans toward acceptance based on the benchmark contribution, provided the explanatory and release claims are tightened. The criticism is aimed at the analysis we built on top of the benchmark: whether the behavioral metrics measure what their names say, whether the explanations were tested or only argued, and whether one Arena world can carry the competition claims. We think that criticism was largely right, and most of our work during the discussion period went into it.

Every FM-Bench run replays bit-identically from its action log, so we could re-analyse the raw logs directly and run new experiments, with predictions written down before launch (in the supplementary archive, rebuttal/PREREGISTRATION.md). What came out of it:

  1. Reproducibility. All 45 solo runs replay with zero drift in two passes, and every Arena world reproduces its final state hash. Recomputing Appendix J from the raw logs found 8 factual errors and 7 aggregation inconsistencies, which we will correct.
  2. Robustness of the score. Under 2,000 random joint settings of the score's weights, discounts, asset valuation and revival penalty, claude-fable-5 stays first in 97.6% of them and the order of the scripted anchors never changes.
  3. A stronger blind baseline. A rule-based policy that sees only what the agents see scores 50.1 (the old heuristic: 17.1), still below 14 of the 15 models.
  4. Three more Arena worlds on the solo seeds. claude-fable-5 has the best mean rank over the three worlds (2.3) but wins only one of them, so we will report mean rank rather than a winner. The title rotates in every world.
  5. Price learning, measured directly. In the three Arena worlds we compared every bid with the seller's exact acceptance threshold. Bids do not get closer to it over 20 seasons, and completed purchases paid a median 2.3 times the lowest price the seller would have taken. Within a single negotiation, however, models do move toward the threshold after a counter-offer.
  6. Memory and horizon interventions. Removing the notebook lowered gpt-5.6-terra's score in both repeats (89.7 to 85.8, and 91.3 to 85.9). For the other models the effect was within run-to-run noise, which matches how differently the models use the notebook. With a 10-year horizon, models stop long-term investment at the new end. Rerunning the same model on the same seed can move its score by up to 16 points, which we will report.
  7. Withdrawn and narrowed claims. We withdraw the attribution of title rotation to a counter-adaptive market (identical scripted managers rotate the title too), we will state behavior–score links as associations, and we will describe the memory target 0.35 as a descriptive reference chosen after seeing seed 1.
  8. Release. We replaced the supplementary archive. It now holds the logs and manifests of every reported run, including the new ones, the solo transcripts, the analysis scripts, and a map from each table and figure to its runs. The anonymous repository holds the engine code only, which is why its README excludes run logs.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.