- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces FM-Bench, a purpose-built football-management simulator for evaluating long-horizon decision-making under hidden information, delayed consequences, a reactive market and competing objectives. The benchmark has 2 modes: Solo, where agents manage a club against scripted opponents, and Arena, where agents compete directly against each other. The benchmark evaluates a wide selection of LLMs and measures their performance in terms of a composite score as well as various capability/behavioral metrics. The benchmark is well defined and the code is well structured and extendable.
Critical strengths
- The benchmark follows established principles for reproducibility and includes seeded deterministic simulation, engine-computed scoring, reference policies and thorough logging. The analysis goes beyond just a leader board and includes a wide range of behavioural metrics that are relevant to the benchmark’s objectives.
- The paper is well written and structured. Introduced concepts are clearly defined in the main body and the appendix provides thorough documentation of most additional aspects of the benchmark and evaluation.
Critical weaknesses
- The behavioural metrics are not yet sufficiently validated as measures of the capabilities they name. Offers per completed signing does not distinguish accurate pricing from overpayment, while notebook similarity does not directly measure retention of useful information. The memory score additionally rewards proximity to an empirically selected similarity value, and other metrics were discarded based on their observed correlations. This makes it difficult to distinguish independently validated capability measures from exploratory descriptions chosen after examining outcomes. Ideally I would like analysis of the correlation and contribution of each of the capabilities to the overall score, as well as a more thorough statistical analysis of the hypotheses and documentation of experimental hypotheses setup.
- One 1 seed is used in the arena case which limits the conclusions that can be drawn. The paper acknowledges this and reports descriptive statistics and correlations, but the ability to do a more thorough statistical analysis is limited.
- Many figures contains text that is significantly smaller than latex "\small", and some figures in the appendix have formatting issues.
Questions for authors
- Why was consistency of correlation with the final score used to retain or reject capability metrics, and how does this selection affect the interpretation of the reported capability–score associations?
- Would it be possible to investigate the contribution of each of the capabilities to the overall score?
- A mix of sd, mean and mean (with min/max) is used across tables and figures. I think the min/max makes sense, but in some cases a 95CI interval might be more useful. Can the authors clarify their choice of reporting metrics and consider using a more consistent approach across tables?
- For the price discovery attribute, can this not be hacked by always overpaying for players? Does the game engine have the true (mean) market value of players? Would it not make more sense to define the price discovery attribute as how close the agent is to the true market value of players or some compound of this and transfer success rate?
- Given that models are now trained in persistent harness settings, was using their own harness and allowing them to manage context and how they want to memorise things themselves considered? This may improve results and be more realistic to how models are trained and used in practice.
- How was the memory-similarity target of 0.35 chosen and validated as measuring useful memory curation?
- Can you make it more clear what attributes FM-Bench tests that the other long horizon benchmarks do not? How does it compare to other benchmarks in terms of difficulty and what capabilities are being tested?
- Can it be made more clear that the benchmark is not based on the football manager game but an environment that covers a similar setting? I did not realise this until the end of my first reading.
- Does the game constitute a POMDP? If so maybe some formalization of the problem in terms of POMDP would be useful. Also a RL baseline would be useful to compare against. If not, what is the formalization of the problem?
8 comments
Response to Reviewer PqQj (2/5)
Q2. Contribution of each capability
Yes, and we did so in three ways. The short answer is that the capabilities overlap, and together they explain only a small part of the differences in score.
1. The capabilities are correlated with each other. Spearman correlations between the six capability metrics (15 models, each metric averaged over the three seeds):
| endgame cut | idle cash | renewal lead | offers per signing | distance from 0.35 | tokens | |
|---|---|---|---|---|---|---|
| endgame cut | 1 | 0.29 | −0.70 | 0.57 | −0.20 | 0.30 |
| idle cash | 1 | −0.35 | 0.03 | 0.01 | 0.00 | |
| renewal lead | 1 | −0.21 | −0.02 | 0.26 | ||
| offers per signing | 1 | −0.27 | 0.46 | |||
| distance from 0.35 | 1 | 0.03 | ||||
| tokens | 1 |
The endgame cut and renewal timing correlate at −0.70: models that renew early are largely the same models that wind investment down near the end. Attributing score to one of them separately is therefore not well defined with 15 models.
2. Predictive contribution. In leave-one-model-out prediction of the score (fit on 14 models, predict the 15th), the best single metric (endgame cut) explains 13% of the variance, the three headline metrics together 4%, and all six together do worse than predicting the mean (, overfitting). Behavioral metrics describe how models differ, but they do not add up to a decomposition of the score.
3. Interventional contribution. We tried two ways to change one capability and observe the score. (i) Prompting a model to renew early or to keep less idle cash did not change its behavior enough to pass our pre-registered manipulation check, so we cannot read off an effect. (ii) In a scripted policy we can switch each behavior on and off exactly. In heuristic_v2 (our stronger blind baseline), switching off any single rule, including the survival controller, changes the score over 13 seeds by an amount indistinguishable from zero (every paired 95% interval contains 0). In a script, at least, none of these behaviors has a measurable effect on its own.
Finally, the composite score itself has three channels, and across the 15 models the honors channel accounts for 74% of the variance in score, value added for 13% and squad value for 13%. Most of the separation between models therefore comes from on-pitch results, which the capability metrics reach only indirectly.
We will add these three analyses to Appendix J and state in §4.2.1 that the capabilities are a descriptive profile, not a decomposition of the score.
Q3. Reporting (sd, min/max, CI)
We agree that the mix is confusing. We will use one convention in every results table: the mean, the three seed values and a t-based 95% interval, which is wide with three seeds (for example [78.0, 103.9] for the top model). With three seeds and run-to-run variation of up to 16 points, only 11 of the 105 model pairs are separated by a seed-paired 95% interval, and we will say so next to Table 1. Model comparisons will use seed-paired differences, and Arena results will be mean ranks over the three matched-seed worlds.
Response to Reviewer PqQj (3/5)
Q4. Can price discovery be gamed; does the engine know true values?
We thank the reviewer for this sharp observation, which raises three questions that we answer in turn.
Does the engine have the true market value of players? Yes. Every player has a deterministic market value, and every seller has a deterministic accept threshold, which is the asking price adjusted by the seller's style and financial distress. Both are pure functions of the seed and are hidden from agents, who see only noisy scout bands. The oracle reads the threshold directly from the engine and bids exactly that, so each of its purchases closes on the first offer at the lowest price the seller would take (Appendix H). This is what fixes the reference at 1.0 offers per completed signing.
Can the metric be hacked by always overpaying? Taken in isolation, yes: a policy that bids far above the threshold would also close on the first offer. In the benchmark this cannot be exploited, for two reasons. First, the six capability metrics are post-hoc diagnostics. They do not enter the composite score and are never shown to the agent, so there is nothing for an agent to optimize against. Second, overpaying is penalized by the score that does rank the models. By Proposition A.3, buying a player of market value at price changes net worth by , so every credit paid above fair value lowers the value-added channel. The fee is also not burned but transferred to the selling club, a direct rival in the same league, which weakens the buyer's relative position and hence the honors channel. An always-overpay policy would look good on this one diagnostic and lose on the ranking.
Should the attribute be defined by closeness to true market value instead? We agree with the underlying point: offers per signing measures search cost, that is, how many attempts an agent needs to close a deal, and it does not by itself measure price efficiency. A low value is not evidence of accurate pricing. We will revise §4.2.1 and Appendix J to state this explicitly and remove the wording that read the statistic as a statement about the prices models bid. We keep the diagnostic as a measure of search cost because price efficiency is already what the value-added channel of the composite rewards, and we prefer each diagnostic to capture a behavior the score does not directly measure.
Q5. Letting models use their own harness
We considered it and kept one harness so that the score compares models rather than harnesses. Our new experiment shows the choice matters: removing the notebook lowered gpt-5.6-terra's score in both repeats, while other models were unaffected, and the models manage the same notebook in very different ways (see Q6). A model's own harness would mix these differences with harness engineering. In the Limitations section we will add: "We fix one harness so that scores compare models rather than harnesses; a track in which each model brings its own context and memory management is future work."
Q6. The memory target of 0.35
0.35 was chosen after seeing the seed-1 results, and the submission did not validate it. We have now checked what it does and does not measure.
It does track notebook quality. Two LLM judges (gpt-5.6-terra and claude-fable-5) scored 225 season-end notebooks against what the agent did the next season, without seeing the model name or the similarity value. Notebooks further from 0.35 received lower memory-quality scores from both judges (Spearman and , within models; the two judges agree with each other at 0.67). The effect comes mainly from append-only notebooks (similarity above 0.5), which scored worst.
It does not predict score. The distance does not predict the final score on held-out seeds, and re-fitting the target with each model held out in turn gives unstable targets (0.34 to 0.90) that do not predict the held-out models' scores.
And memory matters for some models but not others. Removing the notebook lowered gpt-5.6-terra's score in both repeats (89.7 to 85.8, and 91.3 to 85.9). gpt-5.6-terra writes at nearly every stop and almost never rewrites, so its notebook is a running record it depends on. claude-fable-5 keeps a short notebook it rewrites often and plays about as well without it; gpt-5.6-sol keeps an append-only archive, and capping it changed nothing; glm-5.2 writes far less often than the other models.
We will therefore keep the distance from 0.35 only as a descriptive measure of curation style, no longer call 0.35 the best value, and present memory as a per-model profile.
Response to Reviewer PqQj (1/5)
Thank you for a thorough review and for the many concrete suggestions. You found the paper well written and the benchmark built on established principles for reproducibility; your concerns are about validation and statistics, and we ran new analyses for each of them. Because several answers include full tables, this reply is split over a few comments.
W1 / Q1. Validation of the behavioral metrics; selection by observed correlation
We agree that choosing metrics by how consistently they correlate with the score makes the results exploratory, and we will say so. Our reason for the rule was that a metric whose sign flips between seeds cannot describe a stable capability, but applying it after looking at the outcome selects for metrics that happen to agree on three seeds. Below we (a) re-test all nine candidates under one procedure, so the dropped metrics are judged exactly like the kept ones, (b) report what each metric does and does not measure, and (c) describe the hypothesis documentation for the new experiments.
(a) All nine candidates, one procedure. For each metric we computed the Spearman correlation with the final score on each seed separately (15 models per seed), a permutation p-value that shuffles model labels within each seed (20,000 permutations), and corrected for multiple comparisons over all nine with Holm (family-wise) and Benjamini–Hochberg (false discovery rate). "Within model" is the correlation after removing each model's mean, that is, whether a model's better seeds are the ones where it shows more of the behavior.
Two metrics survive correction: idle cash and renewal lead time. The endgame cut keeps its sign on every seed but does not survive correction. Of the three dropped metrics, tactic reversals is as strong as the endgame cut before correction, so the earlier selection was not conservative in a useful way: it kept two metrics (offers per signing, distance from 0.35) that are weaker than one it dropped. Idle cash needs a caveat: much of its correlation comes from the score's own discount on hoarded cash (without the discount the per-seed correlations are −0.24, −0.12, −0.17). Renewal lead time is therefore the one behavior with a clean, corrected association with score.
As a result, we will present the capability profile as exploratory description, report this full table (including the dropped metrics) in Appendix J, and limit association claims to renewal timing, with idle cash flagged as coupled to the score.
(b) What the two questioned metrics measure. Offers per completed signing does not separate accurate pricing from overpayment, as you say. We now measure pricing directly against the engine's exact acceptance threshold; see Q4 and our reply to Reviewer 2mbG (W6). Notebook similarity does not measure retention of useful information directly. To test what it does capture, two blinded LLM judges scored 225 season-end notebooks on whether they held the information the agent needed for the next season; distance from 0.35 tracks lower judged quality (−0.30 and −0.32), mainly by flagging append-only notebooks. We will keep it only as a description of that pattern. The notebook intervention (removing or capping it) shows that whether memory matters depends on the model; see Q6.
(c) Hypothesis documentation. For every new experiment in this rebuttal, the hypotheses, the treated and control runs, the decision rule (an effect must exceed twice the same-seed run-to-run SD and replicate in a repeat), and the manipulation checks were written down before launch (in the supplementary archive,
rebuttal/PREREGISTRATION.md); amendments made after launch are dated and listed in the same file. We will add this protocol to the appendix.