- Rating
- 2
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper presents a deterministic football-management benchmark built on a fork of Openfoot Manager. It varies five factors: model, scaffold, responsibility scope, match-control granularity, and horizon. Results are scored with paired seeds and Z-scores normalized to a greedy reference calibrated on 100 seeds.
Key findings: (i) GPT-5.6 loses its recruitment gain at the Manager boundary while its skipped-decision rate rises to 58%; (ii) scaffold choice changes Manager Z by up to 0.48, more than the Flash/Pro model swap; (iii) a 3-year ranking reversal; and (iv) a 10-year trajectory that peaks at year 3.
Critical strengths
- Varying responsibility scope within the same world is a genuinely informative design. The Recruiter rung localizes where GPT-5.6’s regression happens, and the behavioral diagnostics go beyond endpoint scores.
- Deterministic replay, paired seeds, frozen calibration, and treating the scaffold as an experimental factor are sound benchmark-design choices.
Critical weaknesses
- The temporal-persistence findings appear to be artifacts of the benchmark’s own design.
- Appendix A.2 states that the action set contains no contract renewal and that contract end dates are not observable. Players therefore leave automatically on expiry.
- The 10-year decline coincides exactly with expiry-driven squad attrition, and squads below 11 degrade silently.
- This measures an agent’s inability to manage something the interface does not let it see or act on, not long-horizon competence.
- The 10-year result is also a single preselected configuration, and the 3-year “reversal” rests on year-one differences that are not significant.
- The headline composition finding is a single-model effect whose cause is not isolated.
- GPT-5.6’s matchday skip rate rises from 0.2% to 58% on decisions that are identical in scope, which points to a prompt-, observation-length-, or serving-related cause rather than to functional composition.
- No observation-length, context, or reasoning-token analysis is provided.
- Results use 8 seeds per scenario with one hosted-model run per seed, and the model–scaffold cross covers only two models from one family, so the attribution claims are limited.
Questions for authors
- Why is contract renewal excluded? How do the 10-year results change if renewal is added or contract end dates are exposed?
- What are the observation token lengths per decision for Recruiter vs. Manager? Does truncating the Manager observation to Recruiter length restore GPT-5.6’s matchday participation?
- Were GPT-5.6 and Opus 5 called through the official API with documented reasoning settings? Can you replicate the GPT-5.6 ladder via the official endpoint?
- How sensitive are the conclusions to the choice of reference policy used for Z normalization, given that the greedy reference sits near Z ≈ 0?