acceptodds
Under review as a conference paper at ICLR 2027

FromPitch2Board: Benchmarking LLM Agents in Long-Horizon Football Management

Abstract

Long-horizon agent benchmarks typically report how far an agent progresses, but do not identify whether its performance comes from the foundation model, scaffold, responsibility scope, match-control granularity, or horizon. We introduce FromPitch2Board, a deterministic football-management benchmark that studies five configurable factors through controlled comparisons on a single simulator, using paired seeds and a frozen calibration. We evaluate four foundation models and four agent scaffolds. In the Model Track, Coach points Z-scores span 0.19, while Manager points Z-scores span 0.68, with GPT-5.6 showing a sharp rise in passivity under responsibility expansion. Its responsibility ladder rises from 46.1 to 58.1 points with recruitment, then falls to 46.8 under full management, localizing the regression to the final responsibility boundary. Across that boundary, its skipped-decision rate rises from 1.1% to 57.9%. Within the Flash–Pro pair crossed across every scaffold, scaffold choice changes Manager points Z-scores by up to 0.48 relative to the fixed stateless scaffold. The 3Y cohort shows a directional reversal in mean ranking between years one and three, while a selected Claude Code+Pro configuration peaks in year three and remains below that peak, showing that responsibility scope and horizon expose behavior changes that a single headline score conceals.

Primary area:
datasets and benchmarks
Keywords:
LLM agents, agent evaluation, benchmarks, functional composition, long-horizon decision making, football management

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_aDnrOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents a deterministic football-management benchmark built on a fork of Openfoot Manager. It varies five factors: model, scaffold, responsibility scope, match-control granularity, and horizon. Results are scored with paired seeds and Z-scores normalized to a greedy reference calibrated on 100 seeds.

Key findings: (i) GPT-5.6 loses its recruitment gain at the Manager boundary while its skipped-decision rate rises to 58%; (ii) scaffold choice changes Manager Z by up to 0.48, more than the Flash/Pro model swap; (iii) a 3-year ranking reversal; and (iv) a 10-year trajectory that peaks at year 3.

Critical strengths
  1. Varying responsibility scope within the same world is a genuinely informative design. The Recruiter rung localizes where GPT-5.6’s regression happens, and the behavioral diagnostics go beyond endpoint scores.
  2. Deterministic replay, paired seeds, frozen calibration, and treating the scaffold as an experimental factor are sound benchmark-design choices.
Critical weaknesses
  1. The temporal-persistence findings appear to be artifacts of the benchmark’s own design.
  • Appendix A.2 states that the action set contains no contract renewal and that contract end dates are not observable. Players therefore leave automatically on expiry.
  • The 10-year decline coincides exactly with expiry-driven squad attrition, and squads below 11 degrade silently.
  • This measures an agent’s inability to manage something the interface does not let it see or act on, not long-horizon competence.
  • The 10-year result is also a single preselected configuration, and the 3-year “reversal” rests on year-one differences that are not significant.
  1. The headline composition finding is a single-model effect whose cause is not isolated.
  • GPT-5.6’s matchday skip rate rises from 0.2% to 58% on decisions that are identical in scope, which points to a prompt-, observation-length-, or serving-related cause rather than to functional composition.
  • No observation-length, context, or reasoning-token analysis is provided.
  • Results use 8 seeds per scenario with one hosted-model run per seed, and the model–scaffold cross covers only two models from one family, so the attribution claims are limited.
Questions for authors
  1. Why is contract renewal excluded? How do the 10-year results change if renewal is added or contract end dates are exposed?
  2. What are the observation token lengths per decision for Recruiter vs. Manager? Does truncating the Manager observation to Recruiter length restore GPT-5.6’s matchday participation?
  3. Were GPT-5.6 and Opus 5 called through the official API with documented reasoning settings? Can you replicate the GPT-5.6 ladder via the official endpoint?
  4. How sensitive are the conclusions to the choice of reference policy used for Z normalization, given that the greedy reference sits near Z ≈ 0?
Reviewer_mWvnOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper proposes to use soccer manager simuator as LLM agent benchmark. Using three scopes of coach, recruiter manager the action space increaes and various models can be studied. LLM agents consist of model and harness, the impact of both in analysed. A custom harness is considered alongside standard harnesses like Pi.

Critical strengths
  • Soccer management games are well studied category of economocy simulations. With increasing usage of LLM agents in everyday settings, using football management simulators as benchmark is a very timely idea. Wrapping the Openfoot Manager for LLM agents and evaluating model and harness combinations is interesting.
  • Natural ladder to increase the scope of the roles.
Critical weaknesses
  • Why is memory disabled in your harness. You state partially observable. Also the proposed harness isn't particularly performant.
  • Unclear how responsibities of the player outside the agents scope are handeled (see Q1)
  • Simulator is seeded and determinism described, but llm sampling details are missing (as is ablation for external model api that can hardly be seeded)
  • Squad size reduction feels odd. Currently unclear if issue with the setup or LLM capabilities.
Questions for authors
  • In the coach and recruiter case, how are the missing action types handeled? By the default AI as for other players? Are there hidden coordination issues
  • The squad size is decreasing over time, contracts expire. Please double check that you can a human can successfully play without unintended squad size decrease and potentially add the option to extend contracts.
  • Why disable session continuity in multi year. It seems particularly crucial there
  • Which parts of the harness are actually contributing to the performance?
  • Why not handle all decisions at same decision points sequentially. or in parallel instead of prioritising them?
Reviewer_c9VBOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper introduces FromPitch2Board, a deterministic, agent-native benchmark based on a simulation of football club management to evaluate LLM agents on long-horizon tasks. It systematically examines how agent performance is influenced by five configurable factors: foundation model, agent scaffold, responsibility, match-control granularity, and time horizon.

Contributions:

  1. A multi-factor compositional and temporal benchmark framework that isolates attribution in long-horizon agent tasks.
  2. Functional-composition diagnostics that uncover non-monotonic performance changes
  3. Temporal-persistence diagnostics revealing directional ranking reversals in multi-season cohorts and long-horizon performance degradation.
Critical strengths
  1. The paper systematically varies responsibility scope, match-control granularity, and horizon while crossing models and scaffolds. As such, it addresses a blind spot in current agent literature: the conflation of foundation model capability with interface/scaffold effects.

  2. Long-horizon agent simulations are noisy and difficult to reproduce. The paper's methodological rigor makes the observed behavioral signatures credible (e.g., GPT-5.6's shift toward passivity).

Critical weaknesses
  1. The paper is poorly written and organized, thus increasing unnecessarily the cognitive load.. Rather than providing a high-level conceptual overview, the abstract abruptly dives into granular jargon before defining the metrics or explaining the core system. The introduction continues in this spirit: it lacks a clean, traditional roadmap that makes the paper extremely hard to read,
  2. The paper features extensive experiments, the fully crossed Model-Scaffold matrix is restricted to the Flash-Pro model pair.

Candidate Alternative Abstract

Evaluating Large Language Model (LLM) agents on long-horizon tasks typically measures how far an agent progresses, but fails to isolate whether performance stems from the foundation model, the agent scaffold, responsibility scope, decision granularity, or time horizon. To address this limitation, we introduce FromPitch2Board, a deterministic, simulation-based benchmark for professional football management that studies these five configurable factors under controlled comparisons, paired seeds, and frozen calibrations. By systematically varying management responsibilities from a narrow coaching role to full club management across multiple seasons, our evaluation demonstrates that agent competence is non-monotonic: expanding responsibilities can trigger severe behavioral shifts—such as extreme passivity rather than invalid actions—that single headline scores conceal. Furthermore, multi-season horizons expose directional rankings reversals between early and later years. By crossing foundation models with diverse agent scaffolds, FromPitch2Board provides a rigorous diagnostic framework to attribute agent performance and behavior to specific architectural and task design choices.

Questions for authors

n/a

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.