acceptodds
Under review as a conference paper at ICLR 2027

From Clues to Closure: Benchmarking Ultra-Long-Horizon Autonomous Computer Use through Evidence-Driven Gameplay

Abstract

As large language models gain strong GUI skills, computer use is becoming easy for them: recent models such as GPT-6 Astra and Claude Opus 5.5 saturate short-horizon GUI tasks. What still stands between them and real work is autonomy over long horizons, in tasks that set a goal but give no path and run long enough that the agent must maintain its own task state. We show that evidence-driven investigation can be used to build environments that demand this autonomy, and we state seven requirements on a task's evidence that make memory and multi-path exploration unavoidable. We introduce Clue2Closure, an ultra-long-horizon computer-use benchmark of three investigation games about one family: the commercial detective game The Roottrees are Dead, its expansion, and Scattered Roots, a new game that our pipeline builds to the seven requirements by splitting every scored fact into jointly required fragments and hiding them among thousands of documents. Six frontier agents score between 26% and 100% on the original games, where GPT-6 Astra fills all 333 blanks in 2,207 steps, but Scattered Roots remains unsolved: within 3,000 steps, no agent holds more than 19 of its 312 points. Their trajectories show that agents fail to act on what they already know, revise errors by trial and error, and get lost among open paths.

Primary area:
datasets and benchmarks
Keywords:
computer-use agents, long-horizon evaluation, benchmark, autonomy, agent memory, exploration, GUI agents, synthetic environments

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_GKZNOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces Clue2Closure, a benchmark for evaluating ultra-long-horizon autonomous computer use through evidence-driven investigation games. It proposes seven requirements for evidence structure and constructs a new game, Scattered Roots, through a synthesis pipeline. Experiments with frontier agents examine task completion and failure patterns related to memory management and multi-path exploration.

Critical strengths
  1. Benchmark Design. The paper considers memory and exploration challenges in long-horizon computer-use tasks. The seven evidence-structure requirements provide a structured approach to benchmark construction.
  2. Trajectory Analysis. The paper evaluates multiple frontier agents and analyzes their behavior, including delayed use of evidence, incorrect revisions, and exploration stalls.
Critical weaknesses
  1. Limited Benchmark Validation. Scattered Roots is extremely difficult for all evaluated agents, with the highest score reaching only 19 out of 312 points. This raises concerns about its ability to meaningfully distinguish agent capabilities. Moreover, no human evaluation is provided for the new game, and the seven design requirements are not individually validated through controlled ablations. It remains unclear whether the observed failures primarily reflect the intended autonomy challenges or excessive task difficulty.
  2. Concerns about Experimental Reliability and Generalization. Each configuration is evaluated only once, without run-to-run variance. Some evaluations involve additional in-game aids or operator intervention, and parts of Scattered Roots were finalized during the runs (Appendix A.4 and B.1). These factors may affect the reliability and comparability of the reported results. Furthermore, the benchmark focuses on investigation games within a single family-tree setting, limiting evidence of broader applicability.
Questions for authors
  1. Could the authors provide human evaluations or additional difficulty levels to validate the solvability and discriminative ability of Scattered Roots?
  2. Could the authors conduct controlled experiments to isolate the contributions of the seven evidence-structure requirements?
  3. How do the authors account for the effects of in-game aids, operator intervention, and changes made during evaluation?
5 comments
AuthorsOpenReview ↗

Response to Reviewer GKZN (1of4)

We thank Reviewer GKZN for the detailed review and constructive questions, and we appreciate your recognition of our seven evidence-structure requirements and trajectory analysis. Below, we address each of your concerns.

Limited Benchmark Validation. Scattered Roots is extremely difficult for all evaluated agents, with the highest score reaching only 19 out of 312 points. This raises concerns about its ability to meaningfully distinguish agent capabilities.

We agree that the scores in our submission (at most 19 of 312 points) did not separate agents well. What we changed, and the new scores that now separate the models, are in the General Response (Sec. 1–3).

Limited Benchmark Validation. ...Moreover, no human evaluation is provided for the new game...

Q1: Could the authors provide human evaluations or additional difficulty levels to validate the solvability and discriminative ability of Scattered Roots?

We do not have human play data for SR; solvability is covered in General Response B, and discriminative ability in our answer above. Beyond these:

  • Additional difficulty levels: The two hint experiments in General Response B also act as difficulty levels on the same game: GPT-6 Astra goes from 27.6–28.2% with no hints to 89.1% with ideas-only hints at 13.5 SR h, and to 100% in 1.05 SR h with a walkthrough (Appendix A.10).

  • Human reference on the original game: 38.5% of Steam players complete the original game's base tree, in a median of 10.5 hours (Appendix D.2).

AuthorsOpenReview ↗

Response to Reviewer GKZN (2of4)

Limited Benchmark Validation. ...the seven design requirements are not individually validated through controlled ablations.

Q2: Could the authors conduct controlled experiments to isolate the contributions of the seven evidence-structure requirements?

To clarify first, the main focus of the paper is to propose an evidence-based benchmark for ultra-long-horizon computer use. The seven design requirements are not the main novelty here, and only serve to illustrate how we turn an investigative game into a benchmark that exposes the challenges of ultra-long-horizon CUA tasks. We have revised the phrasing in the paper to avoid any misunderstanding of the paper's emphasis.

As for the ablation, the seven design requirements are not meant for single-factor ablation, because they work together as a whole to create the long-horizon challenge. For example, if we take out unavoidable misdirection (R4), then persistent uncertainty (R6) would also disappear. In addition, ablating each requirement would require building and evaluating seven additional games, which we cannot complete due to resource constraints.

Instead, we validate each requirement in two ways:

  • Individual metrics and check scripts (Table 1, Sec. 3.4; Table 4, Appendix A.1): Every requirement has a concrete metric, a threshold, and a check script with a mutation self-test. Tested locally, single steps are easy: our single-step probe with Claude Sonnet 5 (a previous-generation model) passes 82.9% with memory, and a designed misdirection is corrected in 6/6 probes once its refuting evidence is read (Table 4).

  • Matching failure modes in agent trajectories: We find a clear failure mode corresponding to each requirement in the agents' trajectories, showing how each requirement contributes to the overall challenge:

RequirementObserved Failure Mode in Agent Trajectories
R1: Open threadsAn agent notices a married-surname defect on a card but never returns to fix it; another carries the same "to post" list across six compactions without posting it.
R2: No master keyAn agent spends 14 SR h trying to open one locked archive that is not a unique route to any point it missed.
R3: Long-span retrievalA newspaper issue opened at 6 SR h is skipped at 20 SR h; a founding year found at 3.9 SR h is forgotten in compaction and "rediscovered" at 13.9 SR h.
R4: MisdirectionCards keep common name spellings on the board even after the agent reads the birth records giving the required birth-name spelling.
R5: Where to lookTwo months of the wrong newspaper are scanned even though the wedding date is known; a lock page's pointer is ignored; an obituary link is never clicked.
R6: Persistent uncertaintyAgents hold back solved people under self-imposed rules: "so I left it", "Not added (no birth name confirmed)", or "no card until certain" held for 11 SR h.
R7: Memory loadUp to 37% of SR time goes to re-reading the agent's own local files, and context is rebuilt by grepping old outputs after compaction.

Interventions on the same game: Although we cannot build seven separate games, we can supply, on the same game, part of what the long horizon withholds (e.g., the needed memory in single-step probes, or direction in Experiment A). These are not single-requirement ablations, but each time performance rises sharply, consistent with the long horizon, rather than single-step difficulty, being the main obstacle:

What we giveMainly easesWithout itWith it
The memory a player would hold at that step (single-step probe, Claude Sonnet 5)Recalling evidence read long ago (R3)21.5% of steps82.9% of steps
Hints with the open questions and the reasoning behind clues, without answers (GPT-6 Astra, full game)Tracking open threads, deciding where to look (R1, R5); also eases some single steps27.6–28.2% at 13.5 SR h89.1% at 13.5 SR h (one run)

Row 1: % of single steps answered correct and certain (with memory: in ≥80% of attempts; without: first answer) (Sec. 5). Row 2: % of the 312 scored points, Experiment A in General Response B (Appendix A.10).

AuthorsOpenReview ↗

Response to Reviewer GKZN (3of4)

Limited Benchmark Validation. ...It remains unclear whether the observed failures primarily reflect the intended autonomy challenges or excessive task difficulty.

We added a section on exactly this question (Sec. 5, "Decoupling Long-Horizon Autonomy from Short-Task Difficulty"): each short step is easy on its own and needs little domain knowledge, so what remains hard is the long horizon. Its main test is a local single-step probe (Sec. 5; Appendix C.2). For each point closure on the reference route (181 questions covering 243 points), we give a probe model (Claude Sonnet 5, a previous-generation model) the memory a player would already hold at that step and ask it to make the single closing inference:

Probe / Split MeasureResult
Single-step probe (Claude Sonnet 5) with the memory a player would hold82.9% pass (150 / 181)
Single-step probe without that memory (correct and certain; correct only)21.5% (34.3%)
Local pass rate on points no GPT-6 Astra run solved vs. solved points82.4% vs. 82.2%
Solve rate in Astra runs 1–3 on structurally easy (≤6 fragments) vs. hard (>6 fragments) points42.4% vs. 29.7%

This shows that individual steps are not too hard, even for a previous-generation model. Instead, agents fail because they lose track of evidence over the long run, fail to revise earlier mistakes, or fail to post what they already found. We also added Appendix F, which groups failure case studies into different categories, plus (F.4) showing that every run at the default window stopped because its score plateaued while solvable points were still right in front of it.

Concerns about Experimental Reliability and Generalization. Each configuration is evaluated only once, without run-to-run variance.

Repeated runs of GPT-6 Astra and Claude Opus 5.5, including how many points are solved in all three runs, are in the General Response (Sec. 4); because the solved points vary, we analyse single points only across runs. The other models ran once because a run is costly, as Agents' Last Exam also repeats only some configurations.

References

Reviewer_46hxOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces a long-horizon computer-use benchmark for language models that emphasizes three core abilities: manage large amounts of information it is given, revising wrong beliefs when encountering a refutation, and strategically choosing paths to explore and push towards completion of the task. The benchmark is based on an evidence-based investigation game. Part of the benchmark exactly recreates the game in a GUI run in a sandbox, and a novel part of it consists of a more difficult variation of the game with the same structure. Evaluation of frontier models on the benchmark shows their struggle to solve the task even with 3000 steps of execution.

Critical strengths

I found the paper to be fairly well-written and easy to follow. The general topic is significant, and I appreciate the extra steps to investigate the failure modes. Here are the main strengths of the paper in my view:

  • The benchmark targets GUI computer-use capability of the models. This is a critical interface for useful AI agents, and this paper contributes to non-saturated benchmarks in this area.
  • The design principles of the benchmark are very clearly described. The seven requirements for the authors' notion of autonomy are listed, and their presence is quantified in Table 1.
  • The behavior and capability of models are investigated for each of the three aspects of autonomy that is introduced (Section 5). This provides a more actionable insights to improve the models compared to a single performance metric reported.
Critical weaknesses

The main areas that the paper could improve upon, in my opinion, are the following:

Lack of reference solution. It is not clear how many steps is it actually needed to solve Scattered Roots. The performance of the models with 3000 steps is very low, but it is not settled how much this should be attributed to models' capability rather than the budget of 3000 steps. I couldn't find a score vs. steps plot for Scattered Roots and the only result appears to be Table 2. It is very important to see if the model's score is still improving at step 3000 or it has been stalled. If the score is still improving, it will suggest that the low performance is due to the limited budget and unavoidable.

Discussions on relation to more realistic use cases. At a high level, GUI interfaces and long-horizon are real challenges the models face in real-world tasks. However, these characteristics alone do not guarantee a meaningful evaluation of the model for the real world. As an example, one can simply create a graph search problem with a GUI interface that is extremely long-horizon and unsolvable by the models. Improving on this benchmark requires branch selection biases that are aligned with the position of goal node. This, however, is irrelevant to any realistic performance metric. I would really appreciate more discussions on why each of the seven capabilities are necessary in realistic use cases. Ideally, this should be examples on agentic tasks that include these challenges. The current draft doesn't address this very well and only assumes the introduced definition of autonomy is correct.

Fine-grained performance metrics for each capability. In Section 5, the paper nicely investigates the behavior of the model in each specific area. For example, the number of steps it takes a model to solve a task once all of the required knowledge is gathered tests the information retrieval capability of the model. I would really like it, if through similar ideas, additional sub-metrics were being reported that each focus on some capability.

More explanations of the game in the writing. In some places in the text, understanding some phrases required more knowledge of the game that was not provided. I suggest the authors include explanations for terms like "open profiles", "filled profile", etc. I could have a guess, but more clarity would help.

Questions for authors

Please see weaknesses.

4 comments
AuthorsOpenReview ↗

Response to Reviewer 46hx (1of4)

We sincerely thank Reviewer 46hx for the positive assessment and encouraging feedback! We appreciate your recognition that our paper is clear and easy to follow, and that our failure-mode analysis (Section 6) offers actionable insights beyond a single score. Below, we address each of your suggestions.

Lack of reference solution. It is not clear how many steps is it actually needed to solve Scattered Roots. The performance of the models with 3000 steps is very low, but it is not settled how much this should be attributed to models' capability rather than the budget of 3000 steps. I couldn't find a score vs. steps plot for Scattered Roots and the only result appears to be Table 2. It is very important to see if the model's score is still improving at step 3000 or it has been stalled. If the score is still improving, it will suggest that the low performance is due to the limited budget and unavoidable.

Reference solution & budget ratio: The reference solutions (a scripted reference player and two guided experiments) are in General Response B. In the units of your question, a step-by-step walkthrough needs only 111 model calls, 4% of the 3,000 steps, while our unhinted runs may use up to 27 SR h, about 26 times what the walkthrough needs, so the budget is not what limits the score.

Score vs. steps plot and table: We added Figure 4 (Sec. 4.3) plotting score against model calls and SR hours, as well as Table 6 (Appendix A.5) and Appendix F.4:

RunScore at 13.5 SR hScore at 3,000th callScore at stop (SR h)
GPT-6 Astra, run 127.932.732.7 (20.5)
GPT-6 Astra, run 228.2not reached28.8 (16.5)
GPT-6 Astra, run 327.636.239.7 (22.0)
Claude Fable 5.224.7not reached26.3 (17.6)
Claude Opus 5.5, run 19.99.99.9 (14.7)
Claude Opus 5.5, run 27.48.08.0 (17.1)
Claude Opus 5.5, run 39.311.515.7 (20.1)
Claude Opus 54.25.45.4 (18.0)
GPT-5.6 Sol2.62.62.6 (16.5)
Claude Sonnet 5.50.6not reached0.6 (13.5, min. runtime)
GPT-5.6-luna0.0not reached0.0 (13.5, min. runtime)
GPT-6 Astra, 475K36.2not reached44.6 (20.5)
GPT-6.1 Sol, 475K30.4not reached31.1 (17.1)
Claude Opus 5.5, 475K8.7not reached8.7 (17.0)
Claude Sonnet 5.5, 475K1.6not reached1.6 (16.6)
Claude Haiku 5.5, 475K0.0not reached0.0 (16.5)
GPT-5.6-luna, 475K0.00.00.0 (13.5, min. runtime)

Scores in % of 312. Rows marked 475K use a 475K context window (Appendix A.9); the others use the default window.

Most runs had stopped improving by step 3,000: at the default window, 5 of the 7 runs that reached their 3,000th call already had their final score there (e.g., GPT-6 Astra run 1: 32.7%). Two kept gaining for a while (Claude Opus 5.5 run 3: 11.5% → 15.7%; GPT-6 Astra run 3: 36.2% → 39.7%), but every run eventually plateaued well before the 27 SR h cap (Appendix F.4).

AuthorsOpenReview ↗

Response to Reviewer 46hx (2of4)

How we tell "the game is simply too long" apart from "the game poses our intended challenge": We totally agree with the reviewer's logic. We use three criteria to distinguish the two scenarios:

Expected if the game is simply too longExpected if the game poses our desired challenge (observed)
More time → steady score gain.With more time, the score plateaus (no run reached the 27 SR h cap; every run stopped on the plateau rule or at the 13.5 SR h minimum; Appendix F.4).
The agent makes clear progress toward new evidence/points, with just too many actions left to go.The agent has very low productivity (e.g., repeatedly searching what it already knows, or blindly enumerating the action space even when the clue is in its memory).
A large amount of material is never reached.The missing evidence is reachable and the agent's own leads point to it, but it neither follows them nor turns what it holds into points (at Astra run 1's stop, 20 points had all their evidence pages read and still did not score; Table 17).

We provide one case study for each of the last two rows from GPT-6 Astra, run 1 (more in Appendix F.1–F.4):

Case 1 (Low productivity): At the time, GPT-6 Astra is looking for the wedding of a couple. It already read a marriage license with the exact wedding date, and wrote: "The license confirms a [date] marriage between [the couple]. I'm checking wedding notices around that date for the bride's full name." The right way here is to read the wedding announcements in the few days after that date (3–5 issue listings). Instead, it listed every daily issue of one newspaper for two full months (61 issues, including the month before the wedding) and searched the headlines for the couple's names. The two wedding pages it found held only "unrelated couples", so it queued another newspaper, which has no relevant page in that year. Here, the agent not only fails to use the exact date it got earlier, but also chooses a highly inefficient strategy.

Case 2 (Reachable evidence not used): By the end of run 1, GPT-6 Astra conducted 7,438 game operations (7.7× the reference route) and opened every chapter by 8.3 SR h (so no missing evidence sat behind an unopened lock). When it stopped, it had read 54% of all fragments overall, and for 88% of its unsolved points it had read at least one fragment. For 34 unsolved points, the leads it had already read were enough to close them. Yet in its last 1.6 SR h, GPT-6 Astra opened none of those pages, and from 16 SR h on it spent 69%–82% of its operations listing newspaper indexes (Appendix F.3):

SR h0–22–44–66–88–1010–1212–1414–1616–1818–2020–20.5
Index listing23%30%33%15%28%36%33%35%69%76%82%
New page reads34%29%29%67%26%29%18%22%10%7%6%
Board writes112413638176781230
Score gained (%)+1.6+6.1+2.2+1.3+4.8+5.8+6.1+1.0+3.5*+0.30.0

*The 16–18 h gain came from half an hour of posting known facts, not from listing.

AuthorsOpenReview ↗

Response to Reviewer 46hx (3of4)

The bad strategies above are not accidental: as the game continues, the agent spends much more time on useless actions or repeats what it already knows (98.6% of the newspaper dates that Astra listed in its plateau carry no designed content; similarly, GPT-5.6 Sol's index lookups hit a designed date 77% of the time while gaining, vs. 2.6% while stalled). This happens even though the brief explicitly tells agents that every response and operation takes game time, sets a fixed deadline, and states that "a correct card counts for every minute it is on the tree".

Finally, we believe efficiency is also an important part of the evaluation, and major CUA benchmarks also evaluate under bounded budgets (OSWorld 2.0 evaluates at 150-, 300- and 500-step budgets; Agents' Last Exam sets a 5-hour cap per run on tasks built from expert projects that took days or weeks). What we want to avoid is making a task long through repeated, easy action sequences (as in OS-Marathon, Wu et al., 2026); instead, following Gym-Anything (Aggarwal et al., 2026), difficulty in SR comes from multi-step reasoning and long-horizon planning.

Discussions on relation to more realistic use cases. At a high level, GUI interfaces and long-horizon are real challenges the models face in real-world tasks. However, these characteristics alone do not guarantee a meaningful evaluation of the model for the real world. As an example, one can simply create a graph search problem with a GUI interface that is extremely long-horizon and unsolvable by the models. Improving on this benchmark requires branch selection biases that are aligned with the position of goal node. This, however, is irrelevant to any realistic performance metric. I would really appreciate more discussions on why each of the seven capabilities are necessary in realistic use cases. Ideally, this should be examples on agentic tasks that include these challenges. The current draft doesn't address this very well and only assumes the introduced definition of autonomy is correct.

Relation to realistic use cases: We agree that a GUI and a long horizon alone do not make an evaluation meaningful. Why the abilities SR tests matter in real computer-use work, with a real-job example, is in General Response A; below we explain why SR is not a simple graph search problem.

References

AuthorsOpenReview ↗

Response to Reviewer 46hx (4of4)

Why Clue2Closure is not a simple graph search problem: This is a very interesting point. From our experience, if a task is only a long-horizon graph search problem, then given enough budget, a SOTA agent can probably solve it. A key question is whether the problem contains a structural difficulty that makes the model plateau; otherwise it is foreseeable that the model will keep making steady progress. Clue2Closure is not a simple graph search problem, but a combination of abilities common in the real world (information seeking, logical reasoning, using in-game apps, and planning).

One action that may be considered graph search is browsing. However, keyword choice is not a search the agent must brute-force: the hard part is knowing whom or what to look up, which comes from evidence read earlier. A higher-level action that looks like search is "deciding which direction to go," yet that decision is based on what the player learned from earlier context, and the game does not encourage agents to brute-force it.

The game also does not contain many "dead-end" branches designed to waste search; instead, most branches are useful, and the information on each branch is interconnected (each point needs several scattered fragments plus birth-name reconciliation), so agents must make connections to gain points. Playing the game like a graph search problem happens mainly when the agent itself is weak: GPT-5.6-luna spent nearly half of its 10,431 operations paging microfilm issue by issue and scored 0%, whereas a smarter model can plan its schedule from clues.

Fine-grained performance metrics for each capability. In Section 5, the paper nicely investigates the behavior of the model in each specific area. For example, the number of steps it takes a model to solve a task once all of the required knowledge is gathered tests the information retrieval capability of the model. I would really like it, if through similar ideas, additional sub-metrics were being reported that each focus on some capability.

We thank the reviewer for this suggestion. We added Table 17 (Appendix C.3) reporting five per-capability sub-metrics across all eight SR runs (values at the stop, with plateau-start or rising-phase values in brackets):

RunAll evidence read, not scored (Acting)Right content, name fails rule (Revising)Lead read, not followed (Choosing paths)Index lookups hitting a designed date (Choosing paths)Board writes % of ops (Acting)
Astra, run 120 (19)66 (65)42 (40)1.4% (3.5%)0.1% (14.5%)
Astra, run 214 (10)70 (67)34 (34)4.0% (11.5%)5.1% (4.0%)
Fable 5.217 (13)62 (52)53 (49)5.1% (11.5%)1.8% (6.7%)
Opus 5.55 (3)29 (17)57 (29)27.7% (30.6%)1.7% (3.9%)
Opus 52 (2)21 (21)47 (45)36.4% (55.6%)0.6% (7.4%)
GPT-5.6 Sol3 (3)10 (7)63 (63)2.6% (76.9%)7.8% (2.0%)
Sonnet 5.50 (0)16 (3)40 (38)0.0% (2.2%)15.6% (1.4%)
GPT-5.6-luna0 (–)0 (–)39 (–)0.2% (–)0.2% (–)

We found that different models stall on different capabilities:

  • The three highest-scoring runs (Astra ×2, Fable 5.2) stall mainly at revising errors (62–70 points whose content is right but whose name fails the birth-name rule) and acting on knowledge (14–20 points with all evidence read).

  • GPT-5.6 Sol and Sonnet 5.5 stall mainly at choosing paths: at most 2.6% of their index lookups hit a designed date (GPT-5.6 Sol dropped from 76.9% while gaining to 2.6% in the plateau).

  • The two Opus runs sit in between: their lookups still hit designed dates 27.7%–36.4% of the time, but they leave 47–57 read leads unfollowed.

More explanations of the game in the writing. In some places in the text, understanding some phrases required more knowledge of the game that was not provided. I suggest the authors include explanations for terms like "open profiles", "filled profile", etc. I could have a guess, but more clarity would help.

Terminology (Appendix C.1, Table 16): We added a full Glossary table in Appendix C.1 defining all game and evaluation terms, and the terms you named (open and filled profiles) are now explained or pointed to the glossary at their first use in the main text.

Reviewer_6FR8OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces Clue2Closure, a benchmark for evaluating ultra-long-horizon autonomous computer use through evidence-driven investigation games. The paper evaluates frontier computer-use agents and provides trajectory-level analyses of failure modes, including delayed use of acquired knowledge, ineffective belief revision, and unproductive exploration.

Critical strengths
  1. The evaluation of ultra-long-horizon computer-use tasks is an important and meaningful research direction.

  2. The benchmark is thoughtfully designed around seven evidence-structure requirements to better evaluate agents' capabilities in long-horizon tasks.

  3. The detailed trajectory analysis provides valuable insights into agent limitations in long-horizon tasks.

Critical weaknesses
  1. My primary concern is the limited scope of the proposed benchmark. Since it focuses mainly on investigative gameplay, its generalizability to real-world computer-use tasks remains unclear. Consequently, the benchmark may not adequately reflect agents' broader computer-use capabilities in practical scenarios.

  2. Each model is evaluated only once, which may not adequately account for the variability in long-horizon agent performance. Repeated evaluations are needed to ensure the reliability of the reported results.

  3. The extremely low performance of all evaluated agents on Scattered Roots raises concerns about whether the benchmark can meaningfully assess and distinguish agents' ultra-long-horizon capabilities.

  4. The authors do not provide human evaluation results for the proposed games, making it difficult to assess the benchmark's difficulty and practical solvability.

Questions for authors

See weakness.

1 comment
AuthorsOpenReview ↗

Response to Reviewer 6FR8

We thank Reviewer 6FR8 for the thoughtful comments and constructive suggestions, and we appreciate your recognition of our seven evidence-structure requirements and detailed trajectory analysis. Below, we address each of your concerns.

W1: My primary concern is the limited scope of the proposed benchmark. Since it focuses mainly on investigative gameplay, its generalizability to real-world computer-use tasks remains unclear. Consequently, the benchmark may not adequately reflect agents' broader computer-use capabilities in practical scenarios.

This concern is the topic of General Response A: we chose an investigation game on purpose, so that a failure points to long-horizon state management rather than to domain skill.

W2: Each model is evaluated only once, which may not adequately account for the variability in long-horizon agent performance. Repeated evaluations are needed to ensure the reliability of the reported results.

We now run GPT-6 Astra and Claude Opus 5.5 three times each; the results are in the General Response (Sec. 4). Because a single run is very costly, we ran the other models once, which is similar to Agents' Last Exam, where only a subset of configurations is repeated due to compute budget constraints.

W3: The extremely low performance of all evaluated agents on Scattered Roots raises concerns about whether the benchmark can meaningfully assess and distinguish agents' ultra-long-horizon capabilities.

We agree that the low scores in our initial submission did not separate agents well. What we changed, and the new scores that now separate the models, are in the General Response (Sec. 1–3).

W4: The authors do not provide human evaluation results for the proposed games, making it difficult to assess the benchmark's difficulty and practical solvability.

We do not have human play data for SR; General Response B explains how we instead check solvability, difficulty and a reference completion length.

References

Reviewer_Z1y5OpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a benchmark for long-term planning based on a clue-finding indie game. They have both the original game and its expansion, as well as a PCG version based on the same idea. The PCG system follows 7 criteria to create a difficult long-horizon game with a lot of clues. They tested multiple frontier models on the task and compared them to each other. Showing that agentic AI models fails in certain tasks more than others.

Critical strengths

Introducing a benchmark that tests and pushes the capabilities of frontier models. Having a benchmark like that can highlight the shortcomings in the current agentic systems to try to target later.

Critical weaknesses

The paper has some unclear terminology and does not focus on the 7 criteria, generating them and analyzing the generated games as humans, but only using a brute-force agent. This is interesting for correctness, but the generated games should still be solvable by humans and feel coherent, not just designed for agents.

Although having a benchmark is interesting it doesn't add much to the knowledge of the reader since there is a lot of work in long-term planning and multiple benchmarks. While this work showed a new direction in that area, it didn't focus on it.

Questions for authors

Nothing

3 comments
AuthorsOpenReview ↗

Response to Reviewer Z1y5 (1of3)

We thank Reviewer Z1y5 for the feedback, and we appreciate your recognition that our benchmark pushes the capabilities of frontier models and highlights key shortcomings in current agentic systems. Below, we address each of your concerns.

W2: Although having a benchmark is interesting it doesn't add much to the knowledge of the reader since there is a lot of work in long-term planning and multiple benchmarks. While this work showed a new direction in that area, it didn't focus on it.

Clue2Closure is a long-horizon Computer-Use Agent (CUA) benchmark, not a planning benchmark (Table 20): We think the long-term planning works mentioned by the reviewer mainly refer to classical planning benchmarks that do not focus on computer use or GUI use. While planning is part of Clue2Closure, our goal is to push computer-use evaluation to an ultra-long horizon, where the agent must search across thousands of documents, operate in-game apps (email, microfilm, archives, and the family-tree board), and keep hundreds of findings consistent.

BenchmarkTypeLength of one task
PlanBench, TravelPlannerPlanningplans of 2–16 steps (PlanBench); ≤30 agent steps (TravelPlanner); facts given in the prompt or found with a few search tools
WebArenaWeb≤30 steps
SWE-bench VerifiedSoftware455 of 500 tasks within one engineer hour
OSWorld (2024)Computer useabout 30 steps on average
OS-Marathon (2026)Computer use50–200 human steps
Odysseys (2026)Web100-step budget (about 30 min; 200 also tested)
OSWorld 2.0 (2026)Computer use500-step budget
CUA-World-Long (Gym-Anything, 2026)Computer use500-step budget (2,000 in extra runs)
Clue2Closure, original gamesComputer useup to 6,000 steps; GPT-6 Astra scores 100% (333/333)
Clue2Closure, Scattered Roots (SR)Computer use1,785–3,079 model calls in the first 13.5 SR h; runs continue up to 27 SR h

As OSWorld 2.0 also points out, today's agents often "lose track of constraints, miss information that arrives mid-task", and "forget information gathered early when task state is stored only in compressed reasoning or chain-of-thought context". The challenge of ultra-long horizons is unique in a computer-use context and is an important, underexplored direction. However, as discussed in General Response A, it is difficult to decouple long-horizon challenges from software-specific skills (Peng et al., 2026); Clue2Closure is built so that they can be (Sec. 5).

References

AuthorsOpenReview ↗

Response to Reviewer Z1y5 (2of3)

What makes Clue2Closure different: To our knowledge, Clue2Closure is the first computer-use benchmark that stress-tests these challenges together, in single runs of thousands of model calls, namely task-state management, (multimodal) memory management, revising earlier conclusions when new evidence conflicts with them, and choosing where to look among many open leads. In SR, the agent must maintain what it has found, what leads are still open, and which hypotheses were refuted across 30+ context compactions in 13.5+ SR h (GPT-6 Astra; Sec. 4.3). Because each SR point requires connecting several scattered fragments among 33,693 items while single steps remain easy (even Claude Sonnet 5, a previous-generation model, passes 82.9% given the needed memory; Sec. 5), SR directly stress-tests this long-horizon task-state management ability.

Closest related work: The closest benchmark is ApprenticeBench; we compare the two in General Response A.

W2: ...it doesn't add much to the knowledge of the reader... While this work showed a new direction in that area, it didn't focus on it.

We suggest these new findings to look at: Recent job-based benchmarks (e.g., Agents' Last Exam, Gym-Anything, TheAgentCompany) mix domain-specific expertise with the long horizon, making it unclear which one caused an agent to fail (Peng et al., 2026). In contrast, Clue2Closure needs relatively little domain-specific expertise: given the needed memory, even Claude Sonnet 5 solves 82.9% of single steps, so failures point to the long horizon rather than to domain knowledge; we added a section that makes this decoupling explicit (Sec. 5). The paper mainly focuses on building the benchmark and understanding why computer-use agents fail:

  • Agents fail to act on what they already know (Sec. 6.1): Facts that the agent already read often reach the board hundreds of steps late (or never get posted at all), and GPT-6 Astra and Opus 5.5 spend 18–37% of their SR time just re-reading their own saved notes.

  • Agents revise mistakes by blind trial and error (Sec. 6.2): Instead of going back to check the evidence, agents test guesses against the game's feedback and repeat failed searches after compaction.

  • Agents get lost among open paths (Sec. 6.3): Even when unfollowed leads are sitting in their notes, agents stall for long stretches without finding new evidence.

Although these failures might not have been surprising six months ago, with frontier models such as GPT-6 Astra nearly saturating earlier computer-use benchmarks such as OSWorld, it is still unexpected to see them get trapped in front of clues and lose direction simply because the task is longer, much like what happens when people work with agents on a months-long project as context piles up.

A concrete example from the runs (Appendix F.4): By 8.3 SR h, GPT-6 Astra (run 1) had opened every chapter, so no evidence was locked away; by its stop, it had read at least one fragment for 88% of the points it never solved. For 34 of these points, the leads it had already read were enough to close them. Yet in its last 1.6 SR h it opened none of these pages. Instead, the share of its operations spent listing newspaper indexes rose from 15–36% (0–16 SR h) to 69–82% (16–20.5 SR h), and 98.6% of the dates it listed during its plateau held no designed content. The clues were already in its own notes; it simply kept looking in the wrong place.

References

AuthorsOpenReview ↗

Response to Reviewer Z1y5 (3of3)

W1: The paper has some unclear terminology...

Terminology (Appendix C.1, Table 16): We added a full Glossary table in Appendix C.1 defining all game and evaluation terms.

W1: ...does not focus on the 7 criteria...

We agree that the paper does not focus on the seven requirements, and this is intentional: they are not the main novelty we want to promote. We realize that our writing made them look like the main contribution, so we revised how we discuss the role of these criteria (Sec. 1). The seven requirements mainly serve to illustrate how we construct the new game (Sec. 3.2.1), so that its difficulty comes from the long horizon rather than from any single hard step. The focus of Clue2Closure is its length and what it stress-tests. Existing computer-use benchmarks are either too short, mix in specialized software, or do not structurally expose the unique challenges of long-horizon tasks.

W1: ...analyzing the generated games as humans, but only using a brute-force agent. This is interesting for correctness, but the generated games should still be solvable by humans and feel coherent, not just designed for agents.

We admit that we do not have enough human players to play through Scattered Roots (SR) due to resource constraints (details in General Response B). We want to clarify that SR is not validated by a "brute-force agent" alone: we verify that the game is solvable and not just designed for agents (coherence is covered in General Response B):

  • Solvable without relying on an agent: Every point has a certificate listing its exact evidence chain with quoted sentences, and a scripted reference player (which writes each answer only after reading all its required evidence pages) closes all 312/312 points in 967 operations (Appendix A.2). The hint experiments, where agents complete the game given certain levels of guidance, are described in General Response B.

  • Grounded in a human-played game: SR follows the original game's interface (Appendix A.3), and the original commercial game is widely played by humans (38.5% of Steam players complete its base tree, in a median of 10.5 hours; Appendix D.2).

Comments on the paper

3 comments
AuthorsOpenReview ↗

03 General Response B - Concern about Human Players on SR

Raised by: 6FR8 GKZN Z1y5

We admit that we do not have enough human players to play through Scattered Roots (SR). Because SR is designed to be memory-stressful and reasoning-intensive, we are unable to hire human testers due to resource constraints. As we understand this question, human players serve to check whether the game is (1) solvable and coherent, (2) reasonably hard, and (3) bounded by a reference completion length. We address these three points directly:

(1) Solvable and coherent: SR keeps the original game's family tree and people; 182 of its 333 design points match the original game's answer key, and the 151 new ones come from expanding those characters' lives, i.e., 131 relations and 20 sibling orders (Appendix A.2). To verify that the game is solvable and coherent, we first ran two hint experiments with GPT-6 Astra at different levels of help on the same game (Appendix A.10, Table 10):

  • Experiment A (ideas-only hints): The hint file gives only motivation prompts and the reasoning behind clues (e.g., "Open question: which publication of the church would list christening names. The church named in the magazine is the search key, and its own website is the kind of place."). By 4 SR h (31.1%) it already passes the unhinted runs' 13.5 SR h scores (27.6–28.2%), by 6 SR h (46.2%) it passes every unhinted run's stop score at the default window (up to 39.7%), and by 13.5 SR h it reaches 89.1% (278/312).

  • Experiment B (full walkthrough): The walkthrough gives clear step-by-step actions (e.g., "Search the already known name '[name]' and open '[page title]'"). It reaches 100% (312/312 points) in 1.05 SR h (111 model calls, 583 operations).

GPT-6 Astra Configuration4 SR h6 SR h13.5 SR hStop Score & Time
No hints (run 1)7.79.927.932.7 (20.5 h)
Experiment A: Ideas-only hints31.146.289.199.7 (19.9 h)
Experiment B: Full walkthrough100.0 (at 1.05 h)——100.0

Scores in % of 312 points.

In addition, one of the authors walked through the entire game several times with a step-by-step guided review system that shows why a player goes to each page, what it reads, and what follows (stated plainly in Appendix D.2 as a design review for coherence rather than a human difficulty study, since that author knows the answers), alongside two LLM-agent content audits, per-point solvability certificates, and a scripted reference player that closes all 312/312 points in 967 operations (Sec. 3.4; Appendices A.2, A.8).

(2) Reasonably hard: Our local single-step probe, run with Claude Sonnet 5 (a previous-generation model), passes 82.9% of questions when given the memory a player holds at that step (Sec. 5).

(3) Reference length: Because SR puts a large amount of information online, human testers would be bottlenecked by reading speed and cognitive load, just as we would not ask humans to run a deep-research task to measure its reference length. Instead, Experiments A and B (89.1% by 13.5 SR h in A; 100% in 1.05 SR h / 111 calls in B) and the scripted route (967 operations) serve as a direct reference completion length.

AuthorsOpenReview ↗

02 General Response A - Concern about Generalizability to Real-World Computer Use

Raised by: 46hx 6FR8 GKZN

Core abilities and common CUA actions: We agree that our benchmark is an investigative game in a virtual world, not real-world office software. Yet the abilities tested here, including (1) information seeking, (2) logical reasoning across sources, (3) multimodal memory, and (4) long-horizon task-state management, are central to real work, and the actions that agents take are common across benchmarks: web browsing (Mind2Web), reading and sending email (MyPCBench, Jang et al., 2026), and filling structured boards (OSWorld 2.0, OS-Marathon). The main difference (Sec. 7; Table 20, Appendix G) is the long horizon and the interconnected puzzle across thousands of documents.

Why we avoid specialized domain software: During the design of the benchmark, we deliberately avoided specialized software (e.g., 3D CAD or video editing). If an agent fails on a long CAD or video task, it is unclear whether it failed because it does not understand 3D/video, or because of long-horizon state management, memory, and planning. Separating the long-horizon part from specialized domain skill measures the long-horizon residual more directly (Peng et al., 2026). As shown in Sec. 5, each single step is not hard (even Claude Sonnet 5, a previous-generation model, passes 82.9% given the memory a player holds), yet combining them over a long horizon is very challenging (in full runs, even the newer Claude Sonnet 5.5 reaches only 0.6% of the points within 13.5 SR h).

On generalizability to other CUA tasks: First, it is very hard to build an ultra-long-horizon CUA task that is challenging both in long-horizon state management and in a specialized software domain. Even in job-grounded benchmarks like ApprenticeBench (NeoCognition, 2026), doing well on one job environment does not automatically generalize to other CUA tasks, yet it still reflects an agent's long-horizon capability and continual learning. To include more of the diversity of real-world computer use, we can add more common software to the benchmark in the future, and test whether training on SR transfers to other CUA domains.

Second, games are a common way to evaluate agents. For example, Kaggle Game Arena (Google DeepMind, 2025), the PokeAgent Challenge (NeurIPS 2025) and ARC-AGI-3 (2026) are recent, widely used game benchmarks. Among them, ARC-AGI-3 is the closest to our idea: it avoids language and outside knowledge, so its failures reflect adaptive intelligence rather than missing knowledge. These games are not directly tied to real-world tasks, yet they are widely accepted as a standard for evaluation.

Third, our benchmark is closely comparable to ApprenticeBench: both focus on interleaved, general agentic, long-horizon computer-use tasks, much like real office work such as accounting. In ApprenticeBench, the agent works as an accounting clerk who handles about 100 vendor bills in one long run: it reads emails, fixes mistakes with vendors, and enters each bill into an accounting system, reusing what it learned from earlier bills. SR asks for the same kind of work: reading emails and records, cross-referencing thousands of documents, and keeping one structured board consistent over a long run. In addition to these similarities, our benchmark differs in two ways: it offers more varied scenarios (e.g., a 1998-era microfilm reader, a sliding-picture puzzle, and cutting portraits out of photos), and it focuses on the complexity within a single task, i.e., managing task state over a long horizon, rather than on continual learning across tasks.

References

AuthorsOpenReview ↗

01 General Response - Summary of Changes

We thank all reviewers for their thoughtful feedback, and we truly appreciate their suggestions and recognition of the strengths of our work. We have adjusted our benchmark, rerun a few experiments, and revised our paper accordingly to address the concerns mentioned by the reviewers. Here we describe our changes to the benchmark and paper, and provide an overview of the new results and experiments:

1. Difficulty balance 6FR8 GKZN

We relaxed a strict bottleneck at the beginning of SR, so it is easier for agents to enter the later stages of the game.

2. A new budget metric, the SR hour (Sec. 3.3) 46hx

  • Because one SR model call can carry many game operations (Appendix A.4), the step metric used in the original game does not compare resource cost fairly across agents.

  • An SR hour (SR h) is a priced clock: 4s per model call, 1s per 50 output tokens (reasoning included), and 1s per 5,000 uncached input tokens, plus each operation's in-game time (1–3s). Thinking and acting spend the same budget. The design choice and its reasoning are in Appendix A.4.

  • Plateau Rule: Each run gets at least 13.5 SR h, and then continues until it gains at most 2 points in 3 SR h, capped at 27 SR h.

3. New runs (Table 2; Appendix A.5) 6FR8 GKZN

New runs in SR: We evaluate nine models (seven at the default window, six at 475K). The scores separate the models clearly:

ModelOriginalSR: 13.5 SR h → StopSR 475K: 13.5 SR h → Stop
GPT-6 Astra100.027.9 → 32.736.2 → 44.6
GPT-6.1 Sol100.0—30.4 → 31.1
Claude Fable 5.2—24.7 → 26.3—
Claude Opus 5.589.89.9 → 9.98.7 → 8.7
Claude Opus 583.84.2 → 5.4—
GPT-5.6 Sol63.42.6 → 2.6—
Claude Sonnet 5.5100.00.6 → 0.61.6 → 1.6
GPT-5.6-luna14.40.0 → 0.00.0 → 0.0
Claude Haiku 5.5——0.0 → 0.0

Scores in % (original games: The Classics + Roottreemania, out of 333 blanks; SR: out of 312 points). Context window: the original games use the windows in Appendix B (mostly 475K); the SR columns use the default window of 258K (Codex) / 275K (Claude Code), and the SR 475K columns use 475K. GPT-6 Astra and Claude Opus 5.5: run 1 at the default window (all three runs in Sec. 4); 475K: one run.

Full results for every run are in Table 2 and Table 6. We will also run more evaluations on SR (e.g., the Gemini family). Weaker models that score near 0% on SR are still separated on the original games (14.4–100%), so Clue2Closure works as a hierarchy: weaker models compete on the Base game and RTM, and models that saturate them compete on SR.

4. Repeated runs 6FR8 GKZN

To test run-to-run variation, we ran GPT-6 Astra and Claude Opus 5.5 three times each:

ModelRun 1Run 2Run 3Solved in all runsSolved in ≥1 run
GPT-6 Astra27.9 → 32.728.2 → 28.827.6 → 39.712.545.2
Claude Opus 5.59.9 → 9.97.4 → 8.09.3 → 15.73.217.6

Scores in %, 13.5 SR h → Stop. Solved points are counted at 13.5 SR h.

The total score at 13.5 SR h is relatively stable across runs, even though the specific points solved vary. We will run more repeated experiments.

5. A new section: decoupling long-horizon autonomy from short-task difficulty (Sec. 5) GKZN Z1y5

We added a section that shows why Clue2Closure tests the long horizon specifically: each short step is easy on its own and needs little domain knowledge (given the memory a player would hold, even Claude Sonnet 5, a previous-generation model, passes 82.9% of single steps), yet agents fail when these steps must be combined over runs of thousands of operations. The failure modes analysed in Sec. 6 are read through this lens.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.