- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces Clue2Closure, a benchmark for evaluating ultra-long-horizon autonomous computer use through evidence-driven investigation games. It proposes seven requirements for evidence structure and constructs a new game, Scattered Roots, through a synthesis pipeline. Experiments with frontier agents examine task completion and failure patterns related to memory management and multi-path exploration.
Critical strengths
- Benchmark Design. The paper considers memory and exploration challenges in long-horizon computer-use tasks. The seven evidence-structure requirements provide a structured approach to benchmark construction.
- Trajectory Analysis. The paper evaluates multiple frontier agents and analyzes their behavior, including delayed use of evidence, incorrect revisions, and exploration stalls.
Critical weaknesses
- Limited Benchmark Validation. Scattered Roots is extremely difficult for all evaluated agents, with the highest score reaching only 19 out of 312 points. This raises concerns about its ability to meaningfully distinguish agent capabilities. Moreover, no human evaluation is provided for the new game, and the seven design requirements are not individually validated through controlled ablations. It remains unclear whether the observed failures primarily reflect the intended autonomy challenges or excessive task difficulty.
- Concerns about Experimental Reliability and Generalization. Each configuration is evaluated only once, without run-to-run variance. Some evaluations involve additional in-game aids or operator intervention, and parts of Scattered Roots were finalized during the runs (Appendix A.4 and B.1). These factors may affect the reliability and comparability of the reported results. Furthermore, the benchmark focuses on investigation games within a single family-tree setting, limiting evidence of broader applicability.
Questions for authors
- Could the authors provide human evaluations or additional difficulty levels to validate the solvability and discriminative ability of Scattered Roots?
- Could the authors conduct controlled experiments to isolate the contributions of the seven evidence-structure requirements?
- How do the authors account for the effects of in-game aids, operator intervention, and changes made during evaluation?
5 comments
Response to Reviewer GKZN (2of4)
Limited Benchmark Validation. ...the seven design requirements are not individually validated through controlled ablations.
Q2: Could the authors conduct controlled experiments to isolate the contributions of the seven evidence-structure requirements?
To clarify first, the main focus of the paper is to propose an evidence-based benchmark for ultra-long-horizon computer use. The seven design requirements are not the main novelty here, and only serve to illustrate how we turn an investigative game into a benchmark that exposes the challenges of ultra-long-horizon CUA tasks. We have revised the phrasing in the paper to avoid any misunderstanding of the paper's emphasis.
As for the ablation, the seven design requirements are not meant for single-factor ablation, because they work together as a whole to create the long-horizon challenge. For example, if we take out unavoidable misdirection (R4), then persistent uncertainty (R6) would also disappear. In addition, ablating each requirement would require building and evaluating seven additional games, which we cannot complete due to resource constraints.
Instead, we validate each requirement in two ways:
-
Individual metrics and check scripts (Table 1, Sec. 3.4; Table 4, Appendix A.1): Every requirement has a concrete metric, a threshold, and a check script with a mutation self-test. Tested locally, single steps are easy: our single-step probe with Claude Sonnet 5 (a previous-generation model) passes 82.9% with memory, and a designed misdirection is corrected in 6/6 probes once its refuting evidence is read (Table 4).
-
Matching failure modes in agent trajectories: We find a clear failure mode corresponding to each requirement in the agents' trajectories, showing how each requirement contributes to the overall challenge:
| Requirement | Observed Failure Mode in Agent Trajectories |
|---|---|
| R1: Open threads | An agent notices a married-surname defect on a card but never returns to fix it; another carries the same "to post" list across six compactions without posting it. |
| R2: No master key | An agent spends 14 SR h trying to open one locked archive that is not a unique route to any point it missed. |
| R3: Long-span retrieval | A newspaper issue opened at 6 SR h is skipped at 20 SR h; a founding year found at 3.9 SR h is forgotten in compaction and "rediscovered" at 13.9 SR h. |
| R4: Misdirection | Cards keep common name spellings on the board even after the agent reads the birth records giving the required birth-name spelling. |
| R5: Where to look | Two months of the wrong newspaper are scanned even though the wedding date is known; a lock page's pointer is ignored; an obituary link is never clicked. |
| R6: Persistent uncertainty | Agents hold back solved people under self-imposed rules: "so I left it", "Not added (no birth name confirmed)", or "no card until certain" held for 11 SR h. |
| R7: Memory load | Up to 37% of SR time goes to re-reading the agent's own local files, and context is rebuilt by grepping old outputs after compaction. |
Interventions on the same game: Although we cannot build seven separate games, we can supply, on the same game, part of what the long horizon withholds (e.g., the needed memory in single-step probes, or direction in Experiment A). These are not single-requirement ablations, but each time performance rises sharply, consistent with the long horizon, rather than single-step difficulty, being the main obstacle:
| What we give | Mainly eases | Without it | With it |
|---|---|---|---|
| The memory a player would hold at that step (single-step probe, Claude Sonnet 5) | Recalling evidence read long ago (R3) | 21.5% of steps | 82.9% of steps |
| Hints with the open questions and the reasoning behind clues, without answers (GPT-6 Astra, full game) | Tracking open threads, deciding where to look (R1, R5); also eases some single steps | 27.6–28.2% at 13.5 SR h | 89.1% at 13.5 SR h (one run) |
Row 1: % of single steps answered correct and certain (with memory: in ≥80% of attempts; without: first answer) (Sec. 5). Row 2: % of the 312 scored points, Experiment A in General Response B (Appendix A.10).
Response to Reviewer GKZN (3of4)
Limited Benchmark Validation. ...It remains unclear whether the observed failures primarily reflect the intended autonomy challenges or excessive task difficulty.
We added a section on exactly this question (Sec. 5, "Decoupling Long-Horizon Autonomy from Short-Task Difficulty"): each short step is easy on its own and needs little domain knowledge, so what remains hard is the long horizon. Its main test is a local single-step probe (Sec. 5; Appendix C.2). For each point closure on the reference route (181 questions covering 243 points), we give a probe model (Claude Sonnet 5, a previous-generation model) the memory a player would already hold at that step and ask it to make the single closing inference:
| Probe / Split Measure | Result |
|---|---|
| Single-step probe (Claude Sonnet 5) with the memory a player would hold | 82.9% pass (150 / 181) |
| Single-step probe without that memory (correct and certain; correct only) | 21.5% (34.3%) |
| Local pass rate on points no GPT-6 Astra run solved vs. solved points | 82.4% vs. 82.2% |
| Solve rate in Astra runs 1–3 on structurally easy (≤6 fragments) vs. hard (>6 fragments) points | 42.4% vs. 29.7% |
This shows that individual steps are not too hard, even for a previous-generation model. Instead, agents fail because they lose track of evidence over the long run, fail to revise earlier mistakes, or fail to post what they already found. We also added Appendix F, which groups failure case studies into different categories, plus (F.4) showing that every run at the default window stopped because its score plateaued while solvable points were still right in front of it.
Concerns about Experimental Reliability and Generalization. Each configuration is evaluated only once, without run-to-run variance.
Repeated runs of GPT-6 Astra and Claude Opus 5.5, including how many points are solved in all three runs, are in the General Response (Sec. 4); because the solved points vary, we analyse single points only across runs. The other models ran once because a run is costly, as Agents' Last Exam also repeats only some configurations.
References
- Sun et al. Agents' Last Exam. arXiv:2606.05405, 2026.
Response to Reviewer GKZN (1of4)
We thank Reviewer GKZN for the detailed review and constructive questions, and we appreciate your recognition of our seven evidence-structure requirements and trajectory analysis. Below, we address each of your concerns.
We agree that the scores in our submission (at most 19 of 312 points) did not separate agents well. What we changed, and the new scores that now separate the models, are in the General Response (Sec. 1–3).
We do not have human play data for SR; solvability is covered in General Response B, and discriminative ability in our answer above. Beyond these:
Additional difficulty levels: The two hint experiments in General Response B also act as difficulty levels on the same game: GPT-6 Astra goes from 27.6–28.2% with no hints to 89.1% with ideas-only hints at 13.5 SR h, and to 100% in 1.05 SR h with a walkthrough (Appendix A.10).
Human reference on the original game: 38.5% of Steam players complete the original game's base tree, in a median of 10.5 hours (Appendix D.2).