- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper introduces CurioBench-1K, a benchmark of 1,000 deterministic, judge-free agentic tasks whose purpose is to measure, separately from task-solving capability, whether an agent attends to and acts on environmental resources it was not instructed to use, in support of the central claim that default-prompt evaluations capture an agent's default behavior rather than its underlying capability.
Critical strengths
The benchmark has a clean measurement design: paired controls rule out "the model just couldn't do the task," exact-match scoring removes the LLM judge, and the U≤I≤D invariant catches bookkeeping bugs.
Critical weaknesses
-
It remains open whether the sees-but-does-not-open behavior is a mid-tier open-weight phenomenon rather than a general property of current agents.
-
The authors do not clarify how much the gap columns depend on the specific command-string definition of "deliberate," since no human-labeled audit of a stratified trajectory sample is reported and the ladder is not rerun under a looser definition that counts a uniquely-matching grep as deliberate.
-
The authors do not test whether the diagnosed deficit survives harness interventions designed to remove it, such as running the same model under a tool-call harness, a filesystem harness, and a notice-then-decide scaffold on matched tasks, leaving open whether the paper's framing should shift from an agent-level deficit to a harness-level one.
Questions for authors
-
Whether this problem still holds on frontier closed models?
-
How do the authors guarantee that the deliberate-vs-incidental classifier is accurate, given that no human-labeled precision and recall, inter-rater agreement, or borderline-case sensitivity analysis is reported
-
How large the harness's effect on the results is, and whether the problem the paper raises can be resolved by the harness framework itself, given that §A acknowledges serving-side variation and the non-comparability between single-turn text and tool-call modes without running any model on matched tasks across harnesses