acceptodds
Under review as a conference paper at ICLR 2027

SeekOrStop: Benchmarking Tool-Mediated Blue-Team Investigation under Matched Evidence Access

Abstract

Can a model that makes correct decisions from supplied evidence also identify missing evidence, acquire it, and know when to stop? We introduce SeekOrStop, a benchmark that distinguishes evidence acquisition, decision quality, and completion using matched Passive and Active views of the same security investigation episodes. Episodes are generated through controlled execution in a real Kubernetes environment, with ground truth derived independently of sensor labels. Passive provides a registered reference evidence view, whereas Active begins with an initial observation and decides whether to retrieve additional records through finite read-only tools. Across 420 Expansion episodes and four model configurations, end-to-end Active accuracy is consistently 14–32 percentage points lower than Passive accuracy. Moreover, registered-route completion does not guarantee successful investigation: agents may complete the registered route yet make an incorrect decision, fail to submit, or continue retrieving after the registered decision requirements have been met. Controlled contrast and equivalence pairs further test joint correctness under specified relevant and irrelevant changes. SeekOrStop therefore evaluates evidence acquisition, evidence use, completion, and stopping separately, showing why supplied-evidence accuracy alone gives an incomplete account of agentic investigation.

Primary area:
datasets and benchmarks
Keywords:
LLM Agents, Evidence Seeking, Security Investigation, Decision Commitment

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_xJswOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents SeekOrStop, a benchmark for evaluating active and passive reasoning and stopping behavior in the security domain. A Kubernetes environment is constructed with two modes of evaluation. In the active view, the model has to interact with the frozen store environment to collect telemetry data, whereas in the passive view, the data is provided as evidence. Experiments with four models show that models perform much better in the passive rather than active mode. Moreover, following the expected completion path does not guarantee success and vice versa.

Critical strengths
  1. The paper studies an important problem in a practically useful domain.
  2. Results highlight the limitation of latest models on evidence acquisition
Critical weaknesses
  1. Novelty is limited. Prior work like ARBench already evaluates active vs passive reasoning. SeekOrStop mainly extends it to the Kubernetes security tasks, however, presents no unique insights relevant to the domain.
  2. There is only one registered route but models reach the correct answer even without using it. It is unclear whether there are multiple valid routes or are models guessing the answer.
  3. There is limited error analysis. Particularly for GLM 5.3, Qwen and Kimi-K3, the paper simply establishes the gap between active and passive views but doesn't help explain the gap.
Questions for authors

Kimi, GLM, and Qwen hit Gateway errors in 279, 325, and 255 of their 420 Active sessions. Could you explain the reason behind such large number of errors? The frozen store answers only predefined routes; schema-valid queries without a registered route return errors where a real audit or runtime backend would return results. Perhaps a better design could be where any schema-valid query returns its matching records (possibly none) from the frozen store, especially if the predefined route is not the only valid one.

1 comment
AuthorsOpenReview ↗

Response to Reviewer xJsw

We thank the reviewer for the critical feedback. We conducted additional experiments and trace-level audits to address the concerns regarding backend restrictions, alternative retrieval paths, and benchmark novelty.

1. Gateway errors and backend validity.

We acknowledge the limitation of the original exact-catalogue interface. Our audit reproduced 2,392 catalogue-related errors among 6,791 tool queries (35.2%). Replaying these queries against the filtering backend yielded 1,014 nonempty and 1,378 valid-empty responses.

Importantly, the filtering backend executes the model's unchanged, schema-valid queries against the same frozen public records. It does not repair model outputs, alter registered-query responses, or use Gold labels to select evidence.

We conducted supplementary 420-episode evaluations under both backends for Kimi-K3, GLM-5.3, and Qwen3.6-27B. Query-error rates dropped to zero, while accuracy increased from 38.3% to 58.6%, 27.1% to 36.7%, and 15.7% to 24.3%, respectively. GPT-5.5 improved from 61.0% to 63.8% relative to its historical baseline.

These findings confirm that the original interface imposed measurable constraints. We address this concern directly through a less restrictive backend and transparent protocol comparisons, rather than attributing all failures to model reasoning.

2. Are registered routes the only valid evidence paths?

No. We agree that registered-route completion should not be interpreted as the only valid investigation path. However, exact matching provides a deterministic, reproducible, and model-independent measure of reference-workflow completion, avoiding subjective judgments of evidence sufficiency.

Our supplementary trace audit provides concrete evidence of its limitations: among 41 correct Kimi-K3 sessions without unordered-route credit, 24 (58.5%) retrieved the complete registered response content through alternative queries. This analysis covers a retained subset, not the full 420 episodes.

We therefore distinguish registered-workflow completion from evidence-content coverage and final decision accuracy. Importantly, the primary decision-accuracy metric does not require adherence to the registered route, so alternative successful investigations are not automatically scored as incorrect.

3. Novelty beyond AR-Bench: Why is SeekOrStop a distinct benchmark?

We respectfully disagree that SeekOrStop merely transfers AR-Bench to Kubernetes security. While both study active reasoning, they target different evaluation problems.

AR-Bench investigates interactive question-asking across detective cases, situation puzzles, and number-guessing tasks. SeekOrStop instead evaluates whether agents can acquire sufficient operational evidence, make verifiable blue-team security decisions, and stop investigating at the appropriate point.

Its distinctive contributions are threefold:

  • Execution-grounded evaluation: Episodes originate from executed Kubernetes scenarios, with independently established Gold decisions, frozen telemetry evidence, and matched Passive–Active access. This separates the correctness of the security decision from the agent's chosen retrieval path.

  • Controlled behavioral testing: Contrast and Equivalence pairs explicitly test sensitivity to decision-relevant changes and robustness to irrelevant variations. Scenarios with sufficient initial evidence additionally test whether agents can recognize when retrieval is unnecessary.

  • Fine-grained failure analysis: SeekOrStop separates evidence acquisition, route completion, decision correctness, and stopping behavior, revealing failures obscured by end-to-end accuracy alone.

Our supplementary analyses provide concrete examples of these distinctions. Some agents retrieve the required evidence through non-registered queries, while 43 of 231 route-complete GPT-5.5 sessions still produce incorrect decisions despite valid submissions and error-free tool execution.

These findings show that retrieving evidence, correctly interpreting evidence, and determining when to stop are separate challenges. The contribution is therefore not another demonstration of an Active–Passive accuracy gap, but an execution-grounded benchmark that enables these capabilities to be evaluated separately in security investigations.

We appreciate the reviewer's concerns, which motivated substantial additional validation. We believe the new protocol experiments and trace-level evidence directly address the interface criticism and clarify SeekOrStop's distinct contribution.

Reviewer_cW8XOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents SeekOrStop, a benchmark for tool-based security investigation. Each case has two settings. In Passive, the model receives the reference evidence directly. In Active, it starts with an initial observation and can use read-only tools to collect more evidence before making a decision. The cases come from controlled executions in a Kubernetes environment. The benchmark measures decision accuracy, completion of a predefined investigation route, valid submission, and stopping behavior. On 420 cases across four model configurations, Active accuracy is 14–32 percentage points lower than Passive accuracy. The results also show that models can complete the reference route but still answer incorrectly, fail to submit, or continue querying.

Critical strengths
  1. The paper addresses an important research problem of whether LLM agents can proactively identify and acquire missing evidence and determine when to stop investigating and make a decision.
  2. The paper provides a comprehensive experimental evaluation across multiple models and diverse scenarios, with detailed analyses of evidence acquisition, decision-making, and stopping behavior.
Critical weaknesses
  1. The paper focuses on evaluation but offers limited analysis of the observed results. I would like to see deeper analysis of why some models outperform others and what cognitive or behavioral capabilities correlate with performance. For example, GPT-5.5 achieves much higher Active accuracy than Kimi-K3. Is GPT better at identifying the next relevant query from available evidence or integrating information across multiple sources?
  2. Interface failures are unusually frequent for several configurations. In the primary Active evaluation, 383/420 GLM sessions and 290/420 Qwen sessions contain invalid actions. GLM also fails to produce a valid final in 184/420 Passive cases. Such high failure rates raise concerns about the protocol itself. The benchmark needs an evaluation protocol better suited to the tool-use capabilities of current LLMs.
Questions for authors
  1. Why does registered-route completion require a predefined tool-call order? Could alternative valid retrieval paths provide equally sufficient evidence?
  2. A substantial proportion of Active sessions lack valid final submissions, particularly for GLM-5.3 (264/420) and Qwen3.6-27B (246/420). How sensitive are the results to changes in the tool-use protocol?
  3. Why is the investigation budget set to B=8? Was this choice empirically validated, given that the registered reference paths require at most two calls?
  4. Why is post-completion retrieval treated as a stopping failure? Could additional retrieval help verify evidence or improve decision reliability?
1 comment
AuthorsOpenReview ↗

Response to Reviewer cW8X

We thank the reviewer for the constructive feedback. We conducted additional controlled experiments and trace-level analyses to directly address the concerns about protocol validity, model differences, and stopping behavior.

1. Protocol validity and interface failures.

We agree that the original catalogue-based backend imposed unnecessary query restrictions. To address this, we introduced a filtering backend that executes the model's unchanged, schema-valid queries against the same frozen evidence, without repairing model outputs or using Gold labels.

In contemporaneous evaluations under both backends, query-error rates dropped to 0%, while Active accuracy improved by 20.2, 9.5, and 8.6 percentage points for Kimi-K3, GLM-5.3, and Qwen3.6-27B, respectively. Valid-final submission rates also increased substantially (see Shared Results).

These findings directly quantify the interface's impact. Importantly, the remaining errors cannot all be attributed to catalogue restrictions: invalid actions and incorrect decisions persist even after query errors are eliminated.

2. Deeper analysis of model differences.

Our supplementary analysis reveals distinct failure patterns rather than a uniform Active–Passive gap.

For Kimi-K3, accuracy conditional on valid submission increased from 53.3% to 62.8%, indicating improvements beyond submission reliability. In contrast, GLM-5.3 and Qwen3.6-27B showed increased overall accuracy but lower conditional accuracy (52.8%→40.8% and 49.0%→41.9%), suggesting that improved submission reliability contributes substantially to their gains.

We further identified two complementary phenomena:

  • Successful evidence acquisition without route credit: In a retained Kimi-K3 subset, 24/41 correct sessions lacking unordered-route completion had nevertheless retrieved the registered evidence through alternative queries.
  • Incorrect decisions despite complete evidence: In the original GPT-5.5 evaluation, 43/231 unordered-route-complete sessions still produced incorrect decisions despite valid submissions and error-free tool execution.

These findings provide concrete evidence that interface compatibility, evidence acquisition, submission reliability, and decision correctness represent distinct challenges. They offer a more informative explanation of model differences than aggregate accuracy alone, without assuming unverified cognitive mechanisms.

3. Registered routes and alternative retrieval paths.

Exact registered-route matching is intentionally retained as a deterministic, reproducible, and model-independent process metric. However, it is not the sole measure of investigation success: decision accuracy is evaluated independently of retrieval order or route completion.

Our new response-level audit further captures equivalent evidence obtained through alternative query parameterizations, addressing the concern that a single registered route may underestimate valid evidence acquisition.

4. Investigation budget and stopping behavior.

The B=8 budget allows investigation beyond the reference paths (at most two calls), enabling us to observe redundant retrieval and delayed stopping without forcing immediate termination. We do not claim B=8 is optimal; larger-budget experiments are treated as supplementary diagnostics.

Similarly, post-completion retrieval is an operational stopping measure, not an assertion that additional verification is always harmful. It measures retrieval beyond the registered sufficiency point, separately from decision correctness. A6 episodes, where initial evidence is already sufficient, provide an explicit test of whether agents recognize when no retrieval is needed.

Overall, the supplementary experiments address the reviewer's two central concerns: we quantify and mitigate the restrictive-interface effect, and provide fine-grained evidence explaining why successful tool use, evidence acquisition, and correct decisions do not necessarily coincide. We believe these additional results strengthen the benchmark's soundness and informativeness, and hope they support reconsideration of the assessment.

Reviewer_dg51OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies whether language models can acquire sufficient evidence through tools and stop investigating appropriately when making blue-team security decisions. SeekOrStop constructs a controlled benchmark from executed Kubernetes scenarios, with independently established outcome labels and a frozen evidence store. Its central comparison presents the same episodes either passively, with the registered evidence already supplied, or actively, requiring the model to retrieve that evidence through tools. Across 420 episodes and four models, active performance is consistently lower than passive performance. The analysis separates decision accuracy, registered-route completion, submission failures, and unnecessary investigation. Interface audits, repeated runs, oracle-evidence conditions, and larger-budget experiments further characterize the gap. The contribution is a carefully controlled evaluation of evidence acquisition and stopping behavior, rather than a new investigation algorithm.

Critical strengths
  1. The matched-evidence design makes the principal comparison unusually interpretable. Passive and active conditions share episode labels and registered evidence, reducing ambiguity about whether performance differences arise simply from different information availability. GPT’s accuracy decreases from 75.0% to 61.0%, with larger declines for the other evaluated models. This provides a concrete and reproducible characterization of the difficulty introduced by active evidence acquisition.

  2. The diagnostic analysis goes beyond reporting an aggregate performance gap. The paper distinguishes incorrect decisions from missing submissions and route-completion failures, audits interface behavior, and examines reminders, oracle evidence, and larger budgets. These analyses substantially strengthen the benchmark’s value by showing which explanations are supported and which remain unresolved.

Critical weaknesses
  1. The benchmark covers a relatively narrow class of investigative workflows. Episodes use a finite evidence store and registered routes requiring at most two retrieval steps within one underlying application environment. Moreover, route completion measures agreement with an ordered reference route, rather than equivalence among all potentially sufficient investigative paths. The authors correctly retain accurate decisions made without route credit, so this does not invalidate the accuracy comparison. It does limit the interpretation of route-based findings as evidence about broader investigative competence.

  2. Some of the strongest diagnostic conclusions remain difficult to reproduce or interpret causally. The oracle-evidence and larger-budget experiments were conducted after the primary evaluation, and the manuscript documents variation even in conditions with identical zero-retrieval histories. These experiments are therefore informative diagnostics, but not clean estimates of the contribution of evidence acquisition or budget alone. Their exclusion from the described v0.1.0 release also limits independent verification of this part of the analysis.

Questions for authors
  1. Clarification: How sensitive are the route-based conclusions to accepting alternative query orders or equivalent sufficient evidence sets? Can this be assessed separately from the already reported decision-accuracy metric?

  2. Clarification: Can the oracle-evidence and larger-budget traces be included in the reproducibility package, with explicit model-version and execution-date information to clarify their comparability with the primary experiments?

1 comment
AuthorsOpenReview ↗

Response to Reviewer dg51

We thank the reviewer for the positive assessment and constructive suggestions. We conducted additional trace-level analyses to address the concerns about alternative retrieval paths and have expanded the reproducibility information on our project repository homepage.

1. Alternative retrieval paths and route-completion sensitivity.

We agree that multiple retrieval paths may provide equivalent evidence. However, exact registered-route matching is an intentional design choice, providing a deterministic, reproducible, and model-independent measure of reference-workflow completion without relying on subjective evidence-sufficiency judgments. Importantly, our primary decision-accuracy metric does not require following the registered route.

To assess sensitivity, we examined both order-agnostic completion and response-level evidence coverage. In retained Kimi-K3 runs, relaxing query order recovered 5 correct sessions under the original backend and 2 under the filtering backend. More importantly, among 41 correct filtered-backend sessions without unordered-route credit, 24 (58.5%) had already retrieved the complete registered evidence content through alternative queries.

For example, ANY_AUDIT returned the same required audit record as the registered POD_EXEC query, despite differing query parameters.

These results demonstrate that registered-route completion and actual evidence acquisition are complementary rather than interchangeable measures. We retain exact matching as an objective process metric while using response-level coverage and decision accuracy to characterize alternative retrieval behavior. We do not assume that every alternative path is semantically sufficient.

2. Reproducibility of supplementary experiments.

We have already expanded the project repository homepage with additional reproducibility information for the supplementary experiments, addressing the reviewer's concerns about transparency and comparability.

We also explicitly distinguish the post-hoc oracle-evidence and larger-budget experiments from the primary confirmatory evaluation. These analyses provide diagnostic insights rather than causal estimates of evidence acquisition or budget effects.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Shared Supplementary Results: Protocol Sensitivity Analysis

We thank all reviewers for their constructive feedback. To address concerns about the query interface, we conducted supplementary Active evaluations using a filtering backend.

The filtering backend executes the model's unchanged, schema-valid queries against the same frozen public records and returns matching records, rather than requiring exact tool/argument combinations to appear in a pre-registered catalogue. It preserves registered-query responses and neither repairs model outputs nor uses Gold labels to select evidence.

Kimi-K3, GLM-5.3, and Qwen3.6-27B were re-evaluated under both backends during the same experimental period, while GPT-5.5 uses the original paper results as its baseline. All evaluations cover the same 420 episodes.

Supplementary Evaluation Results

ModelBackendAccuracyValid FinalQuery ErrorInvalid ActionUnordered RouteRoute & Correct
GPT-5.5Original61.0%100.0%3.8%0.5%55.0%44.8%
Filtered63.8%100.0%0.0%0.0%81.2%53.3%
Kimi-K3Original38.3%71.9%68.3%36.2%54.8%31.9%
Filtered58.6%93.3%0.0%46.2%36.7%32.1%
GLM-5.3Original27.1%51.4%87.9%88.6%74.3%21.9%
Filtered36.7%89.8%0.0%75.7%57.4%33.8%
Qwen3.6-27BOriginal15.7%32.1%61.0%4.3%60.2%11.0%
Filtered24.3%57.9%0.0%3.8%49.0%16.4%

All rates use 420 episodes per configuration. Query Error and Invalid Action are session-level rates. Unordered Route measures registered-query completion regardless of order; Route & Correct requires both completion and a correct final decision.

Key Findings

Finding 1: Interface sensitivity. The original backend's exact-catalogue restriction introduced avoidable query errors. With the filtering backend, query-error rates dropped to zero, while accuracy improved by 2.9–20.2 percentage points. The GPT-5.5 comparison uses a historical baseline. These results quantify the substantial impact of interface design on agent evaluation.

Finding 2: Exact route matching provides a rigorous and reproducible process metric. Pre-registered query matching ensures deterministic, model-independent, and comparable evaluation of reference-workflow completion without subjective evidence-sufficiency judgments. It is intentionally distinct from assessing all possible valid retrieval paths. Our trace audit found that 24 of 41 correct Kimi-K3 sessions without unordered-route credit (58.5%) retrieved equivalent registered evidence through alternative queries. We therefore retain exact route matching as a standardized process metric, complemented by response-level evidence coverage and decision accuracy.

Finding 3: Evidence acquisition and decision correctness are distinct challenges. In the original GPT-5.5 evaluation, 43 of 231 unordered-route-complete sessions still produced incorrect decisions despite valid submissions and error-free tool execution. Thus, successfully acquiring the registered evidence does not guarantee correct security decisions.

Overall, these results demonstrate that interface compatibility, evidence acquisition, reference-workflow completion, and decision correctness are distinct evaluation dimensions. By separating these dimensions, SeekOrStop provides a controlled framework for identifying investigation failures beyond aggregate Active–Passive accuracy. We have also expanded the reproducibility information on our project repository homepage.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.