- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper presents SeekOrStop, a benchmark for evaluating active and passive reasoning and stopping behavior in the security domain. A Kubernetes environment is constructed with two modes of evaluation. In the active view, the model has to interact with the frozen store environment to collect telemetry data, whereas in the passive view, the data is provided as evidence. Experiments with four models show that models perform much better in the passive rather than active mode. Moreover, following the expected completion path does not guarantee success and vice versa.
Critical strengths
- The paper studies an important problem in a practically useful domain.
- Results highlight the limitation of latest models on evidence acquisition
Critical weaknesses
- Novelty is limited. Prior work like ARBench already evaluates active vs passive reasoning. SeekOrStop mainly extends it to the Kubernetes security tasks, however, presents no unique insights relevant to the domain.
- There is only one registered route but models reach the correct answer even without using it. It is unclear whether there are multiple valid routes or are models guessing the answer.
- There is limited error analysis. Particularly for GLM 5.3, Qwen and Kimi-K3, the paper simply establishes the gap between active and passive views but doesn't help explain the gap.
Questions for authors
Kimi, GLM, and Qwen hit Gateway errors in 279, 325, and 255 of their 420 Active sessions. Could you explain the reason behind such large number of errors? The frozen store answers only predefined routes; schema-valid queries without a registered route return errors where a real audit or runtime backend would return results. Perhaps a better design could be where any schema-valid query returns its matching records (possibly none) from the frozen store, especially if the predefined route is not the only valid one.
Response to Reviewer xJsw
We thank the reviewer for the critical feedback. We conducted additional experiments and trace-level audits to address the concerns regarding backend restrictions, alternative retrieval paths, and benchmark novelty.
1. Gateway errors and backend validity.
We acknowledge the limitation of the original exact-catalogue interface. Our audit reproduced 2,392 catalogue-related errors among 6,791 tool queries (35.2%). Replaying these queries against the filtering backend yielded 1,014 nonempty and 1,378 valid-empty responses.
Importantly, the filtering backend executes the model's unchanged, schema-valid queries against the same frozen public records. It does not repair model outputs, alter registered-query responses, or use Gold labels to select evidence.
We conducted supplementary 420-episode evaluations under both backends for Kimi-K3, GLM-5.3, and Qwen3.6-27B. Query-error rates dropped to zero, while accuracy increased from 38.3% to 58.6%, 27.1% to 36.7%, and 15.7% to 24.3%, respectively. GPT-5.5 improved from 61.0% to 63.8% relative to its historical baseline.
These findings confirm that the original interface imposed measurable constraints. We address this concern directly through a less restrictive backend and transparent protocol comparisons, rather than attributing all failures to model reasoning.
2. Are registered routes the only valid evidence paths?
No. We agree that registered-route completion should not be interpreted as the only valid investigation path. However, exact matching provides a deterministic, reproducible, and model-independent measure of reference-workflow completion, avoiding subjective judgments of evidence sufficiency.
Our supplementary trace audit provides concrete evidence of its limitations: among 41 correct Kimi-K3 sessions without unordered-route credit, 24 (58.5%) retrieved the complete registered response content through alternative queries. This analysis covers a retained subset, not the full 420 episodes.
We therefore distinguish registered-workflow completion from evidence-content coverage and final decision accuracy. Importantly, the primary decision-accuracy metric does not require adherence to the registered route, so alternative successful investigations are not automatically scored as incorrect.
3. Novelty beyond AR-Bench: Why is SeekOrStop a distinct benchmark?
We respectfully disagree that SeekOrStop merely transfers AR-Bench to Kubernetes security. While both study active reasoning, they target different evaluation problems.
AR-Bench investigates interactive question-asking across detective cases, situation puzzles, and number-guessing tasks. SeekOrStop instead evaluates whether agents can acquire sufficient operational evidence, make verifiable blue-team security decisions, and stop investigating at the appropriate point.
Its distinctive contributions are threefold:
Execution-grounded evaluation: Episodes originate from executed Kubernetes scenarios, with independently established Gold decisions, frozen telemetry evidence, and matched Passive–Active access. This separates the correctness of the security decision from the agent's chosen retrieval path.
Controlled behavioral testing: Contrast and Equivalence pairs explicitly test sensitivity to decision-relevant changes and robustness to irrelevant variations. Scenarios with sufficient initial evidence additionally test whether agents can recognize when retrieval is unnecessary.
Fine-grained failure analysis: SeekOrStop separates evidence acquisition, route completion, decision correctness, and stopping behavior, revealing failures obscured by end-to-end accuracy alone.
Our supplementary analyses provide concrete examples of these distinctions. Some agents retrieve the required evidence through non-registered queries, while 43 of 231 route-complete GPT-5.5 sessions still produce incorrect decisions despite valid submissions and error-free tool execution.
These findings show that retrieving evidence, correctly interpreting evidence, and determining when to stop are separate challenges. The contribution is therefore not another demonstration of an Active–Passive accuracy gap, but an execution-grounded benchmark that enables these capabilities to be evaluated separately in security investigations.
We appreciate the reviewer's concerns, which motivated substantial additional validation. We believe the new protocol experiments and trace-level evidence directly address the interface criticism and clarify SeekOrStop's distinct contribution.