- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The contributions of this paper lie along two axes:
-
Theoretical analysis of harness dynamics: The paper models harness interactions as absorbing Markov chains and studies protocol-state dynamics, state aggregation, and predictive fidelity.
-
BasinGuard, an online harness plugin: The authors propose a plugin that uses a counter-based logistic gate and heuristic context management to improve agent performance. BasinGuard consistently improves success rates across three harnesses and two models, increasing GitTaskBench success rates from 9.33% to 22.00% without increasing runtime.
Critical strengths
S1. BasinGuard consistently improves success rates across multiple harness–model combinations without increasing execution time, demonstrating its practical effectiveness and generalizability.
Critical weaknesses
W1. The theoretical results are disconnected from the proposed method. Theorems 1 and 2 essentially restate standard identities and the strong lumpability condition, without providing a theoretical foundation for BasinGuard, which instead relies on a counter-based logistic gate and heuristic context management.
W2. The empirical validation lacks comparisons with existing techniques and component-level ablations.
Questions for authors
Q1. The analysis assumes that the protocol state is Markov (control-sufficient). Appendix D.2 shows that adding the previous state greatly improves next-event prediction (Brier 0.4687 → 0.1887), which suggests that the current state alone is not Markov. How do the authors reconcile this with the first-order kernel used to estimate , , and the residence scales in Section 2.1? Also, can the authors clarify what the spectral and attractor analysis adds beyond a 2 * 2 substochastic matrix? Similarly, strong lumpability (Theorem 2) holds or fails by construction in Section 2.2 rather than being tested on the estimated kernels.
Q2. BasinGuard is compared only against the native harness path; there are no component ablations (M/O/K), no predictor-free control (e.g., random-time or fixed-interval application of the same operations), and no comparison with existing context-management methods such as observation masking, LLM summarization (e.g., ACON), or the host’s own compaction, even though the anonymous code defines arms such as M, O, K, MOK, and O_RANDTIME. Can the authors report these comparisons, and clarify how the shared checkpoint is chosen (task start or mid-trajectory)?
Q3. How often did the gate actually fire (i.e., how often among otherwise eligible boundaries), and which operations were applied? If it fires at nearly every boundary, the predictor adds little beyond a fixed schedule. Please also report how and the threshold and scheduling parameters () were obtained, whether they were tuned on the evaluation tasks, and their values, which are missing from both the paper and the anonymous code.
Additional Comments
A1. Figures 2 and 4 are computed from synthetic constructions (the discrete double-well kernel of Appendix C.2 and the three-variable family of Appendix C.4), not from the interaction logs, but neither the captions nor the main text (L124, L201-207) says so, and Appendix C.2 and C.4 are never cited near figures. Please state explicitly in both captions that these are synthetic illustrations, cite the corresponding appendix sections, and give the numerical kernel of the C.4 family, since Figure 4 cannot currently be reproduced from the text.
A2. Details of figures and experiment setup can be improved. (e.g., number in abstract is only Qwen3.8 results, not the average.)