acceptodds
Under review as a conference paper at ICLR 2027

KS Gov: Long-Running AI Discovery and Adversarial Testing from a System Prompt

Abstract

Users ask coding agents to make a system faster or more robust and let them run for hours. Such a run outlives the context window, and one untested change can undo a day’s work. KS Gov is a coding agent built for such runs: five Python modules and 4,449 significant lines under a 3,851-word system prompt. The prompt states engineering rules and two procedures: an AI discovery and optimization loop that logs tried ideas to files, and adversarial testing by alternating break and fix sub-tasks. One task-prompt sentence adds a read-only reviewer from a second vendor. In two long runs the agent built a key-value store that sustains 5.50 Mops/s on YCSB-A over 17.2 agent hours, and made a TPC-H engine 34.5× faster than its single-threaded artifact. The first ran under an earlier one-paragraph discovery instruction, with the adversarial steps written in its task prompt; only the second ran under the seven-step loop. In both, the reviewer found defects the benchmark hid. Each case study is one run without a baseline agent. Terminal-Bench 2.0 tests the loop and tools without the two procedures: under a shorter coding prompt on the HarnessTax study’s 30 tasks and seven models, the agent solves 75.6% of attempts (74.1% counting solves past the study’s 100-turn cap as failures) against 70.0% for Pi, the best published harness, a success gap within sampling error. Against Pi rerun by us on one model, the paired success gap is +11.1 points, at a cost we cannot tell apart.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
AI agents, long-horizon tasks, agent harness, SWE agents, AI-driven discovery

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 4 reviews, above 0% of the others.
Reviewer_ZWcWOpenReview ↗
Rating
1
Confidence
1
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper presents KS Gov, a coding-agent framework designed for long-running software development tasks. Authors argue that the system combines context management, persistent state, and prompt-based procedures for discovery and adversarial testing. The authors demonstrate the framework through two case studies involving a key-value store and a query engine, along with experiments on Terminal-Bench 2.0

Critical strengths
  • Practical motivation. Supporting long-running coding tasks while maintaining progress and correctness is considered as an important problem in LLM agent research.
Critical weaknesses
  • Writing clarity. The presentation is almost impossible to follow, particularly the relationships between the system components, proposed procedures, and experimental evaluations.
  • Unclear evaluation. The two case studies lack comparable baselines and ablations, making it difficult to attribute the reported improvements to the proposed procedures. Also, the terminal-bench experiments seems to omit the discovery and adversarial-testing procedures, so the results don't seem directly validate the paper's main ideas.
Questions for authors
  • Can the authors provide ablations to isolate the contributions of the proposed procedures and individual system components?

  • Can the authors clarify which findings from the Terminal-Bench experiments support the proposed approach to long-running discovery and adversarial testing?

Reviewer_FURSOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper presents KS Gov, a coding agent designed for tasks that run for hours. The central claim is that by encoding two procedures in plain English inside a system prompt: a seven-step "AI discovery and optimization loop" (which maintains state in Markdown files to prevent retrying failed ideas) and "adversarial testing" (alternating break/fix sub-tasks). A general-purpose LLM can carry out long-horizon engineering tasks spanning multiple context windows without any loop written in code. The system consists of five Python modules covering a ReAct loop, a Relentless Agent that resumes from model-written summaries, a per-task git worktree, a parallel tool dispatcher, and a design that adds a read-only reviewer from a second vendor via a single sentence in the task prompt. Two case studies are reported: HydraKV, a key-value store reaching 5.50 Mops/s on YCSB-A over 17.2 agent hours, and a TPC-H engine made 34.5× faster than its single-threaded artifact. On Terminal-Bench 2.0's 30 tasks across seven models, the pooled success rate of 75.6% is within sampling error of Pi's 70.0% (the best published harness); a paired same-model rerun against Pi shows a +11.1% gap (p = 0.016).

Critical strengths
  1. Clear and practically grounded problem framing. The failure modes such as state loss across context-window boundaries, the bug fixer being the same model that wrote the bug, overfitting to a single benchmark workload, are genuine pain points in long-horizon agent work, and the paper articulates them concretely.

  2. Strong openness and reproducibility. The complete logs, tests, audit reports, and process records for both case studies are released. This is a high transparency standard for agent engineering papers and provides the community with a usable reference implementation.

  3. The paired experiment is comparatively rigorous. The Pi rerun uses the same model, same protocol, and same price table, yielding a +11.1% paired gap ([+3.3, +20.0], p = 0.016), which is the most defensible quantitative result in the paper.

  4. Honest acknowledgment of limitations Section 6 and several other places explicitly state that each case study is a single run without a baseline agent, that the causal effect of the procedures cannot be read from the record, and that the two procedures ran in different wording across the two cases. This self-limitation is accurate and commendable.

Critical weaknesses
  1. Causal evidence for the core claim is entirely absent. This is the fundamental issue. The paper's most important claim that the two procedures embedded in the system prompt drive the long-task successes, has no ablation support in either case study. HydraKV ran under an older one-paragraph instruction; TPC-H ran under the seven-step loop; the two procedures were in different versions; and each ran exactly once. The Terminal-Bench experiment then explicitly removes both procedures. The paper therefore provides no controlled evidence that the procedures produced any effect. The work reads more accurately as an engineering case report than as a method with demonstrated causal impact.

  2. The Terminal-Bench headline number is statistically unsupported. The pooled 75.6% vs Pi's 70.0% across 30 tasks and seven models falls within sampling error, as the authors themselves acknowledge. The problem is that this within-noise result is placed prominently in the abstract and introduction, creating an impression of superiority that the evidence does not support. Only the paired rerun (+11.1%) is statistically defensible. These two numbers need to be distinguished more clearly in presentation.

  3. The boundary between engineering contribution and research contribution is unclear The five-layer architecture, tool design, and parallel dispatcher are careful and valuable engineering, but they are an engineering system rather than a scientific finding. The research question like "can plain-text prompts substitute for orchestration code in long-horizon tasks?" is meaningful, but the current evidence cannot support conclusions beyond "this system completed these two tasks." The paper still lacks generalizable scientific findings.

  4. Case studies lack comparative baselines. Neither long-task case study includes a baseline agent. Without one, it is hard to assess how much of the 5.50 Mops/s and 34.5× speedup reflects the method's contribution versus fable-5's raw capability or a sufficiently large compute budget.

  5. Incomplete cost analysis The paper notes that HydraKV's first task cost $81 but does not systematically report the cost distribution across all tasks or discuss the relationship between the run_parallel sub-agent overhead and the performance gains. The Terminal-Bench section correctly notes that different price lists preclude cost comparison, but the overall cost picture remains incomplete.

  6. Writing structure is not reader-friendly Section 2 presents five layers of architectural detail before readers understand what these components are for. Figure 1, the clearest summary in the paper, appears only in Section 2.3. Maybe placing it earlier would help. Key content such as the full system prompt and process records lives in the appendix, which the main text cites frequently but describes minimally, adding to reading load.

Questions for authors

Q1. Were the two procedures (AI discovery loop and adversarial testing) ablated individually or in combination, even on a small subset of Terminal-Bench? If not, what prevented this, and is there a plan to add it?

Q2. The Relentless Agent resumes from model-written summaries. How is the quality of these summaries maintained? Across five context-window restarts in the 17.2-hour HydraKV task, were there any cases where summary information loss or error caused the task to regress?

Q3. The second-vendor reviewer is instructed "not to invent new problems." How was compliance with this assessed? Across both case studies, what fraction of the reviewer's suggestions were accepted and implemented by the developer, and what fraction were rejected?

Q4. The Terminal-Bench pooled experiment used a "shorter coding prompt" that omits the two procedures. How large is the difference between this shorter prompt and the full 3,851-word system prompt? How much of the success-rate variation in Table 2 across models could be attributable to the prompt difference rather than the harness architecture?

Q5. StateM reports 8–9% same-model gains on Terminal-Bench 2.1, in the same range as the paper's +11.1 paired gap. The key stated distinction is "state and procedures live in a prompt rather than a runtime." How do the authors see this comparison? For which task types would the "state-in-prompt" versus "state-in-runtime" distinction matter most, and in which direction?

Reviewer_9keoOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper introduces "KS Gov," a long-running coding agent designed to solve context exhaustion, state loss, and code breakage in extended software engineering tasks. Its core contributions are a 5-layer architecture with file-based state persistence, and the innovative shift of complex "AI discovery" and "adversarial testing" loops from hard-coded scripts into a comprehensive natural language system prompt. But this contribution is not evaluted rigorisly in the experiments.

Critical strengths

Strengths: This work is going to solve a real-world problem. The topic is a practical and useful topic worthy investigation. The author designs a proper task to test if his prompt really works in a long-running coding task.

Critical weaknesses

Weaknesses & Experimental Flaws This paper is poorly written, is not organised as an research paper, but a Tech blog or Readme.md of his code. For example, Sections 1 & 2: Instead of establishing a theoretical foundation or elaborating on the research gap, the authors abruptly present hard engineering metrics—such as "five Python modules," "4,449 significant lines," and a "3,851-word system prompt" with alongside specific file paths like ./tmp/ideas.md. Section 3: The core case studies (HydraKV and TPC-H) lack scientific rigor and unfold like an engineering post-mortem. The narrative exhaustively details API costs (e.g., "$81"), wall-clock hours, sequential debugging steps, human-in-the-loop "steering messages," and specific crash logs like "out-of-memory" and "deleted-key resurrection". There is highly unconventional subsection title, "Result, and what it does not show"?

Besides, regarding the technical part.

  1. The summary-restart mechanism heavily depends on the LLM's summarization accuracy. Any hallucinations or omitted details during session handoffs could permanently corrupt the project context.

  2. The core case studies (HydraKV and TPC-H) are single runs without controlled baselines or ablation studies, making it impossible to attribute the success strictly to the prompt-driven methodology rather than the underlying model's inherent capabilities.

  3. The quantitative Terminal-Bench 2.0 benchmark explicitly disabled the core discovery prompts, meaning the paper's primary innovation was not quantitatively evaluated. And the performance claim for HydraKV relies on a static reference number for FASTER, rather than a controlled comparison run on identical hardware.

  4. in Section 4: The quantitative benchmark (Terminal-Bench 2.0) appears completely disconnected from the paper's core contributions. The authors explicitly admit that to run this benchmark, they removed the general-purpose instructions, reducing the prompt from 3,851 words to 727 words. Consequently, this benchmark explicitly "leaves out the two procedures" (the AI discovery loop and adversarial testing), meaning the primary innovations of the paper were entirely excluded from the quantitative evaluation.

  5. In Section 6: The Limitations section preemptively lists critical methodological flaws that undermine the paper's scientific validity. The authors concede that their core case studies are merely "single runs" with "no rerun," "no controlled baseline agent," and completely lack any "ablation of the procedure". Furthermore, they admit that their primary performance claim for HydraKV relies merely on comparing their result against a static "figure in its task text," rather than empirical execution on a shared baseline.

Questions for authors
  1. can you summarize where is the research gap and what is your main contribution to fill the research gap?
  2. can you briefing how you design your experiments to prove the effectiveness? Soecifically, the work should rely on translating the "AI discovery loop" and "adversarial testing" into natural language prompts. However, the case studies (HydraKV and TPC-H) are single runs without controlled baselines. Could the authors provide ablation study data (e.g., running the same long-horizon tasks with the discovery and adversarial prompt instructions removed) to empirically prove that the success is due to the prompt-driven workflow rather than the underlying capabilities of the foundation models (Claude Fable 5 / GPT-5.6-Sol)?
  3. In Section 4 (Terminal-Bench 2.0 evaluation), the authors state that the general-purpose instructions were removed, explicitly noting that "the benchmark tests the loop, tools, and coding rules only" and leaves out the AI discovery and adversarial testing procedures. If the quantitative benchmark entirely excludes the paper's primary methodological innovations, what is the scientific value of this specific evaluation? Can the authors design or provide a benchmark evaluation that actually quantifies the effectiveness of the long-running AI discovery loop?
  4. The paper claims a 5.9x performance speedup over FASTER (5.50 Mops/s vs. 0.93 Mops/s) for the HydraKV case study. However, it is explicitly acknowledged that this comparison is against a static figure stated in the prompt, rather than an actual execution of FASTER. Why wasn't FASTER deployed and tested in the exact same hardware environment (GCP n2-standard-64) to provide a scientifically rigorous, apples-to-apples baseline comparison?
  5. The system uses gpt-5.6-sol to review the code generated by claude-fable-5. Given that gpt-5.6-sol generally demonstrates stronger performance in standard benchmarks (as also suggested in Table 2), are the discovered vulnerabilities attributable to the "cross-vendor perspective," or simply because a more capable model was used for the secondary check? Do the authors have control experiments using homogeneous review pairs (e.g., Claude reviewing Claude) to isolate these confounding variables?
Reviewer_rfw3OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper presents KS Gov, an AI coding agent designed to support long-running tasks, including code optimization, debugging and adversarial testing. The proposed system consists of five layers and relies on a system prompt to guide the agent through an iterative process of generating ideas, implementing changes, evaluating results and maintaining a history of previous attempts. The authors evaluate their approach through two case studies involving a key-value store and a TPC-H query engine, as well as a comparison with existing coding agent harnesses on Terminal-Bench 2.0.

Critical strengths

The paper investigates long-running AI agents, which is an interesting and timely research direction. The authors provide experimental results on two practical optimization tasks, demonstrating the capabilities of their system. The evaluation includes a comparison with existing coding agent harnesses on Terminal-Bench 2.0, which makes the setup relatable to most readers.

Critical weaknesses

I have 4 main concerns with this paper. First, the paper lacks a clear problem formulation and scientific contribution, despite the fact there is a contribution sub-section in introduction. I had difficulty understanding the exact research problem, the limitations of existing approaches and what is actually novel in the proposed system. Second, the positioning of the paper is confusing. The title emphasizes AI discovery, but the experiments primarily focus on code optimization and debugging. I do not see sufficient evidence supporting the AI discovery claim. Furthermore, the relationship with existing approaches, such as AlphaEvolve, is not sufficiently established. Third, the experimental evaluation is limited. I would expect additional benchmarks beyond Terminal-Bench 2.0 to better demonstrate the generalizability of the approach. It is also unclear how the different experiments collectively support the main research contribution. Lastly, the organization and presentation need significant improvement. I found the main narrative difficult to follow and some section titles feel generic. Overall, the manuscript reads more like an AI-assisted technical report.

Questions for authors

I have a few questions for the authors:

  1. What is the exact research problem the authors aim to address and what are the main novel contributions of the proposed approach? I suggest providing a clearer problem formulation and explicitly distinguishing the contributions from existing methods.
  2. How do the authors define AI discovery in the context of this work? How do the presented experiments demonstrate discovery rather than code optimization and debugging?
  3. How does KS Gov differ from AlphaEvolve and other existing approaches? What specific limitations of these methods does the proposed approach address?
  4. Why was Terminal-Bench 2.0 selected as the only coding-agent benchmark? Could the authors include additional benchmarks and clarify how the experimental results support the claimed contributions?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.