- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
The paper groups user–agent sessions and repository checkpoints into work episodes, each covering one engineering goal. Several agents then generate the execution environment, task statement, quality rubrics, and unit tests, producing 242 CropChat tasks and 56 CropEnterprise tasks. For terra patches, test refinement reduces the false negative rate from 68.0% to 34.0%, but increases the false positive rate from 8.1% to 24.3%, measured against LLM correctness labels.
Critical strengths
The two test-refinement steps check different types of test errors. Criteria-based refinement checks whether the tests accept the gold patch and reject the base repository and mutants, while solver-based refinement checks a solution written independently by another agent. This addresses an important problem that tests need to catch incorrect implementations without requiring the solution to follow the gold patch.
Critical weaknesses
- The final test verdicts still look too unreliable for evaluation or training. In the selected audit, the refined suites reject up to 48.9% of patches labeled correct and accept 24.3–30.0% of patches labeled incorrect; the user audit checks the episodes, statements, and rubrics, but does not check these verdicts on new patches.
- Real users often begin with unclear requests and then clarify or revise them. If the method combines those later requirements into a self-contained task statement, the evaluated agent never has to ask for clarification. Appendix D proposes clarification benchmarks and training environments, but I do not see a tested user simulator or training results showing how the tasks would support those uses.
Questions for authors
- Is SESSIONFARMER currently used to evaluate or train models in a production workflow? In practice, if I want to use the method in production, how much manual checking and correction does each task need, and what error rates remain after that review?
- For the clarification benchmarks proposed in Appendix D, how would the user simulator answer questions that were not asked in the original sessions, or handle a different implementation proposed by the agent? Have you tested whether it can preserve the user's requirements without simply steering the agent toward the recorded solution?
4 comments
Response to Reviewer fL35 (Part 2/2)
(Part 2/2)
Q1: Have we applied SessionFarmer in production?
Yes, SessionFarmer is actively used in our internal production workflow across multiple repositories to build benchmarks for evaluating foundation models, agent harnesses, and self-evolving strategies:
- Upfront validation plus light spot-checking in daily runs: Most manual effort went into initial validation (about 0.5–1 hour per task, Figure 5). In daily production, task generation runs automatically and engineers spot-check fewer than 10% of tasks that show unusual pass patterns (such as unusually high variance across models), taking 0.5–1 hour per flagged task with roughly 5% residual error.
- Practical suggestion by data scale: For production use, when raw session logs are plentiful, using stricter automated checks avoids routine manual review. When logs in a target domain are scarce and high precision is needed, flagging unusual tasks from model pass statistics and spending about 1 hour per flagged task removes almost all problematic cases.
Q2: What's the performance of a user simulator when facing these situations?
Thanks for your insightful questions! To directly test the interesting scenarios you raised, we built a user simulator, give it different inputs as reference and tested it across 40 tasks (Appendix E):
| Performance of user simulator (40 tasks) | Refer session dialogue and recorded solution | Refer only session dialogue | Refer Task Statement (SessionFarmer) |
|---|---|---|---|
| Can give correct requirement. ↑ | 45.7% | 47.7% | 65.1% |
| Revives dropped requests? ↓ | 15.0% | 20.0% | 5.0% |
| Can accept valid implementations ↑ | 87.5% | 97.5% | 92.5% |
| Facing unasked design choices**:** Leaves choices open ↑ / Pushes gold design ↓ | 7.5% / 75.0% | 20.0% / 62.5% | 40.0% / 37.5% |
| Facing unmet questions: Leaks unprompted gold filenames or symbols ↓ | 35.0% | 15.0% | 2.5% (1/40) |
- Facing unclarified or unasked questions, the simulator answers from whatever details its input exposes: When asked about task requirements left unclear in the opening prompt, the simulator fills in the missing details from later user requests. However, when asked about internal design choices that were never discussed in the original session, a simulator that sees raw dialogues or the recorded solution tends to answer using the original agent's implementation and leak file or symbol names, whereas a simulator given only the
Task Statementis much more likely to leave those choices open to the agent. - The user simulator welcomes different implementations most of the time: Across all three settings, when the agent proposes a valid implementation plan written differently from the recorded solution, the simulator accepts it in the vast majority of cases.
- Can the user simulator preserve requirements without steering? It depends on the reference: When the simulator sees recorded solutions, it somtimes be distracted and steers the agent toward the recorded answer. By separating the solution from the simulator, it can preserve user requirements much more cleanly and largely avoids leaking code names or steering.
Thank you again for your valuable time and insightful review. We look forward to further discussion during the rebuttal period.
References: [1] Huang et al., DeepSWE, arXiv:2607.07946, 2026. [2] Deng et al., SWE-Bench Pro, arXiv:2509.16941, 2025. [3] Lu et al., FrontierCode, Cognition Blog: https://cognition.com/blog/frontier-code, 2026.
Thank you for the response. It helps me evaluate your method.
Thank you for your re-evaluation and constructive feedback
Thank you very much for your re-evaluation and for raising your score after reviewing our response. We are glad that our clarifications and new experiments have addressed your concerns regarding test verdict reliability (further validated through our human expert audit), the practical deployment and spot-checking effort of SessionFarmer in production, and the effectiveness of our Task Statement for user simulation and alternative implementations. We also appreciate your recognition of our two-stage test refinement design for catching incorrect patches without over-constraining valid solutions. Your insightful comments and questions directly motivated our new user simulator evaluation (Appendix E) and strengthened both the experimental evidence and practical discussion in the revised manuscript. Thank you again for your time and constructive feedback throughout the review process.
Response to Reviewer fL35 (Part 1/2)
(Part 1/2) Thank you for the thorough review and insightful questions. Below we address W1 and W2 in Part 1/2, and Q1 and Q2 in Part 2/2.
Thank you for closely examining our evaluation metrics! We address your concern below:
1. Verdict disagreement is in line with human-curated benchmarks
Even widely used human-curated coding benchmarks show similar disagreement rates under agent-based audits: DeepSWE [1] reports 8.5%/24.0% FP/FN on SWE-Bench Pro [2], and FrontierCode [3] reports 1.2%/44.9% on DeepSWE and 6.8%/36.0% on SWE-Bench Pro. Under the exact same audit setup (Table 3), our automated benchmark (
CropChat) sits between SWE-Bench Pro and DeepSWE (DeepSWE>CropChat>SWE-Bench Pro).Moreover, comparing model scores on
CropChatwith DeepSWE (Appendix C.2, Figure 14) gives the similar capability ordering (glm-5.2<gpt-5.5< {gpt-5.6-terra,kimi-k3}), showing that our benchmark reliably separates stronger and weaker models.Finally, please note that Table 3 uses
gpt-6-astraas a relative reference to check whether refinement improves the tests, not as absolute error rates.2. Human expert review on new solver patches reaches high agreement
We added a human expert review on 30
gpt-5.6-terrasolver patches, where four software engineers inspected the repositories and ran commands in the execution environments to check each test verdict:gpt-6-astra)Results show that human experts agree with our test verdicts on most patches (84.4%
BAcc). Compared with the LLM reference, human engineers are less overly strict while still agreeing with the LLM on 80.0% of patches.Thank you for your interest in Appendix D (Future Work). Following your feedback, we added data analyses and user simulator experiments in Appendix E (with full simulator test results in Q2 below):
Task Statementremoves file and symbol names that would otherwise leak the original code (Table 21), which could help user simulator to better focus on the main requirements:User PromptonlyUser Prompt + Agent Answer + Tool CallTask Statement(SessionFarmer)SessionFarmerbuilds reproducible Docker images matched to each task (Section 3.2) together with refined test and rubric rewards, providing ready-to-use RL environments and feedback signals. The training result you mention shows us an exciting next step! We will definitely try the policy training once we have the resources.