acceptodds
Under review as a conference paper at ICLR 2027

Harvesting SWE Benchmarks from Wild Vibe Coding Sessions

Abstract

Development teams need reproducible tasks and trustworthy evaluation signals that reflect their own engineering work to choose and improve coding agents. Meanwhile, as more developers start “vibe coding”, their everyday work leaves behind many in-the-wild coding sessions that record rich development context and could supply the raw material for such tasks. Yet these sessions are fragmented and loosely structured, making the valuable records hard to use directly. We present *SessionFarmer*, a framework that harvests these development records into executable software engineering (SWE) tasks. It uses developer–agent dialogue and code diffs to identify task boundaries, then brings together requirements that emerge across turns and sessions into coherent work episodes, each centered on one engineering goal. From each episode, it reconstructs a self-contained task specification. *SessionFarmer* also builds an executable environment and creates quality rubrics and unit tests for the task. A refinement loop is designed to reduce the chance that the tests accept incomplete implementations or reject valid alternatives. Using *SessionFarmer*, we construct CropChat with 242 tasks from public data and CropEnterprise from proprietary enterprise-level data. On CropEnterprise, evaluations by the original developers confirm the high fidelity and reliability of the reconstructed episodes and associated evaluation components. Furthermore, ablation experiments validate the effectiveness of the refinement stage in our construction pipeline. Together, these results show that *SessionFarmer* offers a promising path for unlocking everyday development records into reusable evaluation resources, with demonstrated applicability in real-world enterprise settings.

Primary area:
datasets and benchmarks
Keywords:
datasets and benchmarks, coding agents, software engineering, large language models, automated benchmark generation, evaluation reliability, developer-agent interaction, repository-level coding

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_fL35OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper groups user–agent sessions and repository checkpoints into work episodes, each covering one engineering goal. Several agents then generate the execution environment, task statement, quality rubrics, and unit tests, producing 242 CropChat tasks and 56 CropEnterprise tasks. For terra patches, test refinement reduces the false negative rate from 68.0% to 34.0%, but increases the false positive rate from 8.1% to 24.3%, measured against LLM correctness labels.

Critical strengths

The two test-refinement steps check different types of test errors. Criteria-based refinement checks whether the tests accept the gold patch and reject the base repository and mutants, while solver-based refinement checks a solution written independently by another agent. This addresses an important problem that tests need to catch incorrect implementations without requiring the solution to follow the gold patch.

Critical weaknesses
  1. The final test verdicts still look too unreliable for evaluation or training. In the selected audit, the refined suites reject up to 48.9% of patches labeled correct and accept 24.3–30.0% of patches labeled incorrect; the user audit checks the episodes, statements, and rubrics, but does not check these verdicts on new patches.
  2. Real users often begin with unclear requests and then clarify or revise them. If the method combines those later requirements into a self-contained task statement, the evaluated agent never has to ask for clarification. Appendix D proposes clarification benchmarks and training environments, but I do not see a tested user simulator or training results showing how the tasks would support those uses.
Questions for authors
  1. Is SESSIONFARMER currently used to evaluate or train models in a production workflow? In practice, if I want to use the method in production, how much manual checking and correction does each task need, and what error rates remain after that review?
  2. For the clarification benchmarks proposed in Appendix D, how would the user simulator answer questions that were not asked in the original sessions, or handle a different implementation proposed by the agent? Have you tested whether it can preserve the user's requirements without simply steering the agent toward the recorded solution?
4 comments
AuthorsOpenReview ↗

Response to Reviewer fL35 (Part 1/2)

(Part 1/2) Thank you for the thorough review and insightful questions. Below we address W1 and W2 in Part 1/2, and Q1 and Q2 in Part 2/2.

W1: The final test verdicts are not reliable enough and there is no user check of them.

Thank you for closely examining our evaluation metrics! We address your concern below:

1. Verdict disagreement is in line with human-curated benchmarks
Even widely used human-curated coding benchmarks show similar disagreement rates under agent-based audits: DeepSWE [1] reports 8.5%/24.0% FP/FN on SWE-Bench Pro [2], and FrontierCode [3] reports 1.2%/44.9% on DeepSWE and 6.8%/36.0% on SWE-Bench Pro. Under the exact same audit setup (Table 3), our automated benchmark (CropChat) sits between SWE-Bench Pro and DeepSWE (DeepSWE > CropChat > SWE-Bench Pro).

Moreover, comparing model scores on CropChat with DeepSWE (Appendix C.2, Figure 14) gives the similar capability ordering (glm-5.2 < gpt-5.5 < {gpt-5.6-terra,kimi-k3}), showing that our benchmark reliably separates stronger and weaker models.

Finally, please note that Table 3 uses gpt-6-astra as a relative reference to check whether refinement improves the tests, not as absolute error rates.

2. Human expert review on new solver patches reaches high agreement
We added a human expert review on 30 gpt-5.6-terra solver patches, where four software engineers inspected the repositories and ran commands in the execution environments to check each test verdict:

Reference SourcePatchesFP ↓FN ↓FPR ↓FNR ↓BAcc ↑LLM–Human Agreement
LLM Reference (gpt-6-astra)3010.0%20.0%23.1%35.3%70.8%80.0%
Human Expert Audit303.3%13.3%7.7%23.5%84.4%80.0%

Results show that human experts agree with our test verdicts on most patches (84.4% BAcc). Compared with the LLM reference, human engineers are less overly strict while still agreeing with the LLM on 80.0% of patches.

W2: Lack results of user simulator or training.

Thank you for your interest in Appendix D (Future Work). Following your feedback, we added data analyses and user simulator experiments in Appendix E (with full simulator test results in Q2 below):

  1. Support for user simulator: Turning raw chat logs into a requirement-focused Task Statement removes file and symbol names that would otherwise leak the original code (Table 21), which could help user simulator to better focus on the main requirements:
Input SourceModified Files Exposed ↓New Code Symbols Exposed ↓
Raw User Prompt only41.6%29.0%
Raw User Prompt + Agent Answer + Tool Call78.0%77.1%
Task Statement(SessionFarmer)2.5%4.5%
  1. Support for alternative solutions: When an unclear initial prompt allows multiple valid designs, which is a common cases for user simulator, our refined tests properly accept alternative solutions on 80.0% of multi-design tasks (Appendix E.2).
  2. Support for RL training: For RL training, SessionFarmer builds reproducible Docker images matched to each task (Section 3.2) together with refined test and rubric rewards, providing ready-to-use RL environments and feedback signals. The training result you mention shows us an exciting next step! We will definitely try the policy training once we have the resources.
AuthorsOpenReview ↗

Response to Reviewer fL35 (Part 2/2)

(Part 2/2)

Q1: Have we applied SessionFarmer in production?

Yes, SessionFarmer is actively used in our internal production workflow across multiple repositories to build benchmarks for evaluating foundation models, agent harnesses, and self-evolving strategies:

  1. Upfront validation plus light spot-checking in daily runs: Most manual effort went into initial validation (about 0.5–1 hour per task, Figure 5). In daily production, task generation runs automatically and engineers spot-check fewer than 10% of tasks that show unusual pass patterns (such as unusually high variance across models), taking 0.5–1 hour per flagged task with roughly 5% residual error.
  2. Practical suggestion by data scale: For production use, when raw session logs are plentiful, using stricter automated checks avoids routine manual review. When logs in a target domain are scarce and high precision is needed, flagging unusual tasks from model pass statistics and spending about 1 hour per flagged task removes almost all problematic cases.

Q2: What's the performance of a user simulator when facing these situations?

Thanks for your insightful questions! To directly test the interesting scenarios you raised, we built a user simulator, give it different inputs as reference and tested it across 40 tasks (Appendix E):

Performance of user simulator (40 tasks)Refer session dialogue and recorded solutionRefer only session dialogueRefer Task Statement (SessionFarmer)
Can give correct requirement. ↑45.7%47.7%65.1%
Revives dropped requests? ↓15.0%20.0%5.0%
Can accept valid implementations ↑87.5%97.5%92.5%
Facing unasked design choices**:** Leaves choices open ↑ / Pushes gold design ↓7.5% / 75.0%20.0% / 62.5%40.0% / 37.5%
Facing unmet questions: Leaks unprompted gold filenames or symbols ↓35.0%15.0%2.5% (1/40)
  1. Facing unclarified or unasked questions, the simulator answers from whatever details its input exposes: When asked about task requirements left unclear in the opening prompt, the simulator fills in the missing details from later user requests. However, when asked about internal design choices that were never discussed in the original session, a simulator that sees raw dialogues or the recorded solution tends to answer using the original agent's implementation and leak file or symbol names, whereas a simulator given only the Task Statement is much more likely to leave those choices open to the agent.
  2. The user simulator welcomes different implementations most of the time: Across all three settings, when the agent proposes a valid implementation plan written differently from the recorded solution, the simulator accepts it in the vast majority of cases.
  3. Can the user simulator preserve requirements without steering? It depends on the reference: When the simulator sees recorded solutions, it somtimes be distracted and steers the agent toward the recorded answer. By separating the solution from the simulator, it can preserve user requirements much more cleanly and largely avoids leaking code names or steering.

Thank you again for your valuable time and insightful review. We look forward to further discussion during the rebuttal period.

References: [1] Huang et al., DeepSWE, arXiv:2607.07946, 2026. [2] Deng et al., SWE-Bench Pro, arXiv:2509.16941, 2025. [3] Lu et al., FrontierCode, Cognition Blog: https://cognition.com/blog/frontier-code, 2026.

Thank you for the response. It helps me evaluate your method.

AuthorsOpenReview ↗

Thank you for your re-evaluation and constructive feedback

Thank you very much for your re-evaluation and for raising your score after reviewing our response. We are glad that our clarifications and new experiments have addressed your concerns regarding test verdict reliability (further validated through our human expert audit), the practical deployment and spot-checking effort of SessionFarmer in production, and the effectiveness of our Task Statement for user simulation and alternative implementations. We also appreciate your recognition of our two-stage test refinement design for catching incorrect patches without over-constraining valid solutions. Your insightful comments and questions directly motivated our new user simulator evaluation (Appendix E) and strengthened both the experimental evidence and practical discussion in the revised manuscript. Thank you again for your time and constructive feedback throughout the review process.

Reviewer_6ocqOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposed a SessionFarmer to store and manage development records for creating some executable software engineering tasks. These tasks are self-contained and have quality rubrics and unit tests. They also built CropEnterprise and CropChat and open-sourced these assets for reproducibility. After extensive experiments, they noticed many interesting findings such as model-dependent harness sensitivity and inconsistent outcomes across runs.

Critical strengths
  1. Compared with some work which only construct benchmark by Github Issues or PRs, it really transfer the interaction between real developer and coding agents into executable tasks.

  2. The SessionFarmer is an end-to-end framework. It is more useful than just generating task statement because the combination of environment, tests, and quality verification make benchmark reuseable.

  3. It evaluates and reports every stage during benchmark construction: requirement reconstruction, developer validation, test refinement, and benchmark evaluation.

Critical weaknesses
  1. For Table 3, how can we make sure of the reliability of the generated tests when we mainly use another LLM to judge the performance, not human judgment?

  2. For Figure 4, does the goal-centered episode reconstruction really beat session-based or heuristic grouping strategies? Is it only providing more context to the task author?

  3. For section 3.5, there remain some unclear things. The most interesting question is whether, given the filtering and limited enterprise evaluation, the resulting tasks are representative of real-world development work?

Questions for authors

Please refer to the Weaknesses.

1 comment
AuthorsOpenReview ↗

Response to Reviewer 6ocq

Thank you for carefully reviewing our paper and raising valuable questions. We address each of your points below.

W1: How to make sure the reliability of generated tests and LLM Judge.

Thanks for your question! In our opinion, we think the key point here is whether the LLM judge is reliable enough (even without human judgement):

  1. Prior research supports the reliability of our LLM Judge: Using an LLM agent to check code patches is well supported by Agent-as-a-Judge [1], which matches human reliability on coding tasks, and DeepSWE [2], which reports 8.5%/24.0% FP/FN on SWE-Bench Pro [3] using the task description, original code, and test outputs. Following the same setup with gpt-6-astra, our LLM Judge clearly separates SWE-Bench Pro (63.8% BAcc) and DeepSWE (76.3% BAcc), matching independent community findings [2, 4].
  2. Human expert review confirms both LLM Judge quality and test reliability: Four experienced software engineers reviewed 30 gpt-5.6-terra patches by inspecting the repositories and running scripts directly to judge whether our generated tests give correct verdicts.
Judge SourcePatchesFP ↓FN ↓BAcc ↑Human–LLM Judge Alignment
gpt-6-astra (LLM Judge)3010.0%20.0%70.8%—
Human Expert303.3%13.3%84.4%80.0% (24/30)

Our refined tests reach 84.4% BAcc against human judgments, meaning that most test results are reliable from humans' view, with the LLM acting stricter. Yet human experts and the LLM Judge still agree on 80.0% of patches, which proves the reliability of our LLM judge.

W2: Lack of comparison between episode-reconstruction and other strategies.

We understand "session-based" as treating each single session as one episode (per-session), and "heuristic grouping" as merging sessions by fixed time windows or branches (2h, 24h, and full-branch aggregation). Comparing 1,194 episodes built from 421 sessions across 28 repositories (Appendix B.7, Table 15) shows that episode-reconstruction does much more than simply adding longer context:

Aggregation MethodMean TokensMain-Goal Share ↑Task Suitability ↑Split Goals ↓
episode-reconstruction(Ours)21,45396.6%55.6%16.7% (22/132)
24h branch aggregation26,08186.4%21.3%71.2% (94/132)
full-branch aggregation29,75081.1%25.4%38.6% (51/132)
2h branch aggregation20,33287.0%28.2%46.2% (61/132)
per-session12,50996.1%22.1%100.0% (132/132)
  1. Adding more raw context alone hurts task quality: 24h and full-branch pass 22%–39% more tokens than our method, yet their share of usable episodes is less than half of ours (21.3%–25.4% vs. 55.6%, judged by a blind LLM with 93.3% human agreement). Coarse branch merging lowers main-goal dialogue from 96.6% to 81.1%–87.0%, making most episodes unusable because they mix unrelated tasks.
  2. Both heuristic and per-session baselines split real tasks: Across 132 cross-session tasks identified from commits and logs in this period, branch rules still split 38.6%–71.2% of them across multiple episodes (vs. 16.7% for ours). Meanwhile, per-session splits 100.0% of cross-session tasks, fails to form standalone tasks on 77.9% of episodes, and produces 26.5% duplicate tasks around the same code edits, vs. 0.0% for ours (Appendix B.7). Thus, episode-reconstruction works by finding clean task boundaries and removing unrelated noise, rather than stacking longer logs.

W3: Are resulting tasks representative of real-world development work?

Thanks for this insightful question! We analyzed all 168 merge requests (MRs) and 375 raw sessions collected over two months in CropEnterprise (Appendix B.6, Table 14), confirming that our filtering keeps representative team work:

  1. Filtering drops chat noise, not core code commits: There are more raw sessions than tasks because many sessions are pure Q&A or unmerged debugging without committed code. Our filtering drops only 15.4% of scale-eligible engineering tasks, producing 56 tasks covering 56 MRs.
  2. Task types and touched modules match overall team development: The type of tasks in CropEnterprise: 41.1% features and 55.4% bug fixes, closely match the task distribution along MRs in the same period (42.0% and 51.1%). The tasks also cover all 10 most active modules in the codebase, accounting for 94.5% (378/400) of module touches across all MRs, and are strongly endorsed by the original developers (Figure 5).

Thank you again for your constructive feedback, and we welcome any further questions during the discussion period.

References: [1] Zhuge et al., Agent-as-a-Judge, ICML 2025. [2] Huang et al., DeepSWE, arXiv:2607.07946, 2026. [3] Deng et al., SWE-Bench Pro, arXiv:2509.16941, 2025. [4] Lu et al., FrontierCode, Cognition Blog: https://cognition.com/blog/frontier-code, 2026.

Reviewer_YCRtOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes SessionFarmer, which is a framework that summarizes and converts the development records in agent-assisted vibe-coding activities to SWE tasks for benchmarking. For each identified task, the framework derives the episode from developer-agent dialogue and code diffs. With SessionFarmer, the authors construct two benchmark, CropChat and CropEnterprise, from public and proprietary agent session data. They further benchmark benchmark agents (popular models + agent harnesses) and analyze the results. Finally, they conduct ablation studies to validate the effectiveness of SessionFarmer’s approach.

Critical strengths

This paper addresses an important problem from an interesting idea: generating SWE benchmark from vibe coding session, which is abundantly available. The major challenge is recovering a coherent engineering goal from conversations whose storage locations are beyond task boundaries. This paper generated a 242-task public benchmark, detailed construction accounting, original-developer audit, and 12 model-harness evaluations make this a useful resource contribution.

Critical weaknesses

The main weakness is how the paper's technical section is written. The main technical session of the paper is a run-through of the approach step-by-step, without discussions of design choices and why the problem is solved that way. As an approach to construct benchmarks, showing the understanding of the problem domain and using it to guide the design of the benchmark is more important.

More conversation context is beneficial, but the episode-reconstruction mechanism is not directly validated against a construction baseline. Figure 4 compares statement generation from the first message, first session, and full episode on 34 multi-session and 20 selected single-session tasks. Coverage is judged by an LLM against the original constructed statement’s requirements. This favors agreement with an existing full-context artifact as the operational target, and cannot reveal requirements omitted from that target. Full-episode regenerated statement cover 75% of reference requirements on multi-session tasks and 78% on single-session tasks. There is no controlled comparison of the proposed relevance/episode procedure against a full-transcript per-session baseline or a simple time/branch aggregation baseline.

The evidence does not establish sufficiently reliable functional verdicts. The refined suites have FPR/FNR of 24.3%/34.0% for terra and 30.0%/48.9% for gpt-5.5. Balanced accuracy is 70.8% and 60.5%. Recovering integer confusion counts from the reported denominators gives 26/87 and 35/87 disagreements, respectively, or 29.9% and 40.2%. These are disagreements with an LLM reference, not independently established real error rates. The reference model sees test results and the gold patch, so hiding suite identities does not establish label independence from the tests’ behavior or the reference implementation.

Questions for authors

How does this approach decide whether two vibe sessions are similar and covering similar challenges?

2 comments
AuthorsOpenReview ↗

Response to Reviewer YCRt (Part 1/2)

(Part 1/2) Thank you for the thoughtful review and constructive feedback. Below we address W1 and Q1 together, followed by W2, with W3 in Part 2/2.

W1: Main technical session lacks discussion of design choices and why problem is solved.

Thank you for this valuable suggestion. Following your advice, we revised Section 3 (marked in blue) to first explain the main challenges in real-world vibe coding logs and coding evaluation, then show why our method can solve these challenges:

  1. Wild chat logs are messy: In daily vibe coding, developers rarely state all requirements at once. Most tasks span multiple turns (87.6%) and sometimes multiple sessions (14.0%), as users add details step by step and often drop earlier ideas. Treating each session as one task will drop ideas for final requirements, while grouping sessions simply by time or branch pulls in unrelated content. We therefore start from the final committed code changes to locate where a task begins and ends, and then read the conversations to recover the requirements that were actually kept (Section 3.1), which help to locate the boudary and relevant context of a task. Figure 5 in our paper shows that the original developers strongly agree with our recovered task boundaries and task descriptions.
  2. Coding solutions are hard to check: Most original code changes come without tests, and generated tests can fail in two ways: accepting incorrect implementations, or overfitting to the original function names and rejecting alternative solutions. We therefore check and refine generated tests from both sides (Section 3.4): they must reject broken solutions (mutant), and must fairly judge a solution written independently by another agent. This method increase the roubtness of our generated tests, lowers overall test verdict errors by 9-13 pp (raising BAcc by 7-9 pp) and halves the rate of FNR (68% → 34%), illustraed in Table3 (Section 4.4) in our paper.

Q1: How do we judge that two vibe sessions are relevant?

Base on the challegens discussed above, we decide to detect the relevance between two sessions by checking whether they contribute to the same final code commit in two steps:

  1. Filter by code contribution: We compare each session's transcript and edits against nearby git commits, and use an LLM to drop Q&A or exploratory chats that did not end up in the committed code.
  2. Group by shared goal: Reading the remaining sessions in time order, an LLM merges a later session with an earlier one only when the user is continuing, debugging, or revising the same engineering task.

More details are in Section 3.1, Appendix A.2 (with a multi-session walkthrough case), and Appendix F.1 (criteria and prompts).

W2: No controlled comparison between more aggregation baselines.

Thank you for this insightful observation. Figure 4 is built to show that task requirements are spread across an episode and not bounded to a single turn or session. Per your suggestion, we added controlled comparisons against construction baselines across 421 sessions (1,194 episodes) from 28 repositories in Appendix B.7 (Table 15):

Aggregation MethodTask Suitability ↑Split Goals ↓Mean TokensMain-Goal Share ↑
episode-reconstruction(Ours)55.6%16.7%21,45396.6%
per-session22.1%100.0%12,50996.1%
2h branch aggregation28.2%46.2%20,33287.0%
24h branch aggregation21.3%71.2%26,08186.4%
full-branch aggregation25.4%38.6%29,75081.1%
  1. Our method constructs episodes with higher quality: Blind LLM-Judge evaluation (93.3% agreement with human annotators, Appendix B.7) shows that 55.6% of episodes built by our method form clear, self-contained tasks. The percentage is nearly double of the baselines (21.3%–28.2%), where most of episodes generated by them fail due to missing a clear task or mixing unrelated goals.
  2. Our method keeps cross-session work together and context clean: Across 132 cross-session goals identified from commits and logs, episode-reconstruction splits only 16.7%, compared with 38.6%–71.2% for branch/time baselines and 100.0% for per-session. Moreover, while our relevance filtering keeps 96.6% of dialogue focused on the main goal, heuristic merging drops to ≤87.0% by pulling in nearly 20% unrelated chat noise.
AuthorsOpenReview ↗

Response to Reviewer YCRt (Part 2/2)

(Part 2/2)

W3: Functional verdicts are not reliable enough and LLM reference is not independent.

Thank you for closely examining our evaluation metrics. We address both verdict reliability and reference-label independence below:

1. Our refined tests are as reliable as mainstream human-curated benchmarks
Prior audits show that even widely used human-curated benchmarks have similar verdict disagreement on complex repository tasks: DeepSWE [1] reports 8.5%/24.0% (FP/FN) on SWE-Bench Pro [2], and FrontierCode [3] reports 1.2%/44.9% on DeepSWE and 6.8%/36.0% on SWE-Bench Pro. Under the exact same LLM-labeling setup (Table 3 in our paper), our refined tests beat SWE-Bench Pro and approach DeepSWE.

What's more, as noted in the paper, gpt-6-astra labels serve as a uniform reference to compare tests before and after refinement, not absolute ground truth.

2. Human experts highly endorse the tests quality.
To check verdict quality directly with humans, four experienced software engineers reviewed 30 gpt-5.6-terra solver patches with full access to the repositories and execution environments. Results show that human experts agree with our refined test verdicts much more often than gpt-6-astra, raising Balanced Accuracy (BAcc) from 70.8% to 84.4% while maintaining 80.0% agreement between human and LLM labels:

Reference SourcePatchesFP ↓FN ↓FPR ↓FNR ↓BAcc ↑LLM–Human Agreement
LLM Reference (gpt-6-astra)3010.0%20.0%23.1%35.3%70.8%80.0%
Human Expert Audit303.3%13.3%7.7%23.5%84.4%80.0%

3. Test refinement gains stay consistent when hiding input evidence from the reference LLM
Your point on label independence is very insightful. Following DeepSWE [1], our default reference LLM saw the original committed code (gold patch) and test outputs. To test whether those inputs bias our conclusion, we re-ran the reference LLM on the same 90 patches from Table 3 while hiding the gold patch and further hiding test outputs. The results shows that the that the benefit of our test refinement does not depend on giving reference LLM the gold patch or test outputs:

Reference LLM InputFP ↓ (w/o → Ours)FN ↓ (w/o → Ours)BAcc ↑ (w/o → Ours)Agreement w/ Original
Original (w/ gold patch & test evidence)3.4% → 10.3%39.1% → 19.5%61.9% → 70.8% (+8.9)Reference
Hide gold patch2.3% → 9.2%43.7% → 23.0%62.3% → 69.3% (+7.0)82.6%
Hide gold patch & test evidence1.2% → 7.1%41.7% → 20.2%65.4% → 74.3% (+8.9)78.6%

As shown in the table, labels from both blinded settings still agree closely with the original setup (78.6%–82.6%). More importantly, when comparing tests before and after refinement, the drop in FN and the BAcc gain (+7.0 to +8.9 pp) remain consistent across all settings.

Thank you again for your time and constructive feedback. We welcome any further questions during the discussion period.

References: [1] Huang et al., DeepSWE, arXiv:2607.07946, 2026. [2] Deng et al., SWE-Bench Pro, arXiv:2509.16941, 2025. [3] Lu et al., FrontierCode, Cognition Blog: https://cognition.com/blog/frontier-code, 2026.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

Dear Area Chair and Reviewers,

We sincerely thank you all for your time and constructive feedback. We are glad that the reviewers recognize our idea of building SWE benchmarks from daily vibe coding records, the end-to-end design of SessionFarmer, and the value of open-sourcing CropChat.

To address the reviewers' main concerns, we have added new experiments and revised our paper (all changes are highlighted in blue). Below is a summary of the key issues and our updates:

Main ConcernsRelated Weaknesses & QuestionsOur Responses & Updates
Technical Section & Design ChoicesYCRt-W1Revise Section 3 around two core challenges (fragmented wild logs and reliable code evaluation) to clearly explain the motivation behind our design choices and why they work.
Episode Reconstruction & DeduplicationYCRt-W2, 6ocq-W2, YCRt (Follow-up)Compared against 4 grouping baselines (Appendix B.7), showing clear gains over simply using longer context; added audits confirming minimal code-level redundancy (Appendix B.8).
Test Suite Reliability & Judge IndependenceYCRt-W3, 6ocq-W1, fL35-W1Verified that CropChat preserves model rankings consistent with DeepSWE (Appendix C.2); added a human expert review (high endorsement) and reference LLM ablations in our individual responses.
Real-World Coverage & Production Use6ocq-W3, fL35-Q1, YCRt (Follow-up)Analyzed all enterprise code (Appendix B.6), showing high coverage of active development modules; detailed our internal production workflow and workspace-grounded design principle.

Discussion Outcomes and Progress:

  1. Positive Consensus: Reviewers 6ocq and fL35 acknowledged our rebuttal and additional experiments, noting that our responses addressed their concerns on test reliability, grouping baselines, enterprise representativeness, and production use.
  2. Addressing Follow-up Concerns: In response to Reviewer YCRt's follow-up on code-level deduplication and dataset design principles, we added overlap audits and clarified the design principle of our benchmark. We believe these updates address the remaining concerns.

All other specific questions have been answered in detail in our individual responses. Thank you again for your valuable feedback, which has helped us greatly improve the quality and clarity of our paper and support the community.

Best regards, Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.