acceptodds
Desk rejected at ICLR 2027

SWE-Journey: Towards More Realistic Evaluation of Coding Assistants through Long-Horizon, Multi-Turn Interaction

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

Coding assistants such as Claude Code and Codex have become a major application of LLM agents, yet existing benchmarks remain far from real-world use, particularly in task horizon and interaction length. Code assistants require completing long chains of development work in continuously evolving repositories, while repeatedly clarifying requirements and adapting implementations through multi-turn interaction. To address these gaps, we introduce SWE-Journey, a benchmark for more realistic evaluation of coding assistants. To address the task-horizon gap, we propose a weak-to-strong synthesis pipeline that automatically constructs long-horizon coding tasks. To address the interaction gap, we mine four representative user personas from real interaction data and build a user-simulation agent to reproduce realistic code-assistance interactions. On average, models pass over 75% of tests for requested functionality with software architects, but fewer than 25% with non-coders. These results show that current coding assistants still fall short of enabling reliable coding for non-coders. We further analyze the reasons for this gap and identify asking right, finding right, and fixing right as key capabilities during interaction. Our code and benchmark are available at https://anonymous.4open.science/r/SWEJourney.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Code Assistants, LLM Agents, Benchmark, Long-Horizon Tasks, Multi-Turn Interaction, User Personas

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_cCNqOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces SWE-Journey, a long-horizon benchmark for interactive coding agents. Authors propose a method for synthetically constructing long-horizon engineering tasks composed of multiple sub-tasks (3.2) and situating a coding agent with a simulated user (3.3), whose persona is grounded in dimensions derived from real-world users from SWE-Chat. The benchmark evaluates agents on 30 independent task journeys at both the process-level (scoring a sub-task whenever the user believes it has been completed) and the final-level (scoring at the end of the session).

Critical strengths
  1. This paper approaches a timely and important problem with a very novel approach. I am particularly impressed with the design of the user simulator: with its ability to read and explore code, persona’s grounded in real user conversations, and a confidence score which drives changes in behavior over the course of a dialogue. Each of these address key gaps in prior user simulators that make this evaluation compelling.
  2. I also really like the separation of process vs. final metrics, for evaluating sub-tasks both at the final state and at the time the user believes them to be done (though I am not sure I would use F2P/P2P within these, and would consider instead having a sub-task resolved % which groups these by sub-task, measured at process/final).
Critical weaknesses

Critical weaknesses:

  1. Only 30 independent task journeys are produced. Since failures can compound within a ‘journey’, I think a greater number of independent task journeys would improve the benchmark. The metrics in part address this, but are actually too granular in my view: while there are 6k F2P and 4k P2P ‘nodes’, the set of those assigned to each of the 594 sub-tasks should be highly correlated, and I’m not sure it's meaningful for a sub-task to be considered 80% correct. I think a sub-task resolution might be more appropriate, but don’t consider that to be the only or best solution to the metric granularity, and think >30 independent task runs would be better in either case.
  2. There remains an inevitable gap between synthetically produced problems (especially those which by construction are solvable by GLM 5.2), and real-world problems, as well as synthetic users and real-world users. Some analysis in the paper supports the value of this synthetic benchmark despite this gap, but I think the current analysis is not sufficiently convincing. There are different ways to address this with varying difficulty (user study, human annotation, etc.) and I am not sure of the best approach, but will just comment on the areas this is analyzed and places I think could be better validated in some way.
    • One thing that I think should be validated is that the inclusion of the evaluation tool on the user side doesn’t substantially bias the results of the benchmark. A pathological agent might simply ask the user to run the evaluation tool and share the corresponding results, and then close the gap by creating whatever patch is needed to pass the tests, whether semantically correct or not. Some analysis of the realisticness of the evaluation tool’s use and resulting feedback would address this. My concern reading the results is that the much higher performance under the software architect persona could be due to unrealistic use of this tool (Table 2), in which the user might state that tests don't pass rather than something a real user could actually be expected to verify.
    • A5 verifies tasks are solvable, and A7 evaluates their authenticity. However the source of the rubric and its validation in A7 is not clear. How is the rubric implemented, and does the rubric correlate with human judgements? How does it score real human tasks? For example SWE-Chat, which contains real coding sessions, scores low on Problem Clarity, but this may actually be a more realistic level of clarity than the higher scoring SWE-Journey.
    • Similarly, the interaction rubric in A7 seems valuable, but its source or implementation isn’t described, and if it is new from this work, it’s not been validated as aligning with human judgements.

Other less-critical weaknesses which I still think should be addressed:

  1. The paper does not clearly contextualize its use or relation to prior work, occasionally making it hard to understand which components are novel. In some cases this is due to citation placement, which is addressable but I think carefully placing this work in context of prior work is important:
    • “Specifically, we start from an existing repository and its environment and ask Claude Opus 4.8 (Desai et al., 2026) to generate a high-level task blueprint and its subtasks, making them as close as possible to realistic software-engineering needs.”
      • As written, this almost looks like the Desai citation is for the model Claude Opus 4.8, but it is actually a citation for the process of producing long-horizon tasks in SWE-Marathon. The novelty of the process in this paper relative to SWE-Marathon is not clearly described but should be in this paper.
    • “For rubric-based verification, a separate model, GPT-5.4 (Lee et al., 2025), scores plan generation against a task-authenticity rubric (realistic motivation, clear task boundaries, no oracle information) and scores the Patch-and-F2P stage for semantic alignment among the request, acceptance criteria, and tests.”
      • Similarly here, as written Lee et al. reads as a citation for GPT 5.4 but is actually for task-authenticity rubrics. The paper should clarify the relation to this paper: does it apply its method, tweak it, or just draw inspiration from its design? How do they differ?
  2. The persona taxonomy mining in 3.3 is novel but not described carefully enough to reproduce or validate (~L44-254). I think an appendix can describe its process more carefully, including the prompt to label interaction styles, any data sub-sampling or filtering, the deduplication process, the setting of k in k-means, the prompt and result to Deep Research, and the human feedback process.
  3. The LLM supporting the user simulator is a bit heavy-weight, which I think would limit adoption of this evaluation. Were any smaller user LLMs evaluated, and is the use of GLM 5.2 actually needed?
Questions for authors
  1. How does the long-horizon task generation process in 3.2 differ from the process in SWE-Marathon in Desai et al. 2026? Is it followed directly, modified, or simply related work? How do the pipelines differ?
  2. Does the user simulator receive any input that could contain or imply the gold solution? My understanding of the evaluation tool (L266) would be that the simulator can actually guide the agent to a correct solution by just running the F2P/P2P tests until they pass. If so, how is this controlled for?
  3. Where do the authenticity rubrics in A7 come from, and how are judges for this validated?
  4. In 3.2, where do seed tasks come from for the initial task in the journey?
6 comments
AuthorsOpenReview ↗

Response to reviewer cCNq (1/6)

W1: Only 30 independent task journeys are produced. [...] The metrics in part address this, but are actually too granular in my view: while there are 6k F2P and 4k P2P ‘nodes’, the set of those assigned to each of the 594 sub-tasks should be highly correlated [...] I think a sub-task resolution might be more appropriate [...]

Thanks for the suggestion. Long-horizon benchmarks are costly to build and test. We therefore spent the limited budget on more personas rather than repeated runs, as four different personas contribute more than repeating one setting. Further, compared with other work, we have partly alleviated this common small-data problem: our synthesis pipeline is fully automatic and open-sourced, so instances can be expanded on demand for further use.

(1) Long-horizon benchmarks are costly to build and test, so comparable benchmarks are similar in scale to ours. For construction, SWE-Marathon's [1] 20 tasks were hand-written by 11 contributors, with an estimated 40 to 400 expert hours per task; even manually annotating one short SWE-bench-style task costs about \100[2].Fortesting,eachlong−horizontaskrequireslongcontinuousmodelwork:oneClaude−Opus−5sessioninourpapercosts100 [2]. For testing, each long-horizon task requires long continuous model work: one Claude-Opus-5 session in our paper costs \\42.13 on average, and each new instance needs one long multi-turn session for each of 12 models and 4 personas; task synthesis and the main experiments have already cost about \16k,excludingpilotexperimentsandanalysis.MirrorCode[3]reports16k, excluding pilot experiments and analysis. MirrorCode [3] reports \\2,600 and 19 days for one run on one of its large tasks. Given these markedly higher costs, comparable long-horizon benchmarks are all similar in scale: SWE-Marathon has 20 tasks, MirrorCode 25 target programs, SlopCodeBench [4] 36 problems, SWE-EVO [5] 48 tasks, SWE-INTERACT [6] 75 tasks (25 from each of three benchmarks), and SWE-Chain [7] 12 upgrade chains. Some benchmarks with more instances mainly evaluate on subsets because of cost: Commit0 [8] has 54 libraries, but "due to the complexity of Commit0 and budget constraints", most evaluations use a 16-library lite subset; Beyond Isolated Tasks [9] notes that "evaluating coding agents on long-horizon tasks is computationally expensive" and additionally provides 50-task and 15-task subsets.

(2) Our synthesis pipeline partly breaks this limitation. Existing long-horizon benchmarks mostly rely on manual writing or curation, so each new instance is costly; our pipeline automates both synthesis and quality assurance at \$56.64 per instance, even less than manually annotating one, and is open-sourced with the code. Users can thus expand instances on demand on any repository with the same pipeline; the resulting long-horizon tasks carry executable F2P/P2P tests, support larger-scale testing, and could even serve to build long-horizon coding training data. This cost will keep falling as stronger, cheaper models are released. We therefore believe our work makes an important contribution to the small data volumes caused by the high cost of long-horizon tasks.

(3) On node granularity, the subtask resolution rate gives the same conclusions. As suggested, we computed the subtask resolution rate from existing evaluation results. A subtask counts as solved when all its F2P and P2P test nodes pass. The two tables below correspond to Table 1 and Table 2, with the same settings as the main text.

ModelSubtask Proc.Subtask FinalSubtask AverageAverage (Table 1)
GPT-5.6-Sol40.4942.5741.5365.05
Claude-Opus-536.3439.8238.0865.41
GLM-5.231.0333.5432.2962.17
Kimi-K330.5532.6331.5960.78
Kimi-K2.723.2126.4524.8358.67
GLM-5.123.1725.3924.2854.62
Qwen3.5-397B-A17B21.0421.6221.3352.89
DeepSeek-V4-Pro19.5922.5421.0752.80
Hy320.4221.5320.9754.33
MiMo-V2.5-Pro19.7422.0120.8748.71
DeepSeek-V4-Flash18.8221.3820.1049.33
Qwen3.5-122B-A10B16.1716.2216.1945.37
PersonaSubtask Proc.Subtask FinalSubtask AverageAverage (Table 2)
Software Architect65.1564.8665.0184.26
New Developer15.0919.6417.3750.21
Product Manager11.8114.8213.3147.08
Non-coder8.149.238.6941.83

Model rankings agree with the node-level metrics: ranking models by mean subtask resolution rate is highly consistent with ranking by Average in Table 1 (Spearman ρ = 0.965, p < 10⁻⁶). The persona gap is also more pronounced at the subtask level. Node-level and subtask-level metrics thus reach the same conclusions, and the latter importantly complements node-level metrics. We thank the reviewer again for this insightful suggestion and will add the full subtask resolution tables to the revision.

AuthorsOpenReview ↗

Response to reviewer cCNq (2/6)

References

  • [1] Desai et al. SWE-Marathon: Can Agents Autonomously Complete Ultra-Long-Horizon Software Work? arXiv:2606.07682, 2026.
  • [2] Oliva et al. SPICE: An Automated SWE-Bench Labeling Pipeline for Issue Clarity, Test Coverage, and Effort Estimation. ASE 2025.
  • [3] Adamczewski et al. MirrorCode: AI can rebuild entire programs from behavior alone. arXiv:2606.30182, 2026.
  • [4] Orlanski et al. SlopCodeBench: Benchmarking How Coding Agents Degrade Over Long-Horizon Iterative Tasks. arXiv:2603.24755, 2026.
  • [5] Thai et al. SWE-EVO: Benchmarking Coding Agents in Long-Horizon Software Evolution Scenarios. arXiv:2512.18470, 2025.
  • [6] Raghavendra et al. SWE-INTERACT: Reimagining SWE Benchmarks as User-Driven Long-Horizon Coding Sessions. arXiv:2606.30573, 2026.
  • [7] Lam et al. SWE-Chain: Benchmarking Coding Agents on Chained Release-Level Package Upgrades. arXiv:2605.14415, 2026.
  • [8] Zhao et al. Commit0: Library Generation from Scratch. ICLR 2025.
  • [9] Shastry et al. Beyond Isolated Tasks: A Framework for Evaluating Coding Agents on Sequential Software Evolution. arXiv:2604.03035, 2026.

W2: There remains an inevitable gap between synthetically produced problems (especially those which by construction are solvable by GLM 5.2), and real-world problems, as well as synthetic users and real-world users. [...] I think the current analysis is not sufficiently convincing. [...]

Thanks for the suggestion, which helps improve our paper. Our extensive rubric-based and code-based verification already guarantees the tasks' semantic correctness, code correctness, and solvability. In response, we add human annotation results:

Two master's students in artificial intelligence performed the annotation on two targets. The first is task authenticity: for each instance's complete task tree, judged with the repository background and complete acceptance requirements, they score 5 task-authenticity dimensions from 1 to 5, defined as in the Appendix A.7 task-authenticity rubric. The second is interaction authenticity: for each instance's complete session under each of the four personas (120 sessions), they score 1–5 on five persona dimensions how well the session's concrete behavior matches the assigned persona definition. Exact agreement is 78.00% for task authenticity, with 100% within 1 point, and 87.50% for interaction authenticity, with 99.67% within 1 point. On the 1–5 scale, the merged means of the two records total 4.68 over the five task-authenticity dimensions and 4.65 over the four personas; per-dimension and per-persona means are below.

Annotation targetDimension or personaMerged mean (1–5)
Task authenticityrepository-role fit4.95
Task authenticityeconomic/professional value4.70
Task authenticityproblem-statement clarity4.20
Task authenticitycross-file collaboration4.83
Task authenticityskill compositionality4.70
Task authenticityTotal, five dimensions4.68
Interaction authenticityNon-coder4.65
Interaction authenticityProduct Manager4.82
Interaction authenticityNew Developer4.92
Interaction authenticitySoftware Architect4.23
Interaction authenticityTotal, four personas4.65

W2.1: [...] A pathological agent might simply ask the user to run the evaluation tool and share the corresponding results [...] My concern reading the results is that the much higher performance under the software architect persona could be due to unrealistic use of this tool (Table 2) [...]

Q2: Does the user simulator receive any input that could contain or imply the gold solution? [...] the simulator can actually guide the agent to a correct solution by just running the F2P/P2P tests until they pass. If so, how is this controlled for?

We thank the reviewer for the suggestion. First, only the Software Architect can run hidden tests, test details are restated as little as possible, and the test code is hidden from the model. Second, we have failed and confidence-zero mechanisms: when the model long fails to solve a problem, we do not give it unlimited chances; like a real human, the user loses confidence in it, skips the problem, or even exits the session. Finally, we do not claim that an LLM can fully simulate real users; we only state that, thanks to persona mining from real interaction data and our detailed simulator design, our simulator is more realistic than prior user simulators for coding assistance and yields effective, useful conclusions specific to this scenario. As Limitations states, the simulator "cannot fully capture human complexity; rather than claiming perfect human-level realism, we push beyond prior work to approximate real user behavior more closely". Specifically:

AuthorsOpenReview ↗

Response to reviewer cCNq (3/6)

(1) Only the Software Architect can run hidden tests, and test details are not leaked. First, only the Software Architect can run hidden tests. The Software Architect runs hidden tests on 89.0% of turns, and 0 for the other three personas. it is designed to simulate a maintainer who reviews code and verifies systematically with the test suite. Second, the simulator does not directly leak test details. It must never reveal test names, test files, assertions, or the existence of tests, and describes only observable behavior; every message also passes a non-leakage quality gate. We exact-matched all 23,010 simulator messages of the four personas against their instances' hidden tests, searching for full test node IDs, test function names, test file paths, and verbatim assertion lines from the test patch. Test function names matched only once, as part of a file name the assistant itself had listed earlier. Test file paths appear in 1.9% of Software Architect messages, mainly asking the assistant to delete redundant files it created on hidden-test paths. Our gate thus indeed prevents leakage. Third, the coding assistant can neither see nor run hidden tests: hidden test files are not in its workspace but are applied only temporarily inside the simulator's test tool. The assistant thus sees nothing it should not, avoiding any direct leakage.

(2) The simulated user does not retry without limit: a subtask that cannot progress is marked failed, or the user loses confidence in the assistant and skips it or even exits. Specifically, as part of its state, the simulated user records each subtask's status during the session: pending means not yet handed to the assistant; done or failed means the user has handed it over, the assistant has responded, and the user has judged the result. At session end, failed subtasks account for 1.7%, 1.2%, 1.1%, and 17.5% under the Non-coder, Product Manager, New Developer, and Software Architect. When the model long fails to make progress, the simulated user marks the subtask failed and skips it rather than retrying without limit; when this becomes widespread, the user lowers User Confidence, and at zero gives up and exits like a real human. The shares of sessions stopped early by User Confidence reaching zero are Non-coder 1.0%, Product Manager 0.3%, New Developer 0.6%, and Software Architect 28.1%. The markedly higher rates of failed subtasks and zero confidence under the Software Architect show precisely that the simulator does not keep running the F2P/P2P tests until they pass; it gives only limited chances and incomplete textual information, not enough to solve every task. This shows that even a professional user cannot make some weaker models complete arbitrary tasks.

(3) The assistant rarely asks the user to run tests. We first check the reviewer's "pathological assistant", i.e., whether the assistant scores high by asking the user to run tests. Across all main-experiment sessions, we flagged every assistant reply asking the user to run or verify something, or to share errors, outputs, or test results. This rule is broad, so the shares below are upper bounds. Under the Software Architect, 6.8% of replies contain such requests; under the other personas, 0.5% to 1.7%. Most requests are generic and relatively harmless, e.g., "Could you share the exact error output you're seeing?". Only 1.6% of Software Architect replies mention tests in such requests, and only 5 replies mention hidden tests, all from one session. More requests do not bring higher scores: across 12 models, the Software Architect request share and Final F2P have Spearman ρ = −0.10, so such tendencies gain no accuracy advantage. The assistant thus rarely asks the user to run tests, and the simulator does not relay test details to it.

W2.2: [...] However the source of the rubric and its validation in A7 is not clear. How is the rubric implemented, and does the rubric correlate with human judgements? How does it score real human tasks? [...]

W2.3: Similarly, the interaction rubric in A7 seems valuable, but its source or implementation isn’t described, and if it is new from this work, it’s not been validated as aligning with human judgements.

Q3: Where do the authenticity rubrics in A7 come from, and how are judges for this validated?

We thank the reviewer for the attention to rubric sources and validation, and explain below. The two rubrics address coding-assistance needs and were built by mining existing literature: every dimension is grounded in established criteria, and the most suitable dimensions were selected manually. As discussed under W2, human annotation serves as the final accurate scoring, and its high accuracy also demonstrates our benchmark's quality, avoiding the unreliability of relying on LLM as a judge and the difficulty of validating it.

Reviewer_9yRdOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

SWE-Journey evaluates coding assistants through dependent repository changes and interactions with simulated users. It combines progressive task synthesis, personas derived from real interaction logs, and executable checks of functionality and regressions in a persistent environment. The experiments compare models across personas and against a setting with all requirements provided upfront, then analyze clarification, code localization, and unaddressed problems.

Critical strengths
  1. The evaluation connects evolving requirements with persistent repository state and checks intermediate completion and final regressions. This captures an important aspect of coding assistance that isolated patch evaluation does not directly measure. The contribution is a useful evaluation setting, beyond simply increasing task length or dialogue turns.

  2. The synthesis pipeline combines explicit acceptance criteria, an independent model checking semantic alignment, and executable functionality and regression tests. These complementary safeguards give the generated tasks a credible basis for evaluation and make the construction procedure a useful methodological contribution.

Critical weaknesses
  1. The novelty is limited at the component level. Sustained repository development and adaptive user interaction already appear in the prior work discussed in Section 2. The clearest contribution is their integration into a new benchmark.

  2. The weak to strong claim is insufficiently substantiated. The synthesis pipeline also relies on stronger model planning and repeated verification, leaving the specific contribution of decomposition and progressive extension unclear.

  3. Persona representativeness remains uncertain. Combining traits that are individually overrepresented within an expertise group does not establish that their combination represents typical users. This limits conclusions about the broader user groups.

  4. The persona gap is difficult to interpret. Final scoring does not distinguish failures on communicated requirements from requirements never reached before the simulator ends the session. This ambiguity affects the central reliability claim.

  5. The clarification analysis remains descriptive. Ask% and Rel% measure questioning frequency and relevance without establishing whether ambiguity was resolved. They provide limited support for explaining differences in completion through clarification.

Questions for authors
  1. Which finding specifically depends on jointly evaluating sustained repository development and adaptive user interaction, beyond the capabilities of the settings discussed in Section 2?

  2. What evidence separates the benefits of decomposition and progressive extension from stronger planning and repeated verification?

  3. Do the selected persona traits commonly occur together in the source logs, and how is variation within each expertise group represented?

  4. Does the persona gap persist when evaluation is restricted to requirements actually communicated before the session ends?

  5. How does the analysis establish whether questions resolve ambiguity and improve subsequent task completion?

5 comments
AuthorsOpenReview ↗

Response to reviewer 9yRd (1/4)

W1: The novelty is limited at the component level. Sustained repository development and adaptive user interaction already appear in the prior work discussed in Section 2. The clearest contribution is their integration into a new benchmark.

Q1: Which finding specifically depends on jointly evaluating sustained repository development and adaptive user interaction, beyond the capabilities of the settings discussed in Section 2?

Thanks for the review, which helps us state our contributions more clearly. Sustained repository development and adaptive user interaction are goals we pursue simultaneously. These goals partly overlap with existing work because they are key problems for today's coding agents, which shows their importance. Technically, however, we differ clearly. First, no existing work achieves both, so coding-assistance findings that only the joint setting reveals remained undiscovered; second, we improve existing techniques to better fit coding assistance. We explain in two points below.

(1) Prior work has not achieved long-horizon development and adaptive interaction at the same time. Our premise is that realistic evaluation of coding assistants requires both long-horizon tasks and multi-turn user interaction, whereas prior work usually achieves only one. Long-horizon benchmarks generally give complete requirements upfront, examining neither gradual disclosure nor adaptation to user feedback; multi-turn benchmarks have short interactions and coarse user models, insufficient to capture assistant–user collaboration. Figure 1 shows that prior works each cover only part of the design space, while SWE-Journey is near the top tier on both axes. Achieving both lets us reach conclusions neither type of benchmark can:

First, the joint setting lets us discover and diagnose interaction patterns and problems that arise only in long-horizon interaction, providing deeper insight. We measure assistants along three consecutive stages, Asking Right, Finding Right, and Fixing Right, whose failure modes are observable only in the joint setting. (i) Only when the user's request is incomplete or vague must the assistant clarify proactively; most models currently ask on fewer than 20% of turns. (ii) Only when vague requests fall on an evolving long-horizon repository does localization often fail: the weakest model mislocalizes on nearly 60% of turns, and weaker models grow increasingly confident in the wrong direction. (iii) Only when user statements leave room for ambiguity and changes must persist over a long horizon does unjustified skipping of discovered problems cascade. Per-persona results also show that more cooperative personas yield better Finding Right and Fixing Right. We also observe models correcting earlier errors in later turns: every model's Final F2P exceeds its Proc. F2P, by 3.41 points on average.

Besides, such long-horizon multi-turn tasks reveal the gap between personas. On average, Final F2P is 77.95% for the Software Architect but only 22.06% for the Non-coder, a 55.90-point gap. In the traditional single-turn setting, we merge all of an instance's subtask requirements, which the user would disclose gradually, into one upfront prompt. Its Final F2P is always below the Software Architect and above the other three personas. The single-turn setting thus overestimates the assistant's performance with non-experts and underestimates it with experts. Existing multi-persona benchmarks with simple tasks cannot observe such a pronounced gap or these conclusions. All of them require long horizons and multiple turns together, offering clear, useful insight for real coding assistance.

(2) We go beyond prior work in both long-horizon task and multi-turn interaction. Prior benchmark designs can hardly meet our goals for coding assistance:

Long-horizon tasks. Prior long-horizon tasks are built in two main ways. One relies on costly expert annotation or fully manual construction: annotating even one short SWE-bench-style task costs about \$100 [1], and SWE-Marathon's [2] 20 tasks were hand-written by 11 contributors at markedly higher cost. The other mines public repository histories, risking leakage. We therefore synthesize tasks automatically, avoiding both. Such synthesis remains challenging: existing methods mostly generate isolated bug fixes or one-off repository builds, and none yet synthesizes coherent long-horizon tasks (Appendix A.1). The initial ideas of each module in our workflow have appeared before, e.g., SWE-Marathon's [2] task planning, SWE-Flow's [9] test-driven incremental synthesis, and RefineBench's [10] criteria-based independent review; these general data-synthesis building blocks inspired our workflow. However, adapting them to coding assistance and combining them into a pipeline that achieves such difficult long-horizon coding-task synthesis is equally hard and has not been done before. Specifically, our design covers four aspects.

AuthorsOpenReview ↗

Response to reviewer 9yRd (2/4)

First, in planning, SWE-Marathon's task proposals are human-written, whereas the stronger Claude Opus 4.8 automatically generates our high-level task blueprint and subtasks from a real repository and seed task; the new task must exceed the seed on at least two of four difficulty dimensions, and each round's task seeds the next, giving three progressively harder plan stages that scale in task number and horizon without human annotation. The tasks are indeed long-horizon: each instance chains 19.8 subtasks on average, with reference patches modifying 10.2 files and 768.3 lines in total. Second, in subtask implementation, SWE-Flow hollows out and restores existing code and tests, so it cannot produce functionality absent from the repository; instead, the weaker GLM-5.2 implements new requirements forward, each subtask building on the previous code state and generating in turn the user request and acceptance criteria, the reference patch and F2P tests, and P2P regression tests. Third, in quality control, RefineBench evaluates refinement with checklists derived from reference answers, but our pipeline has no reference answer in advance and must check the reference answer itself; building on much prior work, we instead designed coding-assistance rubrics to ensure data quality. A separate model, GPT-5.4, uses them to score task authenticity and the semantic consistency among requirements, acceptance criteria, and tests; a failed check triggers a retry of the current step with the feedback. Fourth, on data leakage, Appendix B.5 shows that the four most frequently retrieving models score 22.93 to 37.50 points higher Proc. F2P on seed subtasks with answers online than on our synthesized follow-ups, unlike less-retrieving models; seed subtasks thus carry a leakage risk that we avoid. With these improvements, the pipeline costs only \$56.64 per long-horizon task.

Multi-turn interaction. Prior multi-turn benchmarks have short interactions, coarse user models, and no persona system for coding assistants. We differ in two respects. First, our personas are finer-grained and better suited to coding assistance. Prior personas are either generic, coarse personality descriptions, such as Dialogue SWE-Bench's [3] five hand-written personalities, e.g., cautious, perfectionist, and pragmatic; or developers only, e.g., SWE-INTERACT [4] simulates only the Expert Nitpicker, SWE-Chat's most common senior developer, and all 15 personas induced by ToM-SWE [5] are developers. We instead mine five dimensions and four representative personas, from Non-coder to Software Architect, from SWE-Chat, real coding-assistant interaction data. Second, simulated users decide for themselves whether to keep collaborating, and verification ability differs by persona. Prior simulated users usually cooperate until satisfied; we instead model the user's internal state, including whether they consider a task complete and their confidence in the assistant. The simulator marks each subtask's completion by the persona's subjective judgment and maintains an evolving User Confidence; when it drops to zero, the user gives up midway, as a real user would switch to another product or model. Meanwhile, only the Software Architect can run hidden tests; the other personas cannot verify and must rely on the assistant's reports. Finally, interactions are clearly longer than in prior work: SWE-Journey averages 18.48 turns per session, versus reported averages of 5.44 for SlopCodeBench [7] and 8.73 for EvoCode-Bench [8]; under the Software Architect, it reaches 26.49.

References

  • [1] Oliva et al. SPICE: An Automated SWE-Bench Labeling Pipeline for Issue Clarity, Test Coverage, and Effort Estimation. ASE 2025.
  • [2] Desai et al. SWE-Marathon: Can Agents Autonomously Complete Ultra-Long-Horizon Software Work? arXiv:2606.07682, 2026.
  • [3] King and Flanigan. Dialogue SWE-Bench: A Benchmark for Dialogue-Driven Coding Agents. arXiv:2606.13995, 2026.
  • [4] Raghavendra et al. SWE-INTERACT: Reimagining SWE Benchmarks as User-Driven Long-Horizon Coding Sessions. arXiv:2606.30573, 2026.
  • [5] Zhou et al. TOM-SWE: User Mental Modeling For Software Engineering Agents. arXiv:2510.21903, 2025.
  • [6] Shastry et al. Beyond Isolated Tasks: A Framework for Evaluating Coding Agents on Sequential Software Evolution. arXiv:2604.03035, 2026.
  • [7] Orlanski et al. SlopCodeBench: Benchmarking How Coding Agents Degrade Over Long-Horizon Iterative Tasks. arXiv:2603.24755, 2026.
  • [8] Shen et al. EvoCode-Bench: Evaluating Coding Agents in Multi-Turn Iterative Interactions. arXiv:2605.24110, 2026.
  • [9] Zhang et al. SWE-Flow: Synthesizing Software Engineering Data in a Test-Driven Manner. arXiv:2506.09003, 2025.
  • [10] Lee et al. RefineBench: Evaluating Refinement Capability of Language Models via Checklists. arXiv:2511.22173, 2025.
AuthorsOpenReview ↗

Response to reviewer 9yRd (3/4)

W2: The weak to strong claim is insufficiently substantiated. [...]

Q2: What evidence separates the benefits of decomposition and progressive extension from stronger planning and repeated verification?

Thanks for the suggestion. By a weak-to-strong synthesis framework we do not mean decomposition and progressive extension alone, but progressive extension under effective guidance and feedback. Stronger planning and repeated verification are thus an indispensable part of weak-to-strong synthesis, and we do not believe decomposition and progressive extension can work well without them. Our preliminary experiments showed that stronger planning and repeated verification alone cause a cost surge and a markedly limited horizon, while decomposition and progressive extension alone markedly lower data accuracy, task value, and authenticity. The two modules are thus complementary and inseparable.

Further, "weak-to-strong" here has two senses. The first is letting a weaker model such as GLM-5.2 synthesize data close to that of a strong model such as Claude Opus 4.8, lowering cost. Stronger-model planning and repeated verification are most critical: weak-model plans have clear authenticity and completability problems, and their low quality and success rate actually raised cost in our tests, yet with repeated verification the weak model implements the tasks well. This is how they align weak-model synthesized data with strong-model capability. The second is letting these models synthesize tasks they themselves can hardly solve, achieved through decomposition and progressive extension: the model can solve each subtask, but once tasks are progressively extended and aggregated, it struggles to solve such long-horizon tasks well. Results confirm this: GLM-5.2 performs far below 100%: 67.85% Final F2P in the single-shot setting with all requirements given at once (Appendix B.2). The synthesized tasks thus exceed the solving ability of the model that synthesized them, and stacking further subtasks can keep raising difficulty, horizon, and the degree to which tasks exceed the model's own ability.

W3: [...] Combining traits that are individually overrepresented within an expertise group does not establish that their combination represents typical users. [...]

Q3: Do the selected persona traits commonly occur together in the source logs, and how is variation within each expertise group represented?

Thanks for the suggestion. As a benchmark, our personas aim to cover as many diverse persona types as possible, representing each professional group's most discriminative behavior relative to the whole population, not the behavior of statistically common average users.

We want the benchmark to cover users with widely differing behavior: we divide users into four groups by User Expertise and select one representative persona per group. Taking each group's most frequent values would make the personas highly similar: on the SWE-Chat sessions of Appendix A.3, whether taking each dimension's most common category or the most common joint combination, the Product Manager, New Developer, and Software Architect all become similar Brief / Hands-on / Evolving / Verdict-only personas. Behaviors such as Systematic Verification would go unsimulated, although 40.2% of Software Architects verify this way. Our rule selects, per dimension, the category with the highest within-group frequency multiplier relative to all logs, so each persona reflects its group's most discriminative behavior, covering 15 of the 18 categories across 5 dimensions. Appendix A.3 states that these personas "prioritize coverage of distinct collaboration conditions; they are not intended to reproduce the behavioral frequencies or joint distribution of the corresponding real-world user populations". For authenticity, the persona dimensions and categories are mined from real Claude Code logs; the only manual adjustment sets the Non-coder's Goal Stability to Pivoting, the second-highest multiplier, so that it differs from the Product Manager's collaboration conditions, serving broad coverage (Appendix A.3).

W4: The persona gap is difficult to interpret. [...]

Q4: Does the persona gap persist when evaluation is restricted to requirements actually communicated before the session ends?

Thanks for the suggestion. We set the budget loosely to avoid the problem you mention. Normally, all settings complete exactly the same task. Tasks can remain unfinished only when: 1) after discussion with the model, the user marks the current task as failed; 2) the model long fails to complete the task, and the user loses confidence and exits early. Both are model-capability issues and expected properties of the fixed task and persona, which is exactly what we want to test. Your concern should be 3) exiting after the maximum number of turns, which our statistics show is extremely rare, as discussed below:

Reviewer_WfMUOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes SWE-Journey, a benchmark for testing coding agents in multi-turn conversations over long-horizon repo tasks. Instead of giving the agent a complete issue description upfront, requirements come out gradually through an LLM user simulator. The authors build 30 tasks (with 594 chained subtasks and executable F2P/P2P tests) using a weak-to-strong synthesis pipeline, and define four user personas based on interaction patterns from SWE-Chat. They test 12 models in both interactive and single-shot setups. The main empirical finding is a huge gap across user types: models get 77.95% Final F2P on average when paired with a simulated Software Architect, but only 22.06% with a Non-coder. The paper also looks at process-level metrics and breaks down failures into asking, finding, and fixing.

Critical strengths

I like the problem this paper is trying to measure. A coding agent given a clean, well-specified issue faces a very different challenge in the real case like, figuring out what a user actually means, handling mid-stream revisions, and avoiding breaking earlier work across turns. Combining persistent repository state with an interactive user is a meaningful direction for benchmark design, and it gets much closer to how coding assistants are used in practice.

The benchmark construction and empirical setup are fairly substantial. The staged synthesis procedure, F2P/P2P checks, and process-time tracking make this more concrete than a simple collection of prompted chats. I also appreciate that the authors compare interactive runs against a single-shot setting and provide breakdowns across 12 models rather than only reporting an aggregate score.

Critical weaknesses

The main 56-point gap between Software Architects and Non-coders mixes too many factors to interpret cleanly. All five persona dimensions change at once, so we cannot tie the performance drop just to user expertise. Even more importantly, Non-coder chats are much shorter, yet final evaluation grades the entire task. It is very possible that agents paired with Non-coders lose points simply because the simulator stops early and never mentions later subtasks. A similar budget question exists in the single-shot comparison (250 steps per turn vs. a 2,000-step total cap, without reporting actual steps used).

I am not convinced the user simulator behaves like a real human, especially since it has access to privileged test info. Appendix A.3 shows the personas were hand-tuned to be harder rather than matching real SWE-Chat distributions, and the LLM judge in A.7 only checks if the text reads naturally. Also, the simulator can see the hidden checklist and run hidden tests, even when playing a persona that supposedly does no verification. A message filter stops it from leaking raw test assertions, but knowing which hidden test failed can still easily drive when the simulator complains or what it asks for next.

Questions for authors

Can you report Final F2P separately for subtasks that were actually mentioned to the agent vs. those that were never reached? It would clarify how much of the 56-point gap just comes from early stopping if one can know the average number of disclosed subtasks and termination reasons per persona.

How many command steps do models actually use in interactive runs vs. the 2,000-step single-shot baseline? Also, can you vary one persona dimension at a time on a few tasks to separate user expertise from disclosure and verification?

How often does each persona run the hidden tests, and does that affect its next message? Also, have you checked simulator behavior against real humans on these tasks beyond LLM naturalness scores?

3 comments
AuthorsOpenReview ↗

Response to reviewer WfMU (1/3)

W1: The main 56-point gap between Software Architects and Non-coders mixes too many factors to interpret cleanly. All five persona dimensions change at once, so we cannot tie the performance drop just to user expertise. Even more importantly, Non-coder chats are much shorter, yet final evaluation grades the entire task. It is very possible that agents paired with Non-coders lose points simply because the simulator stops early and never mentions later subtasks. A similar budget question exists in the single-shot comparison (250 steps per turn vs. a 2,000-step total cap, without reporting actual steps used).

Q1: Can you report Final F2P separately for subtasks that were actually mentioned to the agent vs. those that were never reached? It would clarify how much of the 56-point gap just comes from early stopping if one can know the average number of disclosed subtasks and termination reasons per persona.

Q2: How many command steps do models actually use in interactive runs vs. the 2,000-step single-shot baseline? Also, can you vary one persona dimension at a time on a few tasks to separate user expertise from disclosure and verification?

We thank the reviewer for the in-depth suggestion. In fact, this paper controls the task and the user persona, setting the budget as loosely as possible to avoid the problem you mention. Normally, all settings complete exactly the same task. Tasks can remain unfinished only when: 1) after discussion with the model, the user marks the current task as failed, i.e., the model cannot complete it; 2) the model long fails to complete the task, and the user loses confidence and exits early. Both are model-capability issues and expected properties of the fixed task and persona, which is exactly what we want to test. Your concern should be 3) exiting after the maximum number of turns, which our statistics show is extremely rare and also due to model capability: the model long fails to finish even under the relatively loose step limit. Excluding such data would instead introduce bias, as discussed below:

(1) Subtasks are almost all communicated. Our command-step setting gives the model relatively loose room while avoiding overly long paths from infinite loops: the single-shot setting thus allows 2,000 steps in total. Sessions average 401 to 761 actual steps across personas, far below this 2,000-step cap, and 0.40 to 1.35 times as many turns as subtasks, far below the 3-times turn cap. Hence no Non-coder session reaches the turn cap, and only 1.0% stop early because user confidence reaches zero. In addition, as part of its state, the simulated user records each subtask's status during the session: pending means not yet handed to the assistant; done or failed means the user has handed it over, the assistant has responded, and the user has judged the result. Subtasks done or failed at session end count as communicated, i.e., given to the model, which may have failed to complete them; those still pending count as uncommunicated. By this criterion, the communicated share is 99.3%, 99.8%, 99.5%, and 92.2% for the Non-coder, Product Manager, New Developer, and Software Architect. Termination reasons per persona, averaged over 12 models:

PersonaAll subtasks marked doneAll subtasks concluded, at least one failedEarly stop: User Confidence reached zeroTurn cap reached
Non-coder91.6%7.4%1.0%0%
Product Manager95.8%3.9%0.3%0%
New Developer89.1%10.3%0.6%0%
Software Architect37.2%32.4%28.1%2.3%

Subtasks are thus almost all communicated, and only the most expert architect ever reaches the turn cap: manual inspection showed that in each case the model long failed to solve the problem and used too many turns, so these stops reflect model capability and are expected, as are the other early stops from User Confidence reaching zero, which have the same cause. Uncommunicated subtasks arise only in these two cases. Restricting to communicated subtasks would therefore down-weight harder problems and cause potential bias: for weaker models, or personas with more turns, more hard data may be excluded, overestimating accuracy. In the data, Final F2P on communicated subtasks is 22.15% for the Non-coder and 83.44% for the Software Architect, widening the gap from 55.90 to 61.30 points, a bias toward the Software Architect, which more often has its confidence reach zero or exceeds the turn cap because the model long fails to make progress, and is thus further overestimated. Keeping only subtasks communicated in all sessions would instead under-cover hard problems.

We control the task and the persona themselves, the fairest setting and one close to real scenarios: with the same user completing the same task, how do different coding assistance systems perform? This setting is close to real usage and can further reduce bias.

AuthorsOpenReview ↗

Response to reviewer WfMU (2/3)

(2) The number of turns does not represent accuracy; it is a dependent variable under a fixed user and task. With these variables controlled, session turns form naturally as different users interact with the assistant: a dependent variable, not an independent one to control. The number of turns also does not determine accuracy: the New Developer and Software Architect average almost the same 25.90 and 26.49 turns, yet score 35.04% and 77.95% Final F2P. The Non-coder averages more turns than the Product Manager, 13.67 versus 7.87, yet scores lower, 22.06% versus 29.94%.

Weaker personas ending earlier is itself a persona effect: non-experts more readily trust the assistant's reports. The Non-coder, Product Manager, and New Developer cannot run hidden tests, so they often accept deliveries based on the assistant's report or their own judgment and conclude several subtasks in one turn, which is also why they almost never end early from confidence reaching zero. The case studies show this. In case F2 of Appendix C.3.2, DeepSeek-V4-Flash reports only one passing self-test and does not modify the batch path; the Product Manager has doubts but still accepts with "You said you tried it [...] then fine, I'm done", while most hidden tests still fail. In case G3 of Appendix C.4.3, the New Developer accepts Qwen3.5-397B-A17B's implementation report, ending with User Confidence 73 but F2P only 12/103. The low scores of weaker personas thus mainly come from users trusting the assistant and wrongly believing the task is complete. This is exactly the real non-expert scenario and the design purpose of the Proc. metric: it runs the tests when the user believes the task is complete, checking whether the user was "deceived". This also supports the conclusion of Section 5: "current assistants still struggle with reliable task completion, particularly when interacting with users with limited technical expertise".

(3) We measures the gap between typical users. Our unit of comparison is the complete persona, so the gap is not attributable to expertise alone; it is the gap between the typical complete personas of users at different expertise levels. Given differences across models and large gaps in base ability between users of different backgrounds, a complete ablation yielding reliable conclusions is very difficult and requires many costly experiments, so we narrowed the verification scope to avoid insufficiently verified conclusions. However, our framework supports arbitrary dimension combinations and the code is open-sourced, so users can evaluate on demand for their own scenarios. Limited by cost, this experiment is hard to add soon, but we commit to adding it in a later version. We are also very interested in such results, which will importantly reinforce our conclusions.

W2: I am not convinced the user simulator behaves like a real human, especially since it has access to privileged test info. Appendix A.3 shows the personas were hand-tuned to be harder rather than matching real SWE-Chat distributions, and the LLM judge in A.7 only checks if the text reads naturally. Also, the simulator can see the hidden checklist and run hidden tests, even when playing a persona that supposedly does no verification. A message filter stops it from leaking raw test assertions, but knowing which hidden test failed can still easily drive when the simulator complains or what it asks for next.

Q3: How often does each persona run the hidden tests, and does that affect its next message? Also, have you checked simulator behavior against real humans on these tasks beyond LLM naturalness scores?

We thank the reviewer for the suggestion. First, only the Software Architect can run hidden tests, test details are restated as little as possible, and the test code is hidden from the model. Second, we do not claim that an LLM can fully simulate real users; we only state that, thanks to persona mining from real interaction data and our detailed simulator design, our simulator is more realistic than prior user simulators for coding assistance and yields effective, useful conclusions specific to this scenario. As Limitations states, the simulator "cannot fully capture human complexity; rather than claiming perfect human-level realism, we push beyond prior work to approximate real user behavior more closely". Specifically:

AuthorsOpenReview ↗

Response to reviewer WfMU (3/3)

(1) Only the Software Architect can run hidden tests, and test details are not leaked. First, only the Software Architect can run hidden tests. For the Non-coder, Product Manager, and New Developer, the run-hidden-tests tool is removed from both the simulator's visible tool list and the tool dispatch table; logs show 0 calls for these three personas. The Software Architect runs hidden tests on 89.0% of turns; it is designed to simulate a maintainer who reviews code and verifies systematically with the test suite. Second, the simulator does not directly leak test details. It must never reveal test names, test files, assertions, or the existence of tests, and describes only observable behavior; every message also passes a non-leakage quality gate. We exact-matched all 23,010 simulator messages of the four personas against their instances' hidden tests, searching for full test node IDs, test function names, test file paths (full paths or file names containing test/spec), and verbatim assertion lines from the test patch. Test function names matched only once, as part of a file name the assistant itself had listed earlier. Test file paths appear in 1.9% of Software Architect messages, mainly asking the assistant to delete redundant files it created on hidden-test paths. Our gate thus indeed prevents leakage. Third, the coding assistant can neither see nor run hidden tests: it has no test tool, its prompt does not say the user can run tests, and hidden test files are not in its workspace but are applied only temporarily inside the simulator's test tool. The assistant thus sees nothing it should not, avoiding any direct leakage, while the simulator interacts with it like a real user who does code review.

(2) Interaction authenticity is demonstrated with LLM as a judge, human annotation, and case studies. Appendix A.7 uses LLM as a judge to score simulated sessions and real interaction logs on five interaction dimensions, preliminarily demonstrating the authenticity of our user simulation. Further, human annotation shows that the interaction trajectories match the assigned personas. Specifically, two master's students in artificial intelligence annotated each instance's complete session under each of the four personas (120 sessions), scoring 1–5 on five persona dimensions how well the session's concrete behavior matches the assigned persona definition. Every score comes with a reason based on instance content or session evidence. Exact agreement on interaction authenticity is 87.50%, with 99.67% within 1 point. On the 1–5 scale, the merged means of the two records total 4.65 over the four personas; per-persona means are below. The human annotation demonstrates the naturalness of the overall interaction, further corroborated by the ample case studies in Appendix C.

Annotation targetDimension or personaMerged mean (1–5)
Interaction authenticityNon-coder4.65
Interaction authenticityProduct Manager4.82
Interaction authenticityNew Developer4.92
Interaction authenticitySoftware Architect4.23
Interaction authenticityTotal, four personas4.65

(3) Persons adjustment aim for broad coverage. As a benchmark, our personas aim to cover as many diverse persona types as possible, representing each professional group's most discriminative behavior relative to the whole population, not the behavior of statistically common average users.

We want the benchmark to cover users with widely differing behavior. Our rule selects, per dimension, the category with the highest within-group frequency multiplier relative to all logs, so each persona reflects its group's most discriminative behavior and different values of each dimension are covered, yielding 4 distinct personas that cover 15 of the 18 categories across 5 dimensions. The only manual adjustment sets the Non-coder's Goal Stability to Pivoting, the second-highest multiplier, so that it differs from the Product Manager's collaboration conditions, serving broad coverage (Appendix A.3). Therefore, although this costs some authenticity, we made this minimal adjustment for broader coverage. It is worth noting that this value is also the second-ranked user trait for the current persona, so it does not seriously compromise authenticity.

Reviewer_Ao2fOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

SWE-Journey is a benchmark for coding assistants that targets two gaps the paper argues existing benchmarks cover only separately: long task horizons and multi-turn interaction with a user. The authors build 30 long-horizon instances comprising 594 subtasks with 6,332 fail-to-pass and 4,166 pass-to-pass test nodes using a weak-to-strong synthesis pipeline. For the interaction side they mine four user personas across five behavioural dimensions from real coding-assistant logs and drive a persona-conditioned user-simulation agent. Twelve models are evaluated; averaged over models, final fail-to-pass rates are 77.95% under the Software Architect persona and 22.06% under the Non-coder persona. Section 4.2 attributes the gap to asking, finding and fixing behaviours scored by an LLM judge. Code and benchmark are released.

Critical strengths
  1. Rather than a single simulated user, the benchmark uses four personas defined over five behavioural dimensions. This matters because in production, coding assistants are used by very different kinds of users.
  2. The full pipeline, not just the benchmark data, is open-sourced.
  3. Figure 1 gives a clear summary of the whole setup.
Critical weaknesses
  1. Few details are given on verification. For the human verification, it is not stated how many people took part or what their inter-annotator agreement was. The same applies to the metrics in Table 3, which are produced by an LLM judge: no annotator count or agreement is reported for its validation.
  2. Statistical issues. The benchmark has only 30 long-horizon instances, which is a low number for reporting any confidence intervals, and there is one run per model (Tables 1 and 2, report point estimates only), with no confidence intervals given.
Questions for authors

Clarification request:

  1. It is unclear, at least when reading the abstract, what this sentence means: "On average, models pass over 75% of tests for requested functionality with software architects, but fewer than 25% with non-coders. These results show that current coding assistants still fall short of enabling reliable coding for non-coders." Please clarify which metric and aggregation this refers to (I believe Table 2).
3 comments
AuthorsOpenReview ↗

Response to reviewer Ao2f (1/3)

W1: Few details are given on verification. For the human verification, it is not stated how many people took part or what their inter-annotator agreement was. The same applies to the metrics in Table 3, which are produced by an LLM judge: no annotator count or agreement is reported for its validation.

We thank the reviewer for this suggestion, which helps improve our paper. We add the setup and results below:

(1) Human annotation of task authenticity and interaction authenticity. Two master's students in artificial intelligence performed the annotation on two targets. The first is task authenticity: for each instance's complete task tree, judged with the repository background and complete acceptance requirements, they score 5 task-authenticity dimensions from 1 to 5, defined as in the Appendix A.7 task-authenticity rubric. The second is interaction authenticity: for each instance's complete session under each of the four personas (120 sessions), they score 1–5 on five persona dimensions how well the session's concrete behavior matches the assigned persona definition. Exact agreement is 78.00% for task authenticity, with 100% within 1 point, and 87.50% for interaction authenticity, with 99.67% within 1 point. On the 1–5 scale, the merged means of the two records total 4.68 over the five task-authenticity dimensions and 4.65 over the four personas; per-dimension and per-persona means are below.

Annotation targetDimension or personaMerged mean (1–5)
Task authenticityrepository-role fit4.95
Task authenticityeconomic/professional value4.70
Task authenticityproblem-statement clarity4.20
Task authenticitycross-file collaboration4.83
Task authenticityskill compositionality4.70
Task authenticityTotal, five dimensions4.68
Interaction authenticityNon-coder4.65
Interaction authenticityProduct Manager4.82
Interaction authenticityNew Developer4.92
Interaction authenticitySoftware Architect4.23
Interaction authenticityTotal, four personas4.65

(2) Human annotation of "Asking Right, Finding Right, Fixing Right". For Table 3, we check whether individual judgments are reasonable and whether model rankings stay stable under a different judge model.

Judging structure. User Confidence (Conf.) directly aggregates the simulator's self-reports without correctness judging. The other six columns come from three stages; the original judge is DeepSeek-V4-Flash-0731, with initial requests at temperature=0.

StageColumnsJudging material and what is judgedDefinition in Table 3
Asking RightAsk, RelFirst judge from the assistant's user-facing reply whether it requests information, a decision, or a confirmation; for question replies, then judge with the complete Task Checklist whether it is on-planAsk: question turns over all reply turns; Rel: on-plan question turns over all reply turns
Finding RightMiss, OCCombining the task background, the summary of the previous dialogue, the current turn's content, and the whole session's F2P results, judge the turn's final localization and whether confidence is reinforced in a wrong directionMiss: wrongly localized turns over turns with a formed localization, no-locus turns excluded from its denominator; OC: turns reinforcing confidence in a wrong direction over all localization-judged turns
Fixing RightSkip, CritFrom the available internal reasoning and commands, judge whether an already-noticed problem is skipped without grounds; for skipped turns, then judge with the Task Checklist whether the problem is criticalSkip: share of skipped turns; Crit: share of skipped-and-critical turns; both with qualified reasoning turns as the denominator
AuthorsOpenReview ↗

Response to reviewer Ao2f (2/3)

Random-sample human annotation. To further test the automatic annotation pipeline, we randomly sample 50 items per column for the six columns, uniformly without replacement under a fixed seed, covering the 12 evaluated models and 4 personas. A master's student in artificial intelligence annotated these 50×6 = 300 items under the same prompt guidance without ever seeing the model's annotations; only after the judgments were completed independently were they compared item by item with the automatic ones. Human and automatic annotations agree exactly in 87% of cases, and in at least 90% on Ask, Rel, and Skip, demonstrating the reliability of our automatic method and conclusions.

W2: Statistical issues. The benchmark has only 30 long-horizon instances, which is a low number for reporting any confidence intervals, and there is one run per model (Tables 1 and 2, report point estimates only), with no confidence intervals given.

We thank the reviewer for the concern about statistical reliability. However, long-horizon benchmarks are costly to build and test, and comparable ones generally have only a few dozen instances; our 30 are on par. We therefore spent the limited budget on more personas rather than repeated runs, as four different personas contribute more than repeating one setting. Further, compared with other work, we have in fact partly alleviated this common small-data problem, so it is to some extent our strength rather than our weakness: our synthesis pipeline is fully automatic and open-sourced, so instances can be expanded on demand for further use.

(1) Long-horizon benchmarks are costly to build and test, so comparable benchmarks are similar in scale to ours. For construction, SWE-Marathon's [1] 20 tasks were hand-written by 11 contributors, with an estimated 40 to 400 expert hours per task; even manually annotating one short SWE-bench-style task costs about \100[2].Fortesting,eachlong−horizontaskrequireslongcontinuousmodelwork:oneClaude−Opus−5sessioninourpapercosts100 [2]. For testing, each long-horizon task requires long continuous model work: one Claude-Opus-5 session in our paper costs \\42.13 on average, and each new instance needs one long multi-turn session for each of 12 models and 4 personas; task synthesis and the main experiments have already cost about \16k,excludingpilotexperimentsandanalysis.MirrorCode[3]reports16k, excluding pilot experiments and analysis. MirrorCode [3] reports \\2,600 and 19 days for one run on one of its large tasks. Given these markedly higher costs, comparable long-horizon benchmarks are all similar in scale: SWE-Marathon has 20 tasks, MirrorCode 25 target programs, SlopCodeBench [4] 36 problems, SWE-EVO [5] 48 tasks, SWE-INTERACT [6] 75 tasks (25 from each of three benchmarks), and SWE-Chain [7] 12 upgrade chains. Some benchmarks with more instances mainly evaluate on subsets because of cost: Commit0 [8] has 54 libraries, but "due to the complexity of Commit0 and budget constraints", most evaluations use a 16-library lite subset; Beyond Isolated Tasks [9] notes that "evaluating coding agents on long-horizon tasks is computationally expensive" and additionally provides 50-task and 15-task subsets.

AuthorsOpenReview ↗

Response to reviewer Ao2f (3/3)

(2) Our synthesis pipeline partly breaks this limitation. Existing long-horizon benchmarks mostly rely on manual writing or curation, so each new instance is costly; our pipeline automates both synthesis and quality assurance at $56.64 per instance, even less than manually annotating one, and is open-sourced with the code. Users can thus expand instances on demand on any repository with the same pipeline; the resulting long-horizon tasks carry executable F2P/P2P tests, support larger-scale testing, and could even serve to build long-horizon coding training data. This cost will keep falling as stronger, cheaper models are released. We therefore believe our work makes an important contribution to the small data volumes caused by the high cost of long-horizon tasks.

References

  • [1] Desai et al. SWE-Marathon: Can Agents Autonomously Complete Ultra-Long-Horizon Software Work? arXiv:2606.07682, 2026.
  • [2] Oliva et al. SPICE: An Automated SWE-Bench Labeling Pipeline for Issue Clarity, Test Coverage, and Effort Estimation. ASE 2025.
  • [3] Adamczewski et al. MirrorCode: AI can rebuild entire programs from behavior alone. arXiv:2606.30182, 2026.
  • [4] Orlanski et al. SlopCodeBench: Benchmarking How Coding Agents Degrade Over Long-Horizon Iterative Tasks. arXiv:2603.24755, 2026.
  • [5] Thai et al. SWE-EVO: Benchmarking Coding Agents in Long-Horizon Software Evolution Scenarios. arXiv:2512.18470, 2025.
  • [6] Raghavendra et al. SWE-INTERACT: Reimagining SWE Benchmarks as User-Driven Long-Horizon Coding Sessions. arXiv:2606.30573, 2026.
  • [7] Lam et al. SWE-Chain: Benchmarking Coding Agents on Chained Release-Level Package Upgrades. arXiv:2605.14415, 2026.
  • [8] Zhao et al. Commit0: Library Generation from Scratch. ICLR 2025.
  • [9] Shastry et al. Beyond Isolated Tasks: A Framework for Evaluating Coding Agents on Sequential Software Evolution. arXiv:2604.03035, 2026.

Q1: Clarification request: It is unclear, at least when reading the abstract, what this sentence means: "On average, models pass over 75% of tests for requested functionality with software architects, but fewer than 25% with non-coders. These results show that current coding assistants still fall short of enabling reliable coding for non-coders." Please clarify which metric and aggregation this refers to (I believe Table 2).

We thank the reviewer for pointing this out and clarify it here.

(1) The metric is Final F2P, averaged over 12 models per persona. The sentence refers to the per-persona Final F2P in Table 2. The main text states its metric and aggregation, but the abstract does not, so we will revise the abstract. The statement is in the "Metrics" paragraph of Section 3.4, which defines the metric and the first aggregation step. It defines F2P as: "The Fail-to-Pass rate (F2P) counts test nodes that previously failed and pass after the coding assistant's edits, measuring how well the model completes the required functionality". The "tests for requested functionality" in the abstract are these F2P tests. The same paragraph states the first aggregation step: "For each model and persona, we pool passed and recorded test-node counts across instances". That is, for each model and persona, we separately sum passed and recorded F2P test nodes across instances and divide them to obtain the pass rate.

(2) We will revise the abstract and clarify the related statement in the introduction. We will change the abstract sentence "On average, models pass over 75% of tests for requested functionality with software architects, but fewer than 25% with non-coders." to: "Averaged over twelve models, the final fail-to-pass test-node pass rate reaches 77.95% when the simulated user is a software architect but only 22.06% with a non-coder."

Comments on the paper

1 comment

Dear Reviewers,

We have posted our responses to the reviews. As we approach the author response deadline on November 18 (AoE), we would be grateful if you could read our response and briefly indicate whether your main concerns have been addressed. Thank you for your time and consideration.

Thanks,

Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.