- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces SWE-Journey, a long-horizon benchmark for interactive coding agents. Authors propose a method for synthetically constructing long-horizon engineering tasks composed of multiple sub-tasks (3.2) and situating a coding agent with a simulated user (3.3), whose persona is grounded in dimensions derived from real-world users from SWE-Chat. The benchmark evaluates agents on 30 independent task journeys at both the process-level (scoring a sub-task whenever the user believes it has been completed) and the final-level (scoring at the end of the session).
Critical strengths
- This paper approaches a timely and important problem with a very novel approach. I am particularly impressed with the design of the user simulator: with its ability to read and explore code, persona’s grounded in real user conversations, and a confidence score which drives changes in behavior over the course of a dialogue. Each of these address key gaps in prior user simulators that make this evaluation compelling.
- I also really like the separation of process vs. final metrics, for evaluating sub-tasks both at the final state and at the time the user believes them to be done (though I am not sure I would use F2P/P2P within these, and would consider instead having a sub-task resolved % which groups these by sub-task, measured at process/final).
Critical weaknesses
Critical weaknesses:
- Only 30 independent task journeys are produced. Since failures can compound within a ‘journey’, I think a greater number of independent task journeys would improve the benchmark. The metrics in part address this, but are actually too granular in my view: while there are 6k F2P and 4k P2P ‘nodes’, the set of those assigned to each of the 594 sub-tasks should be highly correlated, and I’m not sure it's meaningful for a sub-task to be considered 80% correct. I think a sub-task resolution might be more appropriate, but don’t consider that to be the only or best solution to the metric granularity, and think >30 independent task runs would be better in either case.
- There remains an inevitable gap between synthetically produced problems (especially those which by construction are solvable by GLM 5.2), and real-world problems, as well as synthetic users and real-world users. Some analysis in the paper supports the value of this synthetic benchmark despite this gap, but I think the current analysis is not sufficiently convincing. There are different ways to address this with varying difficulty (user study, human annotation, etc.) and I am not sure of the best approach, but will just comment on the areas this is analyzed and places I think could be better validated in some way.
- One thing that I think should be validated is that the inclusion of the
evaluationtool on the user side doesn’t substantially bias the results of the benchmark. A pathological agent might simply ask the user to run theevaluationtool and share the corresponding results, and then close the gap by creating whatever patch is needed to pass the tests, whether semantically correct or not. Some analysis of the realisticness of theevaluationtool’s use and resulting feedback would address this. My concern reading the results is that the much higher performance under the software architect persona could be due to unrealistic use of this tool (Table 2), in which the user might state that tests don't pass rather than something a real user could actually be expected to verify. - A5 verifies tasks are solvable, and A7 evaluates their authenticity. However the source of the rubric and its validation in A7 is not clear. How is the rubric implemented, and does the rubric correlate with human judgements? How does it score real human tasks? For example SWE-Chat, which contains real coding sessions, scores low on Problem Clarity, but this may actually be a more realistic level of clarity than the higher scoring SWE-Journey.
- Similarly, the interaction rubric in A7 seems valuable, but its source or implementation isn’t described, and if it is new from this work, it’s not been validated as aligning with human judgements.
- One thing that I think should be validated is that the inclusion of the
Other less-critical weaknesses which I still think should be addressed:
- The paper does not clearly contextualize its use or relation to prior work, occasionally making it hard to understand which components are novel. In some cases this is due to citation placement, which is addressable but I think carefully placing this work in context of prior work is important:
- “Specifically, we start from an existing repository and its environment and ask Claude Opus 4.8 (Desai et al., 2026) to generate a high-level task blueprint and its subtasks, making them as close as possible to realistic software-engineering needs.”
- As written, this almost looks like the Desai citation is for the model Claude Opus 4.8, but it is actually a citation for the process of producing long-horizon tasks in SWE-Marathon. The novelty of the process in this paper relative to SWE-Marathon is not clearly described but should be in this paper.
- “For rubric-based verification, a separate model, GPT-5.4 (Lee et al., 2025), scores plan generation against a task-authenticity rubric (realistic motivation, clear task boundaries, no oracle information) and scores the Patch-and-F2P stage for semantic alignment among the request, acceptance criteria, and tests.”
- Similarly here, as written Lee et al. reads as a citation for GPT 5.4 but is actually for task-authenticity rubrics. The paper should clarify the relation to this paper: does it apply its method, tweak it, or just draw inspiration from its design? How do they differ?
- “Specifically, we start from an existing repository and its environment and ask Claude Opus 4.8 (Desai et al., 2026) to generate a high-level task blueprint and its subtasks, making them as close as possible to realistic software-engineering needs.”
- The persona taxonomy mining in 3.3 is novel but not described carefully enough to reproduce or validate (~L44-254). I think an appendix can describe its process more carefully, including the prompt to label interaction styles, any data sub-sampling or filtering, the deduplication process, the setting of k in k-means, the prompt and result to Deep Research, and the human feedback process.
- The LLM supporting the user simulator is a bit heavy-weight, which I think would limit adoption of this evaluation. Were any smaller user LLMs evaluated, and is the use of GLM 5.2 actually needed?
Questions for authors
- How does the long-horizon task generation process in 3.2 differ from the process in SWE-Marathon in Desai et al. 2026? Is it followed directly, modified, or simply related work? How do the pipelines differ?
- Does the user simulator receive any input that could contain or imply the gold solution? My understanding of the
evaluationtool (L266) would be that the simulator can actually guide the agent to a correct solution by just running the F2P/P2P tests until they pass. If so, how is this controlled for? - Where do the authenticity rubrics in A7 come from, and how are judges for this validated?
- In 3.2, where do seed tasks come from for the initial task in the journey?
6 comments
Response to reviewer cCNq (2/6)
References
- [1] Desai et al. SWE-Marathon: Can Agents Autonomously Complete Ultra-Long-Horizon Software Work? arXiv:2606.07682, 2026.
- [2] Oliva et al. SPICE: An Automated SWE-Bench Labeling Pipeline for Issue Clarity, Test Coverage, and Effort Estimation. ASE 2025.
- [3] Adamczewski et al. MirrorCode: AI can rebuild entire programs from behavior alone. arXiv:2606.30182, 2026.
- [4] Orlanski et al. SlopCodeBench: Benchmarking How Coding Agents Degrade Over Long-Horizon Iterative Tasks. arXiv:2603.24755, 2026.
- [5] Thai et al. SWE-EVO: Benchmarking Coding Agents in Long-Horizon Software Evolution Scenarios. arXiv:2512.18470, 2025.
- [6] Raghavendra et al. SWE-INTERACT: Reimagining SWE Benchmarks as User-Driven Long-Horizon Coding Sessions. arXiv:2606.30573, 2026.
- [7] Lam et al. SWE-Chain: Benchmarking Coding Agents on Chained Release-Level Package Upgrades. arXiv:2605.14415, 2026.
- [8] Zhao et al. Commit0: Library Generation from Scratch. ICLR 2025.
- [9] Shastry et al. Beyond Isolated Tasks: A Framework for Evaluating Coding Agents on Sequential Software Evolution. arXiv:2604.03035, 2026.
W2: There remains an inevitable gap between synthetically produced problems (especially those which by construction are solvable by GLM 5.2), and real-world problems, as well as synthetic users and real-world users. [...] I think the current analysis is not sufficiently convincing. [...]
Thanks for the suggestion, which helps improve our paper. Our extensive rubric-based and code-based verification already guarantees the tasks' semantic correctness, code correctness, and solvability. In response, we add human annotation results:
Two master's students in artificial intelligence performed the annotation on two targets. The first is task authenticity: for each instance's complete task tree, judged with the repository background and complete acceptance requirements, they score 5 task-authenticity dimensions from 1 to 5, defined as in the Appendix A.7 task-authenticity rubric. The second is interaction authenticity: for each instance's complete session under each of the four personas (120 sessions), they score 1–5 on five persona dimensions how well the session's concrete behavior matches the assigned persona definition. Exact agreement is 78.00% for task authenticity, with 100% within 1 point, and 87.50% for interaction authenticity, with 99.67% within 1 point. On the 1–5 scale, the merged means of the two records total 4.68 over the five task-authenticity dimensions and 4.65 over the four personas; per-dimension and per-persona means are below.
| Annotation target | Dimension or persona | Merged mean (1–5) |
|---|---|---|
| Task authenticity | repository-role fit | 4.95 |
| Task authenticity | economic/professional value | 4.70 |
| Task authenticity | problem-statement clarity | 4.20 |
| Task authenticity | cross-file collaboration | 4.83 |
| Task authenticity | skill compositionality | 4.70 |
| Task authenticity | Total, five dimensions | 4.68 |
| Interaction authenticity | Non-coder | 4.65 |
| Interaction authenticity | Product Manager | 4.82 |
| Interaction authenticity | New Developer | 4.92 |
| Interaction authenticity | Software Architect | 4.23 |
| Interaction authenticity | Total, four personas | 4.65 |
W2.1: [...] A pathological agent might simply ask the user to run the evaluation tool and share the corresponding results [...] My concern reading the results is that the much higher performance under the software architect persona could be due to unrealistic use of this tool (Table 2) [...]
Q2: Does the user simulator receive any input that could contain or imply the gold solution? [...] the simulator can actually guide the agent to a correct solution by just running the F2P/P2P tests until they pass. If so, how is this controlled for?
We thank the reviewer for the suggestion. First, only the Software Architect can run hidden tests, test details are restated as little as possible, and the test code is hidden from the model. Second, we have failed and confidence-zero mechanisms: when the model long fails to solve a problem, we do not give it unlimited chances; like a real human, the user loses confidence in it, skips the problem, or even exits the session. Finally, we do not claim that an LLM can fully simulate real users; we only state that, thanks to persona mining from real interaction data and our detailed simulator design, our simulator is more realistic than prior user simulators for coding assistance and yields effective, useful conclusions specific to this scenario. As Limitations states, the simulator "cannot fully capture human complexity; rather than claiming perfect human-level realism, we push beyond prior work to approximate real user behavior more closely". Specifically:
Response to reviewer cCNq (3/6)
(1) Only the Software Architect can run hidden tests, and test details are not leaked. First, only the Software Architect can run hidden tests. The Software Architect runs hidden tests on 89.0% of turns, and 0 for the other three personas. it is designed to simulate a maintainer who reviews code and verifies systematically with the test suite. Second, the simulator does not directly leak test details. It must never reveal test names, test files, assertions, or the existence of tests, and describes only observable behavior; every message also passes a non-leakage quality gate. We exact-matched all 23,010 simulator messages of the four personas against their instances' hidden tests, searching for full test node IDs, test function names, test file paths, and verbatim assertion lines from the test patch. Test function names matched only once, as part of a file name the assistant itself had listed earlier. Test file paths appear in 1.9% of Software Architect messages, mainly asking the assistant to delete redundant files it created on hidden-test paths. Our gate thus indeed prevents leakage. Third, the coding assistant can neither see nor run hidden tests: hidden test files are not in its workspace but are applied only temporarily inside the simulator's test tool. The assistant thus sees nothing it should not, avoiding any direct leakage.
(2) The simulated user does not retry without limit: a subtask that cannot progress is marked failed, or the user loses confidence in the assistant and skips it or even exits. Specifically, as part of its state, the simulated user records each subtask's status during the session: pending means not yet handed to the assistant; done or failed means the user has handed it over, the assistant has responded, and the user has judged the result. At session end, failed subtasks account for 1.7%, 1.2%, 1.1%, and 17.5% under the Non-coder, Product Manager, New Developer, and Software Architect. When the model long fails to make progress, the simulated user marks the subtask failed and skips it rather than retrying without limit; when this becomes widespread, the user lowers User Confidence, and at zero gives up and exits like a real human. The shares of sessions stopped early by User Confidence reaching zero are Non-coder 1.0%, Product Manager 0.3%, New Developer 0.6%, and Software Architect 28.1%. The markedly higher rates of failed subtasks and zero confidence under the Software Architect show precisely that the simulator does not keep running the F2P/P2P tests until they pass; it gives only limited chances and incomplete textual information, not enough to solve every task. This shows that even a professional user cannot make some weaker models complete arbitrary tasks.
(3) The assistant rarely asks the user to run tests. We first check the reviewer's "pathological assistant", i.e., whether the assistant scores high by asking the user to run tests. Across all main-experiment sessions, we flagged every assistant reply asking the user to run or verify something, or to share errors, outputs, or test results. This rule is broad, so the shares below are upper bounds. Under the Software Architect, 6.8% of replies contain such requests; under the other personas, 0.5% to 1.7%. Most requests are generic and relatively harmless, e.g., "Could you share the exact error output you're seeing?". Only 1.6% of Software Architect replies mention tests in such requests, and only 5 replies mention hidden tests, all from one session. More requests do not bring higher scores: across 12 models, the Software Architect request share and Final F2P have Spearman ρ = −0.10, so such tendencies gain no accuracy advantage. The assistant thus rarely asks the user to run tests, and the simulator does not relay test details to it.
W2.2: [...] However the source of the rubric and its validation in A7 is not clear. How is the rubric implemented, and does the rubric correlate with human judgements? How does it score real human tasks? [...]
W2.3: Similarly, the interaction rubric in A7 seems valuable, but its source or implementation isn’t described, and if it is new from this work, it’s not been validated as aligning with human judgements.
Q3: Where do the authenticity rubrics in A7 come from, and how are judges for this validated?
We thank the reviewer for the attention to rubric sources and validation, and explain below. The two rubrics address coding-assistance needs and were built by mining existing literature: every dimension is grounded in established criteria, and the most suitable dimensions were selected manually. As discussed under W2, human annotation serves as the final accurate scoring, and its high accuracy also demonstrates our benchmark's quality, avoiding the unreliability of relying on LLM as a judge and the difficulty of validating it.
Response to reviewer cCNq (1/6)
Thanks for the suggestion. Long-horizon benchmarks are costly to build and test. We therefore spent the limited budget on more personas rather than repeated runs, as four different personas contribute more than repeating one setting. Further, compared with other work, we have partly alleviated this common small-data problem: our synthesis pipeline is fully automatic and open-sourced, so instances can be expanded on demand for further use.
(1) Long-horizon benchmarks are costly to build and test, so comparable benchmarks are similar in scale to ours. For construction, SWE-Marathon's [1] 20 tasks were hand-written by 11 contributors, with an estimated 40 to 400 expert hours per task; even manually annotating one short SWE-bench-style task costs about \100[2].Fortesting,eachlong−horizontaskrequireslongcontinuousmodelwork:oneClaude−Opus−5sessioninourpapercosts42.13 on average, and each new instance needs one long multi-turn session for each of 12 models and 4 personas; task synthesis and the main experiments have already cost about \16k,excludingpilotexperimentsandanalysis.MirrorCode[3]reports2,600 and 19 days for one run on one of its large tasks. Given these markedly higher costs, comparable long-horizon benchmarks are all similar in scale: SWE-Marathon has 20 tasks, MirrorCode 25 target programs, SlopCodeBench [4] 36 problems, SWE-EVO [5] 48 tasks, SWE-INTERACT [6] 75 tasks (25 from each of three benchmarks), and SWE-Chain [7] 12 upgrade chains. Some benchmarks with more instances mainly evaluate on subsets because of cost: Commit0 [8] has 54 libraries, but "due to the complexity of Commit0 and budget constraints", most evaluations use a 16-library lite subset; Beyond Isolated Tasks [9] notes that "evaluating coding agents on long-horizon tasks is computationally expensive" and additionally provides 50-task and 15-task subsets.
(2) Our synthesis pipeline partly breaks this limitation. Existing long-horizon benchmarks mostly rely on manual writing or curation, so each new instance is costly; our pipeline automates both synthesis and quality assurance at \$56.64 per instance, even less than manually annotating one, and is open-sourced with the code. Users can thus expand instances on demand on any repository with the same pipeline; the resulting long-horizon tasks carry executable F2P/P2P tests, support larger-scale testing, and could even serve to build long-horizon coding training data. This cost will keep falling as stronger, cheaper models are released. We therefore believe our work makes an important contribution to the small data volumes caused by the high cost of long-horizon tasks.
(3) On node granularity, the subtask resolution rate gives the same conclusions. As suggested, we computed the subtask resolution rate from existing evaluation results. A subtask counts as solved when all its F2P and P2P test nodes pass. The two tables below correspond to Table 1 and Table 2, with the same settings as the main text.
Model rankings agree with the node-level metrics: ranking models by mean subtask resolution rate is highly consistent with ranking by Average in Table 1 (Spearman ρ = 0.965, p < 10⁻⁶). The persona gap is also more pronounced at the subtask level. Node-level and subtask-level metrics thus reach the same conclusions, and the latter importantly complements node-level metrics. We thank the reviewer again for this insightful suggestion and will add the full subtask resolution tables to the revision.