- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduces OS-Bench, a benchmark to evaluate cross-OS code migration abilities of language agents. It consists of tasks migrating from Windows to Linux and vice-versa. It contains 140 tasks from 111 repositories across several programming languages. Tasks are constructed from repositories via differential coverage. Evaluation is done across six models × four harnesses.
EDIT: Increased the score to 3 during rebuttal.
Critical strengths
- The evaluation metric is well defined, spanning across both target and source operating systems.
- Evaluation spans across multiple (then) state-of-the-art models and harnesses, giving insights into failures that are observed even now.
Critical weaknesses
- The agent’s goal is misdefined. The agents are prompted to just migrate the code over the target OS, and make sure that all test-cases pass on the target OS. Agent is never told to retain the capabilities on the source OS, leading to a conflict in task-instruction and evaluation metric.
- The effect of masking on task difficulty is unclear. An LLM replaces the selected target-specific implementations with placeholders designed to make the missing functionality unobvious. Although the authors manually verify the masking, they do not evaluate how much difficulty this introduces through artificial fault localization or whether the resulting tasks resemble realistic migration work.
- Claude-5-sonnet is run twice with Claude Code, while other configurations seem to be run once. The budget is defined differently for each configuration as well. Results under a common attempt budget, together with resource usage and a complete breakdown of budget-related terminations, would make the comparisons more reliable.
- The distinction from prior work should be clearer. SWE-bench-Live, introduced through RepoLaunch [1], already includes Windows-specific implementation tasks using existing Linux implementations.
- Potential training-data contamination is insufficiently discussed. The benchmark is constructed from well-known git repositories. These repositories are used for other SWE-related datasets/benchmarks, and are often included in the training data of the models.
- Minor correction: Harnesses like claude code are not just defined on linux, they can be used cross-platform as well.
[1] Li, Kenan, et al. "RepoLaunch: Automating Build and Management of Code Repositories across Languages and Platforms." arXiv preprint arXiv:2603.05026 (2026).
Questions for authors
See weaknesses above
4 comments
Rebuttal by Authors (2/2)
W4: SWE-bench-Live already includes Windows-specific implementation tasks.
OS-Bench focuses on a specific software engineering task: cross-OS code migration. In contrast, SWE-bench-Live/Windows is a general-purpose benchmark covering a broad range of Windows-related software engineering tasks, including PowerShell operations, rather than specifically targeting cross-OS migration.
Furthermore, SWE-bench-Live/Windows derives its tasks from GitHub issues, making its task coverage dependent on the availability of Windows-specific issues reported by developers on GitHub. If no such issues are raised, corresponding benchmark tasks cannot be constructed. In contrast, OS-Bench proposes a generalizable task generation method that can construct migration tasks from arbitrary repositories with cross-OS implementations, without relying on existing GitHub issues. This enables OS-Bench to generate a broader range of Windows-related tasks and offers greater scalability and generalizability across code repositories.
W5: Potential training-data contamination is insufficiently discussed.
We agree that potential training-data contamination should be discussed more explicitly, and we will address this in the limitations section. Nevertheless, the results show that current state-of-the-art models still achieve low success rates on OS-Bench. This suggests that, even with possible exposure to the underlying repositories, these models still struggle to complete cross-OS migration tasks successfully. This capability remains challenging and underexplored for current models and agents.
W6: Harnesses like claude code can be used cross-platform as well.
Claude Code(cc) can run on Windows hosts and operate on files stored on those hosts. However, in our experiments, it could not run successfully inside the Windows Docker containers used by OS-Bench. When attempting to install cc within the Windows container, an error was immediately reported, informing that cc cannot be installed in the Windows container environment. As cc is proprietary software, we are unable to debug and resolve this incompatibility. Since our Windows tasks are provided in these containers, we were unable to evaluate Claude Code on them.
We also confirmed that SWE-agent and OpenHands did not support the Windows-container execution setup used in our evaluation. OpenHands and SWE-agent hard-coded docker APIs that only work with Linux containers in their source code. Also, the two agents hard-coded Linux bash commands in their source codes, which cannot run in Windows environments. We will clarify that this limitation concerns our containerized evaluation setup. We will state this more clearly in the paper.
My concerns are addressed. I still slightly disagree regarding W4 (novelty concern), where authors agree that prior work is more generalized to what they propose, while covering the migration tasks as well.
However, I believe a more systematic benchmark and a in-depth study is useful. I have increased my score to 3.
Thank you for your prompt response and willingness to reconsider the score. We appreciate your recognition of the value of our benchmark. We wish you all the best in your future research!
Rebuttal by Authors (1/2)
We really appreciate your time and effort in reviewing our paper. We address each of your points below and have revised the paper in response to your feedback.
W1: Agent is never told to retain the capabilities on the source OS.
Thank you for raising this issue. We intended the phrase “code migration” to imply that the code should continue to work on the source platform. However, we agree that the agent’s instructions should state this requirement explicitly. We therefore reran the 28 instances using the same GPT‑5.6-Sol+Claude Code, with the following revised problem statement:
These 28 instances had previously passed evaluation on the target platform but had no validated pass on the source platform. In our new run with the revised prompt, 3 failed on the target platform. Only 2 passed source-platform validation, and both were among those 3 target-platform failures, so none passed on both platforms. Overall, the revised prompt had little effect and didn't improve LM’s internal ability, the differences are consistent with normal run-to-run variability.
W2: The effect of masking on task difficulty is unclear.
Thank you for highlighting this concern. Our benchmark targets end-to-end cross-platform migration, including identifying where target-platform functionality is missing and implementing the required behavior. Fault localization is therefore an intended component of the evaluation. Explicit error-raising placeholders can directly expose the modified locations through error messages, while silent placeholders retain the need to diagnose missing functionalities.
We carried out a preliminary experiment on the difficulty of different masking strategies. Using GPT-5.6-Sol and SWE-agent, the success rates of different masking strategies on Linux:
Regarding if the masked codes resemble realistic placeholders before migration, when using language model to mask code blocks, we explicitly prompted the language model to use those placeholders that are commonly used in real-world development or the native placeholders from the original repo. For example, in Windows-to-Linux task
trunk-rs__trunk-f1ee3d4__rust__linux__f1a2ab:The oracle answer:
The masked code:
The original repository already uses an
Ok(())stub for other unimplemented functions in other places, which means the unimplemented functions will be skipped during execution. The masked Unix version follows that same native pattern adopted in the original repo, so the masked blocks resemble realistic placeholders.W3: Claude-5-sonnet is run twice with Claude Code.
Thank you for spotting this issue. At the time of evaluating Claude-Sonnet-5, the enterprise collaborator funding our work imposed a monthly API budget limit. We therefore had to split its evaluation into two batches to comply with this policy. The two batches contained no overlapping instances, so each instance was evaluated only once. Combining these batches is equivalent to a single run over the full set. We included this detail in the paper for transparency, but we recognize that the wording caused confusion. We have revised the description to make this distinction clearer.