acceptodds
Under review as a conference paper at ICLR 2027

OS-Bench: Evaluating SWE Agents on Cross-OS Code Migration

15 views

Abstract

Language-model (LM) agents are increasingly evaluated on repository-level software engineering (SWE) benchmarks derived from GitHub issues and pull requests. Although these benchmarks cover a broad range of development tasks, they provide limited insight into a common yet technically distinct challenge: migrating an existing implementation from one operating system (OS) to another while preserving behavior on both. We introduce OS-Bench, an executable benchmark for cross-OS code migration between Linux and Windows. Motivated by case studies showing that cross-platform features are frequently implemented on one OS before support is added for another, OS-Bench identifies migration opportunities by comparing execution coverage collected from the same repository on both systems. Platform-specific regions are then verified, organized into five migration scenarios, and transformed into repository-level tasks with target-platform tests and source-platform regression tests. The resulting benchmark contains 140 validated tasks from 111 repositories across C, C++, C#, Go, and Rust, comprising 91 Windows-to-Linux and 49 Linux-to-Windows migrations. We evaluate six models with SWE-agent, Claude Code, OpenHands, and Win-agent. The best configurations achieve only 26.4% success on Linux-target tasks and 20.4% on Windows-target tasks while preserving correctness across both systems. These results indicate that cross-OS migration is a challenging and underexplored capability, and OS-Bench is a testbed for improving platform-aware SWE agents.

Primary area:
datasets and benchmarks
Keywords:
SWE-agent, coding agent, operating system, code migration

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_x6qjOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces OS-Bench, a benchmark to evaluate cross-OS code migration abilities of language agents. It consists of tasks migrating from Windows to Linux and vice-versa. It contains 140 tasks from 111 repositories across several programming languages. Tasks are constructed from repositories via differential coverage. Evaluation is done across six models × four harnesses.

EDIT: Increased the score to 3 during rebuttal.

Critical strengths
  1. The evaluation metric is well defined, spanning across both target and source operating systems.
  2. Evaluation spans across multiple (then) state-of-the-art models and harnesses, giving insights into failures that are observed even now.
Critical weaknesses
  1. The agent’s goal is misdefined. The agents are prompted to just migrate the code over the target OS, and make sure that all test-cases pass on the target OS. Agent is never told to retain the capabilities on the source OS, leading to a conflict in task-instruction and evaluation metric.
  2. The effect of masking on task difficulty is unclear. An LLM replaces the selected target-specific implementations with placeholders designed to make the missing functionality unobvious. Although the authors manually verify the masking, they do not evaluate how much difficulty this introduces through artificial fault localization or whether the resulting tasks resemble realistic migration work.
  3. Claude-5-sonnet is run twice with Claude Code, while other configurations seem to be run once. The budget is defined differently for each configuration as well. Results under a common attempt budget, together with resource usage and a complete breakdown of budget-related terminations, would make the comparisons more reliable.
  4. The distinction from prior work should be clearer. SWE-bench-Live, introduced through RepoLaunch [1], already includes Windows-specific implementation tasks using existing Linux implementations.
  5. Potential training-data contamination is insufficiently discussed. The benchmark is constructed from well-known git repositories. These repositories are used for other SWE-related datasets/benchmarks, and are often included in the training data of the models.
  6. Minor correction: Harnesses like claude code are not just defined on linux, they can be used cross-platform as well.

[1] Li, Kenan, et al. "RepoLaunch: Automating Build and Management of Code Repositories across Languages and Platforms." arXiv preprint arXiv:2603.05026 (2026).

Questions for authors

See weaknesses above

4 comments
AuthorsOpenReview ↗

Rebuttal by Authors (1/2)

We really appreciate your time and effort in reviewing our paper. We address each of your points below and have revised the paper in response to your feedback.

W1: Agent is never told to retain the capabilities on the source OS.

Thank you for raising this issue. We intended the phrase “code migration” to imply that the code should continue to work on the source platform. However, we agree that the agent’s instructions should state this requirement explicitly. We therefore reran the 28 instances using the same GPT‑5.6-Sol+Claude Code, with the following revised problem statement:

You are given a repository at /testbed. The repository was first developed on Windows but now you need to migrate it to Linux. Fix all platform compatibility issues on Linux while preserving all existing functionality on Windows. The repository must continue to compile and build successfully on Windows, and all existing tests that pass on Windows must continue to pass. Your final goal is to pass all regression tests on Linux without breaking compilation or functionality on Windows.

These 28 instances had previously passed evaluation on the target platform but had no validated pass on the source platform. In our new run with the revised prompt, 3 failed on the target platform. Only 2 passed source-platform validation, and both were among those 3 target-platform failures, so none passed on both platforms. Overall, the revised prompt had little effect and didn't improve LM’s internal ability, the differences are consistent with normal run-to-run variability.

W2: The effect of masking on task difficulty is unclear.

Thank you for highlighting this concern. Our benchmark targets end-to-end cross-platform migration, including identifying where target-platform functionality is missing and implementing the required behavior. Fault localization is therefore an intended component of the evaluation. Explicit error-raising placeholders can directly expose the modified locations through error messages, while silent placeholders retain the need to diagnose missing functionalities.

We carried out a preliminary experiment on the difficulty of different masking strategies. Using GPT-5.6-Sol and SWE-agent, the success rates of different masking strategies on Linux:

Masking strategySuccess rate
Directly deleting the selected blocks32.4%
Replacing them with explicit error-raising placeholders44.7%
Replacing them with silent placeholders18.7%
As the third strategy is the most difficult, we adopted it as our final masking strategy.

Regarding if the masked codes resemble realistic placeholders before migration, when using language model to mask code blocks, we explicitly prompted the language model to use those placeholders that are commonly used in real-world development or the native placeholders from the original repo. For example, in Windows-to-Linux task trunk-rs__trunk-f1ee3d4__rust__linux__f1a2ab:

The oracle answer:

#[cfg(unix)]
fn set_file_permissions(
    file: &mut File,
    mode: u32,
    file_path_hint: impl Display,
) -> Result<()> {
    use std::fs::Permissions;
    use std::os::unix::fs::PermissionsExt;

    tracing::debug!("Setting permission of '{file_path_hint}' to {mode:#o}");

    file.set_permissions(Permissions::from_mode(mode))
        .context("failed setting file permissions")?;

    Ok(())
}

The masked code:

#[cfg(unix)]
fn set_file_permissions(
    _file: &mut File,
    _mode: u32,
    _file_path_hint: impl Display,
) -> Result<()> {
    // Kept as a platform hook so extraction code can use one shared path.
    Ok(())
}

The original repository already uses an Ok(()) stub for other unimplemented functions in other places, which means the unimplemented functions will be skipped during execution. The masked Unix version follows that same native pattern adopted in the original repo, so the masked blocks resemble realistic placeholders.

W3: Claude-5-sonnet is run twice with Claude Code.

Thank you for spotting this issue. At the time of evaluating Claude-Sonnet-5, the enterprise collaborator funding our work imposed a monthly API budget limit. We therefore had to split its evaluation into two batches to comply with this policy. The two batches contained no overlapping instances, so each instance was evaluated only once. Combining these batches is equivalent to a single run over the full set. We included this detail in the paper for transparency, but we recognize that the wording caused confusion. We have revised the description to make this distinction clearer.

AuthorsOpenReview ↗

Rebuttal by Authors (2/2)

W4: SWE-bench-Live already includes Windows-specific implementation tasks.

OS-Bench focuses on a specific software engineering task: cross-OS code migration. In contrast, SWE-bench-Live/Windows is a general-purpose benchmark covering a broad range of Windows-related software engineering tasks, including PowerShell operations, rather than specifically targeting cross-OS migration.

Furthermore, SWE-bench-Live/Windows derives its tasks from GitHub issues, making its task coverage dependent on the availability of Windows-specific issues reported by developers on GitHub. If no such issues are raised, corresponding benchmark tasks cannot be constructed. In contrast, OS-Bench proposes a generalizable task generation method that can construct migration tasks from arbitrary repositories with cross-OS implementations, without relying on existing GitHub issues. This enables OS-Bench to generate a broader range of Windows-related tasks and offers greater scalability and generalizability across code repositories.

W5: Potential training-data contamination is insufficiently discussed.

We agree that potential training-data contamination should be discussed more explicitly, and we will address this in the limitations section. Nevertheless, the results show that current state-of-the-art models still achieve low success rates on OS-Bench. This suggests that, even with possible exposure to the underlying repositories, these models still struggle to complete cross-OS migration tasks successfully. This capability remains challenging and underexplored for current models and agents.

W6: Harnesses like claude code can be used cross-platform as well.

Claude Code(cc) can run on Windows hosts and operate on files stored on those hosts. However, in our experiments, it could not run successfully inside the Windows Docker containers used by OS-Bench. When attempting to install cc within the Windows container, an error was immediately reported, informing that cc cannot be installed in the Windows container environment. As cc is proprietary software, we are unable to debug and resolve this incompatibility. Since our Windows tasks are provided in these containers, we were unable to evaluate Claude Code on them.

We also confirmed that SWE-agent and OpenHands did not support the Windows-container execution setup used in our evaluation. OpenHands and SWE-agent hard-coded docker APIs that only work with Linux containers in their source code. Also, the two agents hard-coded Linux bash commands in their source codes, which cannot run in Windows environments. We will clarify that this limitation concerns our containerized evaluation setup. We will state this more clearly in the paper.

My concerns are addressed. I still slightly disagree regarding W4 (novelty concern), where authors agree that prior work is more generalized to what they propose, while covering the migration tasks as well.

However, I believe a more systematic benchmark and a in-depth study is useful. I have increased my score to 3.

Thank you for your prompt response and willingness to reconsider the score. We appreciate your recognition of the value of our benchmark. We wish you all the best in your future research!

Reviewer_N9WaOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces OS-Bench, an executable benchmark for cross-OS code migration between Linux and Windows. The authors identify platform-specific code through differential test coverage, then mask it after LM-based and manual validation. The benchmark contains 140 tasks from 111 repositories across five languages. Patches must restore functionality on the target platform while preserving existing tests on the source platform. Across 6 models and 4 agent harnesses, the best success rates are 26.4% on Linux-target tasks and 20.4% on Windows-target tasks, with substantial drops once source-platform validation is required.

Critical strengths
  1. OS-Bench evaluates repository-level Linux↔Windows migration in native environments. Requiring both target-platform repair and source-platform preservation better reflects cross-platform CI and can detect fixes that break source-only code or remove platform guards.

  2. Scalable task construction. Differential test coverage identifies platform-specific code without manually extracting migration changes from commit histories. This approach can therefore extend to more repositories and languages.

  3. The paper is clear written and easy to follow.

Critical weaknesses
  1. None of the prompts in Appendix D explicitly instructs agents to preserve source-platform behavior, and agents run only in the target environment. In this setting, 28 of the 50 GPT-5.6-Sol + Claude Code patches (56%) that pass Stage I lack a validated Stage II pass. The paper does not analyze these failures. The manual audit in Section 5.4 (Table 4) excludes patches that fail only at Stage II, and Appendix C acknowledges that the Stage I–II gap does not show why each patch fails. It is therefore unclear whether the gap reflects poor cross-platform reasoning or a requirement that the prompts leave unstated.

  2. Linux-target and Windows-target tasks differ in task set, language distribution, agent harness, and shell, so differences in difficulty or in model rankings cannot be attributed to migration direction. Harness prompts also differ. For example, the Claude Code prompt restricts each response to exactly one tool call. Each configuration appears to be run only once, with no confidence intervals, and differences of one task (e.g., 17.6% vs. 16.5% for GPT-5.6-Luna) are interpreted as ranking reversals. Claude-Sonnet-5 uses the union of two runs, so its results are not directly comparable to the single-run results.

Questions for authors
  1. Can the authors break down Stage II failures into compilation failures and test failures, and manually inspect the 28 GPT-5.6-Sol + Claude Code cases? This would help determine why source-platform validation fails.

  2. If agents were explicitly instructed to preserve source-platform functionality, how much would the Stage I–II gap decrease?

  3. How many tasks become uncompilable on the target platform after masking, so that the entire test suite enters TtfixT_t^{\text{fix}}?

  4. Why does Claude-Sonnet-5 use the union of two runs while other configurations use one? Can the authors report run-to-run variance or confidence intervals?

2 comments
AuthorsOpenReview ↗

Rebuttal by Authors (1/2)

We really appreciate your time and effort in reviewing our paper. We address each of your points below and have revised the paper in response to your feedback.

W1 & Q2: None of the prompts in Appendix D explicitly instructs agents to preserve source-platform behavior, and agents run only in the target environment.

Thank you for raising this issue. We intended the phrase “code migration” to imply that the code should continue to work on the source platform. However, we agree that the agent’s instructions should state this requirement explicitly. We therefore reran the 28 instances using the same GPT‑5.6-Sol+Claude Code, with the following revised problem statement:

You are given a repository at /testbed. The repository was first developed on Windows but now you need to migrate it to Linux. Fix all platform compatibility issues on Linux while preserving all existing functionality on Windows. The repository must continue to compile and build successfully on Windows, and all existing tests that pass on Windows must continue to pass. Your final goal is to pass all regression tests on Linux without breaking compilation or functionality on Windows.

These 28 instances had previously passed evaluation on the target platform but had no validated pass on the source platform. In our new run with the revised prompt, 3 failed on the target platform. Only 2 passed source-platform validation, and both were among those 3 target-platform failures, so none passed on both platforms. Overall, the revised prompt had little effect and didn't improve LM’s internal ability, the differences are consistent with normal run-to-run variability.

W2: Linux-target and Windows-target tasks differ in task set, language distribution, agent harness, and shell, so differences in difficulty or in model rankings cannot be attributed to migration direction.

Thank you for raising the concern that Linux-target and Windows-target tasks differ in their task sets and language distributions. We agree that these factors limit the comparability of the Linux and Windows results, and we will explicitly acknowledge this limitation in the revised paper. Nevertheless, we are actively expanding the dataset. As the dataset grows, we expect the differences in language distributions to gradually decrease.

However, we would like to clarify that differences in agent harnesses, shells, and prompts do not necessarily undermine the comparison. As shown in Appendix Table 5, Win-agent and SWE-agent provide the same set of tools, and their prompts are also aligned. Therefore, we believe that the evaluation settings of Win-agent on Windows and SWE-agent on Linux are sufficiently aligned to support a meaningful comparison of their success rates. Moreover, we retain the original prompts of all the four agents under evaluation, meaning that any remaining prompt differences reflect the intrinsic characteristics of their respective agent harness designs rather than arbitrary changes to the evaluation settings.

Regarding shell differences, Linux uses Bash as its CLI, whereas Windows does not have Bash but instead uses PowerShell as the CLI by default. These differences reflect the characteristics of real-world development environments and are therefore an inherent aspect of cross-platform migration tasks. Consequently, we believe that shell-related challenges should be evaluated alongside code modifications as part of the overall task difficulty, rather than treated as separate factors.

AuthorsOpenReview ↗

Rebuttal by Authors (2/2)

Q1: Can the authors break down Stage II failures into compilation failures and test failures, and manually inspect the 28 GPT-5.6-Sol + Claude Code cases?

We manually inspected all 28 cases, these include:

  • 22 compilation errors (casey__just-87efe02__rust__linux__76ccbd, LykosAI__StabilityMatrix-aa7e3a1__cs__linux__0b87fa, bvschaik__julius-34fad90__c__linux__3e08c5, Tyrrrz__CliWrap-64621b0__cs__linux__f69a4a, aquasecurity__tfsec-b692c20__go__linux__b63b52, fsnotify__fsnotify-dfa2f7e__go__linux__b1e05c, bytedance__monoio-e707bce__rust__linux__16016e, rust-lang__rust-clippy-95dd88d__rust__linux__9e0b24, adnanh__webhook-1b13355__go__linux__17f7ca, aws__aws-sdk-go-070853e__go__linux__090d4e, etcd-io__bbolt-7625e06__go__linux__7e9330, cloudwego__netpoll-b0bf57d__go__linux__2e14fe, go-swagger__go-swagger-5446886__go__linux__4af636, goadesign__goa-93c5a9b__go__linux__2130a5, go-playground__validator-53c4d6b__go__linux__f5c183, mtkennerly__ludusavi-772f18b__rust__linux__34551e, AssetRipper__AssetRipper-9b98d6__cs__linux__da993d, google__leveldb-ac69108__cpp__linux__b9bf34, OpenRA__OpenRA-8a0197c__cs__linux__80c53e, google__go-github-d08bec9__go__linux__2ab6e1, probe-rs__probe-rs-14f2e57__rust__linux__4c05c5, watchexec__cargo-watch-58e792b__rust__linux__e33412)
  • 3 test/runtime errors (rs__zerolog-9dacc01__go__linux__1b9746, spf13__viper-4df8a66__go__linux__86b702, xamarin__Xamarin.Forms-2f8f48__cs__linux__3a9494)
  • 2 evaluator errors (AvaloniaUI__Avalonia-995f8ed__cs__linux__6c5c90, tmenier__Flurl-36a3fa2__cs__linux__015f5f)
  • 1 dependency error (go-rod__rod-393ac0d__go__linux__0400ab)

Our manual inspection found that most compilation and test failures (22 out of 25) were caused by agents modifying code structures (or arguments) that were already working correctly on the other platform.

In casey__just-87efe02__rust__linux__76ccbd, the agent renamed the shared Alias method is_public() to is_private() and updated its callers in Justfile. The subsequent Windows build failed with Rust error E0599, reporting that is_private() was missing.

In xamarin__Xamarin.Forms-2f8f48__cs__linux__3a9494, the agent replaced the platform-dependent default of Shell.NavBarHasShadowProperty with a constant true. All 29 Linux tests passed, but these tests do not check this property. The original default is true only on Android, the replacement changes the Windows/UWP default from false to true, failing to preserve existing source-platform behavior.

Q3: How many tasks become uncompilable on the target platform after masking, so that the entire test suite enters?

After masking, 57 of 140 tasks (40.7%) fail to build on the target platform (36 on Linux and 21 on Windows) causing all originally passing tests to enter FAIL_TO_PASS.

Q4: Why does Claude-Sonnet-5 use the union of two runs while other configurations use one?

Thank you for spotting this issue. At the time of evaluating Claude-Sonnet-5, the enterprise collaborator funding our work imposed a monthly API budget limit. We therefore had to split its evaluation into two batches to comply with this policy. The two batches contained no overlapping instances, so each instance was evaluated only once. Combining these batches is equivalent to a single run over the full set. We included this detail in the paper for transparency, but we recognize that the wording caused confusion. We have revised the description to make this distinction clearer.

Reviewer_2438OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces OS-Bench, a benchmark for migrating code between Linux and Windows. The authors compare test coverage on the two systems, use an LM and human checks to identify platform-specific code, and remove the target-platform implementation for an agent to restore. There are 140 tasks from 111 repositories across five languages. A solution has to pass tests on both platforms. The authors evaluate six models with four agent harnesses; the best results are 26.4% on Linux-target tasks and 20.4% on Windows-target tasks.

Critical strengths
  1. This is a problem worth studying. Getting a patch to work on one OS does not tell us whether it still works on the other. Testing the same patch on both systems gives this benchmark a clear purpose beyond another issue-fixing dataset.
  2. The benchmark covers a good range of repositories and languages, with tasks in both migration directions. The checks during construction make sense: the source-platform tests must still pass after masking, at least one target test must stop passing, and the removed code provides a reference solution.
  3. I like that the paper reports target-only and both-platform results separately. This shows what we miss by testing on just one system. The examples in Section 5.4 also help explain what went wrong in individual runs. The authors are clear that differences between the Linux and Windows results also involve different tasks and harnesses.
Critical weaknesses
  1. My main concern is how closely the tasks match real migration work. Sections 3.3-3.4 put all retained platform-specific regions for a repository and migration direction into one task. That could bundle several unrelated features together. An average task spans seven files, and the largest spans 152 files and 5,287 estimated lines, while the instruction in Appendix D.3 just asks the agent to fix platform compatibility. Some of the difficulty may therefore come from finding everything that was removed. The paper says the tasks were manually checked, but I would like to see what those checks found. A few examples of accepted and rejected tasks, the frequency of manual corrections, and a comparison with actual migration changes would help.

  2. The instruction and the evaluation do not quite line up. Appendix D.3 tells the agent to fix compatibility issues and pass regression tests on the target OS. The final score also requires the source-platform tests to pass, but that requirement is not stated in this prompt. Preserving existing behavior is a reasonable expectation; still, it is central enough here to tell the agent directly. Do agents get this instruction elsewhere, and can they run checks on the source OS? Rerunning a small subset with an explicit requirement to preserve both platforms would help show how much the wording affects the results.

  3. I wanted more detail on what happened during source-platform testing. For GPT-5.6-Sol with Claude Code, 50 Linux-target patches pass Stage I, but only 22 pass both stages. How many of the 28 patches actually broke the source build or tests, and how many had timeouts, infrastructure problems, or missing results? This distinction matters when using the gap as evidence that agents struggle to preserve compatibility. A breakdown and a few concrete regression examples would be more informative here, with reruns and reference checks for unclear cases.

  4. I found the motivation harder to follow than it needs to be. Figure 1 gives a libuv example, but the two code diffs assume quite a bit of background. A simpler diagram could show an application calling the same API, the separate OS backends, and the gap that needs to be filled without breaking the working backend. For example, the figure could follow a call to uv_udp_set_broadcast() and explain what behavior users expect on either OS. This would make the reason for migration clearer before the introduction moves on to the difficulties of extracting tasks from commit history.

Questions for authors
  1. What did the manual task checks find? In particular, how often does one task combine separate migration features?
  2. Are agents told to preserve source-platform behavior, and can they test on that platform?
  3. What caused the 28 cases that passed target tests but had no validated source-platform pass in the GPT-5.6-Sol/Claude Code results?
  4. Can you clarify the number of attempts per model and report the Sonnet runs separately? It would also help to give the construction-model and harness versions, inference costs.
3 comments
AuthorsOpenReview ↗

Rebuttal by Authors (1/3)

We really appreciate your time and effort in reviewing our paper. We address each of your points below and have revised the paper in response to your feedback.

W1.1: An average task spans seven files, and the largest spans 152 files and 5,287 estimated lines. Some of the difficulty may therefore come from finding everything that was removed.

Thank you for highlighting this concern. Our benchmark targets end-to-end cross-platform migration, including identifying where target-platform functionality is missing and implementing the required behavior. Fault localization is therefore an intended component of the evaluation.

Recent benchmarks also consider substantial implementation scope. For example, ProgramBench reports reference codebases containing up to approximately 2.7 million lines of code. Although a task spanning 152 files and 5,287 estimated lines is not a small one, we believe that tasks of this scale are valuable for evaluating agents’ ability to carry out complex, repository-level migrations.

Q1.1: What did the manual task checks find? A few examples of accepted and rejected tasks and a comparison with actual migration changes would help.

The manual check of LM-based masking finds:

Manual InspectionPercentageExampleManual action
Incomplete masking0.7%The model forgot to mask some target-specific regions.Mask missed blocks, e.g. void f() {}.
Exceptions or replacement implementations2.3%The language model inserted exceptions or re-implementations into the places that need to be masked. An observed exception-producing placeholder is todo!("wire socket read state") in SocketRead::new, in src/io/sys/unix/net/socket_read.rs, for Xudong-Huang__may-69565dc__rust__linux__fbc539. This Rust macro panics when executed. Its membership in the reported correction group is unverified.Replace todo!(...) with an inert constructor stub (Self { … }, schematic).
Changes to source-platform code or tests2.0%The language model modified source-platform codes and source-platform test suite. In the Windows-to-Linux instance cyanfish__naps2-3991313__cs__linux__193f27, masking changes NAPS2.Lib/Platform/Windows/StillImage.cs: it removes /StiDevice: argument handling and the ShouldScan and DeviceID properties used by Windows scanner integration.Restore DeviceID = arg.Substring(DEVICE_PREFIX.Length); ShouldScan = true; and both properties.
Masking as expected95.0%N/ANo correction required.

Q1: How often does one task combine separate migration features?

We would like to clarify that our task unit is a repository–migration-direction pair. Each repository contributes at most two tasks: Windows→Linux and Linux→Windows. One repository can contain multiple annotated feature groups but no combination of features is performed. Statistically, 114 of 140 tasks (81.4%) contain multiple feature groups, with an average of 23.8 groups per task.

W2 & Q2: The evaluation requires the source-platform tests to pass, but that requirement is not stated in this prompt.

Thank you for raising this issue. We intended the phrase “code migration” to imply that the code should continue to work on the source platform. However, we agree that the agent’s instructions should state this requirement explicitly. We therefore reran the 28 instances using the same GPT‑5.6-Sol+Claude Code, with the following revised problem statement:

You are given a repository at /testbed. The repository was first developed on Windows but now you need to migrate it to Linux. Fix all platform compatibility issues on Linux while preserving all existing functionality on Windows. The repository must continue to compile and build successfully on Windows, and all existing tests that pass on Windows must continue to pass. Your final goal is to pass all regression tests on Linux without breaking compilation or functionality on Windows.

These 28 instances had previously passed evaluation on the target platform but had no validated pass on the source platform. In our new run with the revised prompt, 3 failed on the target platform. Only 2 passed source-platform validation, and both were among those 3 target-platform failures, so none passed on both platforms. Overall, the revised prompt had little effect and didn't improve LM’s internal ability, the differences are consistent with normal run-to-run variability.

AuthorsOpenReview ↗

Rebuttal by Authors (2/3)

W3 & Q3: What caused the 28 cases that passed target tests but had no validated source-platform pass in the GPT-5.6-Sol/Claude Code results?

We manually inspected all 28 cases, these include:

  • 22 compilation errors (casey__just-87efe02__rust__linux__76ccbd, LykosAI__StabilityMatrix-aa7e3a1__cs__linux__0b87fa, bvschaik__julius-34fad90__c__linux__3e08c5, Tyrrrz__CliWrap-64621b0__cs__linux__f69a4a, aquasecurity__tfsec-b692c20__go__linux__b63b52, fsnotify__fsnotify-dfa2f7e__go__linux__b1e05c, bytedance__monoio-e707bce__rust__linux__16016e, rust-lang__rust-clippy-95dd88d__rust__linux__9e0b24, adnanh__webhook-1b13355__go__linux__17f7ca, aws__aws-sdk-go-070853e__go__linux__090d4e, etcd-io__bbolt-7625e06__go__linux__7e9330, cloudwego__netpoll-b0bf57d__go__linux__2e14fe, go-swagger__go-swagger-5446886__go__linux__4af636, goadesign__goa-93c5a9b__go__linux__2130a5, go-playground__validator-53c4d6b__go__linux__f5c183, mtkennerly__ludusavi-772f18b__rust__linux__34551e, AssetRipper__AssetRipper-9b98d6__cs__linux__da993d, google__leveldb-ac69108__cpp__linux__b9bf34, OpenRA__OpenRA-8a0197c__cs__linux__80c53e, google__go-github-d08bec9__go__linux__2ab6e1, probe-rs__probe-rs-14f2e57__rust__linux__4c05c5, watchexec__cargo-watch-58e792b__rust__linux__e33412)
  • 3 test/runtime errors (rs__zerolog-9dacc01__go__linux__1b9746, spf13__viper-4df8a66__go__linux__86b702, xamarin__Xamarin.Forms-2f8f48__cs__linux__3a9494)
  • 2 evaluator errors (AvaloniaUI__Avalonia-995f8ed__cs__linux__6c5c90, tmenier__Flurl-36a3fa2__cs__linux__015f5f)
  • 1 dependency error (go-rod__rod-393ac0d__go__linux__0400ab)

Our manual inspection found that most compilation and test failures (22 out of 25) were caused by agents modifying code structures (or arguments) that were already working correctly on the other platform. In casey__just-87efe02__rust__linux__76ccbd, the agent renamed the shared Alias method is_public() to is_private() and updated its callers in Justfile. The subsequent Windows build failed with Rust error E0599, reporting that is_private() was missing. In xamarin__Xamarin.Forms-2f8f48__cs__linux__3a9494, the agent replaced the platform-dependent default of Shell.NavBarHasShadowProperty with a constant true. All 29 Linux tests passed, but these tests do not check this property. The original default is true only on Android, the replacement changes the Windows/UWP default from false to true, failing to preserve existing source-platform behavior.

W4: Figure 1 is hard to follow.

Thank you for this suggestion. Assume an application wants to send “hello” through an IPv4 UDP socket. Before sending, it can configure the following options:

  • uv_udp_set_multicast_ttl(): Setting ttl = 1 keeps messages sent to a multicast group within the local network. Setting ttl = 2 permits forwarding through at most one router, provided multicast routing and the destination address’s scope allow it.
  • uv_udp_set_broadcast(): Setting on = 0 prevents the socket from sending “hello” to a broadcast address. Setting on = 1 permits that send, addressing devices on the local network.

The application expects the same behavior from these APIs on either operating system. In the example, the migration task is to add the missing Windows implementation while preserving the working Unix implementation.

The highlighted platform differences reflect how Unix and Windows expose their socket APIs:

  1. Unix represents it using an integer file descriptor (fd), whereas Windows uses a socket handle of type SOCKET.
  2. The Unix setsockopt() interface accepts a const void *, so the integer’s memory address can be passed directly as &ttl. The Windows interface expects a const char *, so the address must be cast to that type.

We will refine Figure 1 following your suggestion to show the application calling the same API through separate OS backends. We will use this application example in the diagram and add the implementation explanations to the revised figure caption.

Q4.1: Why is Claude-5-sonnet run twice?

Thank you for spotting this issue. At the time of evaluating Claude-Sonnet-5, the enterprise collaborator funding our work imposed a monthly API budget limit. We therefore had to split its evaluation into two batches to comply with this policy. The two batches contained no overlapping instances, so each instance was evaluated only once. Combining these batches is equivalent to a single run over the full set. We included this detail in the paper for transparency, but we recognize that the wording caused confusion. We have revised the description to make this distinction clearer.

AuthorsOpenReview ↗

Rebuttal by Authors (3/3)

Q4.2: Can you give the construction-model and harness versions, inference costs?

Our construction used GPT‑5.5, we are currently using GPT‑6-Astra to actively expand the size of our dataset. Claude Code version is 2.1.89, SWE-agent version is 1.1.0, OpenHands version is 1.16.1.

Here is the mean inference cost per task (USD), we will add this in the paper:

ModelSWE-agentOpenHandsClaude Code
Claude-Sonnet-5$0.170$1.438$2.647
Claude-Opus-4.8$3.611$3.679$13.440
GPT-5.6-Luna$0.072$0.136$0.217
GPT-5.6-Terra$0.932$1.101$1.826
GPT-5.6-Sol$1.976$3.364$7.136
DeepSeek-V4-Pro$2.008$1.924$1.815

Comments on the paper

1 comment

We sincerely thank the reviewer for your continued support and positive assessment of our work. We greatly appreciate your constructive feedback and thoughtful consideration throughout the review process. We wish you all the best!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.