acceptodds
Desk rejected at ICLR 2027

T2J-BENCH: Benchmarking Coding Agents on Execution-Grounded PyTorch-to-JAX Repository Migration

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

Migrating a machine-learning codebase across frameworks—for example from PyTorch to JAX—is a routine but costly engineering task that is fundamentally cross-paradigm: eager, stateful, object-oriented modules must be rewritten as pure, jittable functions over an explicit parameter pytree. We ask whether coding agents can perform such whole-repository migrations correctly, and argue that existing code benchmarks, which grade unit-test pass rates or surface-level similarity to a reference translation, cannot answer this: they neither require the migrated system to behave the same nor prevent a JAX reimplementation from faking equivalence by secretly importing PyTorch itself. We introduce T2J-BENCH, a benchmark of 63 whole-repository PyTorch→JAX migration tasks spanning 18 model families across both training and serving, where torch-free checkpoint conversion is itself part of the deliverable. T2J-BENCH grades execution-grounded functional equivalence: each candidate is run inside a separate, torch-free, offline verifier and scored against references frozen from the PyTorch original, under a hard independence gate and a continuous, threshold-free reward. Across twenty off-the-shelf agent–model configurations, the strongest recovers only 0.676 of the reward against a 0.980 oracle, with substantial numeric and behavioral divergence despite high spec pass rates. Finally, we contribute TWINHARNESS, a reference- grounded agent harness that turns the agent’s executable PyTorch source into legitimate self-verification feedback; applied to an opus-5 agent it raises the reward from 0.649 to 0.723—the highest of any configuration—using only task- visible resources. We will release the benchmark to support reproducible study of cross-paradigm ML migration.

Primary area:
datasets and benchmarks
Keywords:
coding agents, code migration benchmark, PyTorch to JAX, execution-grounded functional equivalence

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_cqriOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces T2J-Bench, a benchmark of 63 repository-level PyTorch-to-JAX migration tasks covering training and serving. Candidates are evaluated in an offline, torch-free environment using structural, numerical, and behavioral equivalence. The paper evaluates 20 agent-model configurations and proposes TwinHarness, a reference-grounded diagnose-migrate-verify loop that improves reward from 0.649 to 0.723.

Critical strengths

1.Rigorous and practically meaningful evaluation. The torch-free verifier evaluates numerical, gradient, training, and serving behavior rather than superficial code similarity; 2.Broad empirical validation. The study covers 63 tasks, 20 configurations, three seeds, cost analysis, failure taxonomy, and several refinement settings. The benchmark is challenging and appears useful to the community.

Critical weaknesses
  1. Limited task diversity and independence. Although the benchmark contains 63 tasks, many are correlated variants formed by crossing a small number of source repositories, model families, and training or serving configurations. It is therefore unclear how well the conclusions generalize to genuinely diverse real-world migration projects;
  2. Incomplete validation of the evaluation and harness design. The overall ranking depends on a custom weighted reward, but sensitivity to alternative weightings is not reported. Moreover, TwinHarness is evaluated largely as a complete system, without sufficiently isolating the contribution of diagnosis, persistent evidence, and independent verification. This makes it difficult to determine which design choices actually produce the improvement.
Questions for authors
  1. What capability does TwinHarness introduce beyond existing oracle-guided self-debugging methods? 2.How sensitive are system rankings to the Spec/Numeric/Behavioral reward weights? 3.Can the authors ablate the diagnosis, persistent evidence, and independent verification components of TwinHarness separately?
5 comments
AuthorsOpenReview ↗

Response 1

We sincerely thank the reviewer for the constructive comments and insightful suggestions.

W1: Limited task diversity and independence.

Although the benchmark contains 63 tasks, many are correlated variants formed by crossing a small number of source repositories, model families, and training or serving configurations. It is therefore unclear how well the conclusions generalize to genuinely diverse real-world migration projects.

Response. Thanks for the comments. We answer them one by one.

Task diversity. We argue that upstream repository diversity and migration task diversity are distinct dimensions. Although our tasks originate from a limited number of upstream codebases, they exhibit substantial diversity in model architectures, training algorithms, modalities, and execution semantics. Specifically, T2J-Bench covers 18 model families with different architectural structures, parameter organizations, and checkpoint conversion requirements, including both text-only and vision-language models. The 45 training tasks span SFT, DPO, and PPO, which require different optimization objectives, gradient computations, and training dynamics. The 18 serving tasks cover Core, Ragged, and Chunked modes, involving different execution semantics such as dynamic batching, paged KV-cache management, and prefix reuse. These differences are also reflected in our empirical results (Table 4), where different task categories exhibit substantial performance variations and changes in agent rankings. For example, Claude-Code performs best on SFT, OpenCode leads on Core serving, while TwinHarness achieves the highest performance on DPO, PPO, and Chunked serving. These observations suggest that the tasks exercise different migration capabilities rather than merely repeating identical migration problems under different configurations.

A framework that scales to new migration projects. Moreover, our main contribution is not a fixed task set. It is a framework for constructing PyTorch-to-JAX codebase migration tasks with rigorous evaluation, and this framework extends to new migration projects. The framework applies to any PyTorch codebase that can be run end to end. Given an upstream repository pinned to a commit, it works in three steps:

  1. It extracts a self-contained PyTorch codebase and verifies that it reproduces the upstream behavior exactly.
  2. It builds a reference JAX solution.
  3. It generates graders from shared components: outputs frozen from PyTorch, Spec, numeric and behavioral checks, and tolerance bands set from PyTorch's own run-to-run variation. Because the grading library and reward computation are shared, a new codebase does not require a new evaluation design.

Additional codebases. We added one training task from TRL and one serving task from SGLang. For each, we pinned the upstream commit, verified the extracted PyTorch workload against upstream behavior, built a reference JAX port, and applied the shared grading procedure using frozen PyTorch outputs, structural, numerical, and behavioral checks, and reference-calibrated tolerances. Multimodal GRPO uses TRL v1.17.1 with Qwen2.5-VL-3B (4.4k nonblank Python lines). Approximately 20% of its lines match existing tasks, mostly in vendored Hugging Face Qwen-VL modeling code, with little overlap in the trainer, loss, or data pipeline. Sliding-window RadixAttention uses SGLang v0.5.19 with Gemma 4 (12.1k nonblank Python lines), with 6.5% line overlap. The reference JAX ports scored 0.999997 and 0.999656, respectively.

As shown in Table 1, TWINHARNESS achieved the highest observed mean reward among the evaluated configurations on both new tasks. On GRPO, it scored 0.681, compared with 0.632 for the strongest baseline (OpenCode with opus-5) and 0.355 for its matched Claude-Code baseline. On RadixAttention, it scored 0.812, compared with 0.767 for the strongest baseline (Claude-Code with opus-5). Our analysis of the run logs traced many zero-reward baseline runs to two task-specific failures: GRPO ports failed to reproduce TRL’s per-batch PyTorch shuffle, while RadixAttention ports ran out of memory during batched serving. We added the task construction, validation, results, and failure analysis to Appendix E, “Migration Tasks from Independent Upstream Codebases” (manuscript Table 20), and referenced them in Section 5.2.

AuthorsOpenReview ↗

Response 2

Continuation of our response to W1 from the previous page:

Table 1. Reward on the two additional tasks from independent upstream codebases. Values are means ± sample standard deviations over three seeds. Bold indicates the highest observed mean in each column.

HarnessModelMultimodal GRPO (TRL)RadixAttention (SGLang)
Claude-Codeopus-50.355 ± 0.4170.767 ± 0.232
Claude-Codesonnet-50.209 ± 0.2200.000 ± 0.000
OpenCodeopus-50.632 ± 0.4000.752 ± 0.255
OpenCodegemini-3.7-flash0.184 ± 0.2120.267 ± 0.463
OpenCodesonnet-50.273 ± 0.2370.000 ± 0.000
OpenCodeglm-5.20.016 ± 0.0140.138 ± 0.239
OpenCodegemini-3.8-flash0.090 ± 0.1090.126 ± 0.218
OpenCodegemini-3.1-pro0.000 ± 0.0000.000 ± 0.000
Gemini-CLIgemini-3.7-flash0.016 ± 0.0140.000 ± 0.000
Gemini-CLIgemini-3.8-flash0.031 ± 0.0310.000 ± 0.000
Gemini-CLIgemini-3.1-pro0.000 ± 0.0000.000 ± 0.000
TWINHARNESSopus-50.681 ± 0.1230.812 ± 0.091

[W2, Q2]: Sensitivity to reward weightings.

W2: The overall ranking depends on a custom weighted reward, but sensitivity to alternative weightings is not reported.

Q2: How sensitive are system rankings to the Spec/Numeric/Behavioral reward weights?

Response. Thank you for raising this concern. We keep Spec fixed as a multiplicative validity gate and vary the Numeric weight α\alpha. The Behavioral weight is 1−α1-\alpha, giving the per-task reward

Rα=S[αN+(1−α)B].R_{\alpha}=S\bigl[\alpha N+(1-\alpha)B\bigr].

We compute reward per task, average across tasks within each seed, and report the mean ± sample standard deviation over three seeds. Across all five tested values of α\alpha, the three opus-5 configurations retain the same ranking: TWINHARNESS > OpenCode > Claude-Code. TWINHARNESS leads OpenCode by 0.045 at α=0\alpha=0 (Behavioral only), 0.047 at the default α=0.50\alpha=0.50, and 0.048 at α=1\alpha=1 (Numeric only). Table 2 reports these results (manuscript Section 5.2, Table 5).

Table 2. Sensitivity of reward to the Numeric weight α\alpha.

HarnessModelα=0\alpha=0α=0.25\alpha=0.25α=0.50\alpha=0.50 (default)α=0.75\alpha=0.75α=1\alpha=1
Claude-Codeopus-50.697 ± .0110.673 ± .0100.649 ± .0080.624 ± .0080.600 ± .008
OpenCodeopus-50.707 ± .0120.692 ± .0130.676 ± .0140.661 ± .0150.646 ± .017
TWINHARNESSopus-50.752 ± .0240.738 ± .0230.723 ± .0220.709 ± .0220.694 ± .021

[W2, Q3]: Incomplete validation and ablation of TWINHARNESS components.

W2: Moreover, TwinHarness is evaluated largely as a complete system, without sufficiently isolating the contribution of diagnosis, persistent evidence, and independent verification. This makes it difficult to determine which design choices actually produce the improvement.

Q3: Can the authors ablate the diagnosis, persistent evidence, and independent verification components of TwinHarness separately?

Response. Thank you for asking us to clarify how these components contribute to TWINHARNESS. Table 3 summarizes the role ablations on Claude Code with opus-5. We changed one role at a time.

  • Diagnosis. Removing the Diagnoser eliminated the dedicated step that identifies and prioritizes migration gaps. Reward fell from 0.723 to 0.684 (−0.039).
  • Independent verification and persistent evidence. As described in Section 4 and Appendix D.1, the Verifier checks the current JAX candidate and records its findings in E_t. In the next round, the Diagnoser reads E_t and prioritizes repairs, which the Migrator then implements. Removing the Verifier produced the largest observed drop, from 0.723 to 0.671 (−0.052). This ablation measures the combined contribution of independently checking the candidate and carrying those findings forward for repair.
  • Migrator specialization. We also retained the Diagnoser and Verifier but replaced the specialized Migrator with a generic coding agent that performs the JAX edits. Reward fell from 0.723 to 0.698 (−0.025).

We report these results and their analysis in Appendix D.7, “Single-Agent Comparison and Role Ablations”, referenced in Section 5.4 under Structured iteration versus ordinary continuation. The full harness achieved higher reward than all three ablated variants, supporting the contribution of each tested role: the Diagnoser prioritizes repairs, the specialized Migrator implements them, and the Verifier supplies execution evidence to guide the next round.

Table 3. Role ablations on Claude Code (opus-5).

ConfigurationRewardDifference from full
TWINHARNESS (full)0.723—
Without Diagnoser0.684−0.039
Without Verifier0.671−0.052
With a generic Migrator0.698−0.025
AuthorsOpenReview ↗

Response 3

Q1: Novelty beyond oracle-guided self-debugging.

What capability does TwinHarness introduce beyond existing oracle-guided self-debugging methods?

Response. Thank you for asking us to clarify this distinction. TWINHARNESS does not receive feedback from the hidden benchmark oracle. As stated in §4.2 and Appendix D.2 (“Development reference versus benchmark oracle”), development-time feedback comes from executing the task-visible PyTorch source and the candidate JAX repository. Hidden tests, frozen reference outputs, reference JAX solutions, and benchmark scores are inaccessible during development; Appendix D.3 details this information boundary.

We acknowledge that the executable source can be viewed as a behavioral oracle in a broad sense. Our contribution is therefore not the general idea of debugging with execution feedback, but a protocol for evidence-driven, whole-repository cross-framework migration:

  • Interface-sensitive verification (§4.2; Appendix D.2). Comparisons preserve the original caller's access patterns and output contracts, rather than merely matching numerical values after candidate-specific unwrapping. Candidate-side diagnostics additionally examine parameter trees, gradient liveness, JAX tracing, and integrated training/serving execution.
  • Persistent evidence distinct from code state (§4.1; Appendix D.1). The harness tracks established, violated, and unresolved properties separately from the evolving repository, carrying repair targets and preservation constraints into subsequent iterations.
  • Separated repair and verification (§4.1; Appendix D.1). Only the Migrator modifies the repository; the Diagnoser and Verifier operate on read-only candidates. Verification requires execution evidence rather than accepting the implementer's completion claims.

Together, these mechanisms turn source execution into localized repair objectives and explicit regression-preservation constraints across iterations. We position this as a migration-specific harness contribution, rather than claiming that execution-guided self-debugging itself is new.

We would like to thank the reviewer once again for their valuable time and thoughtful feedback. We sincerely hope that our responses and revisions have adequately addressed the concerns raised and further improved the clarity and overall quality of the manuscript. We would greatly appreciate the reviewer's further assessment of these improvements and warmly welcome any additional feedback or discussion.

Reviewer_rGxJOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

T2J-Bench contains 63 PyTorch-to-JAX migration tasks covering LoRA training and paged-attention serving. An isolated, offline verifier compares migrated implementations against frozen PyTorch references using structural, numerical, and behavioral checks.

Critical strengths

The benchmark addresses a useful problem with an evaluation design that goes substantially beyond checking whether translated code runs. Checkpoint conversion, gradients, optimizer behavior, and serving-state invariants are all meaningful migration requirements. Separating the development environment from a PyTorch-free verifier is sensible, and using the original PyTorch implementation as ground truth avoids treating a second implementation as infallible.

The evaluation is substantial: twenty baseline configurations, three seeds, task-subtype breakdowns, reference-solution checks, and an injected-defect study. TwinHarness improves several scaffold-model combinations, not just one. The paper also reports regressions and uneven gains rather than presenting refinement as uniformly beneficial. These are genuine strengths, and the benchmark could be a useful community resource.

Critical weaknesses

1. Task diversity and incomplete multimodal grading.
Although 63 tasks is a good scale, they are generated from two shared codebases—15 model families × 3 training algorithms from LLaMA-Factory and 6 families × 3 serving modes from vLLM/tpu-inference—so the three tasks within each family share most of their model and weight-conversion code. Despite 24 of the 63 tasks involving vision-language models, the paper reports no breakdown by model family or by text vs. multimodal tasks. Furthermore, Appendix C.7 notes that on multimodal tasks, image features flowing all the way into language-model logits are not numerically graded, and Qwen2-Omni's audio path is only checked for presence. Reporting per-family and text-vs-VLM numbers would give a clearer picture of true coverage.

2. TwinHarness bundles several changes without isolating what drives the gain.
TwinHarness combines at least four additions over the baseline: (i) explicit prompt instructions to run /workspace/torch_orig and match caller unwrapping, (ii) pre-executed deterministic {runtime_checks} (and an unexplained {skills} block in Listing 2) injected into the prompt, (iii) a three-role Diagnose–Migrate–Verify split with read-only permissions, and (iv) persistent structured evidence (EtE_t). Comparing this full package against unguided continuation does not show whether the three-agent loop is needed, or whether a single agent given {runtime_checks} and the same source-execution prompt would perform just as well. Notably, in Round 1 on OpenCode (Fig. 3b), TwinHarness (0.632) starts below the single-round baseline (0.676) before overtaking it in Round 2 (0.706), and on Claude-Code (Table 4) it slightly degrades SFT (0.854 → 0.821). How {runtime_checks} constructs matched differential probes across PP and JJ without task-specific test scaffolding also needs explanation.

3. Oracle failures on training tasks and runtime/cost accounting.
The reference JAX solution passes all checks on 18/18 inference tasks (0.997), but fails strict checks on 11 of 45 training tasks (0.973 overall; SFT and DPO oracle rewards are 0.966 and 0.965, lower than PPO's 0.988 in Table 4). It is unclear whether those 11 cases reflect cross-framework training variance, reference bugs, or overly tight checks. In addition, the seeded-defect study (84% check sensitivity) is only run on the 18 inference tasks, not on training. Finally, please clarify whether JAX JIT compilation time is included in the 2×2\times runtime budget at L5L_5–L6L_6, how the 3-hour wall-clock limit applies across multi-round continuations, and provide the exact cost/runtime numbers behind Figure 3a.

Questions for authors
  1. In the grader implementation, how are negative SscalarS_{\text{scalar}} values, zero-tensor ScosS_{\text{cos}} values, partial Spec failures, and crashed downstream levels (L4L_4–L6L_6) actually aggregated?
  2. How are {runtime_checks} and {skills} (Listings 2–3) implemented, and how does a single-agent baseline perform when given those same runtime checks and source-execution instructions?
  3. What causes the reference JAX solution to fail strict checks on 11/45 training tasks (especially on SFT and DPO), and how do agent scores break down between text-only and VLM families?
6 comments
AuthorsOpenReview ↗

Response 1

We sincerely thank the reviewer for the detailed comments on task coverage, component attribution, and grading validity. We group related questions below and address each concern separately.


[W1, Q3]: Task diversity and performance breakdown

Response. Thank you for highlighting the distinction between task diversity and performance breakdown.

  • Task diversity and shared code. Regarding task diversity, we argue that upstream repository diversity and migration task diversity are distinct dimensions. Although our tasks originate from a limited number of upstream codebases, they exhibit substantial diversity in model architectures, training algorithms, modalities, and execution semantics. Specifically, T2J-Bench covers 18 model families with different architectural structures, parameter organizations, and checkpoint conversion requirements, including both text-only and vision-language models. The 45 training tasks span SFT, DPO, and PPO, which require different optimization objectives, gradient computations, and training dynamics. The 18 serving tasks cover Core, Ragged, and Chunked modes, involving different execution semantics such as dynamic batching, paged KV-cache management, and prefix reuse. These differences are also reflected in our empirical results (manuscript Table 4), where different task categories exhibit substantial performance variations and changes in agent rankings. For example, Claude-Code performs best on SFT, OpenCode leads on Core serving, while TwinHarness achieves the highest performance on DPO, PPO, and Chunked serving. These observations suggest that the tasks exercise different migration capabilities rather than merely repeating identical migration problems under different configurations.

    The clearest evidence comes from holding the agent and model family fixed. On Qwen2-VL, Claude-Code with opus-5 scores 0.99 on SFT, 0.96 on DPO, and 0.49 on PPO. On GPT-OSS, OpenCode with opus-5 scores 0.80 on core serving, 0.28 on chunked serving, and 0.10 on ragged serving. Across all 21 groups of three tasks (15 training families and six serving families), the median within-group gap between the highest and lowest reward is 0.29 for Claude-Code and 0.34 for OpenCode. Agent rankings also vary by workload: Claude-Code leads on SFT, OpenCode on core serving, and TWINHARNESS on DPO, PPO, and chunked serving (manuscript Table 4). We clarified the shared code and the architectural, training, and serving differences in Section 3.1.

    To broaden source coverage, we added multimodal GRPO from TRL v1.17.1 (Qwen2.5-VL-3B, 4.4k nonblank Python lines) and sliding-window RadixAttention from SGLang v0.5.19 (Gemma 4, 12.1k lines). Approximately 20% of the GRPO code matches existing tasks, mostly in vendored Qwen-VL modeling code, compared with 6.5% for RadixAttention. We pinned both tasks to upstream commits, checked the extracted PyTorch workloads against upstream behavior, and applied the same execution-based grading principles. The reference JAX ports scored 0.999997 and 0.999656, respectively.

    As shown in Table 1, TWINHARNESS achieved the highest observed mean reward on both additional tasks. On GRPO, it scored 0.681, compared with 0.632 for the strongest baseline (OpenCode with opus-5) and 0.355 for its matched Claude-Code baseline. On RadixAttention, it scored 0.812, compared with 0.767 for the strongest baseline (Claude-Code with opus-5). Our analysis of the run logs traced many zero-reward baseline runs to two task-specific failures: GRPO ports failed to reproduce TRL’s per-batch PyTorch shuffle, while RadixAttention ports ran out of memory during batched serving. Section 5.2 references the task construction, validation, results, and failure analysis in Appendix E (manuscript Table 20).

  • Per-family and text-versus-VLM results: Tables 2 and 3 report the requested breakdown for TWINHARNESS and the two strongest off-the-shelf baseline configurations, OpenCode and Claude-Code, all using opus-5. These results and their analysis have been added to Appendix A.4 (Table 8), with a reference in Section 5.2.

TWINHARNESS achieves the highest reported mean in 10 of 18 model families and in both task groups. Its text-only reward exceeds OpenCode, the stronger baseline in that group, by 0.038; its vision-language reward exceeds Claude-Code, the stronger baseline in that group, by 0.032. The aggregate improvement is therefore present in both groups, although gains are not uniform across families: OpenCode leads in six families and Claude-Code in two. These VLM scores reflect the properties covered by the current grader, rather than complete multimodal equivalence.

AuthorsOpenReview ↗

Response 2

Continuation of our response to [W1, Q3] from the previous page:

Table 1. Reward on the two additional tasks from independent upstream codebases.

HarnessModelMultimodal GRPO (TRL)RadixAttention (SGLang)
Claude-Codeopus-50.355 ± 0.4170.767 ± 0.232
Claude-Codesonnet-50.209 ± 0.2200.000 ± 0.000
OpenCodeopus-50.632 ± 0.4000.752 ± 0.255
OpenCodegemini-3.7-flash0.184 ± 0.2120.267 ± 0.463
OpenCodesonnet-50.273 ± 0.2370.000 ± 0.000
OpenCodeglm-5.20.016 ± 0.0140.138 ± 0.239
OpenCodegemini-3.8-flash0.090 ± 0.1090.126 ± 0.218
OpenCodegemini-3.1-pro0.000 ± 0.0000.000 ± 0.000
Gemini-CLIgemini-3.7-flash0.016 ± 0.0140.000 ± 0.000
Gemini-CLIgemini-3.8-flash0.031 ± 0.0310.000 ± 0.000
Gemini-CLIgemini-3.1-pro0.000 ± 0.0000.000 ± 0.000
TWINHARNESSopus-50.681 ± 0.1230.812 ± 0.091

Table 2. Mean reward by model family. All configurations use opus-5

Model familyTask typeClaude-CodeOpenCodeTWINHARNESS
BLOOMText-only0.7740.8600.821
FalconText-only0.6110.6740.702
Gemma-4Text-only0.3300.5280.417
GPT-OSSText-only0.5140.3920.575
Llama 3Text-only0.5390.5900.540
MistralText-only0.6350.6950.751
Phi-4Text-only0.6870.7340.641
Qwen2Text-only0.4890.5780.817
Qwen3Text-only0.6010.6840.744
StarCoder2Text-only0.6780.7150.691
Gemma 3VLM0.7710.8540.806
InternVLVLM0.7130.8860.907
LLaVA-NeXTVLM0.8150.7380.717
PaliGemmaVLM0.8240.8930.907
Qwen2-VLVLM0.8110.7680.820
Qwen2.5-VLVLM0.7430.4920.665
Qwen3-VLVLM0.7700.7860.839
Qwen2-OmniVLM0.6870.4840.732

Table 3. Mean reward on text-only and vision-language tasks.

Task groupTasksClaude-CodeOpenCodeTWINHARNESS
Text-only390.5760.6390.677
Vision-language240.7670.7380.799
Overall630.6490.6760.723

[W2, Q2]: Component attribution, runtime checks, and the single-agent comparison

Response.

  • Implementation of {runtime_checks} and {skills}. We apologize for leaving the two placeholders unexplained in the original manuscript. Both were intended to convey evidence from the deterministic development-time probes described in Appendix D.2, Table 11. For each of the seven probe types, Table 11 specifies what it checks and what evidence it returns. The harness runs the applicable probes on the current candidate and inserts the resulting observations and failure details into {runtime_checks}. To clarify this, we removed the redundant {skills} placeholder from Listing 2 and revised the captions of Listing 2 (Migrator) and Listing 3 (Verifier) in Appendix D.4. Both captions now define {runtime_checks} and point directly to Table 11.

  • First-round OpenCode performance and the SFT reward decrease. We revised Section 5.4 (Figure 3b) to clarify when TWINHARNESS shows an advantage. On OpenCode, TWINHARNESS trails the baseline after one round (0.632 vs. 0.676). At a matched two-round budget, when the Migrator can use first-round verification findings to revise the candidate, TWINHARNESS achieves higher reward than ordinary continuation on both OpenCode (0.706 vs. 0.629) and Claude Code (0.723 vs. 0.658). We also expanded the analysis in Section 5.2. Claude Code already achieves its highest training reward on SFT (0.854), leaving less room for improvement. Further edits can disrupt working behavior, which may explain the modest decrease to 0.821. The 0.033 gap is averaged over three seeds. For context, manuscript Table 3 reports an across-seed standard deviation of 0.022 for TWINHARNESS’s overall reward. The observed gains are larger on DPO (+0.067) and PPO (+0.082), where the baseline starts lower.

  • Construction of matched PyTorch–JAX differential probes. The baseline serving prompt (Appendix D.5, Listing 5) identifies the PyTorch source repository and specifies the public methods, signatures, and submodule paths that the JAX port must preserve. It also specifies the checkpoint and precision settings, how JAX parameter-tree leaves map to checkpoint tensors, the paged KV-cache layout, and the named output fields for prefill and decode. For example, an agent can call Engine.prefill on both engines with the same input_ids and request_id, then compare the logits field of the two PrefillResult objects. The prompt thus tells the agent what to call on each side, which inputs to pass, and which results to compare.

AuthorsOpenReview ↗

Response 3

Continuation of our response to [W2, Q2] from the previous page:

  • Comparison fairness and the additional single-agent ablation. Thank you for suggesting a single-agent control with the same runtime checks and source-execution instructions. Table 4 compares refinement over two rounds, and Table 5 examines which roles contribute to the full system’s reward. Both experiments use Claude Code with opus-5 and appear in Appendix D.7 (manuscript Tables 17–18), as referenced in Section 5.4.

    Single-agent comparison (Table 4). We compared ordinary single-agent continuation, a single agent given the same runtime checks and source-execution instructions as TWINHARNESS, and full TWINHARNESS. The instructions direct the agent to execute the task-visible PyTorch source and test the JAX port using the original caller’s access pattern. TWINHARNESS organizes this feedback through its Diagnose–Migrate–Verify loop: the Verifier checks the first-round candidate, the Diagnoser uses the recorded discrepancies to prioritize repairs, and the Migrator implements those repairs in Round 2.

    Both single agents started at 0.649. Giving the agent checks and source-execution instructions produced a modest improvement over ordinary continuation, increasing the second-round gain from 0.009 to 0.016. TWINHARNESS gained 0.061 and finished 0.058 ahead of the single agent with the same checks and instructions. The larger gain emerged in the round when the Migrator could first act on independently verified discrepancies. This supports the value of organizing execution feedback into a concrete repair plan.

    Role ablations (Table 5). We changed one role at a time. Removing the Diagnoser eliminated the dedicated step that identifies and prioritizes migration gaps, reducing reward by 0.039. Retaining the Diagnoser and Verifier but replacing the specialized Migrator with a generic coding agent reduced reward by 0.025. Removing the Verifier produced the largest observed drop, from 0.723 to 0.671 (−0.052). Because the Verifier produces the evidence used in the next round, this result measures the combined value of independently checking the candidate and carrying the findings forward for repair. Across all three ablations, the full harness achieved the highest reward, supporting the contribution of each tested role within the Diagnose–Migrate–Verify loop.

    Table 4. Reward by refinement round on Claude Code.

    ConfigurationRound 1Round 2Change
    Claude Code (base)0.6490.658+0.009
    Claude Code + runtime checks and source-execution instructions0.6490.665+0.016
    TWINHARNESS (full)0.6620.723+0.061

    Table 5. Role ablations on Claude Code.

    ConfigurationRewardDifference from full
    TWINHARNESS (full)0.723—
    Without Diagnoser0.684−0.039
    Without Verifier0.671−0.052
    With a generic Migrator0.698−0.025

[W3, Q3]: Reference-solution failures, grader validation, and runtime/cost accounting

Response.

  • Causes of reference-solution failures: All 11 failures come from the two checks that compare complete training runs with PyTorch: the final training loss (L5) and the held-out validation loss (L6). On these 11 tasks, every Spec check passes and every other check scores at least 0.99. Every training run also finishes within the time budget. The reference therefore matches PyTorch step by step, and the gap appears only at the end of a full training run. The 11 cases have three causes:
    • Cross-framework drift on vision-language tasks (8 tasks): SFT and PPO on Gemma 3 and LLaVA-NeXT, all three PaliGemma tasks, and PPO on Qwen2-Omni. We attribute these to small numerical differences that accumulate over a full run on long image-token sequences. We found no specific bug, but cannot rule out reference-side differences that appear only over long runs.
    • Unstable PyTorch training (DPO on Qwen3-VL): PyTorch's own validation loss varies across five identical runs (mean 3.18, standard deviation 0.32), while the reference converges to 1.26. Here the reference trains better than the original, and the check penalizes the difference.
    • StarCoder2 dropout (2 tasks): In DPO, both frameworks apply dropout but draw different random masks, so the final training loss misses the band narrowly (by 0.056 against a tolerance of 0.05). In PPO, the reference omits dropout during policy updates. This is a genuine limitation of the reference.

SFT and DPO score below PPO because of how the tolerance band is set. Each band is twice PyTorch's own run-to-run spread, subject to a floor. PyTorch's SFT and DPO runs are deterministic (except DPO on Qwen3-VL), so their bands sit at the floor of 0.05. The PPO band on held-out reward has a floor of 0.10.

Reviewer_evZ9OpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This work introduces T2J-Bench, 63 PyTorch → JAX/Flax migration tasks for coding agents: 45 training tasks (15 model families × SFT/DPO/PPO LoRA pipelines from LLaMA-Factory) and 18 serving tasks (6 families × 3 vLLM-style paged-attention modes). Agents get the PyTorch source and checkpoint and must produce a torch-free JAX repo, including checkpoint conversion. Grading runs offline with PyTorch removed, comparing against references recorded from the original along a seven-level lattice (structure → operator/forward → gradients/decode → end-to-end + held-out behavior), combined into a spec-gated continuous reward. The reference JAX solution scores 0.980, and the best of twenty agent–model configurations (three seeds each) scores 0.676. The authors also propose TwinHarness, a diagnose–migrate–verify loop that raises opus-5 from 0.649 to 0.723.

Critical strengths
  • [Section 3.2, Table 2, Appendix C] The grading design is a real step up from typical migration/translation evals.
    • The torch-free, offline verifier means "fake" equivalence via importing PyTorch fails loudly.
    • References are recorded from the PyTorch original, not from a hand-written JAX solution.
    • Behavior is graded at multiple depths (forward → gradients → training dynamics → held-out loss, or prefill → paged decode).
    • Validated with an oracle run and a seeded-defect sweep (84% of serving checks tripped).
  • [Section 5, Table 3] Broad, reasonably rigorous evaluation.
    • Twenty configurations across five scaffolds, three seeds each on the full suite, with SDs and a cost analysis.
    • High spec pass rates (≈0.92) coexist with large numeric gaps (≈0.62–0.67): a useful message that "it runs" ≠ "it's correct".
Critical weaknesses
  • [Section 3.1, Appendix C.1] The task diversity is narrower than the framing suggests.
    • All 45 training tasks come from one ~4k-line LLaMA-Factory extract, and all 18 serving tasks from one ~1.5k-line engine in three modes. That's effectively ~6 programs × many architectures.
    • "Whole-repository" is a bit of an overclaim; this is closer to "two extracted codebases × many model configs".
    • Contamination / shortcut risk: serving sources partly come from tpu-inference (vLLM's JAX counterpart), many families have public Flax/JAX ports (HF Flax, MaxText, EasyLM), and agents work in a networked container. The independence gate wouldn't catch an agent adapting an existing JAX implementation.
  • [Section 4, 5.2–5.4] The TwinHarness comparison isn't quite apples-to-apples.
    • 0.723 comes after two rounds with three role-specialized agents. After one round it's 0.662 / 0.632, at or below the OpenCode baseline (0.676).
    • "Highest of any configuration" compares a multi-round, multi-agent pipeline to single-run baselines. Only two baselines get matched continuations, and dollar costs aren't stated in the text.
    • Adversarially, the prompts are written by people who know the grader. E.g., the Migrator prompt's "same container type, same fields, reachable by the same access" lines up closely with the Spec checks. That's not cheating, but baselines don't get it.
    • The {skills} and {runtime_checks} slots in Listing 2 are unspecified.

[Overall] I like this benchmark, and the grading methodology is something others could adopt. My main hesitations are the narrow task set, the plausible network/retrieval shortcut, and a TwinHarness comparison that mixes extra rounds with grader-aware prompting. Clarifying the network question and tempering the framing would address most of this.

Questions for authors

Clarifications:

  1. [Section 5.1] Can agents access the internet during development? Did you check trajectories for downloading or copying existing JAX/Flax implementations? Does any provided source include tpu-inference JAX code?
  2. [Section 3.3, C.7] Who wrote the 63 reference JAX solutions, and were agents involved? Why do 11/45 training references fail strict checks? Are some checks unreachable even for a correct migration?
  3. [Eq. 1] The text says independence "zeros the whole score", but s_spec is a fraction of checks passed. If one module imports PyTorch, is R = 0?
  4. [Table 3, A.3] gemini-3.1-pro scores 0.006–0.086 everywhere, below gemini-3.7-flash in every scaffold. Is this an infrastructure issue (timeouts, context limits, refusals)?
  5. [Section 5.3] What are the actual per-run API costs for TwinHarness vs. each baseline? Can you compare at matched rounds and spend for all scaffolds?

Suggestions:

  1. One or two more independent source codebases per class (a different training framework, a different serving engine).
  2. Pin and report CLI versions (Claude Code / OpenCode / Gemini CLI) and release trajectories, since these tools update constantly.

Minor:

  • [Section 3.2, C.6] "Threshold-free" is a bit strong: the L6 band, the 2× runtime budget, and the 0.5/0.5 weights are all thresholds or hyperparameters.
  • [C.6] Does JAX compile time count against the runtime budget?
  • [Section 5.2] "Leads five of six columns": some leads are within the SDs (±.022 vs. ±.014).
  • [References] SWE-bench is cited twice (2024a / 2024b).

Suggested Citations:

4 comments
AuthorsOpenReview ↗

Response 1

We sincerely thank the reviewer for the thoughtful assessment and detailed suggestions on task diversity, evaluation validity, comparison fairness, and reproducibility. We address related comments together below.


[W1, S1]: Task diversity, repository-level scope, and additional codebases

Response. Thank you for raising this concern and for suggesting that we evaluate migrations from additional upstream projects.

Clarification on task diversity and repository-level scope. First, regarding repository-level scope, we emphasize that "repository-level" refers to the scope and complexity of each migration task, rather than the number of independent upstream repositories. Unlike function-level or snippet-level translation, each task requires migrating an entire self-contained, multi-module PyTorch repository into a functional JAX/Flax implementation. Specifically, our training tasks involve approximately 4k lines of code across 15–20 modules, while serving tasks involve approximately 1.5k lines across 19 modules. These tasks require coordinated migration of model implementations, checkpoint conversion, data processing, training or serving pipelines, and their cross-module dependencies. Importantly, the migrated repository must execute end-to-end and preserve the original PyTorch behavior, including numerical equivalence, gradient computation, optimization dynamics, or serving behavior. Therefore, these tasks involve repository-level migration challenges rather than isolated function translation. We acknowledge that these are self-contained extracts rather than complete upstream projects, and have clarified this distinction in the revised manuscript. Second, regarding task diversity, we argue that upstream repository diversity and migration task diversity are distinct dimensions. Although our tasks originate from a limited number of upstream codebases, they exhibit substantial diversity in model architectures, training algorithms, modalities, and execution semantics. Specifically, T2J-Bench covers 18 model families with different architectural structures, parameter organizations, and checkpoint conversion requirements, including both text-only and vision-language models. The 45 training tasks span SFT, DPO, and PPO, which require different optimization objectives, gradient computations, and training dynamics. The 18 serving tasks cover Core, Ragged, and Chunked modes, involving different execution semantics such as dynamic batching, paged KV-cache management, and prefix reuse. These differences are also reflected in our empirical results (Table 4), where different task categories exhibit substantial performance variations and changes in agent rankings. For example, Claude-Code performs best on SFT, OpenCode leads on Core serving, while TwinHarness achieves the highest performance on DPO, PPO, and Chunked serving. These observations suggest that the tasks exercise different migration capabilities rather than merely repeating identical migration problems under different configurations.

Additional codebases. We added one training task from TRL and one serving task from SGLang. For each, we pinned the upstream commit, verified the extracted PyTorch workload against upstream behavior, built a reference JAX reimplementation, and applied the shared grading procedure using frozen PyTorch outputs, structural, numerical, and behavioral checks, and reference-calibrated tolerances. Multimodal GRPO uses TRL v1.17.1 with Qwen2.5-VL-3B (4.4k nonblank Python lines). Approximately 20% of its lines match existing tasks, mostly in vendored Hugging Face Qwen-VL modeling code, with little overlap in the trainer, loss, or data pipeline. Sliding-window RadixAttention uses SGLang v0.5.19 with Gemma 4 (12.1k nonblank Python lines), with 6.5% line overlap. The reference JAX reimplementations scored 0.999997 and 0.999656, respectively.

As shown in Table 1, TWINHARNESS achieved the highest observed mean reward among the evaluated configurations on both new tasks. On GRPO, it scored 0.681, compared with 0.632 for the strongest baseline (OpenCode with opus-5) and 0.355 for its matched Claude-Code baseline. On RadixAttention, it scored 0.812, compared with 0.767 for the strongest baseline (Claude-Code with opus-5). Our analysis of the run logs traced many zero-reward baseline runs to two task-specific failures: GRPO JAX reimplementations failed to reproduce TRL’s per-batch PyTorch shuffle, while RadixAttention JAX reimplementations ran out of memory during batched serving. We added the task construction, validation, results, and failure analysis to Appendix E, and referenced them in Section 5.2.

AuthorsOpenReview ↗

Response 2

Continuation of our response to [W1, S1] from the previous page:

Table 1. Reward on the two additional tasks from independent upstream codebases. Values are means ± sample standard deviations over three seeds.

HarnessModelMultimodal GRPO (TRL)RadixAttention (SGLang)
Claude-Codeopus-50.355 ± 0.4170.767 ± 0.232
Claude-Codesonnet-50.209 ± 0.2200.000 ± 0.000
OpenCodeopus-50.632 ± 0.4000.752 ± 0.255
OpenCodegemini-3.7-flash0.184 ± 0.2120.267 ± 0.463
OpenCodesonnet-50.273 ± 0.2370.000 ± 0.000
OpenCodeglm-5.20.016 ± 0.0140.138 ± 0.239
OpenCodegemini-3.8-flash0.090 ± 0.1090.126 ± 0.218
OpenCodegemini-3.1-pro0.000 ± 0.0000.000 ± 0.000
Gemini-CLIgemini-3.7-flash0.016 ± 0.0140.000 ± 0.000
Gemini-CLIgemini-3.8-flash0.031 ± 0.0310.000 ± 0.000
Gemini-CLIgemini-3.1-pro0.000 ± 0.0000.000 ± 0.000
TWINHARNESSopus-50.681 ± 0.1230.812 ± 0.091

[W2, Q1]: Internet access, contamination, and retrieval shortcuts

Response. Yes, agents have internet access during development. We agree that a PyTorch-independence check alone cannot establish that a submission was not adapted from public JAX code. We therefore audited both execution trajectories and submitted repositories.

  • Network access and evaluation separation: The agent container is networked because the agent must call its model API. Grading takes place in a separate container with no network access and no PyTorch, for all 65 tasks.

  • Trajectory audit: We audited 3,811 agent trials across 60 campaigns, comprising 544,080 tool calls, including subagent calls where logged.

    • No public JAX model implementations were available locally: The agent images use transformers 5.4, which contains no Flax models, and do not include MaxText, EasyLM, EasyDeL, keras-hub, or tpu-inference.
    • No trial successfully fetched, cloned, or installed a public JAX implementation: 0 of 3,811 trials.
    • Web tools were used in 23 trials (0.6%), with 68 calls: 37 consulted JAX/Flax API documentation or error messages; 18 fetched PyTorch reference code; and 13 calls across five trials searched for task-specific JAX code. None of those 13 calls returned an implementation: ten returned empty results and three returned prose summaries.
    • Two additional trials attempted to download nonexistent HF Flax files.
    • The seven trials seeking task-specific implementations scored 0–0.001 or produced no gradable submission.
  • Submitted-code audit: We compared all 3,567 submitted JAX repositories against nine public JAX/Flax codebases, comprising 4,878 files: tpu-inference, MaxText, EasyLM, EasyDeL, Gemma, big_vision, Levanter, keras-hub, and HF Flax. We count only code shared with these repositories that is not already present in the task’s own PyTorch source. As positive controls, we splice a public file into our reference JAX solution and measure detection.

Table 2. Submitted-code overlap with public JAX/Flax implementations, excluding overlap inherited from the provided PyTorch source.

Audit measureExact tokensIdentifier-normalized tokens
Agent-submitted repositories: median / maximum overlap0.18% / 1.44%1.6% / 10.9%
Positive controls containing an adapted public file≥11.8%≥9.1%, even with every identifier renamed
Repositories flagged (>5% overlap or ≥60-line match)03; all identified as boilerplate on manual review

As shown in Table 2, the audit found no evidence of adaptation from public JAX implementations. The three identifier-normalized flags were reviewed manually and attributed to boilerplate, rather than copied task implementations.

  • Provided source code and tpu-inference provenance: All provided sources are PyTorch: no file imports JAX or Flax, and the 18 serving tasks share at most 0.04% of their code fingerprints with tpu-inference. Some serving-source docstrings name tpu-inference as the upstream origin of the engine. No audited trajectory followed these pointers, and we will remove them from the release.

  • Release and audit limitations: The audit tools will accompany the benchmark. These audits can also gate reward: a trial that retrieves an actual JAX implementation of its task, or whose submission is confirmed to contain such an adaptation, can be scored zero.


[W3, Q5]: Fair comparison, refinement budgets, API costs, and prompt information

Response.

  • Refinement rounds and comparison scope: TWINHARNESS follows a Diagnose–Migrate–Verify loop: the Verifier produces execution-grounded evidence for the current candidate, which guides diagnosis and repair in the next round. The first-round candidate therefore has not yet incorporated its own verification feedback; Round 2 completes this feedback-and-revision cycle.
AuthorsOpenReview ↗

Response 3

Continuation of our response to [W3, Q5] from the previous page:

To evaluate the benefit of this mechanism, Figure 3(b) compares TWINHARNESS with ordinary baseline continuation at matched round counts. From Round 1 to Round 2, TWINHARNESS improves from 0.662 to 0.723 with Claude-Code and from 0.632 to 0.706 with OpenCode, while the corresponding baselines change from 0.649 to 0.658 and from 0.676 to 0.629. TWINHARNESS thus achieves higher reward with both scaffolds after two rounds. Figure 3(a) further shows that, at API expenditure close to TWINHARNESS, continued baselines reach approximately 0.66 and 0.60, respectively, below 0.723. These comparisons support the effectiveness of structured, execution-grounded refinement beyond simply allowing additional continuation.

  • API costs and matched-budget comparisons: We already report API costs in Figure 3(a): the horizontal axis gives cumulative API expenditure in USD per 63-task run, including preceding continuation rounds. We assess budget comparability in two complementary ways. Matched rounds: Figure 3(b) compares both methods after one and two rounds. TWINHARNESS improves substantially with both Claude-Code and OpenCode, whereas ordinary continuation yields only a small gain for Claude-Code and reduces OpenCode’s reward. Comparable API cost: Figure 3(a) shows that, at expenditure close to TWINHARNESS, three-round baseline continuation achieves approximately 0.66 with Claude-Code and 0.60 with OpenCode, compared with TWINHARNESS’s 0.723. Thus, TWINHARNESS’s structured refinement achieves higher reward both at matched round counts and at comparable API expenditure.
  • Prompt information and baseline access: Baselines receive the same categories of task and interface information as TWINHARNESS, with task-specific instructions often providing more detailed API signatures, return fields, and parameter conventions. TWINHARNESS organizes validation of these shared requirements through its Diagnose–Migrate–Verify protocol. To make this information access transparent, Appendix D.5 now includes complete baseline prompts for representative training and serving tasks. These prompts demonstrate that the interface requirements are available to both methods, supporting a fair comparison in task information.
  • Implementation of {skills} and {runtime_checks}: We apologize for the unclear placeholders. Both were intended to accommodate deterministic probe information summarized in Table 11, rather than additional undisclosed knowledge. Table 11 summarizes seven development-time checks—module imports, forbidden dependencies, output contracts, parameter trees, gradient liveness, JAX tracing, and training/serving entry points—along with their grounding, examined properties, and returned evidence. This evidence guides diagnosis and repair. To remove ambiguity, we have removed {skills} and clarified the contents of {runtime_checks} in the corresponding listing captions, with a reference to Table 11. These caption additions are highlighted in blue.

Q2: [Section 3.3, Appendix C.7] Reference JAX solutions

Response.

  • Reference authorship and agent involvement: Yes, agents were involved. We wrote the references with Claude Code (Claude Opus 5), guided by oracle feedback and human review. We graded each draft reference against the PyTorch-derived targets and fixed the shortfalls this revealed, such as missing vision-tower code or slow training steps, then graded again. The references are not the grading target:

    • Targets come from PyTorch. Every graded numeric and behavioral target is frozen from runs of the original PyTorch code.
    • Grading never runs the references. They are excluded from both the agent and the verifier images, so evaluated agents never see them.
    • Their role is to prove each task is solvable and to measure the score a faithful migration actually achieves. The only grading input derived from them is the list of required public names, which is intersected with the PyTorch API and so can only relax requirements.
  • Why 11 of 45 training references fail strict checks: All 11 failures come from the two checks that compare complete training runs with PyTorch: the final training loss (L5) and the held-out validation loss (L6). On these 11 tasks, every Spec check passes and every other check scores at least 0.99. Every training run also finishes within the time budget. The reference therefore matches PyTorch step by step, and the gap appears only at the end of a full training run. The 11 cases have three causes:

    • Cross-framework drift on vision-language tasks (8 tasks): SFT and PPO on Gemma 3 and LLaVA-NeXT, all three PaliGemma tasks, and PPO on Qwen2-Omni. We attribute these to small numerical differences that accumulate over a full run on long image-token sequences. We found no specific bug, but cannot rule out reference-side differences that appear only over long runs.
AuthorsOpenReview ↗

Response 4

Continuation of our response to Q2 from the previous page:

  • Unstable PyTorch training (DPO on Qwen3-VL): PyTorch's own validation loss varies across five identical runs (mean 3.18, standard deviation 0.32), while the reference converges to 1.26. Here the reference trains better than the original, and the check penalizes the difference.
    • StarCoder2 dropout (2 tasks): In DPO, both frameworks apply dropout but draw different random masks, so the final training loss misses the band narrowly (by 0.056 against a tolerance of 0.05). In PPO, the reference omits dropout during policy updates. This is a genuine limitation of the reference. The training references nevertheless achieve a median reward of 0.9999 and a mean of 0.973.
  • Reachability of strict checks: Exact full-run trajectory agreement is not achievable for the stochastic cases described above, even with a faithful migration. This motivates continuous similarity scoring rather than an all-or-nothing task verdict: numerical near-misses receive partial credit.

Q3: [Eq. 1] Independence score

Response. Yes—when a PyTorch import prevents the candidate from loading or executing in the torch-free verifier, the task receives R = 0. This evaluation pipeline is specified in Section 3.2 and Figure 1, whose “Isolated Verifier” explicitly excludes PyTorch and network access, and further described in Appendix C.3 (P1: Independence-first). Independence is thus an execution prerequisite, whereas s_spec is the fraction of specification checks passed, not a binary independence indicator. Passing other Spec checks cannot rescue a candidate that cannot execute. We have clarified this distinction around Eq. 1 and in the corresponding appendix discussion, with the revisions highlighted in blue.


Q4: [Table 3, Appendix A.3] Gemini performance

Response. We address this from external coding evaluations and our execution-log analysis.

  • External coding evidence: Gemini 3.7 Flash is a later release and outperforms Gemini 3.1 Pro Preview in independent coding evaluations: 80.80% versus 78.80% on SWE-bench Verified with Mini-SWE-agent [1], 77.53% versus 70.79% on Terminal-Bench 2.1 with Terminus 2 [2], and 34.80% versus 17.31% on Code Migration v1 [3]. The direction of our results is therefore consistent with external evaluations.

  • Execution-log analysis: We inspected 756 trials per model. Pro repeatedly violated specified interfaces, including incorrectly nested parameter dictionaries in 93 training trials, unsupported arguments, and missing trainer methods. Its self-tests used the required apply({"params": p}) interface less often than Flash (58% versus 84%), while 29% used only the incorrect apply(params) form, compared with 2% for Flash. Across three groups with increasing self-test counts, Pro’s artifact loadability remained 26%, 23%, and 22%, compared with 42%, 84%, and 88% for Flash. These observations suggest that Pro’s repeated testing often failed to exercise the required interface, leaving migration errors unresolved.

[1] Vals AI, SWE-bench Verified

[2] Vals AI, Terminal-Bench 2.1

[3] Vals AI, Code Migration v1


S2: CLI versions and trajectories

Response.

  • Versions. Section 5.1 now specifies the versions of Claude Code (2.1.270), OpenCode (1.18.32), Gemini CLI (0.60.0), OpenHands (0.62.0), Terminus-2 (2.0.0).
  • Trajectories. Appendix D.6 (Tables 13–16) now presents and compares agent trajectories on the same Qwen2.5-VL serving task, including actions, self-tests, detected errors, and final rewards. Complete trajectories will be publicly released upon acceptance.

M1: Minor comments

Response.

  • “Threshold-free” terminology: We have replaced the blanket description with “continuous similarity-based scoring” in Section 3.2 and Appendix C, explicitly acknowledging the L6 band, runtime cap, and reward weights.
  • JAX compilation time: Compilation time is included in the runtime budget. We have clarified this in Section 3.2 and Appendix C.6.
  • Ranking and statistical uncertainty: We have clarified in Section 5.2 that the five-of-six-column ranking refers to reported means, without claiming statistically significant differences.
  • Duplicate SWE-bench citation: We have consolidated the duplicate entries and updated their citations throughout the manuscript.

C1: Suggested citations

Response. We added all four citations to the blue-highlighted paragraph in §2, “Repository-Level Generation and Behavioral Evaluation,” and clarified that T2J-Bench evaluates whether a PyTorch-to-JAX migration preserves checkpoint compatibility, gradients, training dynamics, and stateful decoding behavior.

We hope these clarifications address your concerns and would appreciate your reconsideration of the score. Thank you for your constructive feedback.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Our Sincere Appreciation to the Area Chair and Reviewers: Summary of Rebuttal Improvements

We sincerely thank the Area Chair and all reviewers for their thoughtful and constructive feedback. During the rebuttal, we substantially strengthened T2J-BENCH through additional migration tasks, comprehensive evaluation audits, controlled comparisons, component-level ablations, and further clarifications of our grading methodology. Below, we summarize the major concerns, our responses, and the key findings.

DimensionKey ConcernsOur Main ActionsKey Findings
Task Diversity & GeneralizationLimited upstream repositories and correlated migration tasks.Added two independent migration tasks from TRL (multimodal GRPO) and SGLang (RadixAttention), expanding the benchmark to 65 tasks. Reported per-family and text-only/VLM breakdowns.TWINHARNESS achieves the highest observed mean reward on both new tasks (0.681 and 0.812), supporting generalization beyond the original source codebases.
Evaluation Validity & IndependencePotential contamination from public JAX implementations and insufficient grader validation.Audited 3,811 agent trials and 3,567 submitted repositories for retrieval shortcuts and code overlap. Extended seeded-defect validation to all 45 original training tasks.Found no evidence of copied public JAX implementations. The grader detects 83% of 701 injected training defects, supporting its sensitivity to migration errors.
TWINHARNESS Effectiveness & FairnessWhether improvements come from additional refinement rounds, API budgets, or stronger prompts.Added matched-round and comparable-cost comparisons, including a single-agent baseline with identical runtime checks and source-execution instructions.At two rounds, TWINHARNESS reaches 0.723 vs. 0.658 for Claude Code and 0.706 vs. 0.629 for OpenCode, demonstrating benefits beyond ordinary continuation.
Component AblationsUnclear contributions of diagnosis, specialized migration, and independent verification.Added targeted ablations removing the Diagnoser, Verifier, or specialized Migrator.Removing these components reduces reward by 0.039, 0.052, and 0.025, respectively, supporting the value of the structured Diagnose–Migrate–Verify design.
Reward Robustness & Grading TransparencySensitivity to Spec/Numeric/Behavioral weights and unexplained reference failures.Evaluated five reward weightings, clarified score aggregation and reference limitations, and expanded the analysis of training and multimodal grading.TWINHARNESS maintains the highest mean reward across all tested weightings. Additional analysis identified reference-side limitations without changing the PyTorch-grounded evaluation targets.
Reproducibility & EfficiencyMissing implementation details, runtime accounting, and trajectory transparency.Clarified runtime checks, pinned agent CLI versions, reported detailed API costs and runtimes, and expanded trajectory analyses.At comparable API costs, TWINHARNESS achieves 0.723, compared with 0.660/0.603 for continued Claude Code/OpenCode baselines.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.