- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduces T2J-Bench, a benchmark of 63 repository-level PyTorch-to-JAX migration tasks covering training and serving. Candidates are evaluated in an offline, torch-free environment using structural, numerical, and behavioral equivalence. The paper evaluates 20 agent-model configurations and proposes TwinHarness, a reference-grounded diagnose-migrate-verify loop that improves reward from 0.649 to 0.723.
Critical strengths
1.Rigorous and practically meaningful evaluation. The torch-free verifier evaluates numerical, gradient, training, and serving behavior rather than superficial code similarity; 2.Broad empirical validation. The study covers 63 tasks, 20 configurations, three seeds, cost analysis, failure taxonomy, and several refinement settings. The benchmark is challenging and appears useful to the community.
Critical weaknesses
- Limited task diversity and independence. Although the benchmark contains 63 tasks, many are correlated variants formed by crossing a small number of source repositories, model families, and training or serving configurations. It is therefore unclear how well the conclusions generalize to genuinely diverse real-world migration projects;
- Incomplete validation of the evaluation and harness design. The overall ranking depends on a custom weighted reward, but sensitivity to alternative weightings is not reported. Moreover, TwinHarness is evaluated largely as a complete system, without sufficiently isolating the contribution of diagnosis, persistent evidence, and independent verification. This makes it difficult to determine which design choices actually produce the improvement.
Questions for authors
- What capability does TwinHarness introduce beyond existing oracle-guided self-debugging methods? 2.How sensitive are system rankings to the Spec/Numeric/Behavioral reward weights? 3.Can the authors ablate the diagnosis, persistent evidence, and independent verification components of TwinHarness separately?
5 comments
Response 2
Continuation of our response to W1 from the previous page:
Table 1. Reward on the two additional tasks from independent upstream codebases. Values are means ± sample standard deviations over three seeds. Bold indicates the highest observed mean in each column.
| Harness | Model | Multimodal GRPO (TRL) | RadixAttention (SGLang) |
|---|---|---|---|
| Claude-Code | opus-5 | 0.355 ± 0.417 | 0.767 ± 0.232 |
| Claude-Code | sonnet-5 | 0.209 ± 0.220 | 0.000 ± 0.000 |
| OpenCode | opus-5 | 0.632 ± 0.400 | 0.752 ± 0.255 |
| OpenCode | gemini-3.7-flash | 0.184 ± 0.212 | 0.267 ± 0.463 |
| OpenCode | sonnet-5 | 0.273 ± 0.237 | 0.000 ± 0.000 |
| OpenCode | glm-5.2 | 0.016 ± 0.014 | 0.138 ± 0.239 |
| OpenCode | gemini-3.8-flash | 0.090 ± 0.109 | 0.126 ± 0.218 |
| OpenCode | gemini-3.1-pro | 0.000 ± 0.000 | 0.000 ± 0.000 |
| Gemini-CLI | gemini-3.7-flash | 0.016 ± 0.014 | 0.000 ± 0.000 |
| Gemini-CLI | gemini-3.8-flash | 0.031 ± 0.031 | 0.000 ± 0.000 |
| Gemini-CLI | gemini-3.1-pro | 0.000 ± 0.000 | 0.000 ± 0.000 |
| TWINHARNESS | opus-5 | 0.681 ± 0.123 | 0.812 ± 0.091 |
[W2, Q2]: Sensitivity to reward weightings.
W2: The overall ranking depends on a custom weighted reward, but sensitivity to alternative weightings is not reported.
Q2: How sensitive are system rankings to the Spec/Numeric/Behavioral reward weights?
Response. Thank you for raising this concern. We keep Spec fixed as a multiplicative validity gate and vary the Numeric weight . The Behavioral weight is , giving the per-task reward
We compute reward per task, average across tasks within each seed, and report the mean ± sample standard deviation over three seeds. Across all five tested values of , the three opus-5 configurations retain the same ranking: TWINHARNESS > OpenCode > Claude-Code. TWINHARNESS leads OpenCode by 0.045 at (Behavioral only), 0.047 at the default , and 0.048 at (Numeric only). Table 2 reports these results (manuscript Section 5.2, Table 5).
Table 2. Sensitivity of reward to the Numeric weight .
| Harness | Model | (default) | ||||
|---|---|---|---|---|---|---|
| Claude-Code | opus-5 | 0.697 ± .011 | 0.673 ± .010 | 0.649 ± .008 | 0.624 ± .008 | 0.600 ± .008 |
| OpenCode | opus-5 | 0.707 ± .012 | 0.692 ± .013 | 0.676 ± .014 | 0.661 ± .015 | 0.646 ± .017 |
| TWINHARNESS | opus-5 | 0.752 ± .024 | 0.738 ± .023 | 0.723 ± .022 | 0.709 ± .022 | 0.694 ± .021 |
[W2, Q3]: Incomplete validation and ablation of TWINHARNESS components.
W2: Moreover, TwinHarness is evaluated largely as a complete system, without sufficiently isolating the contribution of diagnosis, persistent evidence, and independent verification. This makes it difficult to determine which design choices actually produce the improvement.
Q3: Can the authors ablate the diagnosis, persistent evidence, and independent verification components of TwinHarness separately?
Response. Thank you for asking us to clarify how these components contribute to TWINHARNESS. Table 3 summarizes the role ablations on Claude Code with opus-5. We changed one role at a time.
- Diagnosis. Removing the Diagnoser eliminated the dedicated step that identifies and prioritizes migration gaps. Reward fell from 0.723 to 0.684 (−0.039).
- Independent verification and persistent evidence. As described in Section 4 and Appendix D.1, the Verifier checks the current JAX candidate and records its findings in
E_t. In the next round, the Diagnoser readsE_tand prioritizes repairs, which the Migrator then implements. Removing the Verifier produced the largest observed drop, from 0.723 to 0.671 (−0.052). This ablation measures the combined contribution of independently checking the candidate and carrying those findings forward for repair. - Migrator specialization. We also retained the Diagnoser and Verifier but replaced the specialized Migrator with a generic coding agent that performs the JAX edits. Reward fell from 0.723 to 0.698 (−0.025).
We report these results and their analysis in Appendix D.7, “Single-Agent Comparison and Role Ablations”, referenced in Section 5.4 under Structured iteration versus ordinary continuation. The full harness achieved higher reward than all three ablated variants, supporting the contribution of each tested role: the Diagnoser prioritizes repairs, the specialized Migrator implements them, and the Verifier supplies execution evidence to guide the next round.
Table 3. Role ablations on Claude Code (opus-5).
| Configuration | Reward | Difference from full |
|---|---|---|
| TWINHARNESS (full) | 0.723 | — |
| Without Diagnoser | 0.684 | −0.039 |
| Without Verifier | 0.671 | −0.052 |
| With a generic Migrator | 0.698 | −0.025 |
Response 3
Q1: Novelty beyond oracle-guided self-debugging.
What capability does TwinHarness introduce beyond existing oracle-guided self-debugging methods?
Response. Thank you for asking us to clarify this distinction. TWINHARNESS does not receive feedback from the hidden benchmark oracle. As stated in §4.2 and Appendix D.2 (“Development reference versus benchmark oracle”), development-time feedback comes from executing the task-visible PyTorch source and the candidate JAX repository. Hidden tests, frozen reference outputs, reference JAX solutions, and benchmark scores are inaccessible during development; Appendix D.3 details this information boundary.
We acknowledge that the executable source can be viewed as a behavioral oracle in a broad sense. Our contribution is therefore not the general idea of debugging with execution feedback, but a protocol for evidence-driven, whole-repository cross-framework migration:
- Interface-sensitive verification (§4.2; Appendix D.2). Comparisons preserve the original caller's access patterns and output contracts, rather than merely matching numerical values after candidate-specific unwrapping. Candidate-side diagnostics additionally examine parameter trees, gradient liveness, JAX tracing, and integrated training/serving execution.
- Persistent evidence distinct from code state (§4.1; Appendix D.1). The harness tracks established, violated, and unresolved properties separately from the evolving repository, carrying repair targets and preservation constraints into subsequent iterations.
- Separated repair and verification (§4.1; Appendix D.1). Only the Migrator modifies the repository; the Diagnoser and Verifier operate on read-only candidates. Verification requires execution evidence rather than accepting the implementer's completion claims.
Together, these mechanisms turn source execution into localized repair objectives and explicit regression-preservation constraints across iterations. We position this as a migration-specific harness contribution, rather than claiming that execution-guided self-debugging itself is new.
We would like to thank the reviewer once again for their valuable time and thoughtful feedback. We sincerely hope that our responses and revisions have adequately addressed the concerns raised and further improved the clarity and overall quality of the manuscript. We would greatly appreciate the reviewer's further assessment of these improvements and warmly welcome any additional feedback or discussion.
Response 1
We sincerely thank the reviewer for the constructive comments and insightful suggestions.
Response. Thanks for the comments. We answer them one by one.
Task diversity. We argue that upstream repository diversity and migration task diversity are distinct dimensions. Although our tasks originate from a limited number of upstream codebases, they exhibit substantial diversity in model architectures, training algorithms, modalities, and execution semantics. Specifically, T2J-Bench covers 18 model families with different architectural structures, parameter organizations, and checkpoint conversion requirements, including both text-only and vision-language models. The 45 training tasks span SFT, DPO, and PPO, which require different optimization objectives, gradient computations, and training dynamics. The 18 serving tasks cover Core, Ragged, and Chunked modes, involving different execution semantics such as dynamic batching, paged KV-cache management, and prefix reuse. These differences are also reflected in our empirical results (Table 4), where different task categories exhibit substantial performance variations and changes in agent rankings. For example, Claude-Code performs best on SFT, OpenCode leads on Core serving, while TwinHarness achieves the highest performance on DPO, PPO, and Chunked serving. These observations suggest that the tasks exercise different migration capabilities rather than merely repeating identical migration problems under different configurations.
A framework that scales to new migration projects. Moreover, our main contribution is not a fixed task set. It is a framework for constructing PyTorch-to-JAX codebase migration tasks with rigorous evaluation, and this framework extends to new migration projects. The framework applies to any PyTorch codebase that can be run end to end. Given an upstream repository pinned to a commit, it works in three steps:
Additional codebases. We added one training task from TRL and one serving task from SGLang. For each, we pinned the upstream commit, verified the extracted PyTorch workload against upstream behavior, built a reference JAX port, and applied the shared grading procedure using frozen PyTorch outputs, structural, numerical, and behavioral checks, and reference-calibrated tolerances. Multimodal GRPO uses TRL v1.17.1 with Qwen2.5-VL-3B (4.4k nonblank Python lines). Approximately 20% of its lines match existing tasks, mostly in vendored Hugging Face Qwen-VL modeling code, with little overlap in the trainer, loss, or data pipeline. Sliding-window RadixAttention uses SGLang v0.5.19 with Gemma 4 (12.1k nonblank Python lines), with 6.5% line overlap. The reference JAX ports scored 0.999997 and 0.999656, respectively.
As shown in Table 1, TWINHARNESS achieved the highest observed mean reward among the evaluated configurations on both new tasks. On GRPO, it scored 0.681, compared with 0.632 for the strongest baseline (OpenCode with
opus-5) and 0.355 for its matched Claude-Code baseline. On RadixAttention, it scored 0.812, compared with 0.767 for the strongest baseline (Claude-Code withopus-5). Our analysis of the run logs traced many zero-reward baseline runs to two task-specific failures: GRPO ports failed to reproduce TRL’s per-batch PyTorch shuffle, while RadixAttention ports ran out of memory during batched serving. We added the task construction, validation, results, and failure analysis to Appendix E, “Migration Tasks from Independent Upstream Codebases” (manuscript Table 20), and referenced them in Section 5.2.