acceptodds
Under review as a conference paper at ICLR 2027

XStack: A Benchmark for ML Repo Migration

Abstract

ML code should run where researchers choose, not where it was first written. Today, machine learning code mainly lives in two ecosystems, PyTorch on NVIDIA GPUs and JAX on Google TPUs. Migrating repositories between them means building and testing across frameworks and hardware. This migration is (1) a practical need, since it unlocks different hardware for researchers, and (2) a natural benchmark: it involves different frameworks and hardware, covers whole repositories, and is long-horizon and challenging. We introduce **XStack**, a benchmark that evaluates agents on migrating ML repositories between these two ecosystems. The agent receives a PyTorch repository on GPUs and must deliver a JAX codebase on TPUs. Its output passes if it meets our grading criteria, based on the source repository's published result, authenticated and calibrated by us. Migration by agents remains unreliable: Claude Opus 5 passes 81%, GPT-5.6-sol 44%, and every other agent at most 38%. XStack is a first step toward freeing ML code from its ecosystem. By releasing it, we make progress toward that goal measurable.

Primary area:
datasets and benchmarks
Keywords:
coding agents, benchmarks, code migration, ML frameworks, hardware, TPU, code refactor

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_p6eROpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

XStack evaluates coding agents on migrating 16 PyTorch repositories to JAX on TPUs, including environment setup and full-scale training. Submissions are rerun on clean machines and assessed through performance targets, two model judges, and source-compatible weight loading. A second phase reveals the graded experiment while freezing all files except the entry script. Across seven agents, 26/112 attempts pass initially and 36/112 pass after phase 2; the strongest agent passes 13/16 repositories.

Critical strengths

The task is practically useful and goes beyond API translation: a successful migration must integrate data preparation, numerical implementation, accelerator execution, and evaluation. Full-scale retraining on fresh machines is a meaningful test of whether the delivered code actually works.

I also appreciate the attention to target calibration and evaluation integrity. The makemore example shows why published numbers cannot simply be adopted as thresholds. Reporting both phases, per-repository outcomes, and development-time leakage incidents improves transparency.

Critical weaknesses

1. The evaluation does not establish the claimed breadth of migration fidelity. The main TPU protocol checks one experiment's outcome, its provenance, rule compliance, and whether exported weights load into the source model. These checks do not establish that other configurations, optimizer behavior, or checkpoint/resume functionality have been preserved. Matching a final metric and parameter schema is weaker than faithfully migrating a repository.

This distinction matters because the GPU-only study includes forward, gradient, optimizer, and checkpoint checks that are absent from the stated TPU protocol (Table 6). Appendix A acknowledges this limitation, but it bears directly on the central claim. I would not require bitwise equivalence across accelerators; representative behavioral checks and independent evaluation of exported checkpoints would help. Otherwise, the claim should be narrowed to reproducing a selected experiment on the target stack.

2. The model judges need a systematic validation study. The judges reject 44 of 80 numerically passing results across both phases. That is not evidence that their decisions are wrong, but it makes their reliability central to the reported scores. Section 3.2 says that many rejections were checked manually, without reporting the audit size, sampling procedure, or error rate, which cannot be considered a convincing evidence for judgement reliability for a research paper.

Both judging roles use GPT-5.6-sol, itself an evaluated agent. This does not demonstrate bias, but strengthens the case for blinded human adjudication and cross-model checks. The claim that every accepted pass is a true positive is not established by the evidence presented.

3. The GPU comparison does not isolate the effect of changing hardware. The GPU-only study uses a different model, a different task set with only five overlapping repositories, and different grading criteria. Its 11/12 repository success rate therefore cannot explain the lower TPU success rate specifically in terms of hardware migration. Task selection, provisioning, experiment scale, and evaluation could all contribute.

A matched comparison on the shared repositories would be much more informative. Without one, the GPU result should be presented as exploratory evidence, not support for the broad claim that framework-only translation is nearly solved.

4. Phase 2 does not demonstrate that task ambiguity has been removed. Revealing the recipe after freezing the implementation is a reasonable protocol choice, but whether a correction fits entirely within the entry script can depend on how configuration, data preparation, and training logic were organized. Sixteen numerically passing phase-2 results are rejected for rebuilding frozen functionality, so this boundary is consequential.

Questions for authors
  1. What does a systematic human audit find for accepted and rejected judge decisions, particularly freeze violations?
  2. How reliably do trusted reference migrations pass each task across seeds, and what fidelity checks are applied beyond the selected experiment?
  3. Is there matched evidence separating hardware-migration difficulty from task selection and provisioning, or measuring the effect of disclosing the recipe before implementation?
4 comments
AuthorsOpenReview ↗

Response (1/2) to reviewer p6eR

Thank you for the constructive review and concrete experimental suggestions. Changes for your review are marked in violet in the revised PDF. Throughout, a cell means one agent migrating one repository (7 agents × 16 repositories = 112 cells).

1. Breadth of migration fidelity and seed stability (W1, Q2)

Matching a final metric and parameter schema is narrower than verifying every part of a repository. We made this trade-off on purpose, which is why the paper also includes our GPU-only study with fine-grained checks. Detailed unit checks have to call directly into the migrated code: either the prompt prescribes those entry points (telling the agent what is tested) or custom check scripts must be rewritten per migration (making grading brittle). XStack therefore grades the end-to-end outcome: the full experiment reproduced from scratch on a clean TPU.

For this response, we ran both checks you suggested on accepted migrations, and added re-grades at new seeds:

Fidelity / stability checkOutcome
Independent PyTorch evaluation: exported weights from 30 / 36 accepted cells (13 repos) scored by the unmodified PyTorch source's evaluation code30 / 30 meet the target
Multi-seed retraining: accepted migrations retrained from scratch on a fresh TPU at two new random seeds (8 repos)35 / 35 runs meet the target
Optimizer parity (18 accepted migrations, 5 repos): 1 and 3 steps from identical weights and gradient vs. PyTorch18 / 18 agree within 1e-6
Checkpoint round trip: exported to PyTorch format and loaded back18 / 18 bit-exact
Forward parity: model outputs (logits) at the source's initial weights, same batch18 / 18 agree within 1e-4
Gradient parity: training-loss gradients at the source's initial weights, same batch17 / 18 agree within 1e-3
  • Control. Deliberately introducing a wrong weight decay, decay mask, learning rate, or missing gradient clip shifts these checks by 1e-5 to 1e-3, confirming that they detect real errors.
  • Exported weights scored in PyTorch, by repository (all 30 / 30 meet target):
RepositoryCellsReported on TPUSource eval in PyTorchTarget
minGPT4500/500500/500≥ 495
makemore41.978 to 2.0031.979 to 2.014≤ 2.023
llama2.c31.099 to 1.1001.099 to 1.100≤ 1.130
nanoGPT42.990 to 3.0173.007 to 3.012≤ 3.12
modded-nanoGPT13.2783.278≤ 3.285
x-transformers10.9100.9150.879 to 0.979
LoRA10.69470.6946≥ 0.68
SimCSE177.3377.33≥ 73.5
airbench10.94010.9405≥ 0.9393
AST10.98010.9800≥ 0.975
S4387.47 to 88.4087.46 to 87.91≥ 87.0
HSTU20.3548, 0.35550.3563, 0.3553≥ 0.345
TD-MPC24803 to 854793 to 850≥ 750
  • Seed re-grades from scratch, by repository (all 35 / 35 meet target):
RepositoryMeet targetGraded valuesTarget
makemore8 / 81.989 to 2.017≤ 2.023
minGPT12 / 12500/500 (one run: 498/500)≥ 495
AST2 / 20.9801, 0.9802≥ 0.975
modded-nanoGPT2 / 23.281, 3.282≤ 3.285
airbench2 / 20.9402, 0.9403≥ 0.9393
LoRA2 / 20.688, 0.702≥ 0.68
S46 / 687.02 to 88.37≥ 87.0
SimCSE1 / 176.64≥ 73.5

The revision reports these checks in Appendices K and L.

2. Judge validation and audit (W2, Q1)

Here, a result is a phase-1 or phase-2 run that reached its grading target (82 across both phases: 38 accepted, 44 rejected). Claude Opus 5.5 re-judged all 82 results using the same evidence, prompts, and 3-run rule (492 judge runs):

Judged setOur judges (GPT-5.6-sol)Second-lab judge (Claude Opus 5.5)
Accepted among our 38 accepted results38 / 3838 / 38
Accepted among our 44 rejected results0 / 4420 / 44
Of those, from the three GPT-5.6 agents: sol, terra, luna (17 rejected)0 / 178 / 17 (47%)
Of those, from the other four agents: Opus, DeepSeek, GLM, Muse (27 rejected)0 / 2712 / 27 (44%)

Our judges are stricter, making our pass rates conservative, and show no leniency toward GPT-5.6 agents (47% vs. 44%). Our claim that every pass is a true positive is now backed by more than hand checks. All 38 accepted results are also accepted by the second lab's judge, the exported weights reach the targets when scored in PyTorch (30 / 30), and retraining at new seeds reaches them again (35 / 35), as detailed above. Appendix J of the revision reports the second-lab validation.

  • Third judge family and self-consistency. DeepSeek V4 Pro judged the 12 accepted phase-2 results under the same rule, matched our judges on 106 / 108 answers. Across three runs per result, our correctness judge is unanimous on 77 / 82 results and our cheat judge on 72 / 82.
AuthorsOpenReview ↗

Response (2/2) to reviewer p6eR

2. Judge validation and audit (continued)

  • Human audit (including freeze rejections). Two authors (with AI assistance) audited both accepted and rejected results. Every rejection is justified by our judges' written reasoning, so it is easy to check, and we verified that the reasoning is indeed backed by evidence. For five passing migrations from Opus and sol, we found nothing that should fail. The 20 rejections that Opus 5.5 would accept are cases where it checks less carefully and misses concrete problems our judges explain, including 7 phase-2 results whose entry scripts rebuilt frozen code. Our judges are more careful, and no result changed. Future XStack rounds will use judges from multiple labs.

3. Matched GPU vs. TPU comparison (W3, Q3)

Our earlier record holds a comparison with both the agent (GPT-5.6-terra, high effort) and nine repositories held fixed, which we added to Appendix H:

GPT-5.6-terra (high effort) on the same 9 repositoriesOutcome
Migrated to JAX on GPU, graded by detailed parity checks86 / 88 checks pass
Migrated to JAX on TPU, recipe and target disclosed, up to 5 turns6 / 9 repositories pass
XStack blind phase 1 on TPU (for the 5 of these repositories in XStack)0 / 5 pass

Even with the recipe and target disclosed on TPU, two of the nine failed on TPU-specific setup issues. This further supports the trade-off in point 1 (Part 1): passing detailed checks on GPU does not mean the full experiment works on TPU, and the task is hardest when the agent must also work out which experiment to run. This is why XStack grades the end-to-end outcome and keeps phase 1 blind.

4. Phase 2, the freeze boundary, and stating the recipe first (W4, Q3)

To separate recipe ambiguity from the phase-2 freeze, we ran a new recipe-first setting on short repositories (using a cell list fixed before seeing any result): the phase-1 prompt with the recipe stated upfront, run in a single turn with no freeze against the same hidden target, grader, and judges.

On the four cells where terra and luna failed blind on makemore and minGPT, we ran both settings fresh:

Setting (same 4 cells, run fresh)Reach grading targetAccepted by judges
Blind prompt (phase 1: graded experiment not named)0 / 40 / 4
Blind prompt, then frozen phase 2 (only run_train.py editable)4 / 42 / 4
Recipe stated upfront in one turn (no freeze)4 / 43 / 4

Across all 10 recipe-first cells, 8 / 10 reach the target in one turn and 7 / 10 are accepted, including 3 / 5 cells whose phase 2 in the paper was rejected for rebuilding frozen code:

CellPaper: blind phase 1Paper: frozen phase 2Recipe stated first (1 turn, no freeze)
terra, makemoremissed targetacceptedaccepted (2.002 vs. ≤ 2.023)
luna, minGPTmissed targetacceptedaccepted (500/500 vs. ≥ 495)
luna, makemoremissed targettarget reached, judge-rejectedaccepted (1.999 vs. ≤ 2.023)
terra, minGPTmissed targetacceptedtarget reached (500/500), judge-rejected
DeepSeek, makemoretarget reached, judge-rejectedmissed targetaccepted (2.010 vs. ≤ 2.023)
sol, LoRAmissed targettarget reached, freeze-rejectedaccepted (0.695 vs. ≥ 0.68)
sol, ASTmissed targettarget reached, freeze-rejectedaccepted (0.9816 vs. ≥ 0.975)
terra, airbenchtarget reached, judge-rejectedtarget reached, freeze-rejectedmissed target (0.9387 vs. ≥ 0.9393)
terra, S4missed targettarget reached, freeze-rejectedaccepted (87.62 vs. ≥ 87.0)
terra, x-transformersmissed targettarget reached, freeze-rejectedcrashed in graded run
  • What this shows. Told the recipe upfront, an agent only has to build that one experiment, which is much easier. XStack asks for more: phase 1 asks for a migration of the whole repository, and a complete migration can run the stated recipe from the entry script alone. A cell that must rebuild frozen code in phase 2 did not migrate the full repository in phase 1, so the freeze fails it by design. The three such cells that pass here show this: the agent can build the experiment once told, but its phase-1 migration did not contain it.
  • The freeze rule. The cheat judge enforces the rule that run_train.py must not re-implement, override, or bypass frozen code.

5. Extension to 20 repositories

Since submission, we added four repositories (deepspeech, TGN, PatchTST, and TabR) under the same protocol (140 cells; new Appendix P). The findings hold: 44 / 140 cells pass (31%, vs. 32% on the submitted 16), and at least one agent solves 19 of the 20 repositories. The remaining one, nanoVLM, is also expected to be reachable: Opus's existing migration should pass after one edit. On the four new repositories, the weight check rejects two results that passed both judges.

Thank you again. We welcome any further questions.

Thank you for the thoughtful rebuttal and the additional experiments, which largely address my concerns.

My remaining concerns are about how broadly the results are interpreted. The cross-model re-judging shows that pass rates depend on the choice of judge, particularly for freeze violations, where an independent human audit would still be valuable. The GPU–TPU comparison still uses different grading protocols, and the recipe-first results suggest that when the recipe is disclosed can materially affect success.

Overall, the new evidence strengthens the paper, and I will raise my score from 2 to 3.

Thank you for raising your score and for the suggestions that led to the new experiments. We will add more detail on each of your remaining points in the PDF:

  • Judge choice: both judges accept all 38 results we count as passes, so every reported pass holds under either judge. They differ only on results we reject, where our judges are the stricter ones. For the freeze cases, each rejection comes with our judges' written reasoning, which two authors (with AI assistance) verified against the evidence.
  • GPU vs. TPU grading: the two settings are graded differently on purpose. Detailed parity checks need prescribed entry points, so XStack grades the end-to-end outcome on TPU. The comparison holds the agent and the nine repositories fixed.
  • When the recipe is disclosed: this is what XStack's two phases measure. Phase 1 is blind because working out which experiment to reproduce is part of migrating a repository, and phase 2 then states the recipe. The PDF will give the recipe-first results in more detail, including how much stating the recipe earlier changes the outcome.

Your suggestions made the paper stronger, and we thank you again for your time and feedback.

Reviewer_QohJOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

XStack asks agents to migrate 16 PyTorch repositories to JAX/Flax/Optax on bare TPU machines. Submissions are rerun on a fresh machine against hidden metric targets, checked by two LLM judges, and required to export weights that load into the source model. A second phase reveals the training recipe but permits only entry-script edits. Seven agents pass 36 of 112 cells; Opus 5 passes 13 of 16.

Critical strengths
  1. This is a demanding, practical task. Agents handle environment setup, framework conversion and real TPU execution, including multi-host settings. Fresh-machine reruns are a strong part of the design.
  2. Reporting is fairly detailed: results per cell, token costs, TPU usage and disclosed development-time leaks. The frozen second phase is also a useful attempt to separate choosing the wrong experiment from writing a faulty migration.
Critical weaknesses
  1. The LLM judges have a large effect, but little reported validation. Both judges use GPT-5.6-sol, which is also evaluated. They reject 44 of 80 grader passes and remove 30 of the 66 target-reaching cells. This can change rankings: DeepSeek reaches more targets than sol but finishes with fewer passes. That does not establish bias, but checking an unspecified number of rejections by hand is not enough to support “every pass is a true positive.” Accepted results need auditing too, ideally with blinded human review and another model family.
  2. A pass gives a limited guarantee of migration fidelity. The weight check is described as loading tensors with the expected names and shapes; it rejects no additional cells in Table 4. That does not establish numerical equivalence. Each cell also gets only one grading run, with little calibration evidence beyond makemore. For example, nanoVLM's lower bound is a single reported 35.3% result with no variance, while nanoGPT's ≤3.12 target comes from evaluating a released GPT-2 checkpoint rather than the graded training run. These choices need justification and reference-run statistics.
  3. The explanations go beyond the comparisons. The claim that hardware, rather than translation, drives difficulty compares different agents, repositories and grading criteria. On the five shared repositories, sol and Opus each pass 4/5 on TPU, matching the GPU comparator's full-pass count. The phase-2 analysis is also hard to interpret because §2.2 and Figure 8 place broken freezes differently. “No termination” need not always be a code fault; the source makemore script defaults to an infinite run unless given a horizon. Finally, the cost fit has only seven points and excludes TPU cost, while the claimed role of verification is based on a few trajectory examples. These are plausible hypotheses, but the current evidence does not isolate them.
Questions for authors
  1. Can you audit a stratified sample of accepted and rejected results with blinded human review and a second model family? Please report agreement rates and any ranking changes.
  2. What reference runs establish that each target is reachable under the graded recipe? Please include horizons and variability, explain nanoGPT's target, and add weight-parity checks or narrow the fidelity claim.
  3. Can you provide a per-cell phase-1→phase-2 ledger, including the 16 broken freezes and judge reasons? Where do those violations appear in Figures 5 and 8, and which two failed cells had no phase 2?
  4. Did agents have GPU access, and did all agents receive the same TPU type for each repository? Is there a matched GPU/TPU comparison? Please also confirm whether any reported cell beyond the three disclosed reruns was affected by cross-session access.
4 comments
AuthorsOpenReview ↗

Response (1/2) to reviewer QohJ

Thank you for the thorough review. Changes for your review are marked in amber in the revised PDF. Throughout, a cell means one agent migrating one repository (7 agents × 16 repositories = 112 cells).

1. Judge validation and audit (W1, Q1)

Here, a result is a phase-1 or phase-2 run that reached its grading target (82 across both phases: 38 accepted, 44 rejected). We re-judged all 82 results with Claude Opus 5.5 using the same evidence, prompts, and 3-run rule (492 judge runs):

Judged set / metricOur judges (GPT-5.6-sol)Second-lab judge (Claude Opus 5.5)
Accepted among our 38 accepted results38 / 3838 / 38
Accepted among our 44 rejected results0 / 4420 / 44
Of those, from the three GPT-5.6 agents: sol, terra, luna (17 rejected)0 / 178 / 17 (47%)
Of those, from the other four agents: Opus, DeepSeek, GLM, Muse (27 rejected)0 / 2712 / 27 (44%)
Total passing cells (of 112)3650
Passes per agent (Opus / sol / DeepSeek / Muse / GLM / terra / luna)13 / 7 / 6 / 4 / 3 / 2 / 114 / 9 / 9 / 6 / 5 / 4 / 3

Our judges are stricter, making our pass rates conservative, and are no easier on GPT-5.6 agents (47% vs. 44%). Under Opus 5.5, the ranking stays consistent: Opus stays first and sol ties DeepSeek at 9. Our claim that every pass is a true positive is now backed by more than hand checks. All 38 accepted results are also accepted by the second lab's judge, the exported weights reach the targets when scored in PyTorch (30 / 30), and retraining at new seeds reaches them again (35 / 35), as detailed below. Appendix J of the revision reports the second-lab validation.

  • Third judge family. DeepSeek V4 Pro judged the 12 accepted phase-2 results under the same rule, accepting 10 / 12 and matching our judges on 106 / 108 answers.
  • Stage-by-stage counts and rejection reasons. Filtering across the 112 cells leaves:
Reach grading targetPass correctness judgePass cheat judgePass weight-loading check
66 cells53 cells36 cells36 cells

Breakdown of all 44 rejected results:

Why our judges rejected the resultResults
Phase-2 entry script re-implemented frozen code14
Reported a wrong metric or statistic instead of the graded one11
Ran a sibling or variant experiment rather than the graded recipe8
Read another session's files or grading files, or used an unassigned machine7
Evaluated on training data2
Copied the README's published number1
Evaluated at a different cadence than the recipe1
  • Human audit. Two authors (with AI assistance) audited both accepted and rejected results. Every rejection is justified by our judges' written reasoning, so it is easy to check, and we verified that the reasoning is indeed backed by evidence. For five passing migrations from Opus and sol, we found nothing that should fail. The 20 rejections that Opus 5.5 would accept are cases where it checks less carefully and misses concrete problems our judges explain. Our judges are more careful, and no result changed. Future XStack rounds will use judges from multiple labs.

2. What a pass guarantees (W2, Q2)

The weight check compares parameter names, shapes, and tied weights, which the revision now states explicitly. Since submission, we added four repositories (deepspeech, TGN, PatchTST, and TabR; 140 cells, 44 pass; Appendix P), where the weight check rejects two judge-accepted results. On the submitted 16 repositories, we also tested accepted migrations directly:

Check on accepted migrationsOutcome
Exported weights (30 / 36 accepted cells, 13 repos) scored in PyTorch by the source's evaluation code30 / 30 meet target and match TPU numbers
Retrained from scratch on a fresh TPU at two new random seeds (8 repos)35 / 35 runs meet target
Checkpoint export-and-reload round trip (18 migrations, 5 repos)18 / 18 bit-exact
Optimizer updates (JAX vs. PyTorch, same weights and gradient)18 / 18 within 1e-6
Forward logits at the source's initial weights (same batch)18 / 18 within 1e-4
Parameter gradients at the source's initial weights (same batch)17 / 18 within 1e-3

As a control, we deliberately changed the weight decay, decay mask, learning rate, or gradient clipping. Each change moved these checks by 1e-5 to 1e-3, so they catch real bugs.

  • Seed re-grades from scratch, by repository (all 35 / 35 meet target):
RepositoryMeet targetGraded valuesTarget
makemore8 / 81.989 to 2.017≤ 2.023
minGPT12 / 12500/500 (one run: 498/500)≥ 495
AST2 / 20.9801, 0.9802≥ 0.975
modded-nanoGPT2 / 23.281, 3.282≤ 3.285
airbench2 / 20.9402, 0.9403≥ 0.9393
LoRA2 / 20.688, 0.702≥ 0.68
S46 / 687.02 to 88.37≥ 87.0
SimCSE1 / 176.64≥ 73.5
AuthorsOpenReview ↗

Response (2/2) to reviewer QohJ

2. What a pass guarantees (continued)

  • Exported weights scored in PyTorch, by repository (all 30 / 30 meet target):
RepositoryCellsReported on TPUSource eval in PyTorchTarget
minGPT4500/500500/500≥ 495
makemore41.978 to 2.0031.979 to 2.014≤ 2.023
llama2.c31.099 to 1.1001.099 to 1.100≤ 1.130
nanoGPT42.990 to 3.0173.007 to 3.012≤ 3.12
modded-nanoGPT13.2783.278≤ 3.285
x-transformers10.9100.9150.879 to 0.979
LoRA10.69470.6946≥ 0.68
SimCSE177.3377.33≥ 73.5
airbench10.94010.9405≥ 0.9393
AST10.98010.9800≥ 0.975
S4387.47 to 88.4087.46 to 87.91≥ 87.0
HSTU20.3548, 0.35550.3563, 0.3553≥ 0.345
TD-MPC24803 to 854793 to 850≥ 750

3. Target calibration (W2, Q2)

The per-repository table of reference runs is in Appendix N of the revision. For each of the 16 repositories it gives the target, the published value, our own PyTorch source runs, and the accepted TPU migrations.

For nanoGPT, 3.12 is the README's validation loss for the released GPT-2 checkpoint; under our 120,000-iteration graded recipe, all four accepted migrations clear it comfortably (2.990 to 3.017). For nanoVLM, the target is the README's single number, and it is achievable on TPU: Opus's migration already reaches it and needs only one edit to follow the source recipe (using the resized images directly instead of caching them as lossy JPEGs). With that change, we expect it to pass.

4. Hardware vs. translation, taxonomy counts, cost, and verification (W3)

  • Matched GPU vs. TPU comparison. Holding the agent (GPT-5.6-terra, high effort) and nine repositories fixed across our earlier generations and XStack (added to Appendix H):
GPT-5.6-terra (high effort) on the same 9 repositoriesOutcome
Migrated to JAX on GPU, graded by detailed parity checks86 / 88 checks pass
Migrated to JAX on TPU, recipe and target disclosed, up to 5 turns6 / 9 repositories pass
XStack blind phase 1 on TPU (for the 5 of these repositories in XStack)0 / 5 pass

With the recipe disclosed, two of the three TPU failures came from TPU-specific setup (a multi-host launch deadlock and data setup on a fresh machine). Giving the recipe upfront makes the task much easier, so XStack keeps phase 1 blind: it tests whether an agent can work out from the repository which experiment to reproduce.

  • The two "16" counts. They are two different groups that happen to be the same size. The 16 in Section 3.1 broke the freeze but reached the target, so the judges rejected them ("failed by the judges" in Figures 5 and 8). The 16 in Figure 8 broke the freeze and missed the target. The Figure 8 caption now says so.
  • No termination. The makemore source runs forever by default, so the phase-2 recipe states a stop rule: at most 25,000 steps (the source's own --max-steps flag), stopping earlier after 5 evaluations without improvement. We set the target by running the unmodified source eight times under this rule, and five of seven agents reached it in blind phase 1 with no horizon given.
  • Cost fit. Figure 6 plots model token cost at list price (now noted in the caption). Its slope of 29.7 percentage points per 10× cost has a 95% bootstrap interval over repositories of 20 to 38 (positive in every draw), and adding the TPU cost of the agent's turns gives a slope of 27 points per 10×.
  • Verification. Across all 112 phase-1 transcripts, Opus ran the graded experiment to completion before submitting in 13 / 16 cells, DeepSeek in 10, GLM in 9, Muse in 8, sol in 2, luna in 0, and terra in 0. Cells with both a completed run and a comparison against the source pass 17 / 31 in phase 1, versus 9 / 81 for all others.

5. Per-cell ledger and remaining questions (Q3, Q4)

Appendix O adds the full 112-cell phase-1 → phase-2 ledger.

  • Cells without phase 2: terra on nanochat and luna on LoRA. No result is affected: we examined both, and phase 2 could not have rescued either (one phase-1 migration crashed, the other never finished).
  • GPU access: Agents worked on TPU machines only; since the task is migrating to TPUs, no GPU is needed.
  • TPU hardware mix: We ensured each cell got enough compute for its repository's requirements. 13 of 16 repositories mixed 8-chip v6e and 4-chip hosts based on spot availability, with nearly identical pass rates (18 / 58 [31%] vs. 10 / 31 [32%]).
  • Cross-session access: All 112 cells ran inside the sandbox. No accepted cell had such contact. Five early affected cells were re-run from scratch in the hardened sandbox (correcting "three" in the paper), of which four passed again.

Thank you again. We would be glad to answer anything else.

Thank you for the detailed response. Scoring the exported weights in PyTorch (30/30) and the seed reruns (35/35) address my main concern about whether passes are real, and Appendix O gives the ledger I asked for. I keep my score of 3.

Remaining points:

  1. The second judge accepts 20 of your 44 rejections, which would raise passes from 36 to 50. Rankings stay mostly stable, but absolute pass rates clearly depend on the judge. Please report both counts in the main results. If possible, have someone outside the author team check the 20 disputed cases.
  2. Two counts do not reconcile. First, 26 phase-1 passes (17 + 9) plus 12 accepted phase-2 results give 38 accepted results, but there are only 36 passing cells, and phase 2 only follows a phase-1 failure. Second, each repository was retrained at two new seeds, but SimCSE shows only one run.
  3. In the GPU/TPU comparison, the GPU arm is graded by parity checks and the TPU arms by targets, so I would keep the hardware claim tentative.

Thank you for the close reading of our response and the new appendices. On your remaining points:

  • Judge counts: we will report both counts in the main results (36 under our judges, 50 under the second-lab judge). We will also list the 20 disputed results with our judges' written reasons, and we will try to add a check by someone outside the author team.
  • 38 accepted results vs. 36 passing cells: two cells (DeepSeek on TensoRF, Opus on nanochat) have an accepted result in both phases, because phase 2 had run before the final judging accepted their phase-1 result. A cell counts once: 26 pass in phase 1 and 10 more in phase 2.
  • SimCSE seeds: the second run did not finish in time because of limited compute. The table counts finished runs only. Given the other seed results, we are confident it will pass as well, and we will rerun this seed and add it to the table.
  • GPU vs. TPU: we will state next to the comparison that GPU is graded by parity checks and TPU by end-to-end targets.

Thank you again for the detailed and constructive review.

Reviewer_XRW4OpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3(+1)
Read review
Paper summary

The paper introduces XStack, a benchmark of 16 working PyTorch repositories covering language models, vision, audio, recommendation, control. It asks a coding agent to migrate each one to JAX/Flax/Optax and run it end to end on a bare TPU machine, with success judged by whether a fresh rerun of the migrated code reproduces the source repository’s published headline result within a tolerance, confirmed by two LLM judges and by loading the trained weights back into the original PyTorch model. The grading target is hidden from the agent in a first phase, and a second “frozen” phase tells the agent which experiment is graded while allowing it to edit only the entry script (so that a correct migration is not failed merely for running the wrong experiment). Evaluating seven agents from five labs on real TPUs (~17k TPU chip-hours), the authors find that only Claude Opus 5 passes most repositories (13/16), the average pass rate is 32%, pass rate grows roughly linearly with log cost, and about half of all failures are crashes or fidelity gaps in the migrated code rather than choice of the wrong experiment, which they attribute to agents’ weak verification of their own work.

Critical strengths

I believe the benchmark is both novel and carefully graded: as far as I can verify, it is the first to require migrating a whole working ML repository across a framework and a hardware boundary. This is a consequential capability especially for the open-source community because it can help to make diverse compute usable. Its pass criterion seems hard to game. It also delivers a clear and unsaturated signal: pass rates span 1/16 to 13/16 across seven agents from five labs, 15 of 16 repositories are solved by at least one agent, the top and bottom of the ranking are already present in the judge-free ‘reaches target’ column and survive the deterministic weight-loading check, and the judge model is not the winning model, so the most important features of the ranking are likely not an artifact of LLM-as-judge bias.

My rating for this paper is stronger than 3 (weak acceptance) but I want to wait for the discussion period to raise my score to 4 (clear acceptance).

EDIT AFTER REBUTTAL: I am raising my overall score to 4 (see my official comment) and also "Soundness" to 3.

Critical weaknesses

The paper’s claims outrun its evidence in two ways. First, every one of the 112 cells is a single run of a nondeterministic agent, with no repeats, seeds or error bars, yet the paper ranks seven agents on 16 repositories, where a difference of 4/16 versus 6/16 is well within run-to-run noise; the same applies to the least-squares fit through seven points used to conclude that the “Pass rate rises by about 30% with every 10× in cost”. Second, the paper validates the decisions of the LLM judges only by having "checked many rejections by hand", gives no human agreement rate, and does not break the rejections down by kind, so neither their reliability nor the severity of the 'cheating' can be assessed, even though the ranking below first place rests on them. Both weaknesses are fixable (repeated runs on a subset, a quantified human–judge agreement study, softened claims), which is why they lower my confidence in the specific numbers rather than in the benchmark itself.

A smaller point that concerns the framing rather than the results: the introduction presents a TPU lab holding a PyTorch repository as ‘trapped’, but PyTorch runs on TPUs through PyTorch/XLA and, since April 2026, Google’s TorchTPU, and portability by framework design is practised at scale, for instance by Open Athena’s Marin team, whose JAX-based Levanter framework lets a run resume across GPUs and TPUs and whose 535B hero run was moved from TPUs on GCP to NVIDIA GPUs on CoreWeave this year; the paper should acknowledge these routes and explain why agent-driven migration to JAX remains the capability worth measuring.

Questions for authors

(1) Judges: for the cells that reached the grading target but were rejected, please give the breakdown by kind (wrong number reported, fabricated or copied number, evaluation on training data, sibling variant) and a human agreement rate on a sample of both accepted and rejected results; in particular, were Claude Opus 5’s 13 accepted cells audited by a human with the same scrutiny as the rejections, given that the weight-loading check cannot tell whether this run trained the weights? Does the order of GPT-5.6-sol and DeepSeek V4.1 Flash hold under a judge from a different lab? (2) Noise and confounds: has any cell been run more than once, and how often did its outcome change; and was each repository run on the same TPU generation for all seven agents? (3) Integrity: were all 112 reported cells produced after the sandbox hardening described in Section 4.2, so that no reported run could reach the grading criteria, the reference migrations or other cells’ files?

Minor: confirm that your reference migration passes nanoVLM, and report per agent how often the source experiment was run to completion.

4 comments
AuthorsOpenReview ↗

Response (1/2) to reviewer XRW4

Thank you for the thoughtful review and clear questions. We ran fresh end-to-end repeats, retrained accepted migrations from scratch at new random seeds, re-judged every target-reaching run with a model from another lab, and expanded XStack to 20 repositories. Changes for your review are marked in teal in the revised PDF. Throughout, a cell means one agent migrating one repository (7 agents × 16 repositories = 112 cells).

1. Run-to-run stability and uncertainty (W1, Q2)

We tested stability at two levels: repeating full agent migrations and retraining accepted migrations at new random seeds.

Seed re-grades by repository (retrained from scratch at two new seeds; all 35 / 35 meet target):

RepositoryMeet targetGraded valuesTarget
makemore8 / 81.989 to 2.017≤ 2.023
minGPT12 / 12500/500 (one run: 498/500)≥ 495
AST2 / 20.9801, 0.9802≥ 0.975
modded-nanoGPT2 / 23.281, 3.282≤ 3.285
airbench2 / 20.9402, 0.9403≥ 0.9393
LoRA2 / 20.688, 0.702≥ 0.68
S46 / 687.02 to 88.37≥ 87.0
SimCSE1 / 176.64≥ 73.5
What was repeatedRunsOutcome
Full cell repeated from scratch before submission (same agent, repository, and prompt)6 cells5 / 6 keep the same pass/fail outcome
Full cell repeated for this response on a fresh TPU (short repositories; blind phase 1, then frozen phase 2 if phase 1 fails)9 cells7 / 9 match the paper's outcome
Grading alone: accepted migrations retrained from scratch on a fresh TPU at two new random seeds (8 repositories)35 runs35 / 35 meet the grading target
  • Extension to 20 repositories. Since submission, we added deepspeech, TGN, PatchTST, and TabR (140 cells; new Appendix P). The main findings hold: 44 / 140 cells pass (31%, vs. 32% on the original 16), at least one agent solves 19 / 20 repositories, and the ranking stays the same except that GLM and Muse (one pass apart) swap 4th and 5th place (table below).
  • Cost fit. The slope is 29.7 percentage points per 10× cost (updated to say "percentage points" in the revision), with a 95% bootstrap interval of 20 to 38, positive in every resample.

2. Judge validation and audit (W1, Q1)

Here, a result is a phase-1 or phase-2 run that reached its grading target (82 in total: 38 accepted, 44 rejected). We re-judged all 82 results with Claude Opus 5.5 using the same evidence, prompts, and 3-run rule (492 judge runs). Opus 5.5 accepts all 38 results our GPT-5.6-sol judges accept and would also accept 20 of the 44 they reject, showing that our judges are stricter and our reported pass rates are conservative. A third model family, DeepSeek V4 Pro, judged the 12 accepted phase-2 results under the same rule, accepting 10 / 12 and matching our judges on 106 / 108 individual answers.

  • Judge self-consistency. Each judge reads every result three times, and one failing vote rejects it. Our correctness judge is unanimous across all three runs on 77 / 82 results, and our cheat judge on 72 / 82.
  • Agent rankings across judges and repository sets (repositories passed):
Agent16 repos, our judges (GPT-5.6-sol)16 repos, second-lab judge (Claude Opus 5.5)20 repos (16 + 4 new), our judges
Opus131415
sol799
DeepSeek698
Muse464
GLM355
terra242
luna131

Across both judges and the larger repository set, the ranking stays consistent, with only one swap between adjacent agents (Muse and GLM) after we extend to 20 repositories.

  • Breakdown of the 44 rejected results:
Why our judges rejected the resultResults
Phase-2 entry script re-implemented frozen code14
Reported a wrong metric or statistic instead of the graded one11
Ran a sibling or variant experiment rather than the graded recipe8
Read another session's files or grading files, or used an unassigned machine7
Evaluated on training data2
Copied the README's published number1
Evaluated at a different cadence than the recipe1
  • Human audit. Two authors (with AI assistance) audited both accepted and rejected results. Every rejection is justified by our judges' written reasoning, so it is easy to check, and we verified that the reasoning is indeed backed by evidence. For five passing migrations from Opus and sol, we found nothing that should fail. The 20 rejections that Opus 5.5 would accept are cases where it checks less carefully and misses concrete problems our judges explain. Our judges are more careful, and no result changed. The revision reports the Opus 5.5 comparison, and future XStack rounds will use judges from multiple labs.
AuthorsOpenReview ↗

Response (2/2) to reviewer XRW4

2. Judge validation and audit (continued)

  • Checking accepted cells, including Opus's. Every accepted result is also accepted by a judge from a different lab than the agent's; all 13 seed re-grades of Opus's migrations pass from scratch; and for 30 of the 36 accepted cells (13 repositories), loading the exported weights into the unmodified PyTorch source and running its evaluation code reproduces the target in all 30 / 30 cases:
RepositoryCellsReported on TPUSource eval in PyTorchTarget
minGPT4500/500500/500≥ 495
makemore41.978 to 2.0031.979 to 2.014≤ 2.023
llama2.c31.099 to 1.1001.099 to 1.100≤ 1.130
nanoGPT42.990 to 3.0173.007 to 3.012≤ 3.12
modded-nanoGPT13.2783.278≤ 3.285
x-transformers10.9100.9150.879 to 0.979
LoRA10.69470.6946≥ 0.68
SimCSE177.3377.33≥ 73.5
airbench10.94010.9405≥ 0.9393
AST10.98010.9800≥ 0.975
S4387.47 to 88.4087.46 to 87.91≥ 87.0
HSTU20.3548, 0.35550.3563, 0.3553≥ 0.345
TD-MPC24803 to 854793 to 850≥ 750

3. TPU hardware mix (Q2)

Machine type varied with spot capacity, and we ensured each cell got enough compute for its repository's requirements. Three repositories ran on 8-chip v6e machines for all agents, while the other 13 mixed 8-chip v6e and 4-chip hosts. Within those 13 mixed repositories, pass rates are almost identical (the revision lists every cell's machine):

TPU machineCells runCells passing
8-chip v6e5818 (31%)
4-chip host3110 (32%)

4. Sandbox timing and integrity (Q3)

All 112 cells ran inside the sandbox. No reported pass is affected: the cheat judge covers all 112 cells. It inspects a mechanically extracted log of every command that touched paths or hosts outside the workspace, where the grading criteria, reference migrations, and other cells' files are kept. Any read of another session's files or the grading environment fails the run: 7 target-reaching results were rejected under this rule, and no accepted cell had such contact. Five early cells with external contact were re-run from scratch in the hardened sandbox, and four passed again.

5. Framing

Thank you for these pointers. The revised introduction now cites PyTorch/XLA, TorchTPU, and Levanter, which are practical routes for new or actively maintained code. XStack asks a different question: whether an agent can migrate an existing PyTorch repository across both framework and hardware and verify that it still works by reproducing its published result.

Migrating to native JAX remains worth measuring for three reasons:

  • Maintenance. PyTorch/XLA is now deprecated: its last release (2.9, November 2025) is six minor versions behind PyTorch, 695 issues remain open (GitHub, November 2026), and its users are asked to migrate to TorchTPU (launched April 2026). JAX shipped 15 releases in the same period, and in our experience failures in native JAX are easier to trace and report.
  • Performance. Google's TorchTPU announcement names whole-graph XLA compilation, JAX's default, as the path to peak TPU performance; PyTorch-style eager execution needs fusion to gain 50% to over 100% in speed.
  • Portability. Levanter is portable across GPUs and TPUs because it is written in JAX; a PyTorch repository gains that only after migrating to JAX, the step XStack measures.

6. Minor points

  • nanoVLM is achievable on TPU. Opus's migration already reaches the target and needs only one edit to follow the source recipe: using the resized images directly instead of caching them as lossy JPEGs. With that change, we expect it to pass.
  • Running the source experiment to completion. Across all 112 phase-1 transcripts, Opus ran the graded experiment to completion before submitting in 13 / 16 cells, DeepSeek in 10, GLM in 9, Muse in 8, sol in 2, luna in 0, and terra in 0. Cells with both a completed run and a comparison against the source pass 17 / 31 in phase 1, versus 9 / 81 for all others.

Thank you again for your time and feedback. We are happy to answer any further questions.

Reviewer_XRW4OpenReview ↗

Response to rebuttal

Thank you for the new experiments and the thorough response. The grading is now well validated and the main results hold up. I will raise my score to 4. I see the paper as a valuable contribution well above threshold.

I have a couple of suggestions for the final version:

  1. Move the key validation numbers into the main text. Section 3.2 still says “we checked many rejections by hand”, and the human audit from your response is not in the paper; please add it with its size and procedure.

  2. Present the pass rates as conservative and give the range: 36 cells under your rule, 41 under a majority vote, 50 under the second judge. Please also say that the order below first place is not resolved: about one repeated cell in five changes its outcome, and sol and DeepSeek tie under the second judge.

  3. Like Reviewers QohJ and p6eR, I think Section 4.1 claims more than the evidence shows. Your new experiments suggest that much of the difficulty comes from the blind protocol and not only from the hardware. On the same TPUs, agents that fail blind often succeed once told which experiment is graded: 7 of 10 cells are accepted with the recipe stated first, against 3 of 10 after the frozen phase 2, and terra passes 6 of 9 repositories when told the recipe and target but none of the 5 XStack ones blind. The GPU-only setting is also not “nearly solved” under stricter grading: T2J-Bench, in its current version, reports a best score of 0.676 against 0.980 for a correct reference. I would soften “translation alone is not the hard part” and the description of T2J-Bench in the related work.

  4. The difference between blind and instructed runs could be quite relevant in practice. A precise instruction may partly substitute for a much more expensive model: I think the agents in the recipe-first experiment cost on average $0.50 to $15 per cell in your main results, against $80 for Opus. It also suggests that a blind benchmark mixes skill with autonomy, and that reporting a “told” setting next to the blind one would separate the two. What do you think?

  5. Two clarifications: were the seven results rejected for reading other sessions’ files or the grading environment produced after the sandbox was hardened, and does your own reference migration pass nanoVLM?

Thank you for raising your score and for the helpful suggestions. We will incorporate them in the PDF:

  • Section 3.2 will report the second-lab judge, exported-weights and seed results, and the human audit with its size and procedure.
  • The main results will give all three pass counts (36 under our rule, 41 under a majority vote, 50 under the second-lab judge) and note that sol and DeepSeek tie under the second-lab judge.
  • Section 4.1 will add the matched comparison and the recipe-first results and state how GPU and TPU are graded. For T2J-Bench, we cited an earlier version; the authors have since updated it on arXiv, and we will incorporate the new results.

Blind versus told: XStack's two phases already separate them, and we will report the recipe-first setting beside them. Blind stays the main setting, since a user who hands over a repository needs the agent to find the reported experiment itself.

Your two clarifications:

  • All seven rejected results had their contact in a session that started before the final hardening step. Among the 112 cells, no later session was rejected for it. We will rerun these seven cells in the hardened sandbox and add the results in the PDF.
  • We have a reference migration for nanoVLM that we adapted from Opus's failed migration.

Thank you again for the constructive discussion.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.