- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
XStack evaluates coding agents on migrating 16 PyTorch repositories to JAX on TPUs, including environment setup and full-scale training. Submissions are rerun on clean machines and assessed through performance targets, two model judges, and source-compatible weight loading. A second phase reveals the graded experiment while freezing all files except the entry script. Across seven agents, 26/112 attempts pass initially and 36/112 pass after phase 2; the strongest agent passes 13/16 repositories.
Critical strengths
The task is practically useful and goes beyond API translation: a successful migration must integrate data preparation, numerical implementation, accelerator execution, and evaluation. Full-scale retraining on fresh machines is a meaningful test of whether the delivered code actually works.
I also appreciate the attention to target calibration and evaluation integrity. The makemore example shows why published numbers cannot simply be adopted as thresholds. Reporting both phases, per-repository outcomes, and development-time leakage incidents improves transparency.
Critical weaknesses
1. The evaluation does not establish the claimed breadth of migration fidelity. The main TPU protocol checks one experiment's outcome, its provenance, rule compliance, and whether exported weights load into the source model. These checks do not establish that other configurations, optimizer behavior, or checkpoint/resume functionality have been preserved. Matching a final metric and parameter schema is weaker than faithfully migrating a repository.
This distinction matters because the GPU-only study includes forward, gradient, optimizer, and checkpoint checks that are absent from the stated TPU protocol (Table 6). Appendix A acknowledges this limitation, but it bears directly on the central claim. I would not require bitwise equivalence across accelerators; representative behavioral checks and independent evaluation of exported checkpoints would help. Otherwise, the claim should be narrowed to reproducing a selected experiment on the target stack.
2. The model judges need a systematic validation study. The judges reject 44 of 80 numerically passing results across both phases. That is not evidence that their decisions are wrong, but it makes their reliability central to the reported scores. Section 3.2 says that many rejections were checked manually, without reporting the audit size, sampling procedure, or error rate, which cannot be considered a convincing evidence for judgement reliability for a research paper.
Both judging roles use GPT-5.6-sol, itself an evaluated agent. This does not demonstrate bias, but strengthens the case for blinded human adjudication and cross-model checks. The claim that every accepted pass is a true positive is not established by the evidence presented.
3. The GPU comparison does not isolate the effect of changing hardware. The GPU-only study uses a different model, a different task set with only five overlapping repositories, and different grading criteria. Its 11/12 repository success rate therefore cannot explain the lower TPU success rate specifically in terms of hardware migration. Task selection, provisioning, experiment scale, and evaluation could all contribute.
A matched comparison on the shared repositories would be much more informative. Without one, the GPU result should be presented as exploratory evidence, not support for the broad claim that framework-only translation is nearly solved.
4. Phase 2 does not demonstrate that task ambiguity has been removed. Revealing the recipe after freezing the implementation is a reasonable protocol choice, but whether a correction fits entirely within the entry script can depend on how configuration, data preparation, and training logic were organized. Sixteen numerically passing phase-2 results are rejected for rebuilding frozen functionality, so this boundary is consequential.
Questions for authors
- What does a systematic human audit find for accepted and rejected judge decisions, particularly freeze violations?
- How reliably do trusted reference migrations pass each task across seeds, and what fidelity checks are applied beyond the selected experiment?
- Is there matched evidence separating hardware-migration difficulty from task selection and provisioning, or measuring the effect of disclosing the recipe before implementation?
4 comments
Response (2/2) to reviewer p6eR
2. Judge validation and audit (continued)
- Human audit (including freeze rejections). Two authors (with AI assistance) audited both accepted and rejected results. Every rejection is justified by our judges' written reasoning, so it is easy to check, and we verified that the reasoning is indeed backed by evidence. For five passing migrations from Opus and sol, we found nothing that should fail. The 20 rejections that Opus 5.5 would accept are cases where it checks less carefully and misses concrete problems our judges explain, including 7 phase-2 results whose entry scripts rebuilt frozen code. Our judges are more careful, and no result changed. Future XStack rounds will use judges from multiple labs.
3. Matched GPU vs. TPU comparison (W3, Q3)
Our earlier record holds a comparison with both the agent (GPT-5.6-terra, high effort) and nine repositories held fixed, which we added to Appendix H:
| GPT-5.6-terra (high effort) on the same 9 repositories | Outcome |
|---|---|
| Migrated to JAX on GPU, graded by detailed parity checks | 86 / 88 checks pass |
| Migrated to JAX on TPU, recipe and target disclosed, up to 5 turns | 6 / 9 repositories pass |
| XStack blind phase 1 on TPU (for the 5 of these repositories in XStack) | 0 / 5 pass |
Even with the recipe and target disclosed on TPU, two of the nine failed on TPU-specific setup issues. This further supports the trade-off in point 1 (Part 1): passing detailed checks on GPU does not mean the full experiment works on TPU, and the task is hardest when the agent must also work out which experiment to run. This is why XStack grades the end-to-end outcome and keeps phase 1 blind.
4. Phase 2, the freeze boundary, and stating the recipe first (W4, Q3)
To separate recipe ambiguity from the phase-2 freeze, we ran a new recipe-first setting on short repositories (using a cell list fixed before seeing any result): the phase-1 prompt with the recipe stated upfront, run in a single turn with no freeze against the same hidden target, grader, and judges.
On the four cells where terra and luna failed blind on makemore and minGPT, we ran both settings fresh:
| Setting (same 4 cells, run fresh) | Reach grading target | Accepted by judges |
|---|---|---|
| Blind prompt (phase 1: graded experiment not named) | 0 / 4 | 0 / 4 |
Blind prompt, then frozen phase 2 (only run_train.py editable) | 4 / 4 | 2 / 4 |
| Recipe stated upfront in one turn (no freeze) | 4 / 4 | 3 / 4 |
Across all 10 recipe-first cells, 8 / 10 reach the target in one turn and 7 / 10 are accepted, including 3 / 5 cells whose phase 2 in the paper was rejected for rebuilding frozen code:
| Cell | Paper: blind phase 1 | Paper: frozen phase 2 | Recipe stated first (1 turn, no freeze) |
|---|---|---|---|
| terra, makemore | missed target | accepted | accepted (2.002 vs. ≤ 2.023) |
| luna, minGPT | missed target | accepted | accepted (500/500 vs. ≥ 495) |
| luna, makemore | missed target | target reached, judge-rejected | accepted (1.999 vs. ≤ 2.023) |
| terra, minGPT | missed target | accepted | target reached (500/500), judge-rejected |
| DeepSeek, makemore | target reached, judge-rejected | missed target | accepted (2.010 vs. ≤ 2.023) |
| sol, LoRA | missed target | target reached, freeze-rejected | accepted (0.695 vs. ≥ 0.68) |
| sol, AST | missed target | target reached, freeze-rejected | accepted (0.9816 vs. ≥ 0.975) |
| terra, airbench | target reached, judge-rejected | target reached, freeze-rejected | missed target (0.9387 vs. ≥ 0.9393) |
| terra, S4 | missed target | target reached, freeze-rejected | accepted (87.62 vs. ≥ 87.0) |
| terra, x-transformers | missed target | target reached, freeze-rejected | crashed in graded run |
- What this shows. Told the recipe upfront, an agent only has to build that one experiment, which is much easier. XStack asks for more: phase 1 asks for a migration of the whole repository, and a complete migration can run the stated recipe from the entry script alone. A cell that must rebuild frozen code in phase 2 did not migrate the full repository in phase 1, so the freeze fails it by design. The three such cells that pass here show this: the agent can build the experiment once told, but its phase-1 migration did not contain it.
- The freeze rule. The cheat judge enforces the rule that
run_train.pymust not re-implement, override, or bypass frozen code.
5. Extension to 20 repositories
Since submission, we added four repositories (deepspeech, TGN, PatchTST, and TabR) under the same protocol (140 cells; new Appendix P). The findings hold: 44 / 140 cells pass (31%, vs. 32% on the submitted 16), and at least one agent solves 19 of the 20 repositories. The remaining one, nanoVLM, is also expected to be reachable: Opus's existing migration should pass after one edit. On the four new repositories, the weight check rejects two results that passed both judges.
Thank you again. We welcome any further questions.
Thank you for the thoughtful rebuttal and the additional experiments, which largely address my concerns.
My remaining concerns are about how broadly the results are interpreted. The cross-model re-judging shows that pass rates depend on the choice of judge, particularly for freeze violations, where an independent human audit would still be valuable. The GPU–TPU comparison still uses different grading protocols, and the recipe-first results suggest that when the recipe is disclosed can materially affect success.
Overall, the new evidence strengthens the paper, and I will raise my score from 2 to 3.
Thank you for raising your score and for the suggestions that led to the new experiments. We will add more detail on each of your remaining points in the PDF:
- Judge choice: both judges accept all 38 results we count as passes, so every reported pass holds under either judge. They differ only on results we reject, where our judges are the stricter ones. For the freeze cases, each rejection comes with our judges' written reasoning, which two authors (with AI assistance) verified against the evidence.
- GPU vs. TPU grading: the two settings are graded differently on purpose. Detailed parity checks need prescribed entry points, so XStack grades the end-to-end outcome on TPU. The comparison holds the agent and the nine repositories fixed.
- When the recipe is disclosed: this is what XStack's two phases measure. Phase 1 is blind because working out which experiment to reproduce is part of migrating a repository, and phase 2 then states the recipe. The PDF will give the recipe-first results in more detail, including how much stating the recipe earlier changes the outcome.
Your suggestions made the paper stronger, and we thank you again for your time and feedback.
Response (1/2) to reviewer p6eR
Thank you for the constructive review and concrete experimental suggestions. Changes for your review are marked in violet in the revised PDF. Throughout, a cell means one agent migrating one repository (7 agents × 16 repositories = 112 cells).
1. Breadth of migration fidelity and seed stability (W1, Q2)
Matching a final metric and parameter schema is narrower than verifying every part of a repository. We made this trade-off on purpose, which is why the paper also includes our GPU-only study with fine-grained checks. Detailed unit checks have to call directly into the migrated code: either the prompt prescribes those entry points (telling the agent what is tested) or custom check scripts must be rewritten per migration (making grading brittle). XStack therefore grades the end-to-end outcome: the full experiment reproduced from scratch on a clean TPU.
For this response, we ran both checks you suggested on accepted migrations, and added re-grades at new seeds:
The revision reports these checks in Appendices K and L.
2. Judge validation and audit (W2, Q1)
Here, a result is a phase-1 or phase-2 run that reached its grading target (82 across both phases: 38 accepted, 44 rejected). Claude Opus 5.5 re-judged all 82 results using the same evidence, prompts, and 3-run rule (492 judge runs):
Our judges are stricter, making our pass rates conservative, and show no leniency toward GPT-5.6 agents (47% vs. 44%). Our claim that every pass is a true positive is now backed by more than hand checks. All 38 accepted results are also accepted by the second lab's judge, the exported weights reach the targets when scored in PyTorch (30 / 30), and retraining at new seeds reaches them again (35 / 35), as detailed above. Appendix J of the revision reports the second-lab validation.