- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
LaDiM is an agent framework for migrating deep-learning code across frameworks and languages. It combines translation, staged diagnosis, and repair, using comparisons of forward values, gradients, and parameter updates to guide debugging. Repository-level extensions manage dependencies and reconstruct relevant context. The paper reports improved acceptance or reduced token usage on several migration collections.
Critical strengths
-
The diagnostic signals address a real failure mode. The Tanh example in Figure 4 illustrates that nearly identical forward values and losses can hide a broken differentiation path. Comparing per-parameter gradients provides actionable information that execution-only testing would miss.
-
Some efficiency results are substantial. On the main MindSpore collection, LaDiM and MatchFixAgent both achieve complete acceptance, while LaDiM uses 57.4% fewer tokens. The time-series repository also shows considerable token savings at equal acceptance.
Critical weaknesses
- The evidence for the proposed orchestration is inconsistent. The method mainly combines LLM role separation, persistent repair history, numerical testing, and context management. These are reasonable engineering choices, but the ablations do not consistently justify the complete design. In Table 7, removing repair history preserves 50/50 acceptance while reducing tokens from 5.159M to 4.566M. Although history helps the separate ten-program repair study, this benefit does not extend to the main collection.
- The evaluation is small and heavily constructed. The advertised 50 MindSpore tasks correspond to only 29 distinct source-program/evaluation combinations, and 20 of these already pass before repair. Thus, the main repair comparison involves only nine distinct failing inputs. Repository evaluation covers just two repositories, with 865 and 2,086 nonblank lines. The training-signal ablation uses four deliberately constructed fault types that align directly with the four feedback categories. These experiments demonstrate feasibility, but provide limited evidence of reliable gains on a broader migration workload.
- The gains do not consistently transfer to other settings. In the six-case JAX repair study, both LaDiM and Direct repair achieve 6/6 acceptance, but LaDiM uses 0.674M tokens versus 0.353M. In the native JAX study in Appendix H, all methods reach 9/12 overall acceptance, while LaDiM uses 1.980M tokens versus 0.901M for Direct repair. The headline 80.8% saving comes from the time-series repository; on the recommendation repository, LaDiM consumes more tokens than either main baseline. The evidence therefore supports workload-specific advantages rather than a broadly effective accuracy–cost improvement.
- The comparisons do not sufficiently isolate diagnostic value from basic agent execution behavior. In the Tanh case, both baselines identify the suspected operator, and MatchFixAgent even drafts the relevant registration, but neither applies the patch within the budget. LaDiM’s advantage here therefore includes enforcing that an edit is actually applied and tested, rather than uniquely identifying the fault. The paper does not compare with coding-agent harnesses such as Claude Code or Codex supplied with the same numerical checks. Such a comparison would help establish whether specialized multi-agent coordination adds value beyond providing useful tests and a competent edit–test loop.
Questions for authors
Can the authors compare LaDiM with a coding-agent harness (e.g., codex, claude code) given the same source programs, initial translations, numerical checks, and repair budget? Please distinguish a same-model orchestration comparison from an end-to-end system comparison.
2 comments
Response to Reviewer TR5W (2/2): Orchestration Design, Evaluation Scale, and Transfer
This comment continues our response with W1, W2, and W3.
W1. Evidence for the orchestration design, and repair history in Table 7.
Repair history takes effect when a program needs more than one submission. In the main collection, 20 of the 29 inputs pass before repair and seven of the nine failing inputs are repaired at the first submission, which leaves only two inputs where history can act. Removing it saves tokens at 29/29 acceptance. By contrast, on the Table 2a programs, where repairs take up to three submissions, removing history lowers the repaired programs from 4/5 to 0/5 (Table 3b). On the public programs, where 25 of the 38 repairs by LaDiM are accepted at the second or a later submission, it lowers the repaired attempts from 38/63 to 20/63. On recommendation, the conditions without repository context management stop at the context limit after 59 to 68 calls, and with it the run uses the full 80-call budget (131/145 checks in every condition of Table 11), whereas the token saving of the repository components appears on time series (Table 3c). The revised paper states these conditions next to Tables 7 and 11.
W2. Evaluation scale and constructed tasks.
We agree that the nine failing inputs of the main collection form a small repair set. The repair evaluation now spans 52 failing translations from three collections (composition and sources are in our comment to all reviewers). On the 21 failing translations of public PyTorch programs, LaDiM repairs 38/63 attempts versus 3, 2, and 3 of 21 in one run each for Direct repair, SWE-agent, and MatchFixAgent. These failures arise during translation and therefore complement the four constructed fault types of Table 3a. Among the 52 failing translations, 18 are silent failures (11 first caught by the gradient comparison). For repository scale, we add a third public repository with 69 code files and 7,827 nonblank lines, almost four times the recommendation repository. On this repository, LaDiM passes 116 of 131 behavior checks, MatchFixAgent 113, and SWE-agent 99 (Table 21).
W3. Transfer of the gains to other settings.
The repair gain is large on the Table 2a programs and the public failing translations, where the baselines of Table 2a repair none of the five programs and at most 3 of 21, and the token saving is largest where LaDiM finishes before the budget. In Table 2b, both methods repair all six supplied JAX faults, which leaves the Verifier stage no headroom for further repairs while it still adds tokens. On 16 failing JAX translations from 63 public workloads, LaDiM repairs 9, 8, and 9 in three runs versus 7, 8, and 7 for Direct repair. The repair-token ratio is 1.9 there, close to the 2.4 and 1.9 of the separate studies of Table 13 and Table 2b, so on JAX the additional repairs come at a higher token cost. On time series, LaDiM stops after 35 of the 80 calls (Table 1c). On recommendation, all three agents use the full 80 calls and their token totals lie within 6% of each other. LaDiM restores reward model inference and retrieval, which stay unresolved in both baselines, and reaches the highest coverage of 130/145 checks (130, 131, and 130 in three independent runs versus 129, 130, and 128 for MatchFixAgent). The third repository of W2 shows the same picture: all three agents use the full 80 calls, their token totals lie within 11% of each other in every run, and LaDiM reaches the highest coverage in all three runs. The revised abstract removes the single 80.8% figure and distinguishes the three main comparisons that reduce token use from the one that reaches the highest coverage at comparable token use.
Response to Reviewer TR5W (1/2): Comparison with Claude Code and the Value of Diagnosis
Thank you for the constructive feedback. In the uploaded revision, modified text is in blue and the added tables appear in Appendix J. This comment answers Q1&W4, and Part 2 answers W1, W2, and W3.
Q1&W4. Comparison with a coding-agent harness, and isolating diagnostic value from the edit and test loop.
Following your suggestion, we compare LaDiM with Claude Code (version 2.1.322) given the same source programs, initial translations, and numerical checks. Its repair budget is the same 40 model calls, 120,000 output tokens, 1,800 s, and four submissions. A wrapper ends a session at the first limit reached and counts each run of the acceptance script as a submission. Both systems call the same model on one service with the reasoning setting of Table 14, every model call made by Claude Code counts toward the 40, and scratch tests do not count as submissions for either. Because our claim concerns orchestration, we hold the LLM fixed across LaDiM and Claude Code, so every number below is a same-model result. Pairing Claude Code with its native model would change the model and the orchestration together, and a difference could then come from either. The diagnosis stage does not rely on a particular model, and we leave its evaluation with stronger models to future work. We also include two LaDiM controls with identical signals and tools: flat diagnosis presents the measurements as one flat shuffled list without signal-type labels or check names, while the second control removes the Verifier stage. Unlike Direct repair in Table 2a, which allows two calls and one edit per attempt in one conversation and has no completion condition, this second control retains the repair stages, repair history, and completion condition of LaDiM. Every failing translation receives three independent runs, and the table counts repaired attempts.
On the public programs, the repair rate of LaDiM is 22.2 points above that of Claude Code (95% paired bootstrap interval over source files, 11.1 to 31.7). Table 15 in Appendix J sets the configuration of LaDiM, the two controls, Claude Code, and Direct repair side by side, and Table 17 lists the repaired runs of every program under each method. On these 63 attempts, Claude Code uses 126.3M tokens versus 104.6M for LaDiM, and auxiliary calls outside its main loop make up 4% of its calls.
We agree that the Tanh result includes applying and testing an edit, which is the completion condition of the Repair Agent in Section 3.3. The variant without the diagnosis stage keeps the repair stages, repair history, and completion condition and removes the Verifier, so it measures the gain from adding the diagnosis stage. On the Table 2a programs it repairs 6 of 15 attempts, while Direct repair, with two calls per attempt, repairs 1 of 15 in three runs and none in the single run of Table 2a. The two differ in the length of the repair stages and in the completion condition. The diagnosis stage raises the count from 6 to 12, and over the three rows of the table, LaDiM repairs 29 more attempts than the variant without it. Against flat diagnosis, LaDiM repairs 7 more attempts. On the autoencoder of Figure 4, LaDiM repairs 3 of 3 runs, flat diagnosis 2, the variant without the diagnosis stage 1, and Claude Code 1. Claude Code applies an edit and runs the checks after it in 151 of its 156 sessions (97%), so sessions without a tested edit explain few of its failures. Of its 55 unrepaired sessions, 35 end at the call limit and 8 at the submission limit (Appendix J). We also measure fault localization before any edit under the layered and flat records. Of the 50 controlled faults of Appendix C, the injection changes one function in 38 and several functions in the other 12. On these 38, with one run per fault, the Verifier names the faulty function in 34 cases for LaDiM and 32 for flat diagnosis.