- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper presents VECTRA, an outer control framework for repository-level code translation. It wraps an existing coding agent with repository-wide bootstrap, failure-driven scheduling, verification and rollback, and a persistent ledger across episodes. On RepoTransBench, VECTRA reports a 67.9% average success rate over all 13 translation directions, compared with 55.6% for direct OpenCode.
Critical strengths
The evaluation covers all 13 translation directions and 1,864 projects in RepoTransBench.
Critical weaknesses
-
The evaluation setup is absolutely a major concern. The target tests seem to be visible to the agent during translation and repair, while those same tests are also used to define final success. This makes it possible to optimize directly for the evaluation cases, or even hard-code behavior for visible inputs. Passing these tests alone does not show general source-target behavioral equivalence. It is obvious that test cases or evaluation cases can not be visible during the processing of converting the code.
-
The presentation is very hard to follow, no key point or gap from the introduction part, and method, expirement part are full of defensive writing style which is not a good acdemic writing. The Introduction does not clearly explain what prior work already solves, what is still missing, and what specific gap VECTRA addresses. Figure 1 is also not very informative (or information density is low, too big and not clear with no information), it shows the four components, but does not make the actual control flow, state updates, or accept/reject process clear.
-
The baseline comparison is too limited. On the full benchmark, VECTRA is mainly compared with RepoTransAgent and direct OpenCode. ReCodeAgent is only tested on two directions, and several closely related recent systems are not compared experimentally.
-
Harness matters, so we cannot say VECTRA is a good process than existing harness. Prior work [1,2] shows that the coding-agent harness itself can have a large effect on both performance and efficiency. It is therefore unclear whether the gains come from VECTRA itself or from how it interacts with OpenCode.
-
The comparison is also not fair. VECTRA uses much more runtime and many more tokens per project, so part of the gain may simply come from giving the agent more computation.
References
[1] Naman Vats and Oleg Golev. "The Scaffold Effect in Coding Agents: Harness Choice as a Hidden Variable in Coding-Agent Evaluation." arXiv:2607.22585, 2026.
[2] Red Hat Emerging Technologies. "coding_agent_bench: Reproducible Benchmarks for Coding Agents and Models using Harbor." GitHub repository, 2026.
Questions for authors
I have multiple questions for authors, I pick the most critical 4 Qs:
Are the tests shown to the agent the same tests used for final evaluation? If yes, how do you rule out overfitting?
Why are stronger recent repository-translation systems not compared more broadly?
Do the gains still hold with other coding-agent harnesses?
What exactly is new here compared with prior work on planning, repair, rollback, and memory?