- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes Solvita, a closed-loop four-agent framework with Planner, Solver, Oracle and Hacker for competitive programming. The LLM backbone stays frozen, and three trainable knowledge networks update their retrieval weights and memory contents from task feedback, a three-layer QMS graph for the Solver trained with contrastive REINFORCE and contextual bandits for the Oracle and Hacker. Evaluation covers CodeContests, APPS, and AetherCode across five backbones against commercial and open-source agents, plus component ablations, retrieval controls, patch-vs-regenerate, and a CodeElo-based Codeforces estimate.
Critical strengths
- The evaluation is broad and fairly careful. All methods share the same five backbones, gateway, and decoding settings across three benchmarks. The paper reports paired bootstrap CIs with McNemar/BH correction and candidly states that Solvita is statistically indistinguishable from Codex CLI and Claude Code on their native backbones, and slightly worse in single-run success there.
- The matched-budget controls in Tables 3–4 separate graph growth from weight learning and compare against BM25 and LLM skill selection. They give the most direct support for the central claim that learned routing beats static retrieval.
Critical weaknesses
- The source of gains is unclear. Each Solvita run executes three branches and submits the one passing the most internal tests, while baselines submit one program per run. On native backbones Solvita shows no advantage, so its lead elsewhere may largely reflect commercial agents being tuned to their own models rather than experience learning. Further, the learning gain itself (scaffold-only → full system, about +14 points in Table 2) comes from single training runs with no variance estimates (acknowledged in App. J), and the QMS-over-LLM-pick margin (about 4–5 points) is shown only on GPT-5.4.
- The paper does not state whether these contests (IDs 2219–2237) postdate each backbone’s training cutoff. APPS, CodeContests, and AetherCode 2024 face the same pretraining-contamination risk. The benchmark-isolation procedure covers the retrieval corpus but not the backbone itself.
- Some pages show Solvita scoring below the bare model (e.g., GPT-5.4 on contest 2220), and per-page scores are nearly identical across backbones. Both of them need explanation.
Questions for authors
- Figure 1 lists LeetCode, HackerRank, and 25 sites, while Section 2.1 says the corpus comes only from the APPS, CodeContests, and CodeContests+ training splits. How do these reconcile?
2 comments
Thank you again for your thoughtful review. As the discussion period draws to a close, we wanted to ask whether our response addressed your concerns. If any questions remain, we’d be happy to discuss them further.
Response to Reviewer t7f4
We thank the reviewer for the close reading of our benchmark comparisons and training data sources. We address the four questions in the order raised.
Q1 The source of gains is unclear. Each Solvita run executes three branches and submits the one passing the most internal tests, while baselines submit one program per run. On native backbones Solvita shows no advantage, so its lead elsewhere may largely reflect commercial agents being tuned to their own models rather than experience learning. Further, the learning gain itself (scaffold-only → full system, about +14 points in Table 2) comes from single training runs with no variance estimates (acknowledged in App. J), and the QMS-over-LLM-pick margin (about 4–5 points) is shown only on GPT-5.4.
A1 The comparison evaluates complete agent strategies. Generating several candidates and selecting one through internal tests is part of Solvita's strategy. Commercial agents retain their default workflows and can also generate multiple candidate programs, compile them, and run their own tests before choosing a solution. The rule of one program per run applies to the final submission, and our token accounting includes all Solvita branches. Solvita achieves comparable accuracy on the commercial agents' native models with lower reported token use. The contribution of learning is assessed separately in Table 2, where the backbone, branching, and selection rule remain fixed and training improves performance across three backbones. Across three independent GPT-5.4 training runs with different problem orders and policy sampling seeds, the gain over the scaffold, averaged across the three benchmarks, is 13.79 ± 0.40 percentage points (mean ± standard deviation across runs). The comparison with LLM skill selection in Table 4 isolates the retrieval mechanism on GPT-5.4 under matched corpus and retrieval budgets; the evidence that learning helps across backbones comes from Table 2.
Q2 The paper does not state whether these contests (IDs 2219–2237) postdate each backbone’s training cutoff. APPS, CodeContests, and AetherCode 2024 face the same pretraining-contamination risk. The benchmark-isolation procedure covers the retrieval corpus but not the backbone itself.
A2 The model versions used in our Codeforces experiments were released before or around the contest period; the DeepSeek results use the V4 Pro preview. For APPS, CodeContests, and AetherCode 2024, we use the datasets' designated test sets and exclude evaluation problems from Solvita's training corpus and memories. The underlying LLM weights remain frozen. Any inclusion of these public test sets in model pretraining would come from the provider's data collection, not from Solvita's training or retrieval pipeline. We additionally audited all 165 CodeContests and 1,000 APPS evaluation problems against the 12,017 problems used for initialization and online training. Semantic similarity ≥ 0.85 flagged 27 CodeContests pairs and 115 APPS pairs. Manual checks of sample inputs and outputs and abstract syntax trees found zero paraphrased duplicate problems among these candidates.
Q3 Some pages show Solvita scoring below the bare model (e.g., GPT-5.4 on contest 2220), and per-page scores are nearly identical across backbones. Both of them need explanation.
A3 GPT-5.4 on contest 2220 is the only lower score among the 36 displayed comparisons of model and contest page; Solvita scores higher in the other 35. Each method submits one independently generated program per problem. Retrieval and repair can sometimes steer a solution toward an unsuitable approach, as discussed in Appendix J, so improvements need not occur on every problem. Overall, Solvita with GPT-5.4 solves 72 of 79 distinct problems, compared with 50 for the bare model. The repeated scores largely reflect full marks: on contests 2224, 2231, 2232, and 2237, all three backbones solve every problem on the page. Our scoring sums the initial values of accepted problems and omits runtime, so identical solved sets yield identical scores, and different sets can also have the same total. CodeElo estimates can coincide even when scores differ because the scores produce the same inserted rank in the official standings.
Q4 Figure 1 lists LeetCode, HackerRank, and 25 sites, while Section 2.1 says the corpus comes only from the APPS, CodeContests, and CodeContests+ training splits. How do these reconcile?
A4 The “25 sites” label in Figure 1 refers to the programming websites we surveyed when looking for candidate training data. The final corpus draws only on the APPS and CodeContests training splits and the training split of CodeContests+, as described in Section 2.1. The website names identify where individual problems originated. For example, APPS includes problems from LeetCode and HackerRank. We will revise Figure 1 to distinguish the initial survey from the three dataset splits we used.