- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces AlgoSAGE, a code generation framework that uses structured problem representations to retrieve algorithmic knowledge and solved examples. It combines this guidance with planning, coding, auditing, and debugging, reporting improved accuracy across multiple benchmarks and language models.
Critical strengths
The method addresses a practical challenge: selecting suitable algorithms before generating and debugging code.
Combining reusable algorithmic knowledge with solved examples provides useful guidance for planning.
Results show consistent improvements across several backbones and benchmarks, supported by component ablations and token-usage reporting.
Critical weaknesses
The paper presents a well-motivated framework, with encouraging results across several backbones and benchmarks. I also appreciate the token-usage reporting and component ablations. A few additional comparisons and clarifications would strengthen the findings:
-
Comparisons with retrieval-based methods would help establish the main contribution. Since structured retrieval is central to AlgoSAGE, comparisons with PERC, SolveRank, or standard retrievers using the same memory and downstream workflow would be particularly useful.
-
More evaluation and memory details would improve reproducibility. Please report the exact benchmark subsets and problem counts, the number of knowledge entries and exemplars, and the model used to construct the memory.
-
The overlap-removal procedure could be clearer. It would help to explain how duplicate and near-duplicate problems are detected, how many examples are removed, and whether knowledge derived from removed examples is also checked.
-
The role of Folding could be isolated more directly. Since its labels can already encode inferred algorithmic mechanisms, a Folding-without-retrieval baseline would help distinguish the benefit of initial algorithm analysis from that of retrieval.
-
Budget-controlled comparisons would clarify the efficiency trade-off. Token reporting is helpful, but some settings use substantially more total tokens for relatively small accuracy gains. Comparisons under equal token or monetary budgets would strengthen the analysis.
-
The auditing policy deserves further explanation. Repaired programs are checked against examples without repeating the audit. How does the system establish that the algorithmic or complexity issues raised by the initial audit have been addressed?
-
Direct retrieval-quality analysis would complement the ablations. Measuring algorithm relevance, exemplar usefulness, and failures caused by incorrect first-stage retrieval would help explain when the approach works well and where it struggles.
Overall, the results are promising. More controlled retrieval comparisons and clearer experimental details would make the contribution more convincing.
Questions for authors
See Weaknesses
11 comments
**Question 1:** Comparisons with retrieval-based methods would help establish the main contribution. Since structured retrieval is central to AlgoSAGE, comparisons with PERC, SolveRank, or standard retrievers using the same memory and downstream workflow would be particularly useful.
(3) SolveRank [2]: Reproduction and Overlap-Controlled Evaluation.
We reproduce SolveRank and find that cross-dataset overlap in its retrieval corpus substantially affects its initial results. After removing overlapping candidates and re-evaluating the full benchmark subsets, AlgoSAGE still outperforms SolveRank across all nine backbone–benchmark combinations.
Reproduction. We follow SolveRank's three-stage pipeline: synthetic data construction, contrastive retriever training, and retrieval-augmented code generation.
Using 4,930 xCodeEval training problems as anchors, DeepSeek-Flash generates synthetic queries, and GPT-5.4 mini verifies their logical consistency, yielding 24,650 positive query–anchor examples. We train a BERT-base-uncased bi-encoder using InfoNCE loss with BM25 and randomly sampled negatives. The trained retriever selects the top-3 problem–solution references from approximately 10,480 CodeContests training entries. The same retriever is used across all three generation backbones. The synthetic-data models replace those used in the original SolveRank paper, while preserving its overall pipeline.
Initial evaluation (Run-1). We first evaluate the reproduced SolveRank without additional overlap filtering. With Qwen3.8-27B, it achieves 33.3%, 56.7%, and 73.6% Pass@1 on CodeContests, APPS, and xCodeEval, respectively. Similar patterns appear with the other backbones, with APPS and xCodeEval generally yielding higher accuracy than CodeContests.
Overlap investigation. Given these initial results, we further examine potential overlap between the CodeContests training corpus used for retrieval and the evaluation benchmarks. An initial top-1 analysis reveals mean statement Jaccard similarities of 0.25, 0.91, and 0.59 on CodeContests, APPS, and xCodeEval, respectively. Notably, 131/150 APPS problems have a top-1 Jaccard similarity of at least 0.85, suggesting substantial near-duplicate retrieval.
To investigate this systematically, we apply exactly the same four overlap-detection rules used for AlgoSAGE: normalized problem-ID matching, exact normalized statement matching, full-statement containment, and high lexical overlap based on shared 8-grams.
The top-3 analysis identifies:
- APPS: 144/150 evaluation problems (96.0%) have at least one overlapping candidate, involving 232/450 retrieved candidate occurrences (51.6%).
- xCodeEval: 88/106 problems (83.0%) are affected, involving 104/318 candidates (32.7%).
These findings indicate that the initial retrieval results contain substantial cross-dataset overlap, potentially inflating performance by providing near-duplicate problem–solution references.
Overlap-controlled re-evaluation (Run-2). To eliminate these identified overlaps, we exclude overlapping entries from the retrieval corpus and rerun the trained retriever over the remaining candidates to obtain a new top-3. We then re-evaluate the entire APPS (150 problems) and xCodeEval (106 problems) subsets, rather than only the affected problems. CodeContests results remain unchanged.
The initial and overlap-controlled results are compared below:
| Backbone | Benchmark | SolveRank Run-1 | SolveRank Run-2 | AlgoSAGE |
|---|---|---|---|---|
| Qwen3.8-27B | CodeContests | 33.3 | 33.3 | 70.3 |
| Qwen3.8-27B | APPS | 56.7 | 38.7 | 76.0 |
| Qwen3.8-27B | xCodeEval | 73.6 | 65.1 | 78.3 |
| Claude Haiku 4.5 | CodeContests | 32.7 | 32.7 | 48.5 |
| Claude Haiku 4.5 | APPS | 66.0 | 56.0 | 62.7 |
| Claude Haiku 4.5 | xCodeEval | 65.1 | 57.5 | 62.3 |
| GPT-5.4 mini | CodeContests | 22.4 | 22.4 | 52.7 |
| GPT-5.4 mini | APPS | 56.0 | 26.0 | 49.3 |
| GPT-5.4 mini | xCodeEval | 64.2 | 43.4 | 60.4 |
All values are Pass@1 (%).
Filtering overlapping retrieval candidates leads to substantial performance decreases on APPS and xCodeEval, confirming that cross-dataset near-duplicate retrieval materially contributed to SolveRank's initially elevated results. After filtering, AlgoSAGE outperforms SolveRank in all nine settings, with improvements ranging from 4.8 to 37.3 percentage points.
Together, the controlled retrieval comparisons establish the benefit of structure-grounded matching and Knowledge-conditioned hierarchical retrieval under the same expertise memory, while PERC and overlap-controlled SolveRank provide complementary comparisons against independently developed retrieval-based frameworks.
References [2] Zhang et al., Beyond the Surface: A Solution-Aware Retrieval Model for Competition-level Code Generation, Findings of EMNLP 2025.
**Question 2:** More evaluation and memory details would improve reproducibility. Please report the exact benchmark subsets and problem counts, the number of knowledge entries and exemplars, and the model used to construct the memory.
2. Evaluation and Memory Construction Details
We appreciate the request for additional reproducibility details.
Our evaluation uses 165 CodeContests, 150 APPS, and 106 xCodeEval problems. All methods are evaluated on the same subsets using Python 3 and a common execution-based evaluation framework. We will provide the exact problem identifiers and subset-selection procedure in the supplementary materials.
The final Algorithmic Expertise Memory contains 118 Algorithmic Knowledge entries, organized under an OI Wiki-derived taxonomy, and 2,018 Application Exemplars, retained after execution validation and overlap filtering.
The offline memory-construction model is Claude Haiku 4.5 (claude-haiku-4-5). Reference solutions from CodeContests training problems guide solution-plan generation. Only execution-validated cases are retained, and reference solutions for evaluation problems are unavailable during online evaluation.
**Question 1:** Comparisons with retrieval-based methods would help establish the main contribution. Since structured retrieval is central to AlgoSAGE, comparisons with PERC, SolveRank, or standard retrievers using the same memory and downstream workflow would be particularly useful.
We thank the reviewer for the constructive feedback. We have added retrieval comparisons, component ablations, and inference-cost analyses, and clarified the evaluation and memory-construction protocols.
1. Comparisons with Retrieval-Based Methods
We add two controlled retrieval comparisons using the same expertise memory and reproduce two existing retrieval-based frameworks, PERC and SolveRank. AlgoSAGE consistently outperforms these alternatives, including SolveRank after identifying and removing overlapping retrieval candidates and conducting a full re-evaluation.
(1) Controlled retrieval comparisons.
We first compare AlgoSAGE against simpler retrieval strategies using the same Algorithmic Expertise Memory and downstream generation workflow.
Dense RAG replaces the eight-field representation and structural matching with the original problem statement embedded by Qwen3-Embedding-0.6B, using cosine similarity for retrieval.
Exemplars-then-Knowledge retains the eight-field representation and both types of expertise but retrieves Exemplars globally before attaching their associated Knowledge, removing the Knowledge-conditioned retrieval constraint.
All values are Pass@1 (%) using Qwen3.8-27B.
AlgoSAGE outperforms both lexical and dense retrieval alternatives, achieving improvements of 3.0/6.0/5.7 percentage points over Token-F1 and 6.1/12.7/4.7 points over Dense RAG. More importantly, it outperforms Exemplars-then-Knowledge by 3.0/4.0/3.8 points. Since the latter preserves the structural representation and both expertise types, this comparison more directly supports the contribution of Knowledge-conditioned hierarchical retrieval.
(2) PERC [1].
AlgoSAGE substantially outperforms PERC across all three benchmarks. We reproduce PERC following its plan-based retrieval and example-guided generation workflow. For each target problem, its statement and initial solution plan are used to retrieve a top-1 reference using MPNet-based cosine similarity. The retrieved statement, plan, and code are incorporated into the generation prompt.
With Qwen3.8-27B, PERC achieves 43.0%, 41.3%, and 56.6% Pass@1 on CodeContests, APPS, and xCodeEval, respectively, compared with AlgoSAGE's 70.3%, 76.0%, and 78.3%. Results with other backbones are discussed in our response to Question 5.
References [1] Yoo et al., PERC: Plan-As-Query Example Retrieval for Underrepresented Code Generation, COLING 2025.