acceptodds
Under review as a conference paper at ICLR 2027

AlgoSAGE: Formal Problem Representation and Grounded Expertise for Algorithmic Code Generation

Abstract

Large language models (LLMs) have demonstrated strong capabilities in automated code generation; however, achieving reliable performance on algorithmic programming problems remains challenging, as implicit problem abstraction and free-form planning can lead to inappropriate algorithm choices, underspecified solution plans, and constraint-violating implementations. Our key insight is that reliable code generation requires making the computational structure of a problem explicit, using it to select appropriate algorithmic expertise, and translating that expertise into concrete plans before coding. We propose AlgoSAGE, which constructs a formal problem representation and retrieves expertise at two complementary levels: reusable Algorithmic Knowledge and Algorithmic Application Exemplars that demonstrate how the retrieved knowledge is applied. AlgoSAGE then adapts this evidence into target-specific, executable, and auditable plans. Extensive experiments across multiple benchmarks and LLM backbones demonstrate that AlgoSAGE consistently improves algorithmic code generation. With Qwen3.8-27B-Coder, AlgoSAGE achieves 70.3%, 76.0%, and 78.3% Pass@1 on CodeContests, APPS, and xCodeEval, respectively, outperforming existing approaches by up to 6.7 percentage points. Experiments with additional backbones and function-level programming benchmarks further demonstrate the robustness and general applicability of AlgoSAGE. Code is available at https://anonymous.4open.science/r/AlgoSage-82F9/.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
Algorithmic Code Generation, Large Language Models, Algorithmic Knowledge Memory, Multi-Agent Systems

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_ouvwOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces AlgoSAGE, a code generation framework that uses structured problem representations to retrieve algorithmic knowledge and solved examples. It combines this guidance with planning, coding, auditing, and debugging, reporting improved accuracy across multiple benchmarks and language models.

Critical strengths

The method addresses a practical challenge: selecting suitable algorithms before generating and debugging code.

Combining reusable algorithmic knowledge with solved examples provides useful guidance for planning.

Results show consistent improvements across several backbones and benchmarks, supported by component ablations and token-usage reporting.

Critical weaknesses

The paper presents a well-motivated framework, with encouraging results across several backbones and benchmarks. I also appreciate the token-usage reporting and component ablations. A few additional comparisons and clarifications would strengthen the findings:

  1. Comparisons with retrieval-based methods would help establish the main contribution. Since structured retrieval is central to AlgoSAGE, comparisons with PERC, SolveRank, or standard retrievers using the same memory and downstream workflow would be particularly useful.

  2. More evaluation and memory details would improve reproducibility. Please report the exact benchmark subsets and problem counts, the number of knowledge entries and exemplars, and the model used to construct the memory.

  3. The overlap-removal procedure could be clearer. It would help to explain how duplicate and near-duplicate problems are detected, how many examples are removed, and whether knowledge derived from removed examples is also checked.

  4. The role of Folding could be isolated more directly. Since its labels can already encode inferred algorithmic mechanisms, a Folding-without-retrieval baseline would help distinguish the benefit of initial algorithm analysis from that of retrieval.

  5. Budget-controlled comparisons would clarify the efficiency trade-off. Token reporting is helpful, but some settings use substantially more total tokens for relatively small accuracy gains. Comparisons under equal token or monetary budgets would strengthen the analysis.

  6. The auditing policy deserves further explanation. Repaired programs are checked against examples without repeating the audit. How does the system establish that the algorithmic or complexity issues raised by the initial audit have been addressed?

  7. Direct retrieval-quality analysis would complement the ablations. Measuring algorithm relevance, exemplar usefulness, and failures caused by incorrect first-stage retrieval would help explain when the approach works well and where it struggles.

Overall, the results are promising. More controlled retrieval comparisons and clearer experimental details would make the contribution more convincing.

Questions for authors

See Weaknesses

11 comments
AuthorsOpenReview ↗

**Question 1:** Comparisons with retrieval-based methods would help establish the main contribution. Since structured retrieval is central to AlgoSAGE, comparisons with PERC, SolveRank, or standard retrievers using the same memory and downstream workflow would be particularly useful.

We thank the reviewer for the constructive feedback. We have added retrieval comparisons, component ablations, and inference-cost analyses, and clarified the evaluation and memory-construction protocols.

1. Comparisons with Retrieval-Based Methods

We add two controlled retrieval comparisons using the same expertise memory and reproduce two existing retrieval-based frameworks, PERC and SolveRank. AlgoSAGE consistently outperforms these alternatives, including SolveRank after identifying and removing overlapping retrieval candidates and conducting a full re-evaluation.

(1) Controlled retrieval comparisons.

We first compare AlgoSAGE against simpler retrieval strategies using the same Algorithmic Expertise Memory and downstream generation workflow.

Dense RAG replaces the eight-field representation and structural matching with the original problem statement embedded by Qwen3-Embedding-0.6B, using cosine similarity for retrieval.

Exemplars-then-Knowledge retains the eight-field representation and both types of expertise but retrieves Exemplars globally before attaching their associated Knowledge, removing the Knowledge-conditioned retrieval constraint.

MethodCodeContestsAPPSxCodeEval
w/o Folding (Token-F1)67.370.072.6
Dense RAG64.263.373.6
Exemplars-then-Knowledge67.372.074.5
Full AlgoSAGE70.376.078.3

All values are Pass@1 (%) using Qwen3.8-27B.

AlgoSAGE outperforms both lexical and dense retrieval alternatives, achieving improvements of 3.0/6.0/5.7 percentage points over Token-F1 and 6.1/12.7/4.7 points over Dense RAG. More importantly, it outperforms Exemplars-then-Knowledge by 3.0/4.0/3.8 points. Since the latter preserves the structural representation and both expertise types, this comparison more directly supports the contribution of Knowledge-conditioned hierarchical retrieval.

(2) PERC [1].

AlgoSAGE substantially outperforms PERC across all three benchmarks. We reproduce PERC following its plan-based retrieval and example-guided generation workflow. For each target problem, its statement and initial solution plan are used to retrieve a top-1 reference using MPNet-based cosine similarity. The retrieved statement, plan, and code are incorporated into the generation prompt.

With Qwen3.8-27B, PERC achieves 43.0%, 41.3%, and 56.6% Pass@1 on CodeContests, APPS, and xCodeEval, respectively, compared with AlgoSAGE's 70.3%, 76.0%, and 78.3%. Results with other backbones are discussed in our response to Question 5.

References [1] Yoo et al., PERC: Plan-As-Query Example Retrieval for Underrepresented Code Generation, COLING 2025.

AuthorsOpenReview ↗

**Question 1:** Comparisons with retrieval-based methods would help establish the main contribution. Since structured retrieval is central to AlgoSAGE, comparisons with PERC, SolveRank, or standard retrievers using the same memory and downstream workflow would be particularly useful.

(3) SolveRank [2]: Reproduction and Overlap-Controlled Evaluation.

We reproduce SolveRank and find that cross-dataset overlap in its retrieval corpus substantially affects its initial results. After removing overlapping candidates and re-evaluating the full benchmark subsets, AlgoSAGE still outperforms SolveRank across all nine backbone–benchmark combinations.

Reproduction. We follow SolveRank's three-stage pipeline: synthetic data construction, contrastive retriever training, and retrieval-augmented code generation.

Using 4,930 xCodeEval training problems as anchors, DeepSeek-Flash generates synthetic queries, and GPT-5.4 mini verifies their logical consistency, yielding 24,650 positive query–anchor examples. We train a BERT-base-uncased bi-encoder using InfoNCE loss with BM25 and randomly sampled negatives. The trained retriever selects the top-3 problem–solution references from approximately 10,480 CodeContests training entries. The same retriever is used across all three generation backbones. The synthetic-data models replace those used in the original SolveRank paper, while preserving its overall pipeline.

Initial evaluation (Run-1). We first evaluate the reproduced SolveRank without additional overlap filtering. With Qwen3.8-27B, it achieves 33.3%, 56.7%, and 73.6% Pass@1 on CodeContests, APPS, and xCodeEval, respectively. Similar patterns appear with the other backbones, with APPS and xCodeEval generally yielding higher accuracy than CodeContests.

Overlap investigation. Given these initial results, we further examine potential overlap between the CodeContests training corpus used for retrieval and the evaluation benchmarks. An initial top-1 analysis reveals mean statement Jaccard similarities of 0.25, 0.91, and 0.59 on CodeContests, APPS, and xCodeEval, respectively. Notably, 131/150 APPS problems have a top-1 Jaccard similarity of at least 0.85, suggesting substantial near-duplicate retrieval.

To investigate this systematically, we apply exactly the same four overlap-detection rules used for AlgoSAGE: normalized problem-ID matching, exact normalized statement matching, full-statement containment, and high lexical overlap based on shared 8-grams.

The top-3 analysis identifies:

  • APPS: 144/150 evaluation problems (96.0%) have at least one overlapping candidate, involving 232/450 retrieved candidate occurrences (51.6%).
  • xCodeEval: 88/106 problems (83.0%) are affected, involving 104/318 candidates (32.7%).

These findings indicate that the initial retrieval results contain substantial cross-dataset overlap, potentially inflating performance by providing near-duplicate problem–solution references.

Overlap-controlled re-evaluation (Run-2). To eliminate these identified overlaps, we exclude overlapping entries from the retrieval corpus and rerun the trained retriever over the remaining candidates to obtain a new top-3. We then re-evaluate the entire APPS (150 problems) and xCodeEval (106 problems) subsets, rather than only the affected problems. CodeContests results remain unchanged.

The initial and overlap-controlled results are compared below:

BackboneBenchmarkSolveRank Run-1SolveRank Run-2AlgoSAGE
Qwen3.8-27BCodeContests33.333.370.3
Qwen3.8-27BAPPS56.738.776.0
Qwen3.8-27BxCodeEval73.665.178.3
Claude Haiku 4.5CodeContests32.732.748.5
Claude Haiku 4.5APPS66.056.062.7
Claude Haiku 4.5xCodeEval65.157.562.3
GPT-5.4 miniCodeContests22.422.452.7
GPT-5.4 miniAPPS56.026.049.3
GPT-5.4 minixCodeEval64.243.460.4

All values are Pass@1 (%).

Filtering overlapping retrieval candidates leads to substantial performance decreases on APPS and xCodeEval, confirming that cross-dataset near-duplicate retrieval materially contributed to SolveRank's initially elevated results. After filtering, AlgoSAGE outperforms SolveRank in all nine settings, with improvements ranging from 4.8 to 37.3 percentage points.

Together, the controlled retrieval comparisons establish the benefit of structure-grounded matching and Knowledge-conditioned hierarchical retrieval under the same expertise memory, while PERC and overlap-controlled SolveRank provide complementary comparisons against independently developed retrieval-based frameworks.

References [2] Zhang et al., Beyond the Surface: A Solution-Aware Retrieval Model for Competition-level Code Generation, Findings of EMNLP 2025.

AuthorsOpenReview ↗

**Question 2:** More evaluation and memory details would improve reproducibility. Please report the exact benchmark subsets and problem counts, the number of knowledge entries and exemplars, and the model used to construct the memory.

2. Evaluation and Memory Construction Details

We appreciate the request for additional reproducibility details.

Our evaluation uses 165 CodeContests, 150 APPS, and 106 xCodeEval problems. All methods are evaluated on the same subsets using Python 3 and a common execution-based evaluation framework. We will provide the exact problem identifiers and subset-selection procedure in the supplementary materials.

The final Algorithmic Expertise Memory contains 118 Algorithmic Knowledge entries, organized under an OI Wiki-derived taxonomy, and 2,018 Application Exemplars, retained after execution validation and overlap filtering.

The offline memory-construction model is Claude Haiku 4.5 (claude-haiku-4-5). Reference solutions from CodeContests training problems guide solution-plan generation. Only execution-validated cases are retained, and reference solutions for evaluation problems are unavailable during online evaluation.

Reviewer_MXGmOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

AlgoSAGE represents problems with eight fields of structural labels and retrieves algorithmic knowledge and linked exemplars for planning, coding and auditing. Its memory combines OI Wiki knowledge with reference-guided solution plans from CodeContests training problems, retained after their implementations pass execution tests. Main experiments span three backbones and three algorithmic benchmarks.

Critical strengths
  1. Linking reusable algorithmic knowledge to execution-tested solution cases gives planning explicit guidance beyond model recall.
  2. Table 1 reports gains across all nine main backbone–benchmark combinations. The Qwen ablation shows drops of 3.0, 6.0 and 5.7 percentage points without Folding, alongside losses from removing knowledge, exemplars or auditing. These results support a useful system contribution, although uncertainty estimates are absent.
Critical weaknesses
  1. The claimed advantage of structural retrieval is not fully assessable from the reported baseline. Table 4 shows a meaningful drop without Folding, but §4.5 does not specify the text retriever or whether memory and retrieved-context budgets are held fixed. These details matter for attributing the gains to representation quality rather than differences in retrieved information.
Questions for authors
  1. What query representation and retriever replace Folding in Table 4? Are the memory, retrieved-context budget and downstream generation budget unchanged?
3 comments

We thank the reviewer for pointing out the need for greater clarity in the w/o Folding ablation. The ablation replaces the eight-field formal representation with the original full problem statement as the retrieval query, using lexical Token-F1 retrieval. The expertise memory, retrieval limits, retrieved-context token budget, and downstream generation configurations remain unchanged. Therefore, the comparison isolates the effect of the query representation and its associated retrieval mechanism.

Specifically, the replacement retriever is a lexical Token-F1 similarity function, rather than BM25 or a neural embedding model. Both query and indexed documents are lowercased and tokenized using the pattern [a-z0-9_]+, with individual CJK characters treated as tokens. Similarity is calculated as:

Sim(Q,D)=2∣Q∩D∣∣Q∣+∣D∣\mathrm{Sim}(Q,D)=\frac{2|Q\cap D|}{|Q|+|D|}

where QQ and DD denote the normalized query and document token sets, respectively.

For Algorithmic Knowledge, the indexed text combines the knowledge name, definition, planning template, and applicability conditions. For Application Exemplars, it combines the problem identifier, summary, statement, and stored planning information. Both retrieval stages retain at most three results. Since lexical Token-F1 and structural similarity have different score distributions, the lexical retrieval thresholds are set to 0.02, compared with the structural thresholds of 0.19 and 0.38.

All other experimental conditions are kept identical:

  • Memory: The same Algorithmic Expertise Memory is used, containing 118 Knowledge entries and 2,018 Exemplars.
  • Retrieval budget: Both settings retrieve up to three Knowledge entries and three Exemplars, with the same retrieved-context token budget.
  • Downstream generation budget: Planning, Coding, Auditing, and Debugging configurations remain unchanged. Both settings allow up to three planning trajectories, with up to two debugging attempts per trajectory.
  • Other configurations: The same downstream prompts, model configurations, and evaluation protocol are used.

The retrieved items are naturally allowed to differ because the query representation and retrieval mechanism are the variables being examined. However, the underlying memory and all retrieval and downstream computation budgets are held fixed.

Thus, the performance difference is attributable to replacing structure-grounded retrieval with raw-text lexical retrieval, rather than changes in memory availability or computational budgets.

AuthorsOpenReview ↗

Kind Request for Reviewer Discussion

Dear Reviewer MXGm,

Thank you very much for your time and for your constructive feedback on our submission. As the discussion period approaches its deadline, we would like to follow up and ask whether you have had a chance to review our detailed rebuttal and the additional experimental results. We would be grateful for any indication of whether our response has addressed your concerns.

We fully understand that your schedule may be busy. If you have any remaining questions or would like further clarification on any point, we would be very happy to continue the discussion and address them promptly.

Thank you again for your time and consideration.

Best regards,

The Authors of Submission 44333

Reviewer_MXGmOpenReview ↗

maintain my positive score

The author’s response has addressed my concerns, and I therefore maintain my positive score.

Reviewer_aohuOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes a new algorithmic code generation framework called AlgoSAGE. By decomposing the natural language programming problems into formal problem representations and strategically retrieve through a prepared algorithmic expertise memory, the framework can discover the right algorithm choice and solution plan more accurately. Experiments across multiple programming benchmarks and LLM backbones demonstrate consistent improvements over existing code-generation approaches.

Critical strengths
  1. The paper identifies an important failure mode in current LLM that selecting an inappropriate algorithm early in the reasoning process can lead to unsuccessful implementations. The ChainWord case study in the appendix effectively illustrates this issue and motivates the need for more reliable algorithm selection.
  2. The proposed framework achieves consistent improvements over existing methods, including MapCoder and CodeSIM, across multiple algorithmic programming benchmarks and LLM backbones.
Critical weaknesses
  1. The main components of the proposed framework, including structured problem representation, RAG, knowledge memory and iterative refinement, are well-established concepts in the existing literature. The primary contribution of using the formal representation to support hierarchical algorithm-then-example retrieval seems more of an engineering design choice.
  2. The proposed multi-stage framework incorporates substantial external algorithmic knowledge and requires significantly more inference tokens than direct generation. It remains unclear to what extent the observed gains can be attributed specifically to the proposed formal representation and hierarchical retrieval, rather than access to curated algorithmic knowledge, additional reasoning computation, or more extensive solution exploration.
Questions for authors
  1. What is Qwen3.8-27B-Coder? Is it an independently fine-tuned coding model based on Qwen3.8-27B, or does it refer to the standard Qwen3.8-27B checkpoint?
  2. Were reasoning modes enabled for the evaluated backbone models? How would AlgoSAGE compare with direct generation or simpler baselines when given comparable total inference-token budgets, potentially through increased reasoning effort or multiple solution attempts?
  3. The Algorithmic Expertise Memory is specifically constructed from competitive programming problems and provides additional guidance for algorithm selection. How would the proposed framework compare against simpler approaches that use the same knowledge base, such as directly injecting retrieved algorithmic knowledge into prompts or applying standard RAG without hierarchical retrieval?
9 comments
AuthorsOpenReview ↗

**Weakness 1:** The main components of the proposed framework, including structured problem representation, RAG, knowledge memory and iterative refinement, are well-established concepts in the existing literature. The primary contribution of using the formal representation to support hierarchical algorithm-then-example retrieval seems more of an engineering design choice.

We thank the reviewer for recognizing the importance of the algorithm-selection failure mode and the consistent improvements of AlgoSAGE across benchmarks and backbones. We appreciate the opportunity to clarify our technical contributions and provide additional controlled evidence.

1. On the Novelty and Technical Contributions

We respectfully disagree with the characterization of AlgoSAGE as merely an engineering design choice. Its contribution lies in a non-trivial problem formulation and the coordinated technical mechanisms developed to realize it, rather than the simple combination of established components.

Conceptually, AlgoSAGE makes the computational information governing algorithm selection explicit and uses it to connect problem structure with applicable algorithmic expertise. It distinguishes two complementary roles: Algorithmic Knowledge identifies reusable methods applicable to the target problem, while Application Exemplars demonstrate how these methods are instantiated under specific operations and constraints. This formulation explicitly separates algorithm selection from solution instantiation, rather than leaving both implicit in free-form reasoning and subsequent code generation.

Technically, realizing this idea requires three interdependent designs: (i) an eight-field formal problem representation with controlled structural labels, together with an expertise memory organized under the same representation schema; (ii) hierarchical structure-grounded retrieval, where retrieved Algorithmic Knowledge constrains the candidate space for Application Exemplar retrieval, with additional knowledge-coverage diversification; and (iii) an expertise-grounded generation framework that adapts the retrieved knowledge–exemplar pairs into target-specific solution plans and carries them through coding, execution checking, auditing, and debugging. These mechanisms are carefully designed to work together: the representation provides consistent matching signals, the memory establishes explicit knowledge–application associations, and hierarchical retrieval uses these associations to select relevant evidence for downstream planning.

The ablation results demonstrate that these design choices are consequential rather than interchangeable. Removing Folding, Algorithmic Knowledge, Application Exemplars, or Auditing consistently degrades performance. Furthermore, replacing Skill-first retrieval with global exemplar retrieval followed by linked-skill lookup decreases Pass@1 by 3.0, 4.0, and 3.8 percentage points on CodeContests, APPS, and xCodeEval, respectively, despite retaining the same representation and knowledge–exemplar content.

Therefore, the contribution of AlgoSAGE is not the novelty of individual building blocks, but the problem-specific formulation and coordinated technical design that connect formal problem structure, applicable algorithmic expertise, concrete application examples, and auditable solution generation.

AuthorsOpenReview ↗

**Weakness 2:** The proposed multi-stage framework incorporates substantial external algorithmic knowledge and requires significantly more inference tokens than direct generation. It remains unclear to what extent the observed gains can be attributed specifically to the proposed formal representation and hierarchical retrieval, rather than access to curated algorithmic knowledge, additional reasoning computation, or more extensive solution exploration.

2. On Performance Attribution and Inference Computation

Our results demonstrate that the performance gains of AlgoSAGE cannot be explained solely by access to curated algorithmic knowledge, additional inference tokens, or more extensive solution exploration. We provide controlled ablations to distinguish the contributions of formal problem representation and hierarchical retrieval, together with token-consumption comparisons and additional-budget experiments.

(1) Separating the Effects of Formal Representation and Hierarchical Retrieval from External Knowledge Access

The ablation results demonstrate that the improvements of AlgoSAGE arise not merely from introducing external algorithmic knowledge, but also from how problem structure is represented and how retrieved knowledge is selected and utilized.

We first report the original component ablations from Table 4, which examine the individual contributions of Auditing, Folding, Knowledge, and Exemplars.

MethodCodeContestsAPPSxCodeEval
w/o Audit66.169.374.5
w/o Folding67.370.072.6
w/o Knowledge67.972.074.5
w/o Exemplars63.668.072.6
Full AlgoSAGE70.376.078.3

We further introduce three controlled comparisons to investigate the contributions of structural representation and retrieval design:

MethodCodeContestsAPPSxCodeEval
Folding-only67.974.774.5
Dense RAG64.263.373.6
Exemplars-then-Knowledge67.372.074.5
Full AlgoSAGE70.376.078.3

All results are Pass@1 (%) using Qwen3.8-27B.

The results provide three important observations.

First, both Knowledge and Exemplars contribute independently to the final performance. Removing Knowledge decreases Pass@1 by 2.4, 4.0, and 3.8 percentage points, while removing Exemplars results in larger decreases of 6.7, 8.0, and 5.7 percentage points on CodeContests, APPS, and xCodeEval, respectively. Thus, neither algorithmic principles nor concrete application examples alone can fully account for the performance of AlgoSAGE.

Second, formal problem representation provides benefits independently of external knowledge access. Folding-only supplies the original problem statement and its eight-field representation directly to Planning, without retrieving Knowledge or Exemplars. It achieves 67.9%, 74.7%, and 74.5% Pass@1, demonstrating that explicit structural problem analysis itself provides useful guidance. Full AlgoSAGE further improves these results by 2.4, 1.3, and 3.8 percentage points.

Third, the retrieval hierarchy contributes beyond simply accessing the same knowledge and exemplars. Dense RAG replaces structural retrieval with Qwen3-Embedding-0.6B and cosine similarity over the same expertise memory, resulting in decreases of 6.1, 12.7, and 4.7 percentage points. More importantly, Exemplars-then-Knowledge retains the same formal representation, memory, and both types of knowledge, but retrieves Exemplars globally before attaching their associated Knowledge, removing the Knowledge-first retrieval constraint. This variant decreases Pass@1 by 3.0, 4.0, and 3.8 percentage points.

Together, these results show that AlgoSAGE's improvements cannot be attributed simply to the availability of curated external knowledge. Both knowledge components, explicit structural representation, and Knowledge-conditioned hierarchical retrieval make distinct contributions to the final performance.

AuthorsOpenReview ↗

> **Weakness 2:** The proposed multi-stage framework incorporates substantial external algorithmic knowledge and requires significantly more inference tokens than direct generation. It remains unclear to what extent the observed gains can be attributed specifically to the proposed formal representation and hierarchical retrieval, rather than access to curated algorithmic knowledge, additional reasoning computation, or more extensive solution exploration.

(2) Inference Token Consumption Compared with Existing Methods

AlgoSAGE achieves higher accuracy while consuming fewer total inference tokens than MapCoder across all nine backbone–benchmark combinations. Compared with CodeSIM, AlgoSAGE consistently achieves higher accuracy and generates fewer output tokens, although its total token consumption is higher in most settings.

All three evaluated backbones were tested without Thinking/Reasoning enabled, and all main experiments use temperature = 0. We measure token consumption across the complete online inference workflow, including all intermediate interactions, prompts, and generated responses.

Comparison with MapCoder (Islam et al., 2024).

BackboneBenchmarkMapCoder (Input / Output, K)AlgoSAGE (Input / Output, K)Token ReductionPass@1: MapCoder → AlgoSAGE
Qwen3.8-27BCodeContests40.15 / 25.2638.07 / 25.442.9%56.4 → 70.3
Qwen3.8-27BAPPS29.16 / 19.3727.91 / 18.284.8%51.3 → 76.0
Qwen3.8-27BxCodeEval26.39 / 17.5820.95 / 13.5621.5%67.0 → 78.3
Haiku 4.5CodeContests35.03 / 15.1627.08 / 10.9124.3%44.2 → 48.5
Haiku 4.5APPS20.86 / 9.8518.27 / 7.3016.7%59.3 → 62.7
Haiku 4.5xCodeEval18.79 / 9.7316.26 / 6.9618.6%57.5 → 62.3
GPT-5.4 miniCodeContests24.86 / 9.3824.51 / 7.386.9%46.7 → 52.7
GPT-5.4 miniAPPS19.48 / 7.1219.13 / 5.248.4%47.3 → 49.3
GPT-5.4 minixCodeEval14.38 / 5.7813.87 / 3.9411.7%52.8 → 60.4

Token values are in thousands (K). Token reduction is calculated based on total input and output tokens.

Across all nine settings, AlgoSAGE consumes 2.9%–24.3% fewer total tokens than MapCoder while achieving 2.0–24.7 percentage points higher Pass@1. This demonstrates that AlgoSAGE's accuracy improvements do not come at the expense of increased inference-token consumption relative to MapCoder.

Comparison with CodeSIM (Islam et al., 2025).

AlgoSAGE also achieves higher Pass@1 than CodeSIM across all nine backbone–benchmark combinations and generates fewer output tokens in every setting. However, its total token consumption is higher in eight of the nine settings, with Haiku 4.5 / CodeContests being the exception (37.99K versus 40.30K).

This comparison highlights a more nuanced efficiency trade-off: AlgoSAGE achieves better accuracy with fewer output tokens, but does not consistently minimize total token consumption relative to CodeSIM.

Overall, these comparisons demonstrate that AlgoSAGE's performance improvements are not simply a consequence of consuming more inference tokens. In particular, it consistently outperforms MapCoder while using fewer total tokens across all three backbones and benchmarks.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.