acceptodds
Under review as a conference paper at ICLR 2027

Beyond Semantic Similarity: Abstraction-Based Demonstration Retrieval for Mathematical Reasoning

Abstract

Selecting useful demonstrations is crucial for in-context mathematical reasoning with large language models (LLMs). However, using the original problem for retrieval often focuses too much on surface wording while ignoring the underlying mathematical relations needed for reasoning. To address this, we propose MetaPR, a retrieval-aligned meta-problem framework that learns to generate structured abstractions for retrieving demonstrations with reusable solution structures. For each problem, MetaPR generates a structured abstraction that combines a compact meta-problem with a concept list, preserving key constraints and mathematical relations while filtering out irrelevant details. The structured abstraction is combined with the original problem to form a multi-view retrieval representation. To ensure the generated abstractions genuinely help retrieval rather than merely being well-formed, the generator is initialized via supervised fine-tuning (SFT) and then optimized with Group Relative Policy Optimization (GRPO). During GRPO, we assign a Correctness-transition reward to each sampled abstraction by comparing answer correctness with and without the demonstrations it retrieves. Experiments across two NuminaMath-derived datasets and three downstream solvers demonstrate that MetaPR outperforms lexical, dense, skill-based, and reasoning-graph retrieval baselines in answer accuracy.

Primary area:
foundation or frontier models, including LLMs
Keywords:
in-context learning, demonstration retrieval, mathematical reasoning, large language models, retrieval-augmented generation, reinforcement learning, structured abstraction

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_Aq9NOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

MetaPR uses generated abstractions to retrieve demonstrations for mathematical reasoning. A generator produces a meta-problem and a concept list; a frozen embedding model encodes these and the original question. Retrieval combines the three similarity views with equal weights to select problems and solutions for a downstream solver. The generator is initialized with supervised training, then optimized with GRPO using correctness-transition rewards from a solver ensemble and format/length terms. Evaluation covers two mathematical datasets, several solvers, view and training ablations, and a controlled rewrite-retrieval metric. The shared demonstration pool contains solved, structure-preserving rewrites of selected training and test queries. Approximately 30% of test queries have deliberately constructed matches in this pool.

Critical strengths
  1. Using solution-relevant structure to go beyond topical similarity is a sensible direction. The three-view ablation shows that the representations provide complementary information: the full version improves on the best two-view variant by 1.75-2.50 accuracy points on aops forum. This supports the interface within the reported pool construction.

  2. The training pipeline produces measurable gains. With Qwen-7B, moving from SFT to GRPO raises ALL accuracy from 28.00 to 30.88 on aops forum and from 60.81 to 62.29 on olympiads. The tests include multiple solvers, one outside the reward ensemble, and controlled Hit@3 results. These support the pipeline beyond qualitative examples, although they do not isolate the transition reward.

Critical weaknesses
  1. The shared pool limits what can be concluded about generalization. The authors deliberately add solved rewrites of selected test questions to the demonstration pool. About 30% of test queries have constructed matches, and the remaining queries use that same augmented pool. This disclosed design tests recovery of structurally related examples, but subset A still does not test retrieval from an independently built natural pool. The concern is the scope of the evaluation; this construction does not establish that test answers were used to train the generator.

  2. The advance over earlier abstraction-based retrieval needs to be defined more precisely. Skill-KNN and GraphIC already provide relevant precedents. This version also omits https://arxiv.org/abs/2510.13905, https://arxiv.org/abs/2305.14502, and https://github.com/qpalzm36/StepIR. Their abstractions, selection procedures and retrieval granularity differ, but together they narrow the contribution to the specific trained meta-problem/concept interface.

Questions for authors

Does the advantage persist with an unaugmented demonstration pool drawn from training data and separated by source problem family? Please report the main comparisons in that setting alongside the controlled rewrite results, and correct or confirm the reversed A/B labels in the main-table captions.

2 comments
AuthorsOpenReview ↗

Response to Reviewer Aq9N — Weakness 1: Shared pool & corrected A/B convention

We thank the reviewer for the careful reading and the concrete suggestions, which helped us fix the caption and add the controls below. If you feel that your concerns have been adequately addressed, we kindly ask if you would consider raising your rating. We truly appreciate your valuable suggestions.

Weakness 1 (reviewer's point 1): Whether the shared demonstration pool limits the generalization conclusion

Thank you for carefully pointing out the swapped caption and for asking for results that separate the pool from the retrieval method.

(1)Caption and convention. The main-table caption had swapped A and B; it is now corrected. Throughout the paper, we use one convention: A is the uncontrolled test subset, B is the matched test subset (test queries whose constructed rewrite was added to the demonstration pool), and ALL is their union, A∪BA\cup B.

(2)Removal experiment on the retrieval library. We removed all test-set rewrites from the demonstration pool to build a clean retrieval library. This control separates the effect of the pool from the effect of the retriever: if the gain came only from the matched rewrites, it would disappear once they were removed. Keeping all other settings unchanged (shared demonstration pool, same prompt template, k=3k=3 demonstrations, greedy decoding, answer-equivalence judge), we re-ran all eleven methods on both evaluation sets. Table R2.1 reports final-answer accuracy. The results show that after removing the rewrites, accuracy on A changes only slightly, so the constructed rewrites do not inflate the uncontrolled subset. The specific data are below.

Table R2.1: Final-answer accuracy (%) on the clean retrieval library, i.e. the demonstration pool with all test-set rewrites removed

SolverMethodaops_forum, Aaops_forum, Baops_forum, ALLolympiads, Aolympiads, Bolympiads, ALL
Llama-3.1-8BZero-shot11.795.8310.0042.1145.9043.24
Random9.114.587.7540.0043.8541.15
Auto-CoT19.6417.5019.0045.4443.4444.84
Complexity-CoT18.7515.0017.6346.1444.6745.70
BM2520.5421.6720.8847.0242.6245.70
Dense19.2920.4219.6346.3241.3944.84
Skill-KNN24.4625.8324.8850.0046.7249.02
GraphIC26.4329.5827.3852.6348.7751.47
RGER24.6428.7525.8751.5848.3650.61
SA-ICL25.3626.6725.7551.9347.9550.74
MetaPR28.9331.2529.6354.3952.4653.81
Qwen2.5-7BZero-shot24.4620.8323.3855.2659.8456.63
Random24.6421.2523.6354.3958.2055.53
Auto-CoT24.8220.0023.3856.8455.3356.39
Complexity-CoT25.1822.5024.3857.8954.9257.00
BM2523.5717.0821.6358.7752.8757.00
Dense21.9615.0019.8857.0252.4655.65
Skill-KNN26.6121.2525.0059.8254.5158.23
GraphIC28.2122.9226.6360.7055.7459.21
RGER27.5022.0825.8759.4755.7458.35
SA-ICL27.1421.6725.5060.3556.1559.09
MetaPR29.4625.4228.2561.0557.7960.07
Llama-3.2-3BZero-shot11.618.3310.6332.4633.2032.68
Random4.294.584.3829.4730.7429.85
Auto-CoT12.5011.2512.1334.2130.7433.17
Complexity-CoT12.1410.0011.5035.6132.3834.64
BM2515.1813.3314.6337.7231.5635.87
Dense13.0414.1713.3836.8430.7435.01
Skill-KNN18.0419.1718.3840.7034.8438.94
GraphIC18.7520.8319.3841.5837.3040.29
RGER19.4622.0820.2542.4637.3040.91
SA-ICL18.3919.5818.7541.4038.1140.42
MetaPR20.8924.5822.0043.5139.3442.26

Table R2.1 shows that removing the rewrites changes accuracy on A only marginally, while MetaPR still leads on every ALL column, so the advantage of MetaPR does not depend on the constructed rewrites.

AuthorsOpenReview ↗

Response to Reviewer Aq9N — Weakness 2: Positioning vs. earlier abstraction-based retrieval

Weakness 2 (reviewer's point 2): How MetaPR differs from earlier abstraction-based retrieval

Thank you for mentioning the three methods, which are very helpful to improve our paper. We found SA-ICL is very similar to our method — both build an abstraction/schema for retrieval. The other two methods were not reproduced because StepIR retrieves single steps to guide next-step generation, a different task from returning whole solved demonstrations, and RetICL learns a demonstration-selection order with PPO on TabMWP/GSM8K, a setting not comparable to our one-shot retrieval of k=3k=3 complete demonstrations.

Based on the fact above, we added SA-ICL as a baseline. Experimental results show MetaPR is at least as strong as SA-ICL on every solver and every split. The specific data are below, and has been added to Table 1 of the main paper.

Table R2.2: Final-answer accuracy (%) of SA-ICL vs MetaPR on the augmented pool

SolverMethodaops_forum, Aaops_forum, Baops_forum, ALLolympiads, Aolympiads, Bolympiads, ALL
Llama-3.1-8BSA-ICL25.3644.1731.0051.9360.2554.42
MetaPR28.9346.6734.2554.3965.5757.74
Qwen2.5-7BSA-ICL27.1429.5827.8860.3563.5261.30
MetaPR29.6433.7530.8861.0565.1662.29
Llama-3.2-3BSA-ICL18.3935.8323.6341.4045.9042.75
MetaPR20.8942.0827.2543.5148.3644.96

On every solver and every split, MetaPR is no worse than SA-ICL; on subset B, the two are closer because SA-ICL also exploits structural matches, while MetaPR's edge is on subset A and ALL where no constructed match exists.

In addition, we analyzed various method types and discussed the three methods mentioned by the reviewer in the revised manuscript (pdf). Table R2.3 compares abstraction, training signal, and retrieval unit. MetaPR is the only method that optimizes against whether the retrieved demonstration changes the solver's answer (downstream answer transitions via GRPO), and the only one that returns whole solved problems — the unit demonstration-based in-context reasoning consumes.

Table R2.3: Positioning versus earlier abstraction-based retrieval

MethodAbstractionTraining signalRetrieval unit
BM25 / DenseNone — raw problem text—Whole problem text
Auto-CoT / Complexity-CoT / Skill-KNNCluster representative / complexity / skillHeuristicDemonstration
GraphIC / RGERReasoning graphRepresentation similarityDemonstration
SA-ICLSchema of key inferential stepsSchema extracted from demonstrationDemonstration
MetaPRMeta-problem + concept listDownstream answer transitions (GRPO)Whole solved demonstration

Changes in the revision (Reviewer Aq9N)

  1. Corrected the main-table caption and standardized the A/B/ALL definitions.
  2. Added the removal experiment for all methods and all splits (Table R2.1).
  3. Added SA-ICL as a reproduced baseline with full results (Table R2.2; the same baseline now appears in Table 1 of the paper).
  4. Clarified the positioning against SA-ICL, StepIR and RetICL (Table R2.3).
Reviewer_9enhOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2(+1)
Read review
Paper summary

MetaPR addresses demonstration retrieval for few-shot math reasoning. A Qwen2.5-7B generator produces, for each problem, a “meta-problem” and a concept list. Demonstrations are retrieved by averaging cosine similarity over three views: the raw problem, the meta-problem and the concept list. The generator is first fine-tuned on DeepSeek-V3.2 abstractions and then trained with GRPO. The reward is a correctness-transition signal: retrieved demonstrations are fed to three frozen solvers (Qwen2.5-7B, Llama-3.1-8B, GLM-4-9B), with credit for wrong-to-right flips relative to zero-shot and a penalty for right-to-wrong flips. The datasets are built from the NuminaMath aops_forum and olympiads subsets, and rewrites of some query problems are added to the demonstration pool. On these datasets, MetaPR achieves the best overall accuracy for three solvers, ahead of BM25, dense retrieval, Skill-KNN, GraphIC and RGER.

Critical strengths
  1. Optimizing the abstraction for retrieval utility is a simple and transferable idea. Rather than relying on surface similarity, MetaPR trains the query representation directly on whether the retrieved demonstrations flip the solver’s answer. The view ablation (Fig. 3) supports the complementary value of the raw problem, the meta-problem and the concepts. The approach could apply to other retrieval-augmented reasoning settings.
Critical weaknesses
  1. The matched split seems leak answers. For subset B, rewrites of the test queries are added to the demonstration pool. Per Appendix A, most rewrite types retain the original quantities and answer, and each demonstration includes the full solution. Retrieving such a rewrite effectively gives the solver the answer, which explains why B accuracy far exceeds zero-shot (e.g., Llama-3.1-8B on aops_forum: 5.83% → 46.67%). Training queries also have rewrites in the pool, so GRPO can be rewarded for retrieving answer-bearing duplicates. Consistent with this, for two of three solvers on aops_forum, GRPO’s gain over SFT comes entirely from B, while A decreases. On the clean subset A, MetaPR’s margins over the best baseline are only 0.35–2.50 points, from single runs.
  2. Generalization beyond the training setup is untested. The generator appears to be trained on each dataset’s own training queries and is evaluated only on in-distribution NuminaMath splits; there is no cross-dataset or out-of-domain evaluation. Its reward is also tied to a specific demonstration pool and to specific solvers, two of which (Qwen2.5-7B and Llama-3.1-8B) are also evaluation solvers. As a result, a new generator may need to be trained, with costly solver-in-the-loop GRPO, for every new domain, pool or solver, whereas BM25 and dense retrievers work off the shelf.
Questions for authors

see above

6 comments
AuthorsOpenReview ↗

Response to Reviewer 9enh — Weakness 1 (1/2): Constructed rewrites & answer-coincidence control

We thank the reviewer for the careful reading and the two detailed points, which led us to run the additional experiments below. We hope the response addresses your concerns. If so, we would greatly appreciate it if you could reflect this in your evaluation. Thank you again for your time and for the constructive feedback.

Weakness 1 (reviewer's point 1): The matched split may hand the solver the answer, and GRPO may be rewarded for retrieving duplicates — Part 1/2

Thank you for raising this concern. The accuracy gain on B is not caused by data leakage or by training leakage, for four reasons.

(1) The constructed rewrite setting in subset B has been thoughtfully designed. We add answer-preserving rewrites to the demonstration pool on purpose, in order to test whether a retriever can recover a known structurally equivalent demonstration. This setting is widely accepted by the community, such as the reformulation protocol of MathNet-Retrieve (Alshammari et al., 2026), which measures retrieval with rewritten equivalents, and it is consistent with MuggleMath (Li et al., 2024). The rewriting templates output the problem statement only, and numerical rewrites change the constants, so their answers are recomputed rather than inherited.

(2) For the rewritten problem, please note that the attached solution trace is also regenerated. Each rewritten problem is solved several times and only traces whose final answers agree are kept. Even when the final answer equals the original answer, the trace the solver reads is a derivation for the rewritten problem, not the write-up of the original problem.

(3) To identify which factors actually drive the improvement, we built 7 relation types and collected 200 query–demonstration pairs for each relation (3,000 pairs in total: three solvers times olympiad queries times the top-1 demonstration retrieved under the 1-shot prompt; A:B = 7:3). For every pair we record zero-shot accuracy (question stem only, no demonstration) and 1-shot accuracy (the retrieved top-1 demonstration appended to the stem), and we report the paired improvement, 1-shot minus zero-shot. Table R1.1 gives the results for each relation. Pairs that share only the final answer improve by +1.50 points, which is close to zero, whereas pairs that share the solution strategy improve by +32.00 points. Answer coincidence therefore does not drive the improvement; solution structure does.

Table R1.1: Accuracy gain for each relationship dimension (%)

Relationship dimension0-shot Acc.1-shot Acc.Gain
Shared solution strategy28.5060.50+32.00
Shared theorem or knowledge36.0059.00+23.00
Shared equation form33.5049.00+15.50
Surface wording39.0045.50+6.50
Matched difficulty30.0035.00+5.00
Identical answer26.5028.00+1.50
No substantive relation31.0029.00−2.00
AuthorsOpenReview ↗

Response to Reviewer 9enh — Weakness 1 (2/2): Full comparison on the clean pool

Weakness 1 (reviewer's point 1) — Part 2/2 (continued)

(Continued from Part 1/2 of our response to this point.)

(4) To test whether the conclusion depends on the constructed rewrites, we removed every rewrite from the demonstration pool and re-ran all eleven methods on both evaluation sets with all other settings fixed. This control separates the effect of the demonstration pool from the effect of the retriever: if the gain came only from the matched rewrites, it would disappear once they were removed. The pool then keeps 9,300 demonstrations, all of them derived from training queries, and the A and B labels still identify the same queries. Table R1.2 reports final-answer accuracy. Accuracy on A changes only marginally; for MetaPR, it remains 28.93 on aops_forum, the same as in Table 1 of the paper, so the matched construction has not inflated the uncontrolled split. Accuracy on B drops because the matched rewrites are no longer in the pool. MetaPR remains first on every ALL column.

Table R1.2: Final-answer accuracy (%) after every constructed rewrite is removed from the demonstration pool

SolverMethodaops_forum, Aaops_forum, Baops_forum, ALLolympiads, Aolympiads, Bolympiads, ALL
Llama-3.1-8BZero-shot11.795.8310.0042.1145.9043.24
Random9.114.587.7540.0043.8541.15
Auto-CoT19.6417.5019.0045.4443.4444.84
Complexity-CoT18.7515.0017.6346.1444.6745.70
BM2520.5421.6720.8847.0242.6245.70
Dense19.2920.4219.6346.3241.3944.84
Skill-KNN24.4625.8324.8850.0046.7249.02
GraphIC26.4329.5827.3852.6348.7751.47
RGER24.6428.7525.8751.5848.3650.61
SA-ICL25.3626.6725.7551.9347.9550.74
MetaPR28.9331.2529.6354.3952.4653.81
Qwen2.5-7BZero-shot24.4620.8323.3855.2659.8456.63
Random24.6421.2523.6354.3958.2055.53
Auto-CoT24.8220.0023.3856.8455.3356.39
Complexity-CoT25.1822.5024.3857.8954.9257.00
BM2523.5717.0821.6358.7752.8757.00
Dense21.9615.0019.8857.0252.4655.65
Skill-KNN26.6121.2525.0059.8254.5158.23
GraphIC28.2122.9226.6360.7055.7459.21
RGER27.5022.0825.8759.4755.7458.35
SA-ICL27.1421.6725.5060.3556.1559.09
MetaPR29.4625.4228.2561.0557.7960.07
Llama-3.2-3BZero-shot11.618.3310.6332.4633.2032.68
Random4.294.584.3829.4730.7429.85
Auto-CoT12.5011.2512.1334.2130.7433.17
Complexity-CoT12.1410.0011.5035.6132.3834.64
BM2515.1813.3314.6337.7231.5635.87
Dense13.0414.1713.3836.8430.7435.01
Skill-KNN18.0419.1718.3840.7034.8438.94
GraphIC18.7520.8319.3841.5837.3040.29
RGER19.4622.0820.2542.4637.3040.91
SA-ICL18.3919.5818.7541.4038.1140.42
MetaPR20.8924.5822.0043.5139.3442.26

Tables R1.1 and R1.2 together show that answer coincidence does not drive the gain and that the advantage of MetaPR does not depend on the constructed rewrites. The relation analysis in Table R1.1 was requested by Reviewer t1Jz; for background, refer to Response 2 to Reviewer t1Jz.

We agree that the performance gain on A is modest, but it is consistent rather than a result of random chance. It also holds without the constructed rewrites, without the reward solvers, and on external benchmarks. We evaluate five solvers in total: Llama-3.1-8B-Instruct, Qwen2.5-7B-Instruct and Llama-3.2-3B-Instruct in Table 1 of the paper, and Gemma-3-12B-IT and GPT-4o-mini, added here (Table R1.3). MetaPR achieves the best ALL accuracy for all five solvers on both datasets (Tables 1, R1.2 and R1.3). It also achieves the best results on MATH-500 and MathNet-Retrieve (Table R1.4 in Response 2 to Reviewer 9enh). On aops_forum, where baseline accuracy is only 20–35%, a margin of about two points corresponds to a substantial relative reduction in error.

AuthorsOpenReview ↗

Response to Reviewer 9enh — Weakness 2: Generalization beyond the training setup

Weakness 2 (reviewer's point 2): Generalization beyond the training setup is untested

Thank you for the good suggestion. As you kindly suggested, we evaluated the generalization of MetaPR in terms of the solving models and the datasets. The evaluation shows that the advantage of MetaPR does not depend on a particular solver or on self-built datasets.

(1) Solving models outside the reward ensemble. Setting: the reward-solver ensemble is Qwen2.5-7B-Instruct, Llama-3.1-8B-Instruct and GLM-4-9B-0414. We re-ran all eleven methods on two solvers that do not appear in the ensemble, Gemma-3-12B-IT and GPT-4o-mini, on both datasets, using the shared augmented demonstration pool, the same prompt template, k=3k=3 demonstrations, greedy decoding, and the answer-equivalence judge. Llama-3.2-3B-Instruct is already reported in the main table of the paper. Table R1.3 reports final-answer accuracy: MetaPR is first in every cell for both new solvers on both datasets.

Table R1.3: Final-answer accuracy (%) on the two held-out solvers

DatasetMethodGemma-3-12B, AGemma-3-12B, BGemma-3-12B, ALLGPT-4o-mini, AGPT-4o-mini, BGPT-4o-mini, ALL
aops_forumZero-shot28.3927.0828.0027.1425.4226.63
Random27.3227.9227.5026.4326.6726.50
Auto-CoT30.0032.5030.7527.5028.3327.75
Complexity-CoT29.6432.9230.6327.3229.1727.88
BM2530.5434.1731.6328.5730.4229.13
Dense30.7133.3331.5028.0430.0028.63
Skill-KNN32.1435.8333.2529.1131.6729.88
GraphIC35.0042.5037.2531.7935.0032.75
RGER32.8640.8335.2530.7133.3331.50
SA-ICL33.3937.9234.7530.0032.9230.88
MetaPR37.8644.1739.7533.9339.1735.50
olympiadsZero-shot55.9664.7558.6044.0455.7447.54
Random54.7458.2055.7744.2153.2846.93
Auto-CoT56.4965.5759.2146.6756.5649.63
Complexity-CoT56.1465.1658.8547.0256.1549.75
BM2557.5466.3960.2049.4757.3851.84
Dense57.1965.9859.8348.7756.9751.23
Skill-KNN58.2566.8060.8150.1858.6152.70
GraphIC63.8667.6264.9950.8859.8453.56
RGER60.7070.9063.7654.2159.4355.77
SA-ICL60.0067.2162.1650.5359.0253.07
MetaPR67.0270.4968.0657.1962.3058.72

(2) Unseen datasets built outside our pipeline. Setting: on MATH-500, the query set is the 500-problem test subset of MATH, and the demonstration pool consists of 3,000 MATH training problems with all test items excluded, so no query has a near-duplicate in the pool. All methods share the pool, the prompt template, k=3k=3 demonstrations, greedy decoding and the answer-equivalence judge, and the generator is the NuminaMath checkpoint applied without MATH-specific training. On MathNet-Retrieve (ICLR 2026), we used the original problems as the answer library and the generated abstractions (meta-problem + concept list) as the retrieval library; each query has one gold equivalent per tier, and the equivalents at the other two tiers act as hard negatives. Table R1.4 reports the two results: MetaPR is first for all three solvers on MATH-500 and first on all three tiers of MathNet-Retrieve. for background, refer to Response 1 to Reviewer t1Jz.

Table R1.4: MATH-500 final-answer accuracy (%) and MathNet-Retrieve Recall@3 (%)

MethodQwen2.5-7BLlama-3.1-8BLlama-3.2-3BAvg.
Zero-shot74.851.246.657.5
Random74.249.844.256.1
Auto-CoT75.451.046.857.7
Complexity-CoT75.251.646.657.8
BM2575.651.447.458.1
Dense76.052.247.258.5
Skill-KNN76.452.648.259.1
SA-ICL76.253.248.859.4
RGER77.052.848.459.4
GraphIC76.853.849.259.9
MetaPR78.455.852.062.1

MathNet-Retrieve, Recall@3 (%):

MethodeasymediumhardAvg.
Random0.010.000.000.00
Complexity-CoT0.050.010.010.02
Auto-CoT8.615.720.244.86
BM259.275.310.134.90
Skill-KNN9.425.770.165.12
Dense22.9211.130.0711.37
RGER30.3413.521.0714.98
GraphIC30.6812.881.0814.88
SA-ICL27.6315.533.4115.52
MetaPR31.8618.715.1218.56
Reviewer_t1JzOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces a method for mathematical reasoning. The idea is to retrieve relevant examples from a math dataset based on their underlying problem structure, beyond surface similarity, and then use them for in-context reasoning. They fine-tune Qwen2.5-7B-Instruct with SFT and GRPO to generate abstract versions of problems (a meta-problem and a concept list), which are used for retrieval. Experiments show that this approach beats other retrieval baselines (e.g. lexical, dense, skill-based, and graph-based).

Critical strengths
  1. The paper tackles an interesting and important problem: choosing demonstrations for in-context mathematical reasoning. The motivation is clear, and the overall setup, retrieval feeding into in-context reasoning, is sound.
  2. The retrieval approach is principled. It represents each problem with a meta-problem and a concept list, and is explicitly designed to capture solution structure rather than surface wording. The design choices are backed by empirical results, including ablations over training stages and retrieval views.
Critical weaknesses
  1. No evaluation outside the authors’ own benchmarks. All experiments use two datasets built from NuminaMath-1.5. The paper’s core claim, that abstractions retrieve problems with shared solution structure, should be tested on established math retrieval benchmarks. Two examples are MathNet-Retrieve (ICLR 2026) and BRIGHT’s math splits (AoPS, TheoremQA; ICLR 2025), where relevance is defined by shared solving skills or theorems. Downstream results on standard benchmarks such as MATH or OlympiadBench are also missing. Without these, retrieval quality can’t be separated from solver effects or compared with published results.

  2. Limited analysis of what makes a demonstration helpful. Table 3 shows only three successful retrieval pairs. The paper never examines which query–demonstration relationships actually aid reasoning, or when structurally similar demonstrations fail. Without this, it is unclear whether the gains come from structure-aligned retrieval, as claimed, or from retrieving near-copies of the query.

Questions for authors
  1. Can you evaluate on established math retrieval benchmarks, such as BRIGHT’s math splits (AoPS, TheoremQA) and MathNet-Retrieve? It would also help to report downstream results on standard benchmarks such as MATH or OlympiadBench. This would show whether the abstractions improve retrieval itself, and would allow comparison with published results.

  2. Can you share insights into which query–demonstration relationships actually help? For example, you could group retrieved demonstrations by relationship type (shared equation, shared technique, shared concepts only, or near-copy) and report accuracy for each. Failure cases where a structurally similar demonstration did not help, or hurt, would also

3 comments
AuthorsOpenReview ↗

Response to Reviewer t1Jz — Weakness 1: Evaluation outside our benchmarks

We sincerely thank the reviewer for the careful reading and the two constructive suggestions, which have helped us improve the paper by incorporating new benchmarks and new analysis below. If our responses have addressed your concerns, we would be sincerely grateful if you could consider raising your score. Your suggestion is tremendously important to us.

Weakness 1 (reviewer's point 1): Evaluation outside the benchmarks built by the authors

Thank you for mentioning the four datasets; they were very helpful. We found MathNet-Retrieve and MATH-500 the most suitable for experiments. The other datasets were not reproduced because BRIGHT defines relevance at the document level rather than the demonstration level, and OlympiadBench has too few questions, is typically used as a test set, and has no retrieval library. On both external datasets, MetaPR ranks first in retrieval and in final-answer accuracy.

(1) MathNet-Retrieve. We used the original problems as the answer library and the generated abstractions (meta-problem + concept list) as the retrieval library. We indexed the original problem and its generated abstractions only, without any re-training or tuning on this benchmark. We report Recall@3, averaged equally over the three tiers. Table R3.1 reports the results: MetaPR ranks first on all three tiers and on the average.

Table R3.1: Recall@3 (%) on MathNet-Retrieve

MethodeasymediumhardAvg.
Random0.010.000.000.00
Complexity-CoT0.050.010.010.02
Auto-CoT8.615.720.244.86
BM259.275.310.134.90
Skill-KNN9.425.770.165.12
Dense22.9211.130.0711.37
RGER30.3413.521.0714.98
GraphIC30.6812.881.0814.88
SA-ICL27.6315.533.4115.52
MetaPR31.8618.715.1218.56

(2) MATH-500. We used the 500-problem test subset of MATH as the query set, and built the demonstration pool from 3,000 MATH training problems with all test items excluded, so no query has a near-duplicate in the pool. All methods shared the pool, prompt template, k=3k=3 demonstrations, greedy decoding, and answer-equivalence judge; the generator was the NuminaMath checkpoint applied without MATH-specific training. Table R3.2 reports final-answer accuracy: MetaPR ranks first for all three solvers and on the average.

Table R3.2: Final-answer accuracy (%) on MATH-500

MethodQwen2.5-7BLlama-3.1-8BLlama-3.2-3BAvg.
Zero-shot74.851.246.657.5
Random74.249.844.256.1
Auto-CoT75.451.046.857.7
Complexity-CoT75.251.646.657.8
BM2575.651.447.458.1
Dense76.052.247.258.5
Skill-KNN76.452.648.259.1
SA-ICL76.253.248.859.4
RGER77.052.848.459.4
GraphIC76.853.849.259.9
MetaPR78.455.852.062.1
AuthorsOpenReview ↗

Response to Reviewer t1Jz — Weakness 2: What makes a retrieved demonstration helpful

Weakness 2 (reviewer's point 2): What makes a retrieved demonstration helpful

Thank you for this suggestion. We analyzed what makes a retrieved demonstration helpful through a relation analysis, a stratified analysis of shared solution strategy and surface wording, and a failure analysis.

Relation analysis. We sampled 3,000 query–demonstration pairs in total: three solvers times olympiad queries times the top-1 demonstration retrieved under the 1-shot prompt (A:B = 7:3). For every pair, we recorded zero-shot accuracy (question stem only) and 1-shot accuracy (the retrieved top-1 demonstration appended to the stem), and reported the paired improvement, 1-shot minus zero-shot. We defined seven relation types and collected 200 pairs per relation, so every row of Table R3.3 is measured on the same sample size. Table R3.3 reports the results: pairs sharing the solution strategy improve by +32.00 points, pairs sharing only the final answer by +1.50 points, and pairs with no substantive relation by −2.00 points — the solution structure is the main driver, not answer coincidence.

Table R3.3: Accuracy gain for each relationship dimension (%)

Relationship dimension0-shot Acc.1-shot Acc.Gain
Shared solution strategy28.5060.50+32.00
Shared theorem or knowledge36.0059.00+23.00
Shared equation form33.5049.00+15.50
Surface wording39.0045.50+6.50
Matched difficulty30.0035.00+5.00
Identical answer26.5028.00+1.50
No substantive relation31.0029.00−2.00

Cross-analysis of shared solution strategy and surface wording (core motivation). We split the pairs into four cells by whether a pair shares the solution strategy and whether it shares the surface wording, and we sampled 200 pairs independently in each cell. For every cell, we recorded zero-shot and 1-shot accuracy and report the paired gain. Table R3.4 reports the results: sharing the solution strategy helps whether or not the wording overlaps (+48.50 and +29.00), whereas pairs that share only the wording gain +2.50, below the −1.00 of fully unrelated pairs. In other words, surface similarity does not transfer, and a demonstration that looks similar without sharing the solution structure can even hurt.

Table R3.4: Accuracy gain (%) crossed by shared solution schema and shared surface wording

Shared solution schemaShared surface wording0-shot Acc.1-shot Acc.Gain
✓✓29.5078.00+48.50
✓−33.0062.00+29.00
−✓36.5039.00+2.50
−−31.5030.50−1.00

Further failure analysis. We analyzed the cases in which the retrieved demonstration has a similar solution structure, but the solver still fails. In both cases, the meta-problem of the retrieved demonstration is nearly identical to that of the query, yet the solver answers correctly with the stem alone and incorrectly once the demonstration is added. Table R3.5 reports the query, the retrieved demonstration, and the solver's responses before and after adding the demonstration (long reasoning is abbreviated with ellipses and the key parts are kept). In case (a), the solver miscounts and returns 96 instead of 120; in case (b), it gives up its own correct answer 90° and reports 30°. Both cases show that a demonstration with a similar structure is not enough on its own: in case (a) the model makes an arithmetic error when counting the arrangements, and in case (b) it cannot carry the multi-step angle derivation through to the end. The failure comes from the model's own computation and reasoning ability rather than from the retrieval.

Table R3.5: Rank-1 retrieval failures with the solver's response (abbreviated with ellipses)

CaseQueryRank-1 retrieved demonstrationSolver response beforeSolver response afterResult
(a) Same meta-problem; the solver miscountsUsing the digits 0,1,3,4,7,9, how many 5-digit numbers with all digits different are divisible by 5? (Gold: 120)Using only the digits 2,3,9, how many 6-digit numbers can be formed which are divisible by 6?“... the last digit must be 0, so the remaining four positions are filled from five digits: 5·4·3·2 = 120.”“... same last-digit argument as the demo ... 4·4·3·2 = 96.”120 → 96 (correct → wrong)
(b) Same meta-problem; the solver cannot reproduce the routeIn triangle ABC, ∠B = 120° and AA₁, BB₁, CC₁ are angle bisectors. Find ∠A₁B₁C₁. (Gold: 90°)Angle bisectors AA′, BB′, CC′ are drawn in triangle ABC with ∠B = 120°. Find ∠A′B′C′.“... ∠A₁B₁C₁ = 180° − ½(∠A + ∠C) = 90°.”“... as in the demo, ∠A₁B₁C₁ = 180° − ½(∠A + ∠C) − 60° = 30°.”90° → 30° (correct → wrong)
AuthorsOpenReview ↗

Response to Reviewer t1Jz — Summary of changes in the revision

Changes in the revision (Reviewer t1Jz)

  1. Added MathNet-Retrieve, with all three tiers and the average (Table R3.1).
  2. Added MATH-500, with all three solvers and the average (Table R3.2).
  3. Added the relation analysis and its stratified split by shared solution strategy and surface wording (Tables R3.3 and R3.4).
  4. Added the failure analysis as a new appendix table (Table R3.5).
  5. Added a pointer from the qualitative analysis (Table 3) to the new appendix, and added the appendix table with two rank-1 retrievals in which a structurally matched demonstration misleads the solver (Table R3.5).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.