- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
MetaPR uses generated abstractions to retrieve demonstrations for mathematical reasoning. A generator produces a meta-problem and a concept list; a frozen embedding model encodes these and the original question. Retrieval combines the three similarity views with equal weights to select problems and solutions for a downstream solver. The generator is initialized with supervised training, then optimized with GRPO using correctness-transition rewards from a solver ensemble and format/length terms. Evaluation covers two mathematical datasets, several solvers, view and training ablations, and a controlled rewrite-retrieval metric. The shared demonstration pool contains solved, structure-preserving rewrites of selected training and test queries. Approximately 30% of test queries have deliberately constructed matches in this pool.
Critical strengths
-
Using solution-relevant structure to go beyond topical similarity is a sensible direction. The three-view ablation shows that the representations provide complementary information: the full version improves on the best two-view variant by 1.75-2.50 accuracy points on aops forum. This supports the interface within the reported pool construction.
-
The training pipeline produces measurable gains. With Qwen-7B, moving from SFT to GRPO raises ALL accuracy from 28.00 to 30.88 on aops forum and from 60.81 to 62.29 on olympiads. The tests include multiple solvers, one outside the reward ensemble, and controlled Hit@3 results. These support the pipeline beyond qualitative examples, although they do not isolate the transition reward.
Critical weaknesses
-
The shared pool limits what can be concluded about generalization. The authors deliberately add solved rewrites of selected test questions to the demonstration pool. About 30% of test queries have constructed matches, and the remaining queries use that same augmented pool. This disclosed design tests recovery of structurally related examples, but subset A still does not test retrieval from an independently built natural pool. The concern is the scope of the evaluation; this construction does not establish that test answers were used to train the generator.
-
The advance over earlier abstraction-based retrieval needs to be defined more precisely. Skill-KNN and GraphIC already provide relevant precedents. This version also omits https://arxiv.org/abs/2510.13905, https://arxiv.org/abs/2305.14502, and https://github.com/qpalzm36/StepIR. Their abstractions, selection procedures and retrieval granularity differ, but together they narrow the contribution to the specific trained meta-problem/concept interface.
Questions for authors
Does the advantage persist with an unaugmented demonstration pool drawn from training data and separated by source problem family? Please report the main comparisons in that setting alongside the controlled rewrite results, and correct or confirm the reversed A/B labels in the main-table captions.
2 comments
Response to Reviewer Aq9N — Weakness 2: Positioning vs. earlier abstraction-based retrieval
Weakness 2 (reviewer's point 2): How MetaPR differs from earlier abstraction-based retrieval
Thank you for mentioning the three methods, which are very helpful to improve our paper. We found SA-ICL is very similar to our method — both build an abstraction/schema for retrieval. The other two methods were not reproduced because StepIR retrieves single steps to guide next-step generation, a different task from returning whole solved demonstrations, and RetICL learns a demonstration-selection order with PPO on TabMWP/GSM8K, a setting not comparable to our one-shot retrieval of complete demonstrations.
Based on the fact above, we added SA-ICL as a baseline. Experimental results show MetaPR is at least as strong as SA-ICL on every solver and every split. The specific data are below, and has been added to Table 1 of the main paper.
Table R2.2: Final-answer accuracy (%) of SA-ICL vs MetaPR on the augmented pool
| Solver | Method | aops_forum, A | aops_forum, B | aops_forum, ALL | olympiads, A | olympiads, B | olympiads, ALL |
|---|---|---|---|---|---|---|---|
| Llama-3.1-8B | SA-ICL | 25.36 | 44.17 | 31.00 | 51.93 | 60.25 | 54.42 |
| MetaPR | 28.93 | 46.67 | 34.25 | 54.39 | 65.57 | 57.74 | |
| Qwen2.5-7B | SA-ICL | 27.14 | 29.58 | 27.88 | 60.35 | 63.52 | 61.30 |
| MetaPR | 29.64 | 33.75 | 30.88 | 61.05 | 65.16 | 62.29 | |
| Llama-3.2-3B | SA-ICL | 18.39 | 35.83 | 23.63 | 41.40 | 45.90 | 42.75 |
| MetaPR | 20.89 | 42.08 | 27.25 | 43.51 | 48.36 | 44.96 |
On every solver and every split, MetaPR is no worse than SA-ICL; on subset B, the two are closer because SA-ICL also exploits structural matches, while MetaPR's edge is on subset A and ALL where no constructed match exists.
In addition, we analyzed various method types and discussed the three methods mentioned by the reviewer in the revised manuscript (pdf). Table R2.3 compares abstraction, training signal, and retrieval unit. MetaPR is the only method that optimizes against whether the retrieved demonstration changes the solver's answer (downstream answer transitions via GRPO), and the only one that returns whole solved problems — the unit demonstration-based in-context reasoning consumes.
Table R2.3: Positioning versus earlier abstraction-based retrieval
| Method | Abstraction | Training signal | Retrieval unit |
|---|---|---|---|
| BM25 / Dense | None — raw problem text | — | Whole problem text |
| Auto-CoT / Complexity-CoT / Skill-KNN | Cluster representative / complexity / skill | Heuristic | Demonstration |
| GraphIC / RGER | Reasoning graph | Representation similarity | Demonstration |
| SA-ICL | Schema of key inferential steps | Schema extracted from demonstration | Demonstration |
| MetaPR | Meta-problem + concept list | Downstream answer transitions (GRPO) | Whole solved demonstration |
Changes in the revision (Reviewer Aq9N)
- Corrected the main-table caption and standardized the A/B/ALL definitions.
- Added the removal experiment for all methods and all splits (Table R2.1).
- Added SA-ICL as a reproduced baseline with full results (Table R2.2; the same baseline now appears in Table 1 of the paper).
- Clarified the positioning against SA-ICL, StepIR and RetICL (Table R2.3).
Response to Reviewer Aq9N — Weakness 1: Shared pool & corrected A/B convention
We thank the reviewer for the careful reading and the concrete suggestions, which helped us fix the caption and add the controls below. If you feel that your concerns have been adequately addressed, we kindly ask if you would consider raising your rating. We truly appreciate your valuable suggestions.
Weakness 1 (reviewer's point 1): Whether the shared demonstration pool limits the generalization conclusion
Thank you for carefully pointing out the swapped caption and for asking for results that separate the pool from the retrieval method.
(1)Caption and convention. The main-table caption had swapped A and B; it is now corrected. Throughout the paper, we use one convention: A is the uncontrolled test subset, B is the matched test subset (test queries whose constructed rewrite was added to the demonstration pool), and ALL is their union, A∪B.
(2)Removal experiment on the retrieval library. We removed all test-set rewrites from the demonstration pool to build a clean retrieval library. This control separates the effect of the pool from the effect of the retriever: if the gain came only from the matched rewrites, it would disappear once they were removed. Keeping all other settings unchanged (shared demonstration pool, same prompt template, k=3 demonstrations, greedy decoding, answer-equivalence judge), we re-ran all eleven methods on both evaluation sets. Table R2.1 reports final-answer accuracy. The results show that after removing the rewrites, accuracy on A changes only slightly, so the constructed rewrites do not inflate the uncontrolled subset. The specific data are below.
Table R2.1: Final-answer accuracy (%) on the clean retrieval library, i.e. the demonstration pool with all test-set rewrites removed
Table R2.1 shows that removing the rewrites changes accuracy on A only marginally, while MetaPR still leads on every ALL column, so the advantage of MetaPR does not depend on the constructed rewrites.