acceptodds
Under review as a conference paper at ICLR 2027

Recipe-Matching, Not Equivalence

Abstract

MathNet-Retrieve tests whether a retriever can find, for a math problem, a document that states the same problem. It builds those documents itself: an LLM under one fixed prompt writes each gold document and its near-miss distractors, and LLM judges filter the result. We call that procedure the benchmark's recipe, training on pairs built the same way recipe-matching, and ask how much score it buys beyond the ability the benchmark claims to test. Two models trained from the same base on the same number of rows under the same settings differ only in the training file: pairs written with the benchmark's published prompt by a different vendor's LLM and judge, or pairs a computer-algebra system verified with no LLM anywhere. The first beats the second by R@1 points on the easy tier. As far as a non-LLM paraphrase control can tell, half to two thirds of that gap comes from the training pairs being LLM-written at all: LLM rewrites under two prompts unrelated to the benchmark, with the verified model's negatives, recover and of the points, while back-translated paraphrases with the same negatives recover almost none. The remaining to points, depending on the comparison, appear only under the benchmark's own prompt, and they vanish on real duplicates no generator wrote, the same problem printed in two languages. The hard tier rewards the recipe's pair structure, a deep rewrite against a minimal-edit near-miss: LLM rewrites alone score zero on it, attaching negatives unlocks it, and every kind of negative that does so costs cross-language points against the same rewrites trained alone; the training sets that score highest on it discriminate near-misses no LLM wrote worse than plain LLM rewrites with verified negatives. MELD also moves when a model trains on pairs built its way, without losing retention; on SABER-Math the registered attack fails, and the one gain, from training on its LLM-written summaries, is small and confounded; only on MathNet-Retrieve could we pin an inversion, benchmark score up and real retention down, to one edit of a training file. We release the generator-free duplicate evaluations, the near-miss test and three trained models.

Primary area:
datasets and benchmarks
Keywords:
benchmark validity, synthetic benchmarks, mathematical retrieval, dense retrieval, text embeddings, shortcut learning, data contamination, LLM-generated data, evaluation methodology, benchmark gaming

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_6H2kOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper identifies duplicated math problems generated by LLMs. More specifically, it may be possible to detect the LLM procedure and solve the pattern rather than solving the actual problem. The authors call this phenomenon "recipe-matching". MathNet-Retrieve is used as a benchmark because a three-tier system is used to disguise gold documents given a corpus and a query. The authors train an embedding model based on Qwen3 on LLM-generated pairs against a symbolically verified SymPy baseline. The proposed method achieved ~45R@1 advantage on the easy tier and 9.42 R@1 on the hard tier. Similar trends are observed on the Olympiad dataset.

This paper is extremely hard to read and understand, and it has 54 pages of Appendix. Given that I already spent a significant amount of time trying to understand it and I still don't, I decided I will not spend more time on it as long as the manuscript hasn't been entirely revised. I submit this review after having discussed with the ACs.

Critical strengths

N/A

Critical weaknesses

Poorly written, hard to understand.

Questions for authors

Please review the entire manuscript.

2 comments

Thank you for your time, and we're truly sorry the paper was hard to read.

What the paper shows, in plain terms. We do not propose a retrieval method. We audit a benchmark. MathNet-Retrieve writes its own test documents with an LLM under one fixed prompt. We show that a retriever trained on pairs written the same way gains about 45 points on the benchmark but under 10 on real duplicates written by people, and none on same-language reprints. The 45 points are the symptom we study, not a contribution.

What we changed for readability:

  • The abstract, Sections 1 and 3 and the conclusion are rewritten in plain language; they are red in the revised PDF.
  • A summary of the three findings ends Section 1.
  • A reader's guide opens the appendix, mapping each claim to the section that supports it.

We would be grateful if you could look at the revised Section 1, and we are looking forward to your feedback.

A brief follow-up: the paper does not propose a retriever. The 45-point gain is the flaw we expose. It comes from matching the benchmark’s LLM writing style, and it nearly vanishes on real duplicates. The revised Section 1 (in red) explains this in plain language, and we would be grateful if you would take a look and consider revisiting your assessment.

Reviewer_4ni6OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies whether mathematical retrieval benchmarks reward matching their data-generation procedure. It compares retrievers trained on recipe-generated and CAS-verified pairs, using MathNet-Retrieve, organic duplicates, and generator-free near-misses. Prompt variations, training-data ablations, and experiments on additional benchmarks examine whether benchmark gains reflect general equivalence retrieval.

Critical strengths

The main comparison uses eight seeds and includes substantial prompt, negative-sampling, and contamination controls. These help establish that the reported performance differences are repeatable.

The organic duplicate and generator-free near-miss evaluations are useful additions. They test whether benchmark gains carry over beyond the benchmark’s generation pipeline.

Critical weaknesses

Benchmark gains and worse performance on organic duplicates do not establish “recipe-matching, not equivalence.” Task-specific learning can coexist with forgetting. The back-translation control also produces shallower paraphrases, leaving LLM authorship confounded with transformation depth. The experiments demonstrate a distribution-dependent trade-off more clearly than they establish its proposed explanation.

The hard-tier interpretation has an additional labeling confound. MathNet-Retrieve treats the query’s equivalent reformulations at the other tiers as negatives. Low hard-tier R@1 can therefore mean that another equivalent was retrieved. This needs to be accounted for before attributing the floor mainly to near-misses and recipe structure.

minor: I also suggest the authors to include a more detailed literature review and make the title more descriptive. I can't tell what the paper is about when seeing the title.

Questions for authors

Do the reported per-tier metrics count equivalent reformulations from the other tiers as negatives? If so, how do the main comparisons change when all known equivalents count as positives, or when the other-tier equivalents are removed from the candidate corpus?

Which results distinguish recipe-matching from ordinary task adaptation accompanied by forgetting? In particular, what supports attributing the gains to LLM authorship beyond the paraphrase depth covered by the back-translation controls?

2 comments

Thank you; both points changed the paper (blue in the revised PDF).

Q1 (other-tier equivalents count as negatives). Yes. MathNet-Retrieve's official protocol, which our harness reproduces, has one gold per query, and the three tiers share a corpus. We rescored all 109 runs from their rank dumps two ways. With the query's own easy and medium rewrites removed:

  • For the untrained base and the verified arm nothing changes (0.00→0.02, 0.16→0.54). One of the query's own near-misses holds rank 1 on 88% and 76% of queries, so the floor really is the near-misses.
  • For models trained on LLM rewrites, the official metric hides near-miss discrimination: recipe arm 9.42→34.03. On 69% of the queries it misses, rank 1 is the query's own easy rewrite.
  • Every hard-tier comparison keeps its direction except two, now corrected in the text. The reference trails the recipe arm (25.24 vs 34.03, difference 8.79 [7.46, 10.15]); and D4 with the recipe's near-misses leads the recipe arm 1.26×, not about 3×.
  • Counting any of the three rewrites as a hit gives a single "equivalent above near-misses" score: base 10.1, verified 20.4, reference 61.4, recipe 80.7.

All three scorings are now in Table 1 and the new Appendix D.1.

Q2a (recipe-matching vs task adaptation plus forgetting). Three results separate them:

  1. Fixed checkpoints. On the same queries with only the golds rewritten, each model's forgetting is fixed, yet the winner follows the prompt that wrote the golds (D3 −0.19 vs +11.28 against the recipe arm).
  2. Same task, same negatives. Swapping D4's positives for the benchmark prompt's raises the benchmark score by 24.59 and lowers cross-language retrieval. Ordinary task learning would move both the same way.
  3. Forgetting taken out. The appendix's WiSE-FT soups blended only the verified arm with the base. We now blend all three arms with the base at the same weights. At α = 0.25 and 0.5, where the recipe and reference blends keep the base's cross-language retrieval, the recipe blend leads the reference blend by 19.7 and 23.2 points on the benchmark and by −0.8 and 0.0 on real cross-language duplicates (95% intervals over queries: [−2.8, 1.3] and [−4.1, 4.3]). Against the verified arm's blends it does lead on duplicates, by 8.4 and 5.1, under a third of its benchmark lead (29.2 and 46.1). We fixed this test before the soups ran.

So LLM-written positives do teach some real retrieval, and we now say so. What the benchmark's own recipe adds on top of them does not transfer.

Q2b (authorship beyond back-translation depth). At back-translation depth, LLM paraphrases with the same negatives score 26.38 easy R@1 against back-translation's 18.81. Authorship therefore matters at that depth, for about a quarter of the reference's gain over the verified arm. Part of it is general: the same model also finds real cross-language duplicates better (89.4 vs 84.6).

Minor. We added the work showing that neural retrievers prefer LLM-written text (Dai et al., KDD 2024; Cocktail, Findings of ACL 2024). The paper now carries a descriptive title, Synthetic Math-Retrieval Benchmarks Reward Matching Their Own Generation Recipe.

I think the authors' response solved most of my concerns. Given the nature that this paper can serve as an useful toolbox for future research to refer to, I will raise my score to 3. My further suggestions include 1) further polishing the title, the current one is still difficult to understand although is much better than the previous version and 2) adding a opening diagram/illustration/teaser at the beginning of the paper, so that it can be reader-friendlier.

Reviewer_gJtKOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates how training a math retrieval model on synthetic data generated via a benchmark's exact construction pipeline ("recipe-matching") inflates benchmark scores without improving general mathematical equivalence retrieval. Training the same base retriever on LLM-generated pairs following MathNet-Retrieve's recipe beats training on Computer Algebra System (CAS) verified pairs by 45.33 R@1 points on the benchmark's easy tier across 8 seeds. The authors run extensive controls across positive rewrites, negative sampling, prompts, LLM judges, and dataset volume. Crucially, they evaluate models on human-written reprints and a generator-free near-miss test, showing that the recipe-trained model suffers rank reversals and retention drops on independent evaluations. They also test MELD (where recipe gains preserve retention) and SABER-Math (where a pre-registered document attack fails).

Critical strengths
  • The ablation grid is remarkably thorough and intellectually honest. The authors control for negative counts, dataset size, source contamination, judge models, prompt variations, and retriever backbones, showing that a 43–44 point easy-tier gap survives even after tightening their contamination filter. The factorial experiments uncovering the strong interaction between positive and negative construction are great, and I really respect that the authors drop their initial "prompt-distance" hypothesis when the controls don't support it (Tables 8, 11–12).

  • The external evaluations on human reprints and the held-out near-miss probe make the paper's core warning much more compelling. On the near-miss test, the D4 model trained with full recipe negatives gets 26.97 R@1 on the benchmark's hard tier but only 30.50 R@1 on the full-corpus external probe, whereas the baseline gets 10.30 on the benchmark and 53.51 on the probe. Showing that retention drops even on same-language human reprints also rules out the excuse that the model simply forgot cross-lingual alignment.

Critical weaknesses
  • The paper's quantitative breakdown assigning a specific share of the gain to "LLM authorship" isn't well supported by the back-translation control. The single-hop back-translation positives stay much closer to the source text than the D4 LLM rewrites (character 3-gram overlap of 0.783 vs. 0.421), and chaining three translation hops (0.640 overlap) degrades text quality, which the authors admit doesn't tighten the control. Because rewrite depth isn't matched, you can't attribute the residual gap purely to LLM authorship. Furthermore, the effect of using recipe positives depends heavily on the negatives: switching from D4 to recipe positives adds only 3.34 easy-tier points when paired with CAS negatives, but adds 24.59 points when paired with recipe negatives. The abstract and conclusion should frame these as conditional interactions rather than an additive partition between authorship and recipe effects.

  • The title and framing ("Recipe-Matching, Not Equivalence") overreach slightly beyond what the experiments prove. Specializing on any narrow training distribution can cause forgetting on out-of-distribution tests, so a drop on human reprints doesn't automatically prove that the benchmark wins bypass mathematical equivalence. The human reprints don't include constructed near-misses, the CAS probe mostly tests shallow variable renaming, and the recipe-trained model actually beats the reference baseline on that CAS probe anyway. Meanwhile, there is no human audit of the 1,401 hard-tier queries that only the recipe arm got right. Finally, calling the CAS arm strictly "verified" needs some qualification, since some positives rely on numerical spot-checks and some negative counterexamples ignore variable domain constraints stated in the problem text.

Questions for authors
  • Would you be willing to revise the abstract and conclusion to report the authorship and prompt-recipe effects as conditional contrasts (explicitly highlighting the positive–negative interaction) rather than an additive decomposition?

  • What does a human audit of a stratified sample of the 1,401 unique hard-tier wins for the recipe arm show, are those retrieved targets genuinely mathematically equivalent or flawed? Also, how much do the CAS labels change if counterexamples are required to respect the full problem's domain restrictions?

  • In Table 8, you apply the corrected contamination filter to the main two-arm comparison. Do the key D1-vs-D4 comparisons with identical negatives and the factorial cells also hold up under the corrected contamination gate?

3 comments

Thank you for the careful reading.

Q1 (conditional contrasts). Yes, and done. The abstract, introduction and conclusion now give the contrasts with the other factor held fixed. With the verified negatives fixed, LLM-written positives from unrelated prompts recover 30 and 22 of the 45 easy-tier points. The benchmark prompt's positives add 3.34 points with the verified negatives and 24.59 with its own near-misses, an interaction of 21 points. We no longer describe the gap as a sum of shares.

Q2a (human audit of the 1,401 hard-tier wins). Done. One of the authors read 100 queries: 50 wins, sampled from the 847 whose query matches a topic-labelled problem, and 50 others. The gold was judged equivalent for 46 of the 50 wins (92%; 95% interval 81 to 97), or 42 if the 4 golds marked equivalent with a caveat count as flawed, and for 47 of the other 50. No near-miss was judged equivalent (0 of 300). Each flawed gold is a rewrite with a condition dropped or changed. So most of the sampled targets are genuinely equivalent, in one annotator's judgment. The audit does not show why the arm wins them (Section 7, Appendix C).

Q2b (domains of the CAS labels). We re-checked all 9,405 distinct negatives in the training file, requiring the counterexample to respect the type and range the problem states for each variable (integer, positive or bounded, read from the text; every certificate verified exactly or to 50 digits). Relations between variables, such as a+b+c=1, are not imposed (Appendix A).

  • 98.0% have one, and 0.6% cannot be checked (mostly matrix problems).
  • Of the 1.4% without one, four in five agree only vacuously: both statements hold nowhere on the domain (as in "show there is no integer solution") or both hold everywhere.
  • Retraining the verified arm without the unconfirmed 1.4% moves its easy-tier R@1 by +0.36 (17.74 vs 17.38 for a same-environment rerun, 3 seeds; the band fixed in advance was ±2.37).

On positives, only 2 of 14,361 rest on a numeric spot-check alone. Renamings (37%) are checked by parsing them back; algebraic rewrites (63%) are tested at 20 random complex points, and all but 2 are also confirmed by SymPy simplification. Section 3 now says exactly what "verified" means.

Q3 (corrected gate for the D1-vs-D4 comparisons and the factorial). We checked this two ways.

  1. Clean-slice test on every cell. We applied the paper's own clean-slice test to every factorial cell (new Appendix table). Every contrast keeps its sign on every slice except back-translation's, which stays within noise of zero (1.39 ± 1.77); every step of the ladder keeps its sign, seed by seed. On the slice where neither cell trained on a copy of any query:

    • D1 vs D4 at the recipe's own near-misses is 22.31 (24.07 on all queries; 1.54 points is leakage);
    • D1 vs D4 at verified negatives is 3.68 (3.60);
    • the recipe arm's lead over the reference is 14.23 (15.47).
  2. Retraining under the corrected gate. We also retrained the six cells these contrasts use under the corrected gate (3 seeds, 4,101 rows), with pass rules fixed before the jobs ran. All pass (original gate → corrected gate):

    • D1 vs D4 at the recipe's own near-misses: 24.59 → 21.91;
    • D1 vs D4 at verified negatives: 3.34 → 3.01;
    • the reference's lead over the verified arm: 29.85 → 28.97;
    • back-translation over the verified arm: 1.76 → −0.15, still near zero.

    On the hard tier with the query's own rewrites removed, the recipe arm still leads the reference (by 6.15) and still trails D4 with its near-misses (by 9.50).

Back-translation depth (W1). We agree that the original control could not separate authorship from depth. New control: LLM paraphrases matched source by source to the back-translations' depth (character 3-gram overlap with the source), with the same negatives. It recovers 9.33 of the 29.85 points, where back-translation recovers 1.76 (26.38 vs 18.81 easy R@1; the band fixed in advance was ±2.40). So authorship matters at that depth, and we have dropped the claim that back-translation attributes the share to authorship rather than depth. Part of this gain is not specific to the benchmark: the same model also finds real cross-language duplicates better (89.4 vs 84.6 R@1), and the back-translations carry translation errors ("Let" comes back as "Leave me alone." in 13 rows).

"Recipe-matching, not equivalence" overreaches (W3). We have narrowed the claim and changed the title. The recipe arm does learn some near-miss discrimination that no generator wrote (+6.98 R@1 over the reference on the generator-free probe). What we claim is that most of the benchmark gain is specific to how the test was built. The cleanest evidence holds forgetting fixed. Rebuilding only the golds of the same queries changes the winner while every checkpoint stays fixed: D3 trails the recipe arm by 0.19 on golds rebuilt with the benchmark's prompt and leads it by 11.28 on golds rebuilt with D3's own prompt.

Thanks to the authors for their reply. They have answered all of my questions and I'll my positive assessment of their contribution.

Thank you for reading the response and for your positive assessment.

Comments on the paper

1 comment

Summary of the revision

We thank the reviewers. In the revised PDF, blue marks the changes the reviewers asked for and red marks passages rewritten for readability or shortened ; inside a red passage, a sentence that carries a requested change is blue.

  • Hard-tier scoring (4ni6). The three tiers share one corpus, so on the hard tier the query's own easy and medium rewrites count as wrong answers. We rescored all 109 runs with those two documents removed, and with any of the three rewrites counted as a hit (new Appendix D.1, new row in Table 1). Every hard-tier conclusion holds except two, which we now state: the recipe-free reference no longer matches the recipe arm on the hard tier, and one ratio shrinks from about 3× to 1.26×.
  • Forgetting (4ni6). New: blends of all three arms' weights with the base at the same mixing weights. With forgetting limited this way, the recipe arm's 20–23-point benchmark lead over the recipe-free reference comes with no lead on real duplicates (new appendix table).
  • Conditional contrasts (gJtK). The abstract, introduction and conclusion now report the authorship and prompt effects as conditional contrasts. They interact: the benchmark prompt's positives add 3.3 points with verified negatives and 24.6 with its own near-misses.
  • Depth-matched LLM control (gJtK, 4ni6). New: LLM paraphrases matched source by source to the back-translations' depth, with the same negatives. They recover 9.33 of the reference's 29.85 points over the verified arm, where back-translation recovers 1.76, and they also find real duplicates better. So authorship matters even at that depth, and part of what it buys is a better positive (new appendix paragraph).
  • Corrected contamination gate (gJtK). The paper's clean-slice test, applied to every factorial cell, keeps every contrast's sign, except one that is within noise of zero throughout. The largest shrink is 24.07 → 22.31, and 1.54 points of that is leakage (new Appendix table). We also retrained the cells under the corrected gate (3 seeds each), with pass rules fixed before the runs, and every rule passes: D1 vs D4 at the recipe's own near-misses, for example, goes from 24.59 to 21.91.
  • What "verified" means (gJtK). Section 3 now states exactly what the checks do. Only 2 of the 14,361 positives rest on a numeric spot-check alone: renamings are checked by parsing them back, and the other rewrites are confirmed by SymPy simplification as well as at 20 random points. We re-checked every negative using only values of the type and range the problem states for each variable: 98.0% still have a counterexample, 0.6% cannot be checked, and for 1.4% we found none, mostly because both statements are true everywhere or nowhere. Retraining without those 1.4% moves the verified arm by 0.36 points (new appendix paragraph).
  • Human audit (gJtK). New: one of the authors read 100 hard-tier queries, 50 of them wins of the recipe arm. The gold was judged equivalent for 46 of the 50 wins, or 42 if every gold the annotator marked with a caveat counts as flawed, and for 47 of the other 50; no near-miss was judged equivalent (0 of 300). So most sampled wins have a gold a human judges equivalent. The audit checks labels and does not show why the arm wins them. Its limits are stated: one annotator, who is an author, and a sample drawn from 847 of the 1,401 wins (Section 7; new table in Appendix C).
  • Readability (all). The abstract, Sections 1 and 3 and the conclusion are rewritten in plain language, and Section 1 ends with a short summary of what the paper shows. A reader's guide opens the appendix. This is an audit of a benchmark, not a retrieval method. In the appendix we removed about 1,600 words that repeated a table or another passage and corrected a few stale statements; no result was removed.
  • Title (gJtK, 4ni6). The paper is now titled Synthetic Math-Retrieval Benchmarks Reward Matching Their Own Generation Recipe. The old title, Recipe-Matching, Not Equivalence, claimed more than the experiments show.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.