acceptodds
Under review as a conference paper at ICLR 2027

MetaCurate: Evolving Curation Filters for High-Quality Agentic Coding Data

Abstract

Supervised fine-tuning (SFT) on agent trajectories is an effective way to improve coding agents, but selecting useful training examples requires more than identifying successful runs. A resolved patch may follow inefficient reasoning or unnecessary tool use, while a failed run may still contain useful intermediate steps. Training pipelines therefore rely on rules and qualitative assessments to filter trajectories, yet designing these filters typically requires substantial human expertise, and validating improvements through repeated training and evaluation is expensive. We introduce MetaCurate, a training-free framework that evolves executable trajectory curation filters using pairwise LLM judgments. Disagreements between the filter's rankings and the judge's preferences provide targeted feedback for jointly revising its rules, rubric, and aggregation policy. The resulting filter scores trajectories independently, so pairwise comparisons are required only during search. Starting from a blank filter, MetaCurate autonomously discovers curation strategies that outperform a strong hand-authored filter refined through human expertise. We validate downstream utility through controlled SFT experiments on our uncurated trajectory pool and the Nemotron SWE dataset, with random selection and hand-authored filtering as baselines. In our experiments, MetaCurate yields the best performance among the compared methods on SWEVerified, comparing to hand-authored filter and the filter crafted with 3x expensive filter training-loop.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
data selection for SFT, LLM agents, software engineering agents, rubric-based evaluation, LLM-as-a-judge, pairwise supervision, SWE agents

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_F45nOpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper presents MetaCurate, a framework for automatically improving data curation filters for coding-agent supervised fine-tuning (SFT). Instead of repeatedly training models to evaluate different filters, MetaCurate uses LLM-based pairwise comparisons of agent trajectories as feedback. An LLM coding agent then revises the filtering rules, evaluation criteria, and score aggregation policy. Experiments on two trajectory datasets show that the resulting filters achieve higher average SWE-bench Verified scores than the compared baselines.

Critical strengths

The paper addresses an important practical problem. Selecting high-quality agent trajectories for SFT is challenging, and evaluating different selection strategies through repeated model training is expensive. Developing a cheaper way to improve data selection is valuable.

Critical weaknesses
  1. The proposed objective is not well aligned with the actual goal. The goal of data curation is to select trajectories that improve downstream model performance. However, MetaCurate optimizes agreement with an LLM judge rather than actual training utility. These two objectives are not necessarily aligned. A trajectory preferred by an LLM judge may not be more useful for training a smaller model. The paper does not provide a clear explanation or strong empirical evidence for why improving agreement with the judge should lead to better training data. This leaves a key gap between the proposed method and the original research problem.

  2. The proposed framework is trivial and lacks methodological novelty. The framework essentially follows a simple manual process: compare trajectories, identify problems with the current filtering rules, revise the rules, and repeat. MetaCurate simply uses LLMs to perform these tasks instead of humans. This may reduce manual effort, but it does not introduce a new method or insight into data curation. The framework itself is straightforward, and merely replacing human judgment and rule revision with LLMs is more engineering effort, not a substantial research contribution.

  3. The experimental improvements are relatively small and not sufficiently convincing. Although MetaCurate achieves the highest average scores in the reported experiments, its improvements over strong baselines are modest. On DSv4Pro, it achieves 64.0% compared with 63.1% for the rule-based filter. On Nemotron v3, it achieves 56.5% compared with 54.8% for the LLM-based filter. These gains are relatively small compared with the reported variation, and the paper does not establish their statistical significance. Moreover, the experiments do not clearly isolate how much improvement comes from the proposed optimization process rather than the use of additional LLM calls or rule revisions. The current results are insufficient to demonstrate a substantial advantage over existing approaches.

Questions for authors
  1. Why should agreement with an LLM judge be a reliable measure of downstream training utility? Can the authors provide stronger evidence of the relationship between pairwise preferences and actual improvements from SFT?

  2. Are the improvements over strong baselines statistically significant? Can the authors report results from independent training runs and provide stronger evidence that the observed gains are consistent?

9 comments
AuthorsOpenReview ↗

Response to Reviewer F45n (1/4)

We thank the reviewer for recognizing the importance of high-quality trajectory selection for agentic SFT and the value of developing more cost-efficient approaches to data curation.

We appreciate the reviewer's concerns regarding the alignment between judge preferences and downstream training utility, the methodological contribution of our framework, and the reliability of the reported improvements. We provide detailed responses below.

AuthorsOpenReview ↗

Response to Reviewer F45n (2/4)

1. Judge agreement and actual training utility - Question 1 / Weakness 1

Revised manuscript: Section 4.1 (concordance surrogate, p. 5); Section 5.4 (Table 3, intermediate filters, pp. 8–9); Section 5.3 (Table 2, 27B transfer, p. 8).

We agree that downstream SFT performance is the ultimate criterion for evaluating data curation. However, the search objective need not be identical to the final evaluation metric. The use of quality judgments as a proxy for training utility also has empirical precedents [1] [2]. Directly optimizing downstream performance would require training and evaluating a student for each candidate filter—the computational bottleneck MetaCurate aims to avoid. We therefore use judge concordance as an inexpensive surrogate and downstream SFT as external validation.

Our goal is to construct a trajectory-quality filter without optimizing against a particular student's training feedback. Training utility can depend on student size and learning state, and we do not claim that one quality ranking is optimal for every student. Instead, we target broadly desirable trajectory characteristics, such as accurate diagnosis, appropriate tool use, efficient problem solving, and sufficient verification.

Pairwise judgments provide a practical signal for this purpose: directly comparing trajectories avoids requiring an absolute numerical quality scale, although it does not eliminate judge bias or inconsistency. Applying pairwise comparisons across a large pool would be expensive. MetaCurate therefore uses a limited set of pairwise judgments to improve an explicit pointwise filter that subsequently scores each trajectory independently. This pairwise-to-pointwise transfer enables scalable curation without student training during filter search.

Importantly, we do not assume that judge preferences perfectly represent student-specific training utility, or that every increase in concordance must monotonically improve downstream performance. The relevant empirical question is whether this surrogate enables useful data selection under the training-free search constraint.

Our downstream experiments provide empirical support for this surrogate. In the intermediate-filter analysis, held-out concordance increases from 0.830 to 0.896, while SWE-bench Verified improves overall from 62.4% to 64.0%, although the relationship is not monotonic. Furthermore, the same selected data achieves 74.3% when transferred to a 27B student, compared with 73.3% for the strongest baseline in that transfer experiment. These results support the practical utility of the surrogate in the evaluated settings, without establishing a direct or monotonic relationship between judge agreement and student-specific training utility.

Thus, the distinction between judge concordance and downstream performance is an intentional methodological choice, rather than by itself evidence of a flaw. Its value must be assessed through the downstream effectiveness and efficiency of the resulting filters.

[1] Chen et al., AlpaGasus: Training a Better Alpaca with Fewer Data. ICLR 2024.

[2] Guo et al., CritiQ: Mining Data Quality Criteria from Human Preferences. ACL 2025.

AuthorsOpenReview ↗

Response to Reviewer F45n (3/4)

2. Reliability across independent training and search runs - Question 2 / Weakness 3

Revised manuscript: Appendix D.1 (Table 7, independent SFT, p. 18) and D.2 (Tables 8–9, repeated searches, pp. 18–19).

We agree that the original standard deviations, obtained from repeated evaluations of individual checkpoints, do not quantify variability across independent training runs. We therefore conducted additional training experiments and repeated filter searches.

Independent training. We conducted additional independent training runs on both DSv4Pro and Nemotron v3 for MetaCurate and the strongest baseline identified in Table 1 of the revised manuscript (Rule filter for DSv4Pro and LLM filter for Nemotron v3). The selected trajectories and training recipe remain fixed for each method, with only the training seed changed. Original-run SDs are taken from Table 1; DSv4Pro additional-run SDs reflect repeated evaluations, and Nemotron v3 additional-run SDs are now reported from three evaluations per checkpoint.

DatasetMethodOriginal runAdditional run
DSv4ProMetaCurate64.0 ± 0.763.9 ± 0.9
DSv4ProStrongest baseline (Rule filter)63.1 ± 1.462.1 ± 1.5
Nemotron v3MetaCurate56.5 ± 1.755.9 ± 1.5
Nemotron v3Strongest baseline (LLM filter)54.8 ± 2.251.1 ± 1.6

MetaCurate maintains a higher observed mean than the pre-specified strongest baseline across both training runs on both trajectory pools. Compared with that baseline from Table 1, its observed margin is +0.9 and +1.8 percentage points on DSv4Pro and +1.7 and +4.8 percentage points on Nemotron v3, for the original and additional runs respectively. These results support the consistency of the targeted comparison while not establishing statistical significance across training seeds.

Repeated filter search. We compared two search runs and assessed their resulting filters on the same held-out set of 220 trajectories unused during search.

Search seedHeld-out concordanceSWE-bench Verified (%)
290.89664.0
430.88863.8

The resulting top-1,000 selections share 853 trajectories, or 85.3%, despite differences in rubric formulations and aggregation weights. These observations provide additional evidence that different searches can produce substantially overlapping selections with similar downstream results within the tested pool and judge setting.


3. Methodological and empirical contribution - Weakness 2

Revised manuscript: Section 4 (Figure 3 and Algorithm 1, search procedure, pp. 4–6); Appendix B (promotion details, pp. 16–17); Appendix C (Figures 5–6, filter diagnostics, pp. 17–18).

MetaCurate contributes a concrete search procedure for improving trajectory selection without a pre-existing preference dataset, a separately trained scorer, or student training inside the search loop. Disagreements between the filter and a pairwise LLM judge identify ranking failures and provide rationales for revision. Candidates are then assessed on separate selection pairs inaccessible to the proposer, and the resulting filter independently scores the full pool.

The paper also establishes why this optimization is necessary. Appendix C in the revised version shows that two of six plausible handwritten criteria contribute almost no ranking information, and that different aggregation policies produce substantially different selections even when criterion scores are fixed. Thus, a plausible rubric does not by itself establish an effective selection procedure; its behavior must be tested on actual trajectories.

The significance of this automation becomes particularly clear when applying data curation to a new task without established quality criteria or human preference annotations. In such settings, manually constructing a reliable rubric requires task-specific expertise, while preference-supervised approaches require additional supervision before an effective selection strategy can be developed. MetaCurate instead starts from a blank initialization and uses LLM-generated comparative feedback to discover and iteratively validate the criteria, rules, and aggregation policies needed for trajectory selection.

Our methodological contribution is therefore not simply automating an existing handwritten filtering procedure. It provides a concrete mechanism for discovering and operationalizing curation criteria when those criteria are not available in advance. The evolved filter is explicit, executable, and reusable, and its training utility is validated through downstream experiments without requiring student training during the search process.

Reviewer_FSJsOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors in this paper proposes a new method for evolving a trajectory-curation filter using pairwise LLM judgments instead of expensive downstream training feedback. In particular, an LLM agent iteratively modifies the filter (Python rules, LLM rubric, and aggregation policy) to better match the preferences of an LLM judge. The final filter selects the top 1,000 trajectories for SFT.

Critical strengths
  • The paper studies an important and timely problem, with a practical and relatively inexpensive solution.

  • The final evolved filter is explicit and interpretable rather than a black-box learned model, which is nice.

Critical weaknesses
  • Novelty is my main concern for this paper: In fact, as far as I understand, CritiQ (ACL 2025) and Auto-Rubric (2025) already explore evolving evaluation criteria using preference feedback without any downstream training. The authors here mainly combine this idea with executable filter evolution (as in CurateEvo).

  • Modest gains with low statistical significance and Missing ablations:: While the idea is nice, The gains over the strongest baselines are maximum of 0.7 and 1.7 percentage points on the two datasets, which is within the std margin reported by the authors. Furthermore, the reported standard deviations come from repeated evaluations of the same checkpoint, not independent training runs. Another point that affected my score, is that, at least for me, it is unclear whether jointly evolving rules, rubric, and aggregation is actually necessary. Perhaps, all you need is simply optimizing the rubric and you would achieve similar results. Such analysis are important here but I couldn't find it in the current version of the paper.

Questions for authors

1- What is the precise methodological novelty compared with CritiQ and Auto-Rubric? Is there something fundamentally new beyond combining existing approaches?

2- Have you tested evolving only the rubric versus jointly evolving the rubric, rules, and aggregation policy?

3- What interesting curation criteria did METACURATE discover that were absent from the handwritten filter?

4- How robust are the downstream improvements across independent SFT training runs?

5- Would an LLM proposer that iteratively improves the filter using pointwise scoring diagnostics, rather than pairwise judge disagreements, achieve similar performance under the same inference budget? What evidence demonstrates that pairwise feedback is essential?

6 comments
AuthorsOpenReview ↗

Response to Reviewer FSJs (1/5)

We thank the reviewer for recognizing the importance and timeliness of the problem, the practical and cost-efficient nature of our approach, and the interpretability of the evolved filters.

We appreciate the thoughtful questions regarding methodological novelty, the contributions of individual filter components, and the robustness of our empirical results. We have conducted additional ablation studies and independent training experiments, and further analyzed the evolved filters to address these concerns. We respond to each question below.

AuthorsOpenReview ↗

Response to Reviewer FSJs (2/5)

1. Precise contribution relative to prior work - Weakness 1 / Question 1

Revised manuscript: Section 2 (related work, p. 3), Sections 3.1–3.2 (pairwise-to-pointwise motivation, pp. 3–4), Section 4 (framework, pp. 4–6), and Section 5.2 (Table 1, p. 7).

We thank the reviewer for highlighting these related works.

MetaCurate addresses a different problem: curating coding-agent trajectories for training, whereas CritiQ targets pre-training data selection and Auto-Rubric extracts evaluation criteria for reward modeling. Coding-agent SFT typically uses a much smaller training corpus than pre-training, making the quality and instructional value of individual trajectories particularly consequential. Curation must therefore assess the problem-solving process, rather than treating a successful outcome as sufficient evidence of a good training example.

Beyond this difference in objective, MetaCurate generates comparative supervision directly from an LLM judge on the target trajectory pool, without requiring human preference annotation. CritiQ uses human-annotated preferences, while Auto-Rubric refines evaluation criteria using supplied human- or AI-labeled preferences.

Furthermore, unlike CritiQ, which trains a separate Bradley–Terry scoring model, MetaCurate converts pairwise feedback into a reusable, executable pointwise filter without additional model training. The filter explicitly combines rules, rubric criteria, and aggregation policies, allowing it to independently score trajectories at scale.

Why does this distinction matter? A particularly important setting is trajectory curation for a new task where neither human preference annotations nor reliable task-specific quality criteria are available. Preference-supervised methods require such annotations to be collected or generated before their criteria can be learned or refined. Furthermore, manually specifying a reliable rubric may itself require substantial task-specific expertise, making rubric design part of the problem rather than a readily available input.

MetaCurate addresses this setting by automatically discovering and refining task-specific curation criteria from a blank initialization, using pairwise feedback generated by an LLM judge on the target trajectory pool. Importantly, our contribution goes beyond replacing human preferences with LLM judgments: MetaCurate iteratively translates comparative feedback into executable rules, rubric criteria, and aggregation policies, validates their behavior on actual trajectories, and revises or removes ineffective criteria. The resulting pointwise filter can then independently score the full pool without additional pairwise judgments or a separately trained scorer. Thus, MetaCurate enables curation without requiring either pre-existing preference annotations or a task-specific handwritten filter, while evaluating the resulting selections through downstream coding-agent training.


2. Rubric-only versus joint evolution - Question 2

Revised manuscript: Appendix D.3 (Table 10, restricted-component ablations, p. 19).

We conducted additional ablations restricting the filter to either rubric-based or rule-based scoring. Unlike the full MetaCurate configuration, which combines rules, rubrics, and aggregation, the restricted variants remove the other components entirely rather than keeping them fixed. All configurations were evaluated on the same trajectory pool using Qwen3.5-9B, with 1,000 selected trajectories per method. We used the same search budget and evaluation procedure across configurations, differing only in which filter components were available for evolution.

MethodEvolved ComponentsSWE-bench Verified (%)
MetaCurate (Full)Rules + Rubric + Aggregation64.0 ± 0.7
Rubric-onlyRubric63.4 ± 0.7
Rule-onlyRules62.6 ± 0.9

Both restricted variants achieve competitive performance, while the full configuration obtains the highest observed mean. These results demonstrate that MetaCurate can produce competitive filters even with restricted representations, while the full configuration offers greater flexibility.

We therefore present joint evolution as a capability of the search procedure and interpret its benefit within the scope of this ablation. In other words, our main contribution is a training-free framework that uses pairwise LLM preferences to iteratively improve an explicit, reusable pointwise curation filter, without requiring pre-existing preference annotations or training a separate scoring model.

AuthorsOpenReview ↗

Response to Reviewer FSJs (3/5)

3. Interesting curation criteria discovered by MetaCurate - Question 3

Revised manuscript: Appendix D.5 (Table 12, filter-evolution examples and removed patch-size heuristic, pp. 20–21); Appendix C (Figures 5–6, handwritten-filter analysis, pp. 17–18).

We thank the reviewer for this question. An important observation from the search process is that MetaCurate does not merely generate plausible evaluation criteria: it uses actual trajectories and judge feedback to identify errors in existing assessments, make criteria more specific, and test, revise, or remove proposed rules. Below, we distinguish between criteria introduced from a blank seed and corrections to a hand-authored seed.

The blank seed contains only a generic Overall Training Value criterion, without detailed evaluation dimensions. Although the proposer receives reference notes and the judge’s rationales, the interesting finding is that it does not simply reproduce the provided heuristics; it modifies their applicability conditions or rejects existing criteria based on concrete examples.

Example and Starting PointProblem with the Existing AssessmentCriterion Introduced or Revised During Search
Verification of the actual defect — added from a blank seedValidation credit was given to standalone imitations that did not execute the modified source code, or to syntax checks alone.The rubric was refined to reward regression tests or reproducers that directly exercise the modified behavior and would have failed on the original behavior. Tests that bypass the changed behavior and standalone imitations were no longer considered sufficient verification.
Separating failure frequency from recovery quality — revised from a hand-authored seedDespite anchors intended to distinguish benign exploration errors from unrecovered failures, the initial scoring behavior sometimes penalized normal exploration or failed to reflect eventual recovery. Some trajectories received low scores even after resolving the errors and passing the tests.The criterion was revised to avoid penalizing normal exploration or harmless search failures. In a separate search, the problematic criterion’s weight was set to zero.
Conditional treatment of missing test execution — a heuristic introduced and subsequently refined during blank-seed searchAn earlier iteration introduced a penalty for trajectories without test execution, but applying this penalty unconditionally could penalize otherwise useful trajectories.The rubric introduced “Missing tests alone are neutral.” The aggregation policy was also modified to waive the −12 penalty for missing test execution when Overall Value was greater than zero.
Verifying the final submitted patch — implemented as a new rule from a blank seedA record of test execution alone does not establish whether the final submitted patch was verified. Edits to temporary files could also be confused with edits to the actual source code.The filter cross-referenced source-file paths in the submitted diff with the transcript’s edit history and counted tests as verification only when they were executed after the last actual source-code edit.

The search also produced hypotheses that were subsequently discarded after further validation. For example, after examining previously judged examples, MetaCurate proposed a preference for focused patches of 400–2,000 bytes and introduced a rule awarding a +2 bonus within that range. However, it later judged the heuristic ineffective and removed it. The significance of this example is not that a particular patch size was established as optimal, but that the system formed a hypothesis from the data, implemented it as an executable rule, and withdrew it following further examination.

The key insight from these examples is not the number of newly added criteria, but the specificity with which MetaCurate determines what counts as evidence of quality and when an existing heuristic should not apply. It evolves filters that distinguish substantive evidence of problem solving from superficial behavioral signals, revising or removing proposed criteria based on validation rather than simply accumulating them.

Reviewer_CV3TOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces a framework for improving training data for supervised fine-tuning by selecting better coding-agent trajectories. The system has a proposer, which repeatedly revises a filter. The filter combines deterministic rules, LLM scoring criteria, and a policy for combining scores. The main mechanism is to repeatedly select candidate filters based on agreement with pairwise LLM judgments on a hidden comparison set, without model training. After the filter search, the system selects 1,000 trajectories for training. In the experiments, the authors use Qwen3.5-9B as the base model and train on data selected from two trajectory pools (DSv4Pro and the mini-swe-agent subset of Nemotron v3 SWE-SFT). On SWE-bench Verified, the method achieves 64.0% using DSv4Pro (random selection: 61.7%) and 56.5% using Nemotron v3 (random selection: 52.9%). The authors also use the selected data to train Qwen3.5-27B, achieving 74.3% (random selection: 72.7%). The paper argues that the system, MetaCurate, can improve data selection and reduce the need for expensive model training during filter search.

Critical strengths

The authors carry out a substantial amount of experimental work, including downstream training on two trajectory pools and comparisons with several data-selection methods. They also test the selected data on a larger model and experiment with different initial filtering strategies, proposers, and judges. This level of detail helps us better understand how the system works. The authors also examine the consistency of LLM judgments. The experiments provide some evidence that this method can improve training results while reducing the need for expensive model training during filter search. If the improvements are stable and reliable, this approach would be practical and useful.

Critical weaknesses
  1. The authors repeated the evaluations of each trained checkpoint but did not report independent repetitions of filter search and model training. The gains over the strongest fixed filtering baselines are 0.9 pp and 1.7 pp for Qwen3.5-9B, which are relatively small. So it is unclear whether these gains would be consistent across separate search and training runs.

  2. It seems that the underlying assumption is that the LLM judge can reliably guide the selection of better training data. The results in the paper indeed show an overall improvement. But higher agreement is not always accompanied by better downstream performance.

Questions for authors
  1. Do the authors have results from independent filter searches or training runs that could help us better understand the stability of the gains?

  2. Do the authors have further insights or evidence supporting the use of agreement with the judge to select better training trajectories?

6 comments
AuthorsOpenReview ↗

Response to Reviewer CV3T (1/3)

We thank the reviewer for the positive assessment of our paper, particularly for recognizing the extensive experimental evaluation across multiple trajectory pools, model sizes, and filtering strategies, as well as the practical value of reducing expensive training during filter search.

We appreciate the reviewer's constructive questions regarding the stability of our results and the relationship between judge agreement and downstream training utility. To address these concerns, we conducted additional experiments involving independent training runs, repeated filter searches, and further analyses of intermediate filters, as detailed below.

AuthorsOpenReview ↗

Response to Reviewer CV3T (2/3)

1. Stability across independent training and filter searches - Weakness 1 / Question 1

Revised manuscript: Section 5.2 and Table 1 (p. 7); Appendix D.1 (Table 7, p. 18) and D.2 (Tables 8–9, pp. 18–19).

We thank the reviewer for raising this important question. We agree that the original standard deviations reflected three evaluations of each trained checkpoint, rather than variability across independent training or filter-search runs. We therefore report the additional training and search results separately.

Independent training. We performed one additional independent training run for MetaCurate and the strongest baseline identified in Table 1 of the revised manuscript on each pool: Rule filter on DSv4Pro and LLM filter on Nemotron v3. For each method, we retained the selected trajectories and training recipe and changed only the training seed. The table reports SWE-bench Verified resolved rates. DSv4Pro SDs and Nemotron v3 original-run SDs describe evaluation variability within checkpoints; Nemotron v3 additional-run SDs also reflect three evaluations per checkpoint.

DatasetMethodOriginal runAdditional run
DSv4ProMetaCurate64.0 ± 0.763.9 ± 0.9
Strongest baseline (Rule filter)63.1 ± 1.462.1 ± 1.5
Nemotron v3MetaCurate56.5 ± 1.755.9 ± 1.5
Strongest baseline (LLM filter)54.8 ± 2.251.1 ± 1.6

The independent training repetitions show that MetaCurate maintains a higher observed mean than the strongest baseline identified in the original Table 1 on both trajectory pools. The observed advantage is +0.9 and +1.8 percentage points on DSv4Pro and +1.7 and +4.8 points on Nemotron v3 in the original and additional runs, respectively. These comparisons provide further robustness evidence, although two training seeds do not establish statistical significance.

Reproducibility of filter search. We compared two runs with different search seeds. Both resulting filters were assessed on the same held-out set of 220 trajectories that was not used during filter search.

Search seedHeld-out concordanceSWE-bench Verified (%)
290.89664.0
430.88863.8

The searches produced different rubric formulations and aggregation weights:

AspectSeed 29Seed 43Key difference
Overall training valueRewards strong diagnosis, focused implementation, and convincing verification.Rewards causal diagnosis, focused patches, and supporting evidence, including honest reporting.Similar overall objectives; Seed 43 explicitly emphasizes evidence and honest reporting.
Patch correctnessUnresolved Defect Risk: penalizes concrete remaining defects or regressions. Missing tests alone incur no penalty.Patch Soundness: evaluates robustness and compatibility. Edge-case/API risks prevent a positive score; failed checks incur penalties.Concrete defects versus broader potential risks.
Process qualityRepair Focus: evaluates the causal connection between diagnosis and patch; penalizes symptom-only fixes and overly broad edits.Process Discipline: evaluates exploration–editing–verification efficiency; penalizes detours and runaway behavior.Causal coherence versus workflow efficiency.
Aggregation weightsOutcome: 16; value: 12.5; process: 5; correctness: 2.5.Outcome: 14; value: 9; process: 2; correctness: 2.Both prioritize outcome; Seed 29 gives relatively greater weight to value and process.
Rubric anchor styleComplete sentences.Short phrases.Different wording styles emerge from the same search framework.

The two filters select 853 common trajectories among their respective top-1,000 selections, corresponding to 85.3% overlap. Their held-out concordances differ by 0.008, and their downstream scores differ by 0.2 percentage points. Although the searches produced different rubric formulations and aggregation weights, they identified substantially overlapping training subsets with similar observed downstream performance.

Together, these experiments separately assess the robustness of downstream training and the stability of the filter search process. Importantly, our goal is not merely to obtain a marginal improvement over a strong hand-crafted filter, but to demonstrate that MetaCurate can autonomously discover competitive curation strategies from a blank initialization, without requiring a task-specific handwritten initial filter.

AuthorsOpenReview ↗

Response to Reviewer CV3T (3/3)

2. Evidence for using judge agreement - Weakness 2 / Question 2

Revised manuscript: Section 4.1 (search objective, p. 5); Section 5.4 (Table 3, iteration series, pp. 8–9); Appendix D.5 (Table 12, verification example, pp. 20–21).

We thank the reviewer for this question. We use judge concordance as an inexpensive surrogate for improving trajectory selection, rather than as a direct predictor of student-specific performance gains. Directly optimizing downstream performance would require repeatedly training and evaluating the student for each candidate filter, which is precisely the computational cost MetaCurate aims to avoid.

Our rationale is that relative comparisons can provide meaningful quality feedback even without predefined task-specific quality standards or human-annotated reference judgments. Rather than assigning absolute scores to complex trajectories, an LLM judge compares their relative usefulness as training examples, considering problem-solving behaviors such as diagnosis, repair, and verification. MetaCurate then refines an explicit filter to capture these preferences. For example, the search improved its verification criteria to distinguish genuine regression tests from standalone imitations that did not exercise the modified code (Appendix D.5, Table 12).

The use of quality judgments as a proxy for training utility also has empirical precedents. AlpaGasus [1] demonstrates the effectiveness of LLM-based quality filtering for instruction tuning, while CritiQ [2] uses preference-derived quality criteria for data selection and evaluates their downstream utility. These studies support the premise that quality judgments can guide useful data selection, although they do not establish that higher judge agreement necessarily yields better student performance.

Our results provide empirical support for this surrogate in the evaluated setting. Across intermediate filters, held-out judge concordance increases from 0.830 to 0.896, while SWE-bench Verified performance improves overall from 62.4% to 64.0% (Table 3). Although the relationship is not strictly monotonic, the results indicate that judge concordance can serve as a practical optimization signal for discovering more effective training-data selections without repeatedly retraining the student.

[1] Chen et al., AlpaGasus: Training a Better Alpaca with Fewer Data. ICLR 2024.

[2] Guo et al., CritiQ: Mining Data Quality Criteria from Human Preferences. ACL 2025.


Final Note:

Thank you for your careful review and for highlighting the breadth of the evaluation and the potential value of reducing training costs during filter search. To address your concerns, we added independent training and filter-search repetitions and clarified how judge concordance relates to downstream training utility. We welcome further questions or feedback. If these results and clarifications resolve your concerns, we would sincerely appreciate your consideration of an updated score and support for acceptance.

Comments on the paper

2 comments
AuthorsOpenReview ↗

To AC and All Reviewers

Common Response: Summary of Revisions

We thank all reviewers for their thoughtful and constructive feedback. We appreciate their recognition of the strengths of our work: CV3T highlighted the "substantial amount of experimental work" and noted that the experiments help explain how the system works; FSJs described the work as addressing an "important and timely problem" and praised the final filter as "explicit and interpretable rather than a black-box learned model"; and F45n recognized that the paper "addresses an important practical problem" and that "Developing a cheaper way to improve data selection is valuable". We are grateful for these encouraging assessments and for the constructive feedback that motivated the revisions below.

In response to the reviewers' feedback, we have substantially revised the manuscript to strengthen the empirical evidence, clarify the methodological contributions, and improve reproducibility. The major additions are summarized below.

  1. Additional robustness experiments (Appendix D.1–D.2). We added independent training repetitions for MetaCurate and the strongest baseline from the original Table 1 on each of two trajectory pools (Rule filter on DSv4Pro and LLM filter on Nemotron v3), together with a repeated filter search. MetaCurate retains a higher observed mean than these baselines across both runs on both pools.
  2. Filter component and judge-feedback ablations (Appendix D.3–D.4, Tables 10–11). We added rubric-only and rule-only configurations under the same search budget and evaluation procedure as the full configuration. We also compared pairwise and pointwise judge feedback under the same inference budget, observing 64.0% versus 60.9% SWE-bench Verified, respectively. These experiments clarify the effects of filter representation and judge-feedback formulation.
  3. Concrete examples of filter evolution (Appendix D.5, Table 12). We added examples illustrating how MetaCurate introduces, refines, or removes evaluation criteria and executable rules based on feedback from actual trajectories. These examples make the evolved filters and their decision-making behavior easier to interpret.
  4. Analysis and visualization of filter behavior (Appendix C and Appendix G, Figures 5–8). We expanded the analysis of handwritten filters, showing that seemingly reasonable criteria may provide little ranking information and that aggregation policies can substantially alter data selection. We also added visualizations of the complete search histories for blank and handwritten seeds, illustrating how candidate revisions affect filter rankings and how the search evolves over iterations.
  5. Expanded methodological details (Appendices B, E, and F). We clarified the search configurations, candidate screening and promotion procedures, role prompts, and initial filter definitions, including the complete handwritten and blank-seed configurations, to improve transparency and reproducibility.

For ease of review, substantive additions and revisions are highlighted in blue in the revised manuscript.

We believe these revisions strengthen both the experimental and methodological presentation of MetaCurate. In particular, they demonstrate how comparative LLM feedback can support the automatic discovery, validation, and refinement of executable pointwise curation filters, even when task-specific human preference annotations and reliable handwritten quality criteria are unavailable, without requiring the model training during filter search.

We address the reviewers' individual concerns and questions in detail in the responses below.

AuthorsOpenReview ↗

Final Author Statement: Summary of Rebuttal

Dear Area Chair and Reviewers,

We sincerely thank all reviewers for their thoughtful feedback and constructive discussions throughout the rebuttal period.

We have made substantial efforts to address the concerns raised, including additional experiments conducted in direct response to reviewers' follow-up questions:

  • Robustness (CV3T, FSJs, F45n): Independent training runs and repeated filter searches provide further evidence of consistent performance.
  • Methodological contribution (FSJs): Additional ablations and a direct CritiQ comparison further support our central contribution: an evidence-driven procedure that actively investigates, validates, and refines executable curation filters, rather than merely optimizing predefined criteria.
  • Iterative refinement (F45n): A new Best-of-N comparison, including downstream SFT, demonstrates the benefits of feedback-guided refinement over independent candidate generation at comparable cost.
  • Surrogate objective (CV3T, F45n): Additional analyses provide empirical evidence supporting the usefulness of judge concordance as a training-free optimization signal for trajectory curation.

We appreciate the reviewers' acknowledgment that several concerns have been addressed, including CV3T's positive follow-up and F45n's confirmation that our additional Best-of-N experiment resolved their follow-up question.

We believe these additional experiments substantially strengthen the methodological and empirical contributions of MetaCurate. We hope the Area Chair will consider the complete rebuttal discussions and newly added evidence in the final assessment.

Thank you again for your time and consideration.

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.