- Rating
- 2(+1)
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper presents MetaCurate, a framework for automatically improving data curation filters for coding-agent supervised fine-tuning (SFT). Instead of repeatedly training models to evaluate different filters, MetaCurate uses LLM-based pairwise comparisons of agent trajectories as feedback. An LLM coding agent then revises the filtering rules, evaluation criteria, and score aggregation policy. Experiments on two trajectory datasets show that the resulting filters achieve higher average SWE-bench Verified scores than the compared baselines.
Critical strengths
The paper addresses an important practical problem. Selecting high-quality agent trajectories for SFT is challenging, and evaluating different selection strategies through repeated model training is expensive. Developing a cheaper way to improve data selection is valuable.
Critical weaknesses
-
The proposed objective is not well aligned with the actual goal. The goal of data curation is to select trajectories that improve downstream model performance. However, MetaCurate optimizes agreement with an LLM judge rather than actual training utility. These two objectives are not necessarily aligned. A trajectory preferred by an LLM judge may not be more useful for training a smaller model. The paper does not provide a clear explanation or strong empirical evidence for why improving agreement with the judge should lead to better training data. This leaves a key gap between the proposed method and the original research problem.
-
The proposed framework is trivial and lacks methodological novelty. The framework essentially follows a simple manual process: compare trajectories, identify problems with the current filtering rules, revise the rules, and repeat. MetaCurate simply uses LLMs to perform these tasks instead of humans. This may reduce manual effort, but it does not introduce a new method or insight into data curation. The framework itself is straightforward, and merely replacing human judgment and rule revision with LLMs is more engineering effort, not a substantial research contribution.
-
The experimental improvements are relatively small and not sufficiently convincing. Although MetaCurate achieves the highest average scores in the reported experiments, its improvements over strong baselines are modest. On DSv4Pro, it achieves 64.0% compared with 63.1% for the rule-based filter. On Nemotron v3, it achieves 56.5% compared with 54.8% for the LLM-based filter. These gains are relatively small compared with the reported variation, and the paper does not establish their statistical significance. Moreover, the experiments do not clearly isolate how much improvement comes from the proposed optimization process rather than the use of additional LLM calls or rule revisions. The current results are insufficient to demonstrate a substantial advantage over existing approaches.
Questions for authors
-
Why should agreement with an LLM judge be a reliable measure of downstream training utility? Can the authors provide stronger evidence of the relationship between pairwise preferences and actual improvements from SFT?
-
Are the improvements over strong baselines statistically significant? Can the authors report results from independent training runs and provide stronger evidence that the observed gains are consistent?
9 comments
Response to Reviewer F45n (2/4)
1. Judge agreement and actual training utility - Question 1 / Weakness 1
Revised manuscript: Section 4.1 (concordance surrogate, p. 5); Section 5.4 (Table 3, intermediate filters, pp. 8–9); Section 5.3 (Table 2, 27B transfer, p. 8).
We agree that downstream SFT performance is the ultimate criterion for evaluating data curation. However, the search objective need not be identical to the final evaluation metric. The use of quality judgments as a proxy for training utility also has empirical precedents [1] [2]. Directly optimizing downstream performance would require training and evaluating a student for each candidate filter—the computational bottleneck MetaCurate aims to avoid. We therefore use judge concordance as an inexpensive surrogate and downstream SFT as external validation.
Our goal is to construct a trajectory-quality filter without optimizing against a particular student's training feedback. Training utility can depend on student size and learning state, and we do not claim that one quality ranking is optimal for every student. Instead, we target broadly desirable trajectory characteristics, such as accurate diagnosis, appropriate tool use, efficient problem solving, and sufficient verification.
Pairwise judgments provide a practical signal for this purpose: directly comparing trajectories avoids requiring an absolute numerical quality scale, although it does not eliminate judge bias or inconsistency. Applying pairwise comparisons across a large pool would be expensive. MetaCurate therefore uses a limited set of pairwise judgments to improve an explicit pointwise filter that subsequently scores each trajectory independently. This pairwise-to-pointwise transfer enables scalable curation without student training during filter search.
Importantly, we do not assume that judge preferences perfectly represent student-specific training utility, or that every increase in concordance must monotonically improve downstream performance. The relevant empirical question is whether this surrogate enables useful data selection under the training-free search constraint.
Our downstream experiments provide empirical support for this surrogate. In the intermediate-filter analysis, held-out concordance increases from 0.830 to 0.896, while SWE-bench Verified improves overall from 62.4% to 64.0%, although the relationship is not monotonic. Furthermore, the same selected data achieves 74.3% when transferred to a 27B student, compared with 73.3% for the strongest baseline in that transfer experiment. These results support the practical utility of the surrogate in the evaluated settings, without establishing a direct or monotonic relationship between judge agreement and student-specific training utility.
Thus, the distinction between judge concordance and downstream performance is an intentional methodological choice, rather than by itself evidence of a flaw. Its value must be assessed through the downstream effectiveness and efficiency of the resulting filters.
[1] Chen et al., AlpaGasus: Training a Better Alpaca with Fewer Data. ICLR 2024.
[2] Guo et al., CritiQ: Mining Data Quality Criteria from Human Preferences. ACL 2025.
Response to Reviewer F45n (3/4)
2. Reliability across independent training and search runs - Question 2 / Weakness 3
Revised manuscript: Appendix D.1 (Table 7, independent SFT, p. 18) and D.2 (Tables 8–9, repeated searches, pp. 18–19).
We agree that the original standard deviations, obtained from repeated evaluations of individual checkpoints, do not quantify variability across independent training runs. We therefore conducted additional training experiments and repeated filter searches.
Independent training. We conducted additional independent training runs on both DSv4Pro and Nemotron v3 for MetaCurate and the strongest baseline identified in Table 1 of the revised manuscript (Rule filter for DSv4Pro and LLM filter for Nemotron v3). The selected trajectories and training recipe remain fixed for each method, with only the training seed changed. Original-run SDs are taken from Table 1; DSv4Pro additional-run SDs reflect repeated evaluations, and Nemotron v3 additional-run SDs are now reported from three evaluations per checkpoint.
| Dataset | Method | Original run | Additional run |
|---|---|---|---|
| DSv4Pro | MetaCurate | 64.0 ± 0.7 | 63.9 ± 0.9 |
| DSv4Pro | Strongest baseline (Rule filter) | 63.1 ± 1.4 | 62.1 ± 1.5 |
| Nemotron v3 | MetaCurate | 56.5 ± 1.7 | 55.9 ± 1.5 |
| Nemotron v3 | Strongest baseline (LLM filter) | 54.8 ± 2.2 | 51.1 ± 1.6 |
MetaCurate maintains a higher observed mean than the pre-specified strongest baseline across both training runs on both trajectory pools. Compared with that baseline from Table 1, its observed margin is +0.9 and +1.8 percentage points on DSv4Pro and +1.7 and +4.8 percentage points on Nemotron v3, for the original and additional runs respectively. These results support the consistency of the targeted comparison while not establishing statistical significance across training seeds.
Repeated filter search. We compared two search runs and assessed their resulting filters on the same held-out set of 220 trajectories unused during search.
| Search seed | Held-out concordance | SWE-bench Verified (%) |
|---|---|---|
| 29 | 0.896 | 64.0 |
| 43 | 0.888 | 63.8 |
The resulting top-1,000 selections share 853 trajectories, or 85.3%, despite differences in rubric formulations and aggregation weights. These observations provide additional evidence that different searches can produce substantially overlapping selections with similar downstream results within the tested pool and judge setting.
3. Methodological and empirical contribution - Weakness 2
Revised manuscript: Section 4 (Figure 3 and Algorithm 1, search procedure, pp. 4–6); Appendix B (promotion details, pp. 16–17); Appendix C (Figures 5–6, filter diagnostics, pp. 17–18).
MetaCurate contributes a concrete search procedure for improving trajectory selection without a pre-existing preference dataset, a separately trained scorer, or student training inside the search loop. Disagreements between the filter and a pairwise LLM judge identify ranking failures and provide rationales for revision. Candidates are then assessed on separate selection pairs inaccessible to the proposer, and the resulting filter independently scores the full pool.
The paper also establishes why this optimization is necessary. Appendix C in the revised version shows that two of six plausible handwritten criteria contribute almost no ranking information, and that different aggregation policies produce substantially different selections even when criterion scores are fixed. Thus, a plausible rubric does not by itself establish an effective selection procedure; its behavior must be tested on actual trajectories.
The significance of this automation becomes particularly clear when applying data curation to a new task without established quality criteria or human preference annotations. In such settings, manually constructing a reliable rubric requires task-specific expertise, while preference-supervised approaches require additional supervision before an effective selection strategy can be developed. MetaCurate instead starts from a blank initialization and uses LLM-generated comparative feedback to discover and iteratively validate the criteria, rules, and aggregation policies needed for trajectory selection.
Our methodological contribution is therefore not simply automating an existing handwritten filtering procedure. It provides a concrete mechanism for discovering and operationalizing curation criteria when those criteria are not available in advance. The evolved filter is explicit, executable, and reusable, and its training utility is validated through downstream experiments without requiring student training during the search process.
Response to Reviewer F45n (1/4)
We thank the reviewer for recognizing the importance of high-quality trajectory selection for agentic SFT and the value of developing more cost-efficient approaches to data curation.
We appreciate the reviewer's concerns regarding the alignment between judge preferences and downstream training utility, the methodological contribution of our framework, and the reliability of the reported improvements. We provide detailed responses below.