- Rating
- 2
- Confidence
- 3(−1)
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper presents PACE, a data selection method for multimodal GRPO training. The main idea is to use prompt optimization to identify potentially useful training samples. With the model fixed, PACE compares the answers generated under the original and optimized prompts. Samples whose correctness changes are then filtered through semantic clustering and a balanced selection strategy. The selected subset is used for GRPO training with the original prompt. On multimodal math benchmarks, PACE achieves performance comparable to full-data training while using only around 9% of the available training samples.
Critical strengths
-
Research Idea: What I find interesting about PACE is the use of prompt optimization as a probe rather than as a way to improve inference directly. The authors look at cases where the same model gives different answers under two prompts, and use these changes to decide which samples to train on. I think this is a clever idea.
-
Data Efficiency: The reduction in training data is quite substantial. Table 1 reports an average accuracy of 51.1 with 5K samples, close to the 50.7 obtained by Full-Data GRPO with nearly 55K. PACE also does better than the other methods tested at the same 5K budget. The results on Qwen2.5-VL-3B and InternVL3-2B add some support to the finding.
Critical weaknesses
-
Reliability of the Selection Signal: The main selection signal in PACE comes from just two greedy generations per sample, one with each prompt. This avoids sampling noise, but the result may still depend heavily on the optimized prompt itself. I did not find an analysis of whether different GEPA runs or development sets would produce similar selected subsets. There is also a simpler alternative worth examining: sampling multiple responses with the original prompt. This could identify problems that the model can solve, but not consistently. The paper does compare against Pass Rate, though the comparison does not isolate the value of prompt optimization under an equal probing budget. It is therefore difficult to tell whether the proposed signal provides an advantage over ordinary rollout-based selection.
-
Limited Novelty and Practical Gains: Using prompt optimization as a probe is a good idea. Beyond that, much of PACE relies on familiar filtering and clustering procedures. I also have reservations about the practical gains. The method reduces the training set from roughly 55K to 5K examples, but GRPO still runs for 60 steps rather than 107. The end-to-end runtime drops from 14 to 10 hours, a much smaller reduction than the sample count suggests. Meanwhile, the accuracy difference against Full-Data GRPO is only 0.34 points, with a confidence interval that includes zero. The current results make a reasonable case for sample efficiency, but a less convincing one for the added prompt optimization and probing pipeline.
Questions for authors
Rollout-Based Selection: The paper includes a Pass Rate baseline with four rollouts per sample. However, this does not account for the cost of GEPA. What happens if the same compute budget is used for more rollouts under the original prompt? A comparison with DAPO-style dynamic sampling would also be interesting.
Runtime and Cost: Appendix C reports 10 hours for PACE, compared with 14 hours for Full-Data GRPO. How much time is spent on GEPA, probing, clustering, and GRPO training? Does this include all model calls and API costs? The runtime savings seem quite small compared with the reduction in training data.
Development Set: GEPA uses 150 examples to optimize the prompt. Are these examples also included in the final selection pool? How much do the optimized prompt and selected samples change with a different development set or random seed?
Role of Prompt Optimization: Is GEPA necessary here? Could a few manually written prompts or simple prompt rewrites produce similar results? I would like to see whether these alternatives lead to similar selected samples and downstream accuracy.
Generalization: All seven benchmarks focus on multimodal math. Does PACE also work on general VQA or chart understanding? The paper uses an LLM judge for open-ended answers. How sensitive is the selection process to errors from this judge?
5 comments
W2. Limited Novelty and Practical Gains: Using prompt optimization as a probe is a good idea. Beyond that, much of PACE relies on familiar filtering and clustering procedures. I also have reservations about the practical gains. The method reduces the training set from roughly 55K to 5K examples, but GRPO still runs for 60 steps rather than 107. The end-to-end runtime drops from 14 to 10 hours, a much smaller reduction than the sample count suggests. Meanwhile, the accuracy difference against Full-Data GRPO is only 0.34 points, with a confidence interval that includes zero. The current results make a reasonable case for sample efficiency, but a less convincing one for the added prompt optimization and probing pipeline.
We thank the reviewer for recognizing the potential of prompt optimization as a behavioral probe. We would like to further clarify the methodological contribution and practical benefits of PACE.
1. Methodological Novelty.
The central contribution of PACE is a new data-selection signal derived from prompt-induced correctness transitions, rather than the clustering or filtering techniques themselves. By holding the policy fixed and varying only the prompting condition, PACE identifies examples whose correctness changes across elicitation conditions, capturing both unlocks and regressions.
Our analysis provides empirical evidence that this signal captures information beyond conventional difficulty measures:
- Complementary information: The prompt-sensitive boundary exhibits weak correlations with eight conventional selection signals (all |ρ| < 0.25), with subset overlaps close to random expectation.
- Bidirectional transitions: Both unlocks and regressions contribute to downstream improvements, and their combination achieves stronger performance than either direction alone.
- General applicability: Applying the prompt-sensitive boundary as a candidate filter improves four existing selection strategies by 0.2–0.8 percentage points (Table 4).
These findings demonstrate that prompt-induced behavioral changes provide a distinct and reusable selection signal, with clustering and filtering serving as mechanisms to operationalize it.
2. Practical Efficiency.
PACE achieves full-data-level performance (51.1 vs. 50.7) using only 9.1% of the training examples, while outperforming fixed-5K selection baselines by 1.2–1.5 percentage points.
Importantly, the efficiency gains extend beyond sample count:
| Metric | Full-Data GRPO | PACE |
|---|---|---|
| Training examples | 54.9K | 5K |
| GRPO optimization steps | 107 | 60 |
| End-to-end runtime | 14 h | 10 h |
| AVG7 | 50.7 | 51.1 |
The end-to-end runtime reduction is 28.6%, including the additional prompt optimization, probing, and selection overhead. PACE also requires 43.9% fewer GRPO optimization steps.
We emphasize that PACE is designed to improve training-data selection efficiency, rather than to eliminate all computational overhead. The results demonstrate that prompt-induced correctness transitions enable more effective selection under a fixed data budget, while preserving full-data-level accuracy and reducing overall runtime.
We will further emphasize the distinction between our novel selection signal and its supporting implementation components in the revised manuscript.
Q1. Rollout-Based Selection: The paper includes a Pass Rate baseline with four rollouts per sample. However, this does not account for the cost of GEPA. What happens if the same compute budget is used for more rollouts under the original prompt? A comparison with DAPO-style dynamic sampling would also be interesting.
We thank the reviewer for this insightful suggestion. We further examined the computational costs of PACE and rollout-based selection, and clarify the relationship with dynamic sampling.
1. Comparison with rollout-based selection.
Our existing Pass Rate-4 baseline uses approximately 167K sampled generations, compared with approximately 110K probing generations for PACE. PACE additionally uses GEPA's lightest optimization preset, capped at 580 metric calls. Under the same 5K training budget, PACE achieves 51.1 vs. 49.9 AVG7, an improvement of +1.2 percentage points (95% CI [0.6, 1.8]).
Although this is not a strictly compute-matched comparison, Pass Rate-4 uses more policy generations while achieving lower downstream accuracy. These results provide evidence that prompt-induced correctness transitions offer useful information beyond repeated sampling under the original prompt.
2. Relationship to DAPO-style Dynamic Sampling.
DAPO and PACE identify informative training examples through different mechanisms. DAPO uses multiple rollouts under the same prompt and dynamically filters zero-variance reward groups during RL training. In contrast, PACE uses prompt optimization to construct an alternative elicitation condition and identifies correctness transitions under a frozen policy before training.
Our experiments also include VADE, an online difficulty-based selection method, achieving 50.0 AVG7 compared with 51.1 for PACE using 5K examples. These results further demonstrate the effectiveness of prompt-induced selection relative to existing online selection strategies.
The two mechanisms are complementary: PACE determines which examples enter the training subset, while DAPO-style dynamic sampling determines which rollout groups contribute to policy updates.
Q2. Runtime and Cost: Appendix C reports 10 hours for PACE, compared with 14 hours for Full-Data GRPO. How much time is spent on GEPA, probing, clustering, and GRPO training? Does this include all model calls and API costs? The runtime savings seem quite small compared with the reduction in training data.
We thank the reviewer for this comment. For PACE, GEPA optimization takes approximately 1.2 hours, running concurrently with original-prompt probing. The optimized-prompt probing pass takes approximately 0.8 hours, while clustering and data selection require less than one minute. Downstream GRPO training takes approximately 7.9 hours (60 steps), resulting in an end-to-end wall-clock runtime of approximately 10 hours, compared with 14 hours for Full-Data GRPO. The runtime reduction is smaller than the data reduction because GRPO training does not scale directly with the number of selected examples under our fixed optimization schedule, and PACE incurs approximately 2 hours of offline selection overhead. The reported runtime measures wall-clock time. GEPA incurs approximately $2 in reflection-model API costs. Downstream benchmark evaluation costs are excluded, as they are part of the shared evaluation protocol.
Q3. Development Set: GEPA uses 150 examples to optimize the prompt. Are these examples also included in the final selection pool? How much do the optimized prompt and selected samples change with a different development set or random seed?
No. The 150 GEPA development examples (100 optimization and 50 validation) are explicitly excluded from the final PACE selection pool to avoid overlap. To assess stability, we conducted three independent GEPA optimizations using different development sets. Although the resulting prompt-sensitive boundaries differed (pairwise Jaccard similarity: 0.49–0.53), downstream accuracy remained consistent at 51.1, 50.8, and 50.8, respectively. These results suggest that PACE's downstream performance is relatively stable across different development sets, despite variations in the prompt-sensitive boundaries.
We thank the reviewer for the comments. We address each concern below.
We thank the reviewer for these thoughtful questions. We conducted additional analyses to examine the robustness of the prompt-sensitive boundary and its advantages over rollout-based selection.
1. Robustness across optimized prompts.
We evaluated PACE with three independently obtained optimized prompts using different development-set configurations:
The corresponding prompt-sensitive boundaries have pairwise Jaccard similarities of 0.49–0.53. Despite identifying partially different examples, these optimized prompts yield consistent downstream performance, demonstrating the robustness of PACE's selection signal.
2. Comparison with rollout-based selection.
Our existing Pass Rate-4 baseline uses approximately 167K sampled generations, compared with approximately 110K probing generations for PACE. Under the same 5K training budget, PACE achieves 51.1 vs. 49.9 AVG7, an improvement of +1.2 percentage points (95% CI [0.6, 1.8]).
Together, these results provide evidence that prompt-induced correctness transitions offer useful information beyond repeated sampling under a fixed prompt. We will include these additional analyses in the revision.