acceptodds
Under review as a conference paper at ICLR 2027

Prompt Optimization as a Probe for Data-Efficient Multimodal GRPO

Abstract

Group Relative Policy Optimization (GRPO) can substantially improve multimodal reasoning, but its reliance on multiple rollouts per training problem makes data-efficient training important. Existing data-selection methods typically estimate sample utility from signals observed under a fixed prompt, providing only one view of the current policy's behavior. We introduce PACE (Prompt-Induced Answer Change Estimation), which repurposes prompt optimization as a controlled behavioral probe for GRPO data selection. PACE keeps the initial policy fixed and compares answer correctness under the original and optimized prompts. The resulting bidirectional transitions—unlocks from incorrect to correct and regressions from correct to incorrect—define a prompt-sensitive boundary that captures how the policy responds across elicitation conditions. PACE aggregates these transitions within semantic clusters, retains regions where optimized prompting has a positive aggregate effect, and applies balanced selection to construct a compact training subset. The optimized prompt is used only for probing and selection; downstream GRPO training and evaluation use the original prompt. Experiments using MM-Eureka for training and seven multimodal mathematical reasoning benchmarks for evaluation show that PACE selects only 5,000 examples, approximately 9.1% of the full training set. On Qwen2.5-VL-7B-Instruct, PACE achieves an average accuracy of 51.1, compared with 50.7 for Full-Data GRPO trained on 54,931 examples, while outperforming fixed-budget, larger-budget, and online selection baselines. PACE remains effective across Qwen2.5-VL-3B and InternVL3-2B. Further analyses show that the prompt-sensitive boundary captures information complementary to conventional fixed-prompt signals, that both transition directions provide useful training signals, and that prompt-sensitive filtering improves selection strategies. These results demonstrate that prompt-induced behavioral responsiveness provides an effective signal for data-efficient multimodal GRPO.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Prompt Optimization, Data Selection, Data-Efficient Reinforcement Learning, Multimodal Reasoning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_VTCbOpenReview ↗
Rating
2
Confidence
3(−1)
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents PACE, a data selection method for multimodal GRPO training. The main idea is to use prompt optimization to identify potentially useful training samples. With the model fixed, PACE compares the answers generated under the original and optimized prompts. Samples whose correctness changes are then filtered through semantic clustering and a balanced selection strategy. The selected subset is used for GRPO training with the original prompt. On multimodal math benchmarks, PACE achieves performance comparable to full-data training while using only around 9% of the available training samples.

Critical strengths
  1. Research Idea: What I find interesting about PACE is the use of prompt optimization as a probe rather than as a way to improve inference directly. The authors look at cases where the same model gives different answers under two prompts, and use these changes to decide which samples to train on. I think this is a clever idea.

  2. Data Efficiency: The reduction in training data is quite substantial. Table 1 reports an average accuracy of 51.1 with 5K samples, close to the 50.7 obtained by Full-Data GRPO with nearly 55K. PACE also does better than the other methods tested at the same 5K budget. The results on Qwen2.5-VL-3B and InternVL3-2B add some support to the finding.

Critical weaknesses
  1. Reliability of the Selection Signal: The main selection signal in PACE comes from just two greedy generations per sample, one with each prompt. This avoids sampling noise, but the result may still depend heavily on the optimized prompt itself. I did not find an analysis of whether different GEPA runs or development sets would produce similar selected subsets. There is also a simpler alternative worth examining: sampling multiple responses with the original prompt. This could identify problems that the model can solve, but not consistently. The paper does compare against Pass Rate, though the comparison does not isolate the value of prompt optimization under an equal probing budget. It is therefore difficult to tell whether the proposed signal provides an advantage over ordinary rollout-based selection.

  2. Limited Novelty and Practical Gains: Using prompt optimization as a probe is a good idea. Beyond that, much of PACE relies on familiar filtering and clustering procedures. I also have reservations about the practical gains. The method reduces the training set from roughly 55K to 5K examples, but GRPO still runs for 60 steps rather than 107. The end-to-end runtime drops from 14 to 10 hours, a much smaller reduction than the sample count suggests. Meanwhile, the accuracy difference against Full-Data GRPO is only 0.34 points, with a confidence interval that includes zero. The current results make a reasonable case for sample efficiency, but a less convincing one for the added prompt optimization and probing pipeline.

Questions for authors

Rollout-Based Selection: The paper includes a Pass Rate baseline with four rollouts per sample. However, this does not account for the cost of GEPA. What happens if the same compute budget is used for more rollouts under the original prompt? A comparison with DAPO-style dynamic sampling would also be interesting.

Runtime and Cost: Appendix C reports 10 hours for PACE, compared with 14 hours for Full-Data GRPO. How much time is spent on GEPA, probing, clustering, and GRPO training? Does this include all model calls and API costs? The runtime savings seem quite small compared with the reduction in training data.

Development Set: GEPA uses 150 examples to optimize the prompt. Are these examples also included in the final selection pool? How much do the optimized prompt and selected samples change with a different development set or random seed?

Role of Prompt Optimization: Is GEPA necessary here? Could a few manually written prompts or simple prompt rewrites produce similar results? I would like to see whether these alternatives lead to similar selected samples and downstream accuracy.

Generalization: All seven benchmarks focus on multimodal math. Does PACE also work on general VQA or chart understanding? The paper uses an LLM judge for open-ended answers. How sensitive is the selection process to errors from this judge?

5 comments

We thank the reviewer for the comments. We address each concern below.

W1. Reliability of the Selection Signal: The main selection signal in PACE comes from just two greedy generations per sample, one with each prompt. This avoids sampling noise, but the result may still depend heavily on the optimized prompt itself. I did not find an analysis of whether different GEPA runs or development sets would produce similar selected subsets. There is also a simpler alternative worth examining: sampling multiple responses with the original prompt. This could identify problems that the model can solve, but not consistently. The paper does compare against Pass Rate, though the comparison does not isolate the value of prompt optimization under an equal probing budget. It is therefore difficult to tell whether the proposed signal provides an advantage over ordinary rollout-based selection.

We thank the reviewer for these thoughtful questions. We conducted additional analyses to examine the robustness of the prompt-sensitive boundary and its advantages over rollout-based selection.

1. Robustness across optimized prompts.

We evaluated PACE with three independently obtained optimized prompts using different development-set configurations:

ConfigurationAVG7
PACE (Run 1)51.1
PACE (Run 2)50.8
PACE (Run 3)50.8
Mean ± SD50.9 ± 0.1

The corresponding prompt-sensitive boundaries have pairwise Jaccard similarities of 0.49–0.53. Despite identifying partially different examples, these optimized prompts yield consistent downstream performance, demonstrating the robustness of PACE's selection signal.

2. Comparison with rollout-based selection.

Our existing Pass Rate-4 baseline uses approximately 167K sampled generations, compared with approximately 110K probing generations for PACE. Under the same 5K training budget, PACE achieves 51.1 vs. 49.9 AVG7, an improvement of +1.2 percentage points (95% CI [0.6, 1.8]).

Together, these results provide evidence that prompt-induced correctness transitions offer useful information beyond repeated sampling under a fixed prompt. We will include these additional analyses in the revision.

W2. Limited Novelty and Practical Gains: Using prompt optimization as a probe is a good idea. Beyond that, much of PACE relies on familiar filtering and clustering procedures. I also have reservations about the practical gains. The method reduces the training set from roughly 55K to 5K examples, but GRPO still runs for 60 steps rather than 107. The end-to-end runtime drops from 14 to 10 hours, a much smaller reduction than the sample count suggests. Meanwhile, the accuracy difference against Full-Data GRPO is only 0.34 points, with a confidence interval that includes zero. The current results make a reasonable case for sample efficiency, but a less convincing one for the added prompt optimization and probing pipeline.

We thank the reviewer for recognizing the potential of prompt optimization as a behavioral probe. We would like to further clarify the methodological contribution and practical benefits of PACE.

1. Methodological Novelty.

The central contribution of PACE is a new data-selection signal derived from prompt-induced correctness transitions, rather than the clustering or filtering techniques themselves. By holding the policy fixed and varying only the prompting condition, PACE identifies examples whose correctness changes across elicitation conditions, capturing both unlocks and regressions.

Our analysis provides empirical evidence that this signal captures information beyond conventional difficulty measures:

  • Complementary information: The prompt-sensitive boundary exhibits weak correlations with eight conventional selection signals (all |ρ| < 0.25), with subset overlaps close to random expectation.
  • Bidirectional transitions: Both unlocks and regressions contribute to downstream improvements, and their combination achieves stronger performance than either direction alone.
  • General applicability: Applying the prompt-sensitive boundary as a candidate filter improves four existing selection strategies by 0.2–0.8 percentage points (Table 4).

These findings demonstrate that prompt-induced behavioral changes provide a distinct and reusable selection signal, with clustering and filtering serving as mechanisms to operationalize it.

2. Practical Efficiency.

PACE achieves full-data-level performance (51.1 vs. 50.7) using only 9.1% of the training examples, while outperforming fixed-5K selection baselines by 1.2–1.5 percentage points.

Importantly, the efficiency gains extend beyond sample count:

MetricFull-Data GRPOPACE
Training examples54.9K5K
GRPO optimization steps10760
End-to-end runtime14 h10 h
AVG750.751.1

The end-to-end runtime reduction is 28.6%, including the additional prompt optimization, probing, and selection overhead. PACE also requires 43.9% fewer GRPO optimization steps.

We emphasize that PACE is designed to improve training-data selection efficiency, rather than to eliminate all computational overhead. The results demonstrate that prompt-induced correctness transitions enable more effective selection under a fixed data budget, while preserving full-data-level accuracy and reducing overall runtime.

We will further emphasize the distinction between our novel selection signal and its supporting implementation components in the revised manuscript.

Q1. Rollout-Based Selection: The paper includes a Pass Rate baseline with four rollouts per sample. However, this does not account for the cost of GEPA. What happens if the same compute budget is used for more rollouts under the original prompt? A comparison with DAPO-style dynamic sampling would also be interesting.

We thank the reviewer for this insightful suggestion. We further examined the computational costs of PACE and rollout-based selection, and clarify the relationship with dynamic sampling.

1. Comparison with rollout-based selection.

Our existing Pass Rate-4 baseline uses approximately 167K sampled generations, compared with approximately 110K probing generations for PACE. PACE additionally uses GEPA's lightest optimization preset, capped at 580 metric calls. Under the same 5K training budget, PACE achieves 51.1 vs. 49.9 AVG7, an improvement of +1.2 percentage points (95% CI [0.6, 1.8]).

Although this is not a strictly compute-matched comparison, Pass Rate-4 uses more policy generations while achieving lower downstream accuracy. These results provide evidence that prompt-induced correctness transitions offer useful information beyond repeated sampling under the original prompt.

2. Relationship to DAPO-style Dynamic Sampling.

DAPO and PACE identify informative training examples through different mechanisms. DAPO uses multiple rollouts under the same prompt and dynamically filters zero-variance reward groups during RL training. In contrast, PACE uses prompt optimization to construct an alternative elicitation condition and identifies correctness transitions under a frozen policy before training.

Our experiments also include VADE, an online difficulty-based selection method, achieving 50.0 AVG7 compared with 51.1 for PACE using 5K examples. These results further demonstrate the effectiveness of prompt-induced selection relative to existing online selection strategies.

The two mechanisms are complementary: PACE determines which examples enter the training subset, while DAPO-style dynamic sampling determines which rollout groups contribute to policy updates.

Q2. Runtime and Cost: Appendix C reports 10 hours for PACE, compared with 14 hours for Full-Data GRPO. How much time is spent on GEPA, probing, clustering, and GRPO training? Does this include all model calls and API costs? The runtime savings seem quite small compared with the reduction in training data.

We thank the reviewer for this comment. For PACE, GEPA optimization takes approximately 1.2 hours, running concurrently with original-prompt probing. The optimized-prompt probing pass takes approximately 0.8 hours, while clustering and data selection require less than one minute. Downstream GRPO training takes approximately 7.9 hours (60 steps), resulting in an end-to-end wall-clock runtime of approximately 10 hours, compared with 14 hours for Full-Data GRPO. The runtime reduction is smaller than the data reduction because GRPO training does not scale directly with the number of selected examples under our fixed optimization schedule, and PACE incurs approximately 2 hours of offline selection overhead. The reported runtime measures wall-clock time. GEPA incurs approximately $2 in reflection-model API costs. Downstream benchmark evaluation costs are excluded, as they are part of the shared evaluation protocol.

Q3. Development Set: GEPA uses 150 examples to optimize the prompt. Are these examples also included in the final selection pool? How much do the optimized prompt and selected samples change with a different development set or random seed?

No. The 150 GEPA development examples (100 optimization and 50 validation) are explicitly excluded from the final PACE selection pool to avoid overlap. To assess stability, we conducted three independent GEPA optimizations using different development sets. Although the resulting prompt-sensitive boundaries differed (pairwise Jaccard similarity: 0.49–0.53), downstream accuracy remained consistent at 51.1, 50.8, and 50.8, respectively. These results suggest that PACE's downstream performance is relatively stable across different development sets, despite variations in the prompt-sensitive boundaries.

Reviewer_b2gkOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Motivated by the observation that the same policy may fail on an example under one prompt while solving it correctly under another, this paper proposed PACE (Prompt-Induced Answer Change Estimation), an off-line data-selection framework that repurposes prompt optimization as a controlled behavioral probe. With only 9.1% selected data, PACE outperforms GRPO trained on the full original training set based on Qwen2.5-VL-7B-Instruct, which demonstrates the effectiveness of proposed RL data selection method.

Critical strengths
  1. The formulation and implementation are explicit. The paper is generally well-written and easy to follow, with a clear description of the proposed method PACE. This paper also provides intuitive visual demonstrations (Fig.2) to help better understand the paper.
  2. The proposed method shows great performance gains on diverse multimodal math reasoning benchmarks base on the Qwen2.5-VL-7B-Instruct with only 5K (~10%) budget, achieving highest Gain/1K compared with LIMR, Full-Data GRPO and other baselines.
Critical weaknesses
  1. PACE is currently evaluated mainly on Qwen2.5-VL-3B/7B-Instruct and InternVL3-2B, using MM-Eureka for data selection. To better assess the generalizability of the proposed method, could the authors evaluate it on more advanced open-source models, such as Qwen3-VL, and training datasets, such as MMFineReason?
  2. Is the proposed method applicable beyond multimodal data selection? It would be helpful to include additional experiments on text-only tasks. Alternatively, the authors could provide more details on whether PACE has explored other strategies specifically designed for multimodal data selection, together with the corresponding experimental results.
Questions for authors

See Weaknesses part.

1 comment

We thank the reviewer for the careful review. We address each concern below.

W1. PACE is currently evaluated mainly on Qwen2.5-VL-3B/7B-Instruct and InternVL3-2B, using MM-Eureka for data selection. To better assess the generalizability of the proposed method, could the authors evaluate it on more advanced open-source models, such as Qwen3-VL, and training datasets, such as MMFineReason?.

We thank the reviewer for this valuable suggestion. We agree that evaluating PACE on newer models such as Qwen3-VL and additional training corpora such as MMFineReason would further strengthen the generalizability evidence. We would like to highlight that our current evaluation already covers three model configurations across two model families (Qwen2.5-VL-3B/7B and InternVL3-2B), with evaluations on seven independent benchmarks. Moreover, MM-Eureka is a diverse, multi-source training corpus covering heterogeneous multimodal reasoning problems, rather than a narrowly specialized dataset.

Extending the comparison to another training corpus would require rerunning computationally expensive GRPO baselines. As reported in Appendix C, the end-to-end runtimes are approximately 76 hours for LIMR, 74 hours for ThinkLite-VL, and 14 hours for Full-Data GRPO. Reproducing these comparisons on additional models and corpora is therefore beyond our current rebuttal computational budget. We acknowledge that these experiments do not establish generalization to newer model generations or independent training corpora. We will clarify this limitation and include Qwen3-VL and MMFineReason as important directions for future evaluation.

W2. Is the proposed method applicable beyond multimodal data selection? It would be helpful to include additional experiments on text-only tasks. Alternatively, the authors could provide more details on whether PACE has explored other strategies specifically designed for multimodal data selection, together with the corresponding experimental results.

We thank the reviewer for this insightful suggestion. PACE is built on a modality-independent principle: measuring correctness transitions under different prompting conditions while keeping the model fixed. Since this signal depends on answer correctness rather than visual-specific features, the framework naturally extends to text-only reasoning tasks.

Our experiments also include comparisons with multimodal-specific data selection methods, particularly ThinkLite-VL, which uses MCTS-based reasoning difficulty, and VADE, which performs online difficulty estimation.

MethodSelection StrategySamplesAVG7
ThinkLite-VLMCTS-based difficulty5K49.7
ThinkLite-VLMCTS-based difficulty21K49.8
VADEOnline difficulty estimationAdaptive50.0
PACE (Ours)Prompt-induced correctness transitions5K51.1

Beyond these comparisons, Section 6.5 demonstrates that the prompt-sensitive boundary can improve four additional selection strategies, including k-Center, Facility Location, Entropy, and Random, with gains of 0.2–0.8 percentage points. Together, these results highlight PACE's effectiveness as both a standalone data-selection method and a complementary selection signal. We will further clarify the connections to multimodal-specific selection methods and the broader applicability of prompt-induced correctness transitions.

Reviewer_njFqOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work proposes PACE, an offline data-selection method for multimodal GRPO that turns prompt optimization into a controlled behavioral probe. With the policy frozen, it compares answer correctness under the original and an optimized prompt; the transitions, unlocks (wrong --> right) and regressions (right --> wrong), define a "prompt-sensitive boundary" of examples the policy can solve under some prompts but not consistently. PACE aggregates these within semantic clusters, keeps clusters where the optimized prompt helps, and selects a compact subset via balanced allocation. The optimized prompt is only for probing; training uses the original. On Qwen2.5-VL-7B with MM-Eureka data, PACE picks 5K of 54,931 examples (9.1%), hits 51.1 vs 50.7 for full-data GRPO, and beats fixed-budget and adaptive baselines. It transfers to Qwen2.5-VL-3B and InternVL3-2B, and the boundary is nearly orthogonal to eight fixed-prompt signals.

Critical strengths
  • The idea is fresh and clean: use prompt optimization as a controlled probe, and treat prompt-induced correctness flips (unlock/regression) as the signal for "can solve but not consistently" examples, exactly what GRPO needs. Existing selectors all work under a fixed prompt; PACE is the first to vary the elicitation condition.
  • The complementarity is demonstrated, not just claimed. The boundary's correlation with eight fixed-prompt signals is all ∣ρ∣|\rho| < 0.25, and subset overlap is near chance (fold 0.97–1.26, κ\kappa −0.018 to 0.146), while those same signals strongly agree with each other (entropy vs log-prob κ\kappa = 0.84). So PACE is finding something new, not relabeling difficulty.
  • The efficiency is concrete: 5K examples (9.1%) matches or edges out 54.9K full-data GRPO, and beats the best fixed-5K baseline by ~1.2–1.6 points.
Critical weaknesses
  • The demonstrated payoff is marginal, and this is what decides it for me. Against full data it's 0.4 points (51.1 vs 50.7) — parity, not a win, with no significance evidence in the main text. Against the other 5K selectors it's 1.2–1.6 points, a real but narrow, incremental gain. For a data-selection method to justify acceptance I'd want a clearer, larger advantage than this.
  • The method depends on the prompt optimizer (GEPA), but there's no ablation of optimizer strength. If the boundary quality tracks P* quality, a weaker optimizer would degrade the signal, and I can't tell how much.
  • Only one training dataset (MM-Eureka). The efficiency claim isn't reproduced on any other training corpus.
  • The budget curve is non-monotonic: 5K gives 51.1, but 10K and 15K give 50.6 and 50.8. If the signal were clean, adding more selected examples shouldn't hurt, which makes me suspect the half-to-top-cluster allocation is tuned to 5K and doesn't generalize.
Questions for authors
  • How sensitive is the boundary to the choice of prompt optimizer? Would a weaker GEPA variant noticeably degrade the signal?
  • Why does PACE drop from 51.1 at 5K to 50.6/50.8 at 10K/15K? Is the balanced allocation tuned specifically to 5K?
2 comments

We thank the reviewer for the careful review. We address each concern below.

W1. The demonstrated payoff is marginal, and this is what decides it for me. Against full data it's 0.4 points (51.1 vs 50.7) — parity, not a win, with no significance evidence in the main text. Against the other 5K selectors it's 1.2–1.6 points, a real but narrow, incremental gain. For a data-selection method to justify acceptance I'd want a clearer, larger advantage than this.

We thank the reviewer for this comment. We agree that the 0.4-point difference between PACE and Full-Data GRPO (51.1 vs. 50.7) should be interpreted as performance parity rather than statistically significant superiority. Our primary contribution is achieving full-data-level performance using only 9.1% of the training data.

Regarding the gains over fixed-budget selectors, Appendix B already provides statistical evidence from 10,000 paired bootstrap resamples. PACE achieves improvements of +1.2 pp over Pass Rate (95% CI [0.6, 1.8]), +1.5 pp over LIMR ([0.8, 2.1]), and +1.3 pp over ThinkLite-VL ([0.6, 2.0]). All three confidence intervals exclude zero.

We further report results across three runs using different seeds:

MethodAVG7
PACE (seed 1)51.1
PACE (seed 2)50.8
PACE (seed 3)50.8
PACE (mean ± SD)50.9 ± 0.1

These results demonstrate consistent downstream performance across the three runs, rather than relying on a single favorable optimized prompt. Importantly, PACE improves the base model by 2.7 points, compared with 1.5 points for Pass Rate, representing approximately 80.0% greater improvement over the base model under the same 5K data budget. We will highlight these statistical and robustness results in the main text. We emphasize that PACE's contribution is more effective data selection at a fixed training budget, while retaining full-data-level performance with approximately 11.0× fewer training examples.

W2. The method depends on the prompt optimizer (GEPA), but there's no ablation of optimizer strength. If the boundary quality tracks P* quality, a weaker optimizer would degrade the signal, and I can't tell how much.

We thank the reviewer for this suggestion and apologize for not clearly reporting the GEPA configuration in the original submission. In fact, PACE already uses GEPA's lightest preset (auto="light") on a development set of 150 examples (100 training and 50 validation). Thus, our method does not rely on extensive prompt optimization. While further reducing the already limited budget is possible, it would leave very few opportunities to evaluate alternative prompts, making such a comparison less informative. We will explicitly report the GEPA configuration and optimization budget in the revised manuscript. We appreciate the reviewer for pointing out this omission.

W3. Only one training dataset (MM-Eureka). The efficiency claim isn't reproduced on any other training corpus.

We thank the reviewer for highlighting this limitation. We would like to clarify that MM-Eureka is itself a diverse, multi-source training corpus, rather than a narrowly specialized dataset. Its 54,931 training examples are drawn from multiple datasets covering mathematical and geometric reasoning, chart comprehension, and scientific reasoning, including GeoQA, ChartQA, and ScienceQA. This diversity allows us to evaluate data selection across heterogeneous reasoning problems within a unified training corpus.

Furthermore, we evaluate PACE on seven independent benchmarks and three model configurations, covering different model scales and architectures. We agree that evaluating on an additional training corpus would further strengthen the generalization evidence. However, a rigorous comparison would require repeating data selection and GRPO training for all competing methods. As reported in Appendix C, the end-to-end runtimes are approximately 76 hours for LIMR, 74 hours for ThinkLite-VL, and 14 hours for Full-Data GRPO, compared with 10 hours for PACE. Reproducing these baselines on another corpus would therefore entail substantial additional computational cost beyond the available rebuttal budget.

We acknowledge that diversity within MM-Eureka does not replace independent cross-corpus validation. We will clarify this limitation.

W4. The budget curve is non-monotonic: 5K gives 51.1, but 10K and 15K give 50.6 and 50.8. If the signal were clean, adding more selected examples shouldn't hurt, which makes me suspect the half-to-top-cluster allocation is tuned to 5K and doesn't generalize.

We thank the reviewer for this observation. The 50% allocation was chosen as a simple, fixed heuristic rather than tuned specifically for the 5K budget, and the same allocation rule was applied across all budgets. To investigate this concern, we conducted an additional allocation sensitivity experiment at the 5K budget. Using 25%, 50%, and 75% allocations to the top-ranked cluster yields average accuracies of 50.7, 51.1, and 50.9, respectively.

Regarding the non-monotonic budget curve, we use a fixed training budget of 60 GRPO optimization steps across all subset sizes. Thus, selecting more examples does not increase the number of training updates, potentially reducing the training exposure per example. Consequently, larger subsets do not necessarily yield higher accuracy under the same training budget. We will include these sensitivity results in the revision.

Q1. How sensitive is the boundary to the choice of prompt optimizer? Would a weaker GEPA variant noticeably degrade the signal?

As clarified in W2, PACE already uses GEPA's lightest preset (auto="light"), with a maximum of 580 metric calls on 150 development examples. Furthermore, our three optimized-prompt variants achieve 51.1, 50.8, and 50.8 AVG7 accuracy, respectively (50.9 ± 0.1), suggesting that PACE does not depend on a single favorable optimized prompt. While these experiments do not directly measure the effect of further reducing GEPA's budget, they demonstrate consistent downstream performance under an already lightweight optimization configuration. We will explicitly document these settings and robustness results in the revision.

Q2. Why does PACE drop from 51.1 at 5K to 50.6/50.8 at 10K/15K? Is the balanced allocation tuned specifically to 5K?

As discussed in W4, the 50% allocation was chosen as a simple, fixed heuristic rather than tuned specifically for the 5K budget, and the same rule was applied across all subset sizes. To examine its sensitivity, we additionally tested 25%, 50%, and 75% allocations at 5K, obtaining average accuracies of 50.7, 51.1, and 50.9, respectively.

Regarding the non-monotonic performance, we use a fixed budget of 60 GRPO optimization steps across all subset sizes. Increasing the number of selected examples therefore does not increase the number of training updates, potentially reducing the training exposure per example. This may contribute to the slight performance drops at 10K and 15K.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.