acceptodds
Under review as a conference paper at ICLR 2027

HOP: Harness Optimization with Provable Guarantees

Abstract

A language-model agent's harness governs its reasoning and tool use, but its effects on performance depend on complex task interactions. This complexity often leads harness optimization to be treated as a black-box problem. We take a novel view of harness optimization by distinguishing structured harness configurations from unstructured task contexts. Building on this distinction, we introduce HOP, a semiparametric framework that models outcomes parametrically in configurations and nonparametrically in task contexts. This structure allows HOP to estimate performance across an exponential configuration space while testing only a linear number of configurations, substantially reducing the data collection burden. Under suitable conditions, HOP's outcome estimators are consistent, asymptotically normal, and asymptotically efficient, and HOP yields consistently optimal harness selection. On -Bench (Shi et al., 2026), HOP achieves lower regret and fewer constraint violations than leading harness optimization methods under a common evaluation protocol.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Harness Optimization, Sample-Efficient Optimization, Language Model Agents

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_UTDSOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

HOP selects agent harness configurations by estimating reward, cost, and latency under resource constraints. It combines a structured configuration model with flexible models of task context, using cross-fitting and an orthogonal correction for estimation. Under stated structural and regularity assumptions, the paper establishes identification, asymptotic inference, and consistent selection, with an additional condition for efficiency. Experiments use a full panel of 32 banking agent configurations as a reference and evaluate selection from ten observed configurations across resource budgets.

Critical strengths
  1. Separating configuration structure from task context gives harness optimization a clear statistical formulation. The identification result explains when a linear number of observed configurations is sufficient, and the theory makes the assumptions underlying inference and selection explicit.

  2. The full factorial reference panel, paired task draws, ablation of the orthogonal correction, and sensitivity analyses make the evaluation transparent. Reaggregating the supplied records reproduces the main selection metrics and interval coverage results, strengthening reproducibility.

Critical weaknesses
  1. The empirical results raise concerns about inference for unobserved configurations. The model represents main effects on the link scale, and retrieval appears in only one observed configuration. In Table 10, nominal 95% latency intervals achieve about 85% coverage overall and 79% for unobserved configurations. The paper attributes this shortfall to extrapolation bias but provides limited guidance on detecting or mitigating it before selection. This limits the practical reliability of the inferential guarantees.

  2. The baseline comparisons do not establish superiority over the original optimization algorithms. OPRO, GEPA, and Meta-Harness are adapted into offline selectors using a single call, while BO uses a fixed surrogate without sequential acquisition. These disclosed adaptations may suit the offline setting, but the conclusions should be limited to the tested variants. A regularized regression baseline with the same outcome link functions would also help distinguish the contribution of the orthogonal correction from that of the response model.

Questions for authors
  1. Using the full panel, how would a more balanced retrieval design affect interval coverage for unobserved configurations and constraint violation rates? Could checks for interactions or model misspecification help explain the coverage shortfall?

  2. What practical diagnostic would indicate when additional configurations are needed to reduce extrapolation bias? How should readers interpret the calibrated interval length comparisons alongside the empirical coverage shortfall?

  3. How does HOP compare with regularized regression using the same outcome link functions and observed panel? Could the claims about OPRO, GEPA, Meta-Harness, and BO be explicitly limited to the tested offline variants?

4 comments
AuthorsOpenReview ↗

Response to Reviewer (1/2): Weakness 1, Questions 1 and 2

We sincerely thank the reviewer for the constructive comments! All changes in the revised manuscript are marked in blue. Our response is split into two comments because of the length limit.

Response to Weakness 1, Question 1 and Question 2

Thank you for your insightful comments and suggestions! We will reply one by one.

  1. Weakness 1: Coverage shortfall and model misspecification. The primary reason the empirical confidence interval (CI) coverage falls below the nominal 95% level is that our theoretical guarantee is asymptotic, whereas our experiments are conducted with finite samples (800 data points). Your insightful comment actually touches on the fundamental problem of causal inference (Holland, 1986): the outcomes of unobserved configurations cannot be observed, making it difficult to assess whether a confidence interval covers the true value in practice. This limitation is inherent to causal inference and cannot be entirely avoided in practice.

  2. Question 1

  • A more balanced retrieval design. Following your suggestion, we sampled 10 designs at random in which retrieval is on in 5 of the 10 configurations and reran HOP on each. The balanced designs estimate cost and latency more accurately and improve latency coverage, but the violation rate becomes worse. The reason is the direction of the errors. With the original design, HOP overestimates cost and latency on unobserved configurations (by 7.0% and 2.1%), which keeps its selections on the safe side. With a balanced design this overestimation disappears (−0.8%-0.8\% and −1.0%-1.0\%), so the estimates fall below the truth much more often, and more selected configurations exceed the budget. So there is a trade-off between estimation accuracy and budget violations. The full details are in Page 34, Appendix C.14.

Table: Original design and balanced retrieval designs (balanced: mean over 10 random designs).

DesignCost errorLatency errorCost biasLatency biasUnobserved latency coverageViolations
Original (retrieval on in 1 of 10)7.6%6.8%+7.0%+2.1%78.6%13.4%
Balanced (retrieval on in 5 of 10)7.2%6.0%−0.8%−1.0%82.7%28.9%
  • Model misspecification, as you suggested, may indeed further lower the empirical CI coverage in finite samples. To examine the effect of model misspecification, we incorporate pairwise interactions into our model by forward selection (Hastie et al., 2009). We find that the latency coverage on unobserved configurations indeed improves from 78.6% to 82.7%. The full details of the implementation and results are in Page 31, Appendix C.11 of the revised manuscript.
  1. Question 2: A practical diagnostic. Following your previous question, incorporating interaction effects may be an effective way to mitigate potential model misspecification and improve empirical coverage. We propose a simple check in Appendix C.11 of the revised manuscript: run forward selection (Hastie et al., 2009) over the pairwise interactions. If it keeps selecting an interaction, the main-effect model is missing something and more configurations are needed.
AuthorsOpenReview ↗

Response to Reviewer (2/2): Weakness 2 and Question 3

Response to Weakness 2 and Question 3

Thank you so much for your suggestions! Firstly, we agree that the baseline comparisons should be interpreted within the tested offline setting. The primary reason we used a single-call method is to compare the benchmark methods with the same set of data. However, we agree with you that these methods are built to be iterative and it is crucial to compare them in this way. We now added experiments where each method runs its own loop given the same rollout budget, picking and running configurations on its own adaptively (see Section 4, Page 8 of the revised manuscript). The table below summarizes the results: HOP achieves lower regret and lower violation rates compared with both one-shot and adaptive methods.

Table: Constrained-optimization performance.

MethodRegret (×100)PCSPGSViolation rateCost overrun (%)Latency overrun (%)Optimizer cost (USD/draw)
HOP1.09645.1%64.1%13.44%0.523%0.074%–
LR1.50019.9%57.9%14.00%†1.149%0.078%†–
PDL1.37528.9%49.2%21.25%1.084%0.114%–
BO1.4674.1%49.8%24.06%0.815%0.129%†–
OPRO (one-shot)3.0031.2%11.9%26.50%1.753%0.498%0.3272
GEPA (one-shot)2.7570.4%10.3%23.31%1.874%0.110%†0.3276
MH (one-shot)2.7521.4%11.3%24.00%2.071%0.224%0.3279
OPRO (adaptive)2.0540.0%12.4%24.06%1.079%0.054%†0.2238
GEPA (adaptive)2.3380.0%11.4%23.69%1.070%0.058%†2.4908
MH (adaptive)2.8810.0%9.1%27.38%2.224%0.519%0.2267

† No significant paired difference from HOP across the 20 draws (p ≥ 0.05). Every unmarked baseline entry is significantly worse than HOP (p < 0.05). One-shot: a single call on the same 800 observed trajectories. Adaptive: the method is told the budget pair and iteratively chooses which of the 10 design configurations to run, using at most 800 rollouts.

Secondly, we also added the requested regularized regression baseline (see Appendix C.13 of the revised manuscript for full details). HOP achieves lower feasibility-conditional regret and higher correct-selection and good-selection rates. All three differences are statistically significant, while budget-violation rates are comparable.

Table: HOP and the regularized regression baseline.

MethodRegret ×100 ↓PCS ↑PGS ↑Budget violations ↓
HOP1.09645.13%64.06%13.44%
Regularized regression2.06313.75%47.88%13.31%

References

  • Lakshya A Agrawal, Shangyin Tan, Dilara Soylu, Noah Ziems, Rishi Khare, Krista Opsahl-Ong, Arnav Singhvi, Herumb Shandilya, Michael J Ryan, Meng Jiang, et al. GEPA: Reflective prompt evolution can outperform reinforcement learning. In International Conference on Learning Representations, 2026.
  • Trevor Hastie, Robert Tibshirani, and Jerome Friedman. The Elements of Statistical Learning: Data Mining, Inference, and Prediction. Springer, 2nd edition, 2009.
  • Paul W Holland. Statistics and causal inference. Journal of the American Statistical Association, 81(396):945–960, 1986.
  • Yoonho Lee, Roshen Nair, Qizheng Zhang, Kangwook Lee, Omar Khattab, and Chelsea Finn. Meta-Harness: End-to-end optimization of model harnesses. arXiv preprint arXiv:2603.28052, 2026.
  • Chengrun Yang, Xuezhi Wang, Yifeng Lu, Hanxiao Liu, Quoc V Le, Denny Zhou, and Xinyun Chen. Large language models as optimizers. In International Conference on Learning Representations, 2024.

Thanks for providing additional experiments. However, the latency coverage on unobserved configurations remains low. And the violation rate increases significantly after adding interactions, indicating that the final decision is still quite sensitive to the model specification. I still have concerns about whether the extrapolated results can reliably support configuration choices under resource constraints.

AuthorsOpenReview ↗

Response to Reviewer yKRV: Latency coverage and model misspecification

Thank you for the follow-up! Let us clarify how we see the latency coverage and the model misspecification.

On coverage: the 95% level is an asymptotic guarantee, and it assumes the model is right. Where the main-effect model fits well, the intervals work: coverage is 100% on the observed configurations, and 99% and 95% for reward and cost on the unobserved ones. The latency misses are concentrated: 76 of the 94 come from 5 of the 22 unobserved configurations, and coverage on the other 17 is 94.7%.

On the model: you are right, the decision depends on the model, and any model with structure can be misspecified. We still use a semiparametric model for three reasons. First, without structure we cannot say anything about a configuration we never ran. Second, with the orthogonal correction it has a theoretical guarantee, and it is still more flexible than a parametric model. Third, a fully nonparametric model has no such guarantee here. PDL in our paper is one, and it does worse (latency coverage 13% and violations 21.25%, against 79% and 13.44% for HOP).

Violations go up with interactions for a specific reason: the main-effect model slightly overestimates cost and latency, which keeps its choices on the safe side, while the interaction model underestimates them. If staying within budget matters most, the conservative rule in Appendix C.12 helps: of the 1,008 selections that pass the 95% upper-bound check, none violates a budget on our panel, at the cost of some reward.

Reviewer_yKRVOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

HOP represents harness-configuration effects through a prespecified parametric link while allowing its coefficients to vary nonparametrically with task context. It uses cross-fitting and an orthogonal correction to estimate reward, cost, and latency before selecting a configuration under resource constraints. Theoretical results cover asymptotic inference and selection consistency under specification and regularity conditions. On the banking domain of tau3-Bench, the method observes 10 of 32 configurations on 80 tasks and estimates outcomes for the remaining configurations, comparing its selections with LR, PDL, BO, and adapted LLM-based selectors.

Critical strengths
  1. Formulating limited-rollout harness selection as constrained estimation is a useful approach. The combination of structured configuration effects and task-dependent coefficients has a clear motivation and explicit theoretical conditions.
  2. Table 1 reports regret, feasibility, and resource overruns rather than reward alone. The orthogonal-correction ablation in Table 2 shows that this component improves decision quality in the studied setting.
Critical weaknesses
  1. Extrapolation assumptions. Proposition A.1's extrapolation from a linear number of configurations depends on the main-effect link being correctly specified, while the experiment fits it on only 10 of 32 configurations. In Table 10, nominal 95% latency intervals cover just 79% of the unevaluated configurations; the paper attributes this to extrapolation bias and limited support for the retrieval factor. Table 1 still reports a 13.44% budget-violation rate.
  2. Comparison scope. Section 4.3 turns OPRO, GEPA, and Meta-Harness into one-call selectors over fixed rollout data, removing their iterative proposal-and-test mechanism. This is a reasonable comparison of fixed-data selection, but it cannot establish that HOP outperforms the original optimizers under a rollout budget.
Questions for authors
  1. Using the existing complete 32-by-97 panel, can you report systematic main-effect residuals by unevaluated configuration and retrieval-related combination? How do latency coverage, budget violations, and regret change with a prespecified sparse pairwise-interaction model?
  2. What fraction of Table 1 selections are among the 22 unevaluated configurations, and where do budget violations concentrate? How would feasibility and regret change if constraints used conservative confidence bounds rather than point estimates?
  3. Does HOP retain its advantage if OPRO, GEPA, and Meta-Harness keep their iterative proposal-and-test mechanisms under the same 800-rollout budget?
2 comments
AuthorsOpenReview ↗

Response to Reviewer (1/2): Weakness 1, Questions 1 and 2

We sincerely thank the reviewer for the constructive comments! All changes in the revised manuscript are marked in blue. Our response is split into two comments because of the length limit.

Response to Weakness 1

Thank you for pointing this out! The primary reason the empirical confidence interval (CI) coverage falls below the nominal 95% level is that our theoretical guarantee is asymptotic, whereas our experiments are conducted with finite samples (800 data points). Similarly, regarding the constraint violation rate, Corollary 3.4 establishes that it converges to zero as the sample size approaches infinity, but does not guarantee zero violations in finite samples. Such deviations reflect the expected gap between finite-sample performance and asymptotic theoretical guarantees. Although our method does not achieve the 95% nominal coverage level in finite samples, it consistently achieves higher empirical coverage than the benchmark methods.

We also appreciate your insightful observation that model misspecification may further contribute to the lower empirical CI coverage in finite samples. To investigate this possibility, we have conducted additional experiments following your suggestion in Question 1, as detailed below (Full details are in Appendix C.11 of the revised manuscript).

Response to Question 1

We fitted the main-effect model to the full data (32 configurations ×\times 97 tasks) and found that residuals do exist, as shown in the table below. For retrieval, the error with retrieval on is about the same as with it off.

Table: Residuals of the main-effect model.

ConfigurationsRewardCostLatency
Observed (10)6.3%4.7%4.0%
Unobserved (22)7.2%4.3%3.7%
Retrieval on (16)6.8%4.8%3.7%
Retrieval off (16)7.0%4.1%3.8%

We then chose pairwise interactions by forward selection (Hastie et al., 2009), using only the 800 observed rollouts, and added the pre-selected interactions to the main-effect model. We find that the latency coverage on unobserved configurations improves, while the budget violation rate becomes worse. The reason is the direction of the errors: the main-effect model tends to overestimate cost and latency (by 7.0% and 2.1%), which keeps its selections on the safe side, whereas the pairwise-interaction model tends to underestimate them (by 2.8% and 3.7%). So there is a trade-off between coverage and budget violations. The full details of the implementation and results are in Page 31, Appendix C.11.

Table: Performance of the main-effect model and the prespecified pairwise-interaction model.

ModelRegret ×100PCSViolationsUnobserved latency coverage
Main effects (paper)1.09645.1%13.4%78.6%
+ interactions, from the observed data1.03236.0%26.8%82.7%

Response to Question 2

Looking at the 1,600 selections behind Table 2, 1,371 of them (85.69%) are among the 22 unobserved configurations, and 132 of these violate a budget (9.63%). In total there are 215 violations: 120 on cost only, 86 on latency only, and 9 on both. The tightest cost budget accounts for 96 of them. Also, 128 of the 215 happen when no configuration is predicted to be feasible and the rule still has to return one.

We also tested the conservative bound you suggested: a configuration passes only if its one-sided 95% upper bounds for cost and latency are both below the budgets. The results show that this approach reduces the constraint violation rate but increases regret, highlighting a trade-off between feasibility and reward. This is because high-performing configurations often lie close to the budget limits, and the conservative bounds may exclude configurations that are actually feasible. The full details of the implementation and results are in Page 32, Appendix C.12.

Table: Original rule and conservative rule.

MetricOriginal ruleConservative rule
Budget violations13.44%8.00%
Regret ×1001.0961.876
AuthorsOpenReview ↗

Response to Reviewer (2/2): Weakness 2 and Question 3

Response to Weakness 2 and Question 3

Thank you for your insightful comment! The primary reason we used a single-call method is to compare the benchmark methods with the same set of data. However, we agree with you that these methods are built to be iterative and it is crucial to compare them in this way. We now added experiments where each method runs its own loop given the same rollout budget, picking and running configurations on its own adaptively (Section 4, Page 8). The table below summarizes the results: HOP achieves lower regret and lower violation rates compared with both one-shot and adaptive methods.

Table: Constrained-optimization performance.

MethodRegret (×100)PCSPGSViolation rateCost overrun (%)Latency overrun (%)Optimizer cost (USD/draw)
HOP1.09645.1%64.1%13.44%0.523%0.074%–
LR1.50019.9%57.9%14.00%†1.149%0.078%†–
PDL1.37528.9%49.2%21.25%1.084%0.114%–
BO1.4674.1%49.8%24.06%0.815%0.129%†–
OPRO (one-shot)3.0031.2%11.9%26.50%1.753%0.498%0.3272
GEPA (one-shot)2.7570.4%10.3%23.31%1.874%0.110%†0.3276
MH (one-shot)2.7521.4%11.3%24.00%2.071%0.224%0.3279
OPRO (adaptive)2.0540.0%12.4%24.06%1.079%0.054%†0.2238
GEPA (adaptive)2.3380.0%11.4%23.69%1.070%0.058%†2.4908
MH (adaptive)2.8810.0%9.1%27.38%2.224%0.519%0.2267

† No significant paired difference from HOP across the 20 draws (p ≥ 0.05). Every unmarked baseline entry is significantly worse than HOP (p < 0.05). One-shot: a single call on the same 800 observed trajectories. Adaptive: the method is told the budget pair and iteratively chooses which of the 10 design configurations to run, using at most 800 rollouts.

References

  • Lakshya A Agrawal, Shangyin Tan, Dilara Soylu, Noah Ziems, Rishi Khare, Krista Opsahl-Ong, Arnav Singhvi, Herumb Shandilya, Michael J Ryan, Meng Jiang, et al. GEPA: Reflective prompt evolution can outperform reinforcement learning. In International Conference on Learning Representations, 2026.
  • Trevor Hastie, Robert Tibshirani, and Jerome Friedman. The Elements of Statistical Learning: Data Mining, Inference, and Prediction. Springer, 2nd edition, 2009.
  • Yoonho Lee, Roshen Nair, Qizheng Zhang, Kangwook Lee, Omar Khattab, and Chelsea Finn. Meta-Harness: End-to-end optimization of model harnesses. arXiv preprint arXiv:2603.28052, 2026.
  • Chengrun Yang, Xuezhi Wang, Yifeng Lu, Hanxiao Liu, Quoc V Le, Denny Zhou, and Xinyun Chen. Large language models as optimizers. In International Conference on Learning Representations, 2024.
Reviewer_axdVOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper casts agent harness selection as constrained statistical estimation. Harness configurations are binary vectors. Expected outcomes for all configurations, including unobserved ones, are estimated with a DML-style orthogonal score and cross-fitting, followed by constrained selection by enumeration. The experiments use τ3\tau ^{3}-Bench, and HOP is compared with LR, PDL, BO, and one-shot adaptations of OPRO, GEPA, and Meta-Harness on regret, PCS/PGS, and constraint violations.

Critical strengths
  • This research problem is pragmatic, treating harness choice as estimation plus constrained decision-making, with valid confidence intervals, which is a useful perspective for agent engineering.

  • The paper builds a comprehensive experimental setting, which analyzes hyperparameter and design sensitivity in Tables 6–7 and reports interval coverage in Table 10.

  • The exposition and mathematics are clear.

Critical weaknesses
  • The distinction between the proposed method and ("Deep Learning for Individual Heterogeneity", 2020) and ("Deep learning-based causal inference for large-scale combinatorial experiments: Theory and empirical evidence", 2025) seems not fully discussed in the paper, especially the new findings of applying those in the agentic setting.

  • The scale of experiments is relatively small. A larger configuration space may be helpful.

  • The selected baselines are somewhat weak. The LLM-based baselines are reduced to single-call selectors.

Questions for authors

Please refer to the weakness mentioned above.

2 comments
AuthorsOpenReview ↗

Response to Reviewer (1/2): Weaknesses 1 and 2

We sincerely thank the reviewer for the constructive comments! All changes in the revised manuscript are marked in blue. Our response is split into two comments because of the length limit.

Response to Weakness 1

Thank you for pointing this out. We have expanded our discussion of these two papers in the related literature section (Page 3, Section 2 of the updated manuscript). Specifically, our work differs from this literature in the following three aspects:

  1. A novel modeling perspective. Our primary contribution lies in introducing a novel modeling framework for agent behavior that separates structured harness choices from complex task contexts. While our methodology builds on the semiparametric inference literature, including these two papers, the existing literature primarily focuses on econometric inference rather than optimization under resource constraints.
  2. Theoretical contributions. Our theoretical results extend beyond those of Farrell et al. (2020) and Ye et al. (2025) in several important ways. First, while these studies primarily focus on statistical inference, our work addresses a constrained optimization problem. In particular, we establish new theoretical guarantees for the consistency of the resulting constrained optimization solution in Corollary 3.4. Second, in Theorem A.5, we establish conditions under which HOP achieves the smallest asymptotic variance among regular estimators in our model. To the best of our knowledge, this result has not been explicitly established in the existing literature.
  3. An empirical focus on optimization. We focus on how well HOP chooses agent harnesses under resource budget constraints. Rather than evaluating estimation accuracy alone, we examine downstream optimization performance such as regret and budget violations.

Response to Weakness 2

Thank you for your comment! The primary reason for the limited scale of our experiments is that, to rigorously evaluate the performance of our method, we need to establish the ground truth by evaluating every configuration. This exhaustive evaluation is required only for benchmarking purposes, not for applying our method in practice. However, obtaining such ground truth becomes computationally infeasible for higher-dimensional harness configurations. For example, with 50 binary configuration variables, evaluating all configurations would require at least 250≈1.1×10152^{50} \approx 1.1 \times 10^{15} agent runs, far exceeding our available computational resources. Importantly, applying our method in practice does not require evaluating every configuration.

To demonstrate the scalability of our method, we have added a semi-synthetic experiment based on our existing real-world agent dataset. Specifically, we consider an agent with 50 binary configuration variables, resulting in 250≈1.1×10152^{50} \approx 1.1 \times 10^{15} possible configurations. Using rollout data from only 200 configurations, we apply HOP to select among all 2502^{50} configurations under 25 budget pairs. The results show that HOP scales effectively, achieving the lowest regret and lower constraint violation rates than the benchmark methods. Full details of this experiment are provided in Appendix C.9 of the revised manuscript.

AuthorsOpenReview ↗

Response to Reviewer (2/2): Weakness 3

Response to Weakness 3

Thank you for your insightful comment! The primary reason we used a single-call method is to compare the benchmark methods with the same set of data. However, we agree with you that these methods are built to be iterative and it is crucial to compare them in this way. We now added experiments where each method runs its own loop given the same rollout budget, picking and running configurations on its own adaptively (Section 4, Page 8). The table below summarizes the results: HOP achieves lower regret and lower violation rates compared with both one-shot and adaptive methods.

Table: Constrained-optimization performance.

MethodRegret (×100)PCSPGSViolation rateCost overrun (%)Latency overrun (%)Optimizer cost (USD/draw)
HOP1.09645.1%64.1%13.44%0.523%0.074%–
LR1.50019.9%57.9%14.00%†1.149%0.078%†–
PDL1.37528.9%49.2%21.25%1.084%0.114%–
BO1.4674.1%49.8%24.06%0.815%0.129%†–
OPRO (one-shot)3.0031.2%11.9%26.50%1.753%0.498%0.3272
GEPA (one-shot)2.7570.4%10.3%23.31%1.874%0.110%†0.3276
MH (one-shot)2.7521.4%11.3%24.00%2.071%0.224%0.3279
OPRO (adaptive)2.0540.0%12.4%24.06%1.079%0.054%†0.2238
GEPA (adaptive)2.3380.0%11.4%23.69%1.070%0.058%†2.4908
MH (adaptive)2.8810.0%9.1%27.38%2.224%0.519%0.2267

† No significant paired difference from HOP across the 20 draws (p ≥ 0.05). Every unmarked baseline entry is significantly worse than HOP (p < 0.05). One-shot: a single call on the same 800 observed trajectories. Adaptive: the method is told the budget pair and iteratively chooses which of the 10 design configurations to run, using at most 800 rollouts.

References

  • Lakshya A Agrawal, Shangyin Tan, Dilara Soylu, Noah Ziems, Rishi Khare, Krista Opsahl-Ong, Arnav Singhvi, Herumb Shandilya, Michael J Ryan, Meng Jiang, et al. GEPA: Reflective prompt evolution can outperform reinforcement learning. In International Conference on Learning Representations, 2026.
  • Max H Farrell, Tengyuan Liang, and Sanjog Misra. Deep learning for individual heterogeneity. arXiv preprint arXiv:2010.14694, 2020.
  • Yoonho Lee, Roshen Nair, Qizheng Zhang, Kangwook Lee, Omar Khattab, and Chelsea Finn. Meta-Harness: End-to-end optimization of model harnesses. arXiv preprint arXiv:2603.28052, 2026.
  • Chengrun Yang, Xuezhi Wang, Yifeng Lu, Hanxiao Liu, Quoc V Le, Denny Zhou, and Xinyun Chen. Large language models as optimizers. In International Conference on Learning Representations, 2024.
  • Zikun Ye, Zhiqi Zhang, Dennis J. Zhang, Heng Zhang, and Renyu Zhang. Deep learning-based causal inference for large-scale combinatorial experiments: Theory and empirical evidence. Management Science, 2025.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.