- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
HOP selects agent harness configurations by estimating reward, cost, and latency under resource constraints. It combines a structured configuration model with flexible models of task context, using cross-fitting and an orthogonal correction for estimation. Under stated structural and regularity assumptions, the paper establishes identification, asymptotic inference, and consistent selection, with an additional condition for efficiency. Experiments use a full panel of 32 banking agent configurations as a reference and evaluate selection from ten observed configurations across resource budgets.
Critical strengths
-
Separating configuration structure from task context gives harness optimization a clear statistical formulation. The identification result explains when a linear number of observed configurations is sufficient, and the theory makes the assumptions underlying inference and selection explicit.
-
The full factorial reference panel, paired task draws, ablation of the orthogonal correction, and sensitivity analyses make the evaluation transparent. Reaggregating the supplied records reproduces the main selection metrics and interval coverage results, strengthening reproducibility.
Critical weaknesses
-
The empirical results raise concerns about inference for unobserved configurations. The model represents main effects on the link scale, and retrieval appears in only one observed configuration. In Table 10, nominal 95% latency intervals achieve about 85% coverage overall and 79% for unobserved configurations. The paper attributes this shortfall to extrapolation bias but provides limited guidance on detecting or mitigating it before selection. This limits the practical reliability of the inferential guarantees.
-
The baseline comparisons do not establish superiority over the original optimization algorithms. OPRO, GEPA, and Meta-Harness are adapted into offline selectors using a single call, while BO uses a fixed surrogate without sequential acquisition. These disclosed adaptations may suit the offline setting, but the conclusions should be limited to the tested variants. A regularized regression baseline with the same outcome link functions would also help distinguish the contribution of the orthogonal correction from that of the response model.
Questions for authors
-
Using the full panel, how would a more balanced retrieval design affect interval coverage for unobserved configurations and constraint violation rates? Could checks for interactions or model misspecification help explain the coverage shortfall?
-
What practical diagnostic would indicate when additional configurations are needed to reduce extrapolation bias? How should readers interpret the calibrated interval length comparisons alongside the empirical coverage shortfall?
-
How does HOP compare with regularized regression using the same outcome link functions and observed panel? Could the claims about OPRO, GEPA, Meta-Harness, and BO be explicitly limited to the tested offline variants?
4 comments
Response to Reviewer (2/2): Weakness 2 and Question 3
Response to Weakness 2 and Question 3
Thank you so much for your suggestions! Firstly, we agree that the baseline comparisons should be interpreted within the tested offline setting. The primary reason we used a single-call method is to compare the benchmark methods with the same set of data. However, we agree with you that these methods are built to be iterative and it is crucial to compare them in this way. We now added experiments where each method runs its own loop given the same rollout budget, picking and running configurations on its own adaptively (see Section 4, Page 8 of the revised manuscript). The table below summarizes the results: HOP achieves lower regret and lower violation rates compared with both one-shot and adaptive methods.
Table: Constrained-optimization performance.
| Method | Regret (×100) | PCS | PGS | Violation rate | Cost overrun (%) | Latency overrun (%) | Optimizer cost (USD/draw) |
|---|---|---|---|---|---|---|---|
| HOP | 1.096 | 45.1% | 64.1% | 13.44% | 0.523% | 0.074% | – |
| LR | 1.500 | 19.9% | 57.9% | 14.00%† | 1.149% | 0.078%† | – |
| PDL | 1.375 | 28.9% | 49.2% | 21.25% | 1.084% | 0.114% | – |
| BO | 1.467 | 4.1% | 49.8% | 24.06% | 0.815% | 0.129%† | – |
| OPRO (one-shot) | 3.003 | 1.2% | 11.9% | 26.50% | 1.753% | 0.498% | 0.3272 |
| GEPA (one-shot) | 2.757 | 0.4% | 10.3% | 23.31% | 1.874% | 0.110%† | 0.3276 |
| MH (one-shot) | 2.752 | 1.4% | 11.3% | 24.00% | 2.071% | 0.224% | 0.3279 |
| OPRO (adaptive) | 2.054 | 0.0% | 12.4% | 24.06% | 1.079% | 0.054%† | 0.2238 |
| GEPA (adaptive) | 2.338 | 0.0% | 11.4% | 23.69% | 1.070% | 0.058%† | 2.4908 |
| MH (adaptive) | 2.881 | 0.0% | 9.1% | 27.38% | 2.224% | 0.519% | 0.2267 |
† No significant paired difference from HOP across the 20 draws (p ≥ 0.05). Every unmarked baseline entry is significantly worse than HOP (p < 0.05). One-shot: a single call on the same 800 observed trajectories. Adaptive: the method is told the budget pair and iteratively chooses which of the 10 design configurations to run, using at most 800 rollouts.
Secondly, we also added the requested regularized regression baseline (see Appendix C.13 of the revised manuscript for full details). HOP achieves lower feasibility-conditional regret and higher correct-selection and good-selection rates. All three differences are statistically significant, while budget-violation rates are comparable.
Table: HOP and the regularized regression baseline.
| Method | Regret ×100 ↓ | PCS ↑ | PGS ↑ | Budget violations ↓ |
|---|---|---|---|---|
| HOP | 1.096 | 45.13% | 64.06% | 13.44% |
| Regularized regression | 2.063 | 13.75% | 47.88% | 13.31% |
References
- Lakshya A Agrawal, Shangyin Tan, Dilara Soylu, Noah Ziems, Rishi Khare, Krista Opsahl-Ong, Arnav Singhvi, Herumb Shandilya, Michael J Ryan, Meng Jiang, et al. GEPA: Reflective prompt evolution can outperform reinforcement learning. In International Conference on Learning Representations, 2026.
- Trevor Hastie, Robert Tibshirani, and Jerome Friedman. The Elements of Statistical Learning: Data Mining, Inference, and Prediction. Springer, 2nd edition, 2009.
- Paul W Holland. Statistics and causal inference. Journal of the American Statistical Association, 81(396):945–960, 1986.
- Yoonho Lee, Roshen Nair, Qizheng Zhang, Kangwook Lee, Omar Khattab, and Chelsea Finn. Meta-Harness: End-to-end optimization of model harnesses. arXiv preprint arXiv:2603.28052, 2026.
- Chengrun Yang, Xuezhi Wang, Yifeng Lu, Hanxiao Liu, Quoc V Le, Denny Zhou, and Xinyun Chen. Large language models as optimizers. In International Conference on Learning Representations, 2024.
Thanks for providing additional experiments. However, the latency coverage on unobserved configurations remains low. And the violation rate increases significantly after adding interactions, indicating that the final decision is still quite sensitive to the model specification. I still have concerns about whether the extrapolated results can reliably support configuration choices under resource constraints.
Response to Reviewer yKRV: Latency coverage and model misspecification
Thank you for the follow-up! Let us clarify how we see the latency coverage and the model misspecification.
On coverage: the 95% level is an asymptotic guarantee, and it assumes the model is right. Where the main-effect model fits well, the intervals work: coverage is 100% on the observed configurations, and 99% and 95% for reward and cost on the unobserved ones. The latency misses are concentrated: 76 of the 94 come from 5 of the 22 unobserved configurations, and coverage on the other 17 is 94.7%.
On the model: you are right, the decision depends on the model, and any model with structure can be misspecified. We still use a semiparametric model for three reasons. First, without structure we cannot say anything about a configuration we never ran. Second, with the orthogonal correction it has a theoretical guarantee, and it is still more flexible than a parametric model. Third, a fully nonparametric model has no such guarantee here. PDL in our paper is one, and it does worse (latency coverage 13% and violations 21.25%, against 79% and 13.44% for HOP).
Violations go up with interactions for a specific reason: the main-effect model slightly overestimates cost and latency, which keeps its choices on the safe side, while the interaction model underestimates them. If staying within budget matters most, the conservative rule in Appendix C.12 helps: of the 1,008 selections that pass the 95% upper-bound check, none violates a budget on our panel, at the cost of some reward.
Response to Reviewer (1/2): Weakness 1, Questions 1 and 2
We sincerely thank the reviewer for the constructive comments! All changes in the revised manuscript are marked in blue. Our response is split into two comments because of the length limit.
Response to Weakness 1, Question 1 and Question 2
Thank you for your insightful comments and suggestions! We will reply one by one.
Weakness 1: Coverage shortfall and model misspecification. The primary reason the empirical confidence interval (CI) coverage falls below the nominal 95% level is that our theoretical guarantee is asymptotic, whereas our experiments are conducted with finite samples (800 data points). Your insightful comment actually touches on the fundamental problem of causal inference (Holland, 1986): the outcomes of unobserved configurations cannot be observed, making it difficult to assess whether a confidence interval covers the true value in practice. This limitation is inherent to causal inference and cannot be entirely avoided in practice.
Question 1
Table: Original design and balanced retrieval designs (balanced: mean over 10 random designs).