- Rating
- 2
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
In this paper, the authors investigate how to split a fixed experimental budget across several treatments, run in batches, so that the best treatment can be picked at the end. They propose RHOPE, an LLM-augmented adaptive experimental design method in which an LLM reads each participant's profile before every batch and predicts how that participant would respond to each treatment. The predictions are combined with the real outcomes to estimate each treatment's average effect more precisely, and the next batch is allocated to minimize the expected loss from picking the wrong treatment. The authors prove that adding the LLM predictions cannot make this decision worse, and show that their estimate reduces to the existing PPI++ estimator in a special case. In simulations and in a drug-response benchmark built from cancer cell-line data, RHOPE achieves the lowest simple regret among the compared methods when the outcome variances and correlations are known in advance.
Critical strengths
-
Adaptive experiments are widely used, and real outcomes are often expensive, so using an LLM to turn participants' rich profiles into predicted outcomes that make each sample count for more is a reasonable idea for an important problem.
-
The proposed framework is sound: the real outcomes are used to correct the LLM predictions, so a biased LLM does not bias the estimates, and when the outcome variances and correlations are known, an uninformative LLM simply makes the method fall back to standard RHO.
Critical weaknesses
- My main concern is the weak evidence that LLM proxies help. The paper is framed around LLMs extracting useful information from complex context, so the first thing to establish is whether the LLM can actually do this. However, the only experiments that use the LLM outputs as they are are the GDSC runs with v = 0. In those runs, the correlation between the LLM predictions and the true outcomes is 0.07-0.10 on average, and below 0.04 for four of the seven drugs (Table 5). At that level, the variance reduction is only around 1%. All the settings where RHOPE clearly wins have informative proxies by construction. The synthetic experiments do not use an LLM. The v > 0 runs add a scaled copy of the simulated patient noise to the LLM output, and no model could predict that noise from the cell-line features. These runs are a useful sensitivity analysis of how the gains grow with proxy quality, but they say nothing about what proxy quality an LLM can actually deliver.
I also do not think the LLM setup was designed carefully. The authors use only DeepSeek V4 Flash with thinking disabled, ask for raw numeric outputs, use one random draw per patient even though 100 draws were generated, and leave out the gene-expression features. I think the authors should at least try a stronger frontier model and put more care into the prompt. Numeric outputs from LLMs are often poorly calibrated, so asking for response levels instead, and averaging over the multiple draws already available, may better reflect what the LLM can do. Finally, there is no non-LLM baseline, such as a ridge model trained on the calibration cell lines, so it is unclear whether the LLM adds anything over a standard predictor.
- The methodological novelty is limited. Setting the LLM aside, RHOPE plugs a PPI-style control variate [1] into RHO [2]. Most of the theoretical analysis is incremental: it shows that the estimator is efficient, that the Gaussian approximation still holds, and that more information cannot hurt, all of which are expected for this combination. I would have liked a result on how much simple regret improves as a function of the proxy correlation, or on what happens with estimated moments, since that is where the method loses its advantage in practice. I think the genuinely new piece is observing the proxies before allocation, and it shows no clear effect: RHOPE-Delayed is better than RHOPE in 10 of the 30 table entries, and no standard errors are reported.
Minor:
- The introduction motivates the method through heterogeneity across participants, but context is never used to decide who gets which treatment. It only reduces noise in the estimates of each treatment's average effect. Framing the motivation around this would make the paper much easier to follow.
- Ma et al. [3] study best-arm identification with generative proxies, which is very close to this setting. The paper cites it but does not discuss how RHOPE differs from it, and does not compare against it.
References [1] Anastasios N. Angelopoulos, Stephen Bates, Clara Fannjiang, Michael I. Jordan, and Tijana Zrnic. Prediction-powered inference. Science, 382(6671):669–674, 2023.
[2] Ethan Che and Hongseok Namkoong. Adaptive experimentation at scale: A computational framework for flexible batches. arXiv preprint arXiv:2303.11582, 2023.
[3] Tianyi Ma, Hanzhang Qin, Ruihao Zhu, and Jierui Zuo. Best-arm identification with generative proxy. arXiv preprint arXiv:2607.06879, 2026.
Questions for authors
Please see the questions raised in the Weaknesses section.
4 comments
Response to Reviewer MqCL (2/4)
Novelty and the theory
We agree that the estimator is a PPI-style control variate inside RHO, and that its efficiency, the Gaussian approximation and the fact that more information cannot hurt are the expected consequences of this combination. The revision treats these as the groundwork and puts the weight on the questions the reviewer raises: how much proxies help as a function of the correlation, how much of that comes from observing proxies before allocation, and what remains when the moments are estimated. Three new propositions answer them (Section 5.2 and Appendix D.3; proofs in Appendices B.9–B.11).
Regret as a function of the proxy correlation. Proposition 2: with exchangeable arms, the Bayes simple regret guarantees of RHOPE-Delayed and RHO are the same function of their terminal posterior variances, and under a flat prior their ratio converges to . Proxies thus act as a variance reduction by the fraction . The synthetic grid over (Tables 1 and 3) and the six GDSC proxies, for which the regret of RHOPE increases as decreases (Table 2), follow this dependence.
Advance information. We agree that no clear effect is visible. All tables now report standard errors, and none of the paired differences between RHOPE and RHOPE-Delayed is significant. Proposition 3 explains why: under a flat prior, the advance-information gain at batch is at most which decays over the horizon, whereas the estimation benefit removes a fixed fraction of the posterior variance in every batch. Proxies are observed for every unit under every arm, so the proxy means are learned about times faster than the residual means, and the current batch's proxies move the posterior little once a few batches have been observed. The abstract, contributions and Section 6 now describe this benefit as small and secondary to estimation precision.
Estimated moments. Proposition 4: when the projection coefficient is replaced by an estimate from pilot pairs, the guarantee of RHOPE contains in place of , and it stays below the guarantee of RHO if and only if , that is, if and only if exceeds a threshold of order . This is the pattern of the synthetic ablation with first-batch estimates (Appendix D, Table 6): with small batches RHOPE often falls behind RHO, and with large batches a RHOPE variant is best in every setting. This is the precise sense in which the method loses its advantage when moments are estimated from little data.
Response to Reviewer MqCL (3/4)
Evidence that LLM proxies help
We agree with the diagnosis. In the submitted version, the only runs that used the LLM outputs as generated () had average correlations of 0.07–0.10, and the runs injected simulated noise that no predictor could recover. We therefore re-diagnosed the original semi-synthetic experiment and rebuilt it (Appendix C.2). Every proxy now enters exactly as generated, with no augmentation, and the setup differs in three further ways.
- Data. The re-diagnosis showed that 247 of the 896 GDSC cell lines carry imputed response entries (all imputed entries of a drug share one value), which distort the proxy–outcome correlations. We use the 627 cell lines with complete measured responses that are not used as in-context examples.
- Proxies. Claude Opus 5.5, GPT-6.1 Sol and DeepSeek V4 Pro with reasoning enabled, a redesigned prompt (see "The LLM setup" below) and the mean of 20 draws per cell line. DeepSeek V4 Flash with its original prompt and settings, now also averaged over 20 draws instead of one draw per patient, is kept as a control, and ridge regression and gradient-boosted trees (GBT) trained on the same 16 labeled cell lines are the non-LLM references.
- Noise. The patient-noise variance is 4 instead of 9. The purpose of the benchmark is to show how proxies of different quality affect the decisions. With variance 9, the proxy–outcome correlations of all models were uniformly low, so proxies of different quality could not be told apart; the smaller noise variance is one component of the redesign. The cell-line responses and the proxies are not affected by this choice.
Results (Table 2; relative simple regret, uniform , mean se over 10,000 replications; RHO, which uses no proxies, is ):
| Proxy | mean | RHOPE-Delayed | RHOPE |
|---|---|---|---|
| Claude Opus 5.5 | 0.31 | ||
| GPT-6.1 Sol | 0.29 | ||
| DeepSeek V4 Pro | 0.26 | ||
| DeepSeek V4 Flash | 0.20 | ||
| GBT | 0.19 | ||
| Ridge | 0.16 |
With each of the three reasoning LLMs, the lowest regret in Table 2 is attained by RHOPE or RHOPE-Delayed, and both are significantly better than RHO; the improvement shrinks with the weaker Flash proxy. Moreover, with every LLM proxy, the gap between RHOPE and RHOPE-Delayed is much smaller than the gap between RHOPE-Delayed and RHO. This confirms the characterization of estimation precision and advance information in the new theory above: relative to estimation precision, the benefit of advance information is small and secondary. Since every proxy is used as generated, Table 2 now measures what current LLMs deliver on this task.
Response to Reviewer MqCL (4/4)
The LLM setup
(a) Stronger models and a more careful prompt. The revised experiment uses Claude Opus 5.5, GPT-6.1 Sol and DeepSeek V4 Pro with reasoning enabled (effort "high" for the first two, "low" for DeepSeek V4 Pro). The prompt (Appendix C.2, with a complete example) lists the drugs' targets and pathways, gives the 16 in-context cell lines with their observed responses, and describes each cell line by cancer type, growth mode, microsatellite status, the status of 18 drug-relevant biomarkers, coding mutations in 300 cancer genes, copy-number gains and losses in 126 regions with the genes they contain, and, as the reviewer suggested, gene-expression features: the expression percentiles of 18 genes related to the drugs' mechanisms.
(b) Raw numeric outputs versus response levels. We kept numeric outputs for a structural reason. RHOPE uses a proxy only through its correlation with the true outcome: the residual correction by the observed outcomes makes the working-posterior mean invariant to any affine transformation of the proxy, so a miscalibrated numeric scale costs nothing, and only the correlation matters. Coarsening the output into levels would discard the within-level ordering without removing anything harmful. The reviewer is right that the numeric outputs are poorly calibrated (their RMSE against the measured responses is close to that of predicting each drug's mean, although their correlations with the true outcomes can still reach 0.3–0.4); this does not affect RHOPE.
(c) Averaging over draws. Every LLM proxy is now the mean of 20 independent draws per cell line (Appendix C.2).
(d) A non-LLM baseline. Ridge regression and GBT are trained, separately for each drug, on exactly the 16 in-context cell lines, with a numeric encoding of the same features that the prompt shows and hyperparameters chosen by leave-one-out cross-validation within the 16 lines (Appendix C.2). Their proxy–outcome correlations are below those of every LLM, and RHOPE with each reasoning LLM is significantly better than RHOPE with either of them (Table 2). With the same inputs and the same 16 labels, the LLMs extract more usable information than the standard predictors.
Minor 1. Motivation through heterogeneity
Agreed. Context is never used to decide who receives which treatment; the target is the arm with the highest mean outcome. Context explains part of the variation in a unit's outcome, and the LLM turns it into a proxy that reduces the noise in each arm's mean estimate. The introduction now motivates the method in these terms.
Minor 2. Relation to Ma et al. (2026)
Ma et al. study fixed-confidence BAI: an elimination procedure with a -PAC guarantee minimizes the number of true samples needed to certify the best arm, and it has no budget parameter. We study fixed-budget BAI with Bayes simple regret, so the two methods address different problems; running theirs under a fixed budget would require choosing an arbitrary and truncating it, outside its design and guarantees, which is why we do not compare against it. Appendix A now states this distinction.
We hope these changes address the concerns, and we are glad to answer further questions.
Response to Reviewer MqCL (1/4)
We thank the reviewer for a detailed and accurate assessment of both the theory and the experiments. The review identified the two central weaknesses of the submitted version: the theory did not say how much proxies help, and the GDSC experiment did not establish what an LLM can actually contribute. The revision adds three quantitative results and rebuilds the experiment.
Main revisions. Section, table and result numbers refer to the revised PDF.
We respond point by point below, starting with novelty and the theory.