acceptodds
Under review as a conference paper at ICLR 2027

In-Context Pure Exploration in Continuous Decision Spaces

Abstract

In this work, we develop a practical and reusable framework for active sequential testing, also termed pure exploration, with continuous recommendations. In this setting, a learner adaptively acquires information to identify an unknown hypothesis with as few queries as possible while satisfying a prescribed prior-averaged accuracy level. Existing methods are predominantly problem-specific, while learned approaches have been limited to finite recommendation spaces. We formulate this objective as Bayesian fixed-confidence pure exploration and characterize the Bellman structure of an ideal learner. Guided by this characterization, we introduce C-ICPE, which meta-trains sequential neural architectures over a task prior to jointly learn exploration, stopping, and recommendation. At inference, C-ICPE gathers evidence and returns a recommendation without parameter updates. We conclude by analyzing the method with experiments across localization, identification and value estimation problems.

Primary area:
reinforcement learning
Keywords:
active hypothesis testing, pure exploration, reinforcement learning, black-box optimization

Reviews

Among:
Shape:
1234this paper, Top 17%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.75 over 4 reviews, above 83% of the others.
Reviewer_wooDOpenReview ↗
Rating
4
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces C-ICPE, a meta-learning framework for Bayesian fixed-confidence pure exploration with continuous recommendation spaces. It jointly learn exploration, stopping, and inference policies through sequential neural architectures. The theoretical contributions include a Bellman characterization, zero-duality-gap results, and Bayesian (ϵ,δ)(\epsilon,\delta)-correctness under regularity conditions. Experiments on synthetic benchmarks and a geochemical exploration task demonstrate promising accuracy and sample efficiency.

Critical strengths
  1. Meaningful theoretical and methodological contributions. the extension of Bayesian fixed-confidence pure exploration to continuous recommendations is significant. The Bellman characterization and correctness analysis under a weaker local closedness condition are technically interesting.

  2. Broad and promising empirical evaluation. The experiments cover several distinct continuous exploration problems, including decoupled query / recommendation spaces. Results demonstrate competitive sample efficiency and successful adaptation across different tasks.

Critical weaknesses

Limitations in empirical validation and generality. Most comparisons are against fixed-budget optimization methods rather than directly comparable fixed-confidence algorithms; the specialized Lazy Track-and-Stop baseline is more sample-efficient on the linear benchmark. The geochemical experiment uses GP-fitted surrogate functions rather than direct sequential field measurements. Furthermore, reliance on task-prior simulators and known optimal targets limits applicability, while robustness under substantial prior shift remains uncertain.

Questions for authors

Could the authors strengthen the comparison with fixed confidence methods, particularly BO equipped with probabilistic stopping rules, to better isolate the benefits of learned exploration?

5 comments
AuthorsOpenReview ↗

Response (1/2)

We thank the reviewer for the positive assessment, and in particular for finding the continuous extension significant, and the analysis technically interesting. Below, we address your concerns. We have uploaded a revised version of the paper, where the changes are in blue, and each answer points to the changes it makes; this reply cites the revised version.

"Could the authors strengthen the comparison with fixed confidence methods, particularly BO equipped with probabilistic stopping rules, to better isolate the benefits of learned exploration?"

We thank the reviewer for this suggestion. We implemented Wilson's stopping rule, and the revised version adds BO with this probabilistic stopping rule on Ackley, GP max-value and geochem, with GP-UCB and GP-logEI. Furthermore, we also added probabilistic bisection to binary search (Horstein, 1963).

Wilson stopping rule. For each function gg sampled from the GP posterior, Wilson's rule checks if the recommendation x^\hat x has a small value regret, max⁡ag(a)−g(x^)≤ϵ\max_a g(a)-g(\hat x)\le\epsilon. This is the loss of geochem, so on geochem we use the rule as published. On Ackley and GP max-value the loss is different, and a small value regret does not imply a small loss (new, App. C.12, Prop. 14). So on these two tasks we change the check. On Ackley we check ∥x^−xg⋆∥2≤ϵ\Vert\hat x-x^\star_g\Vert_2\le\epsilon, with xg⋆x^\star_g the optimum of gg. On GP max-value the recommendation x^\hat x is an estimate of the max value, and we check ∣x^−max⁡ag(a)∣≤ϵ|\hat x-\max_a g(a)|\le\epsilon.

In general, this rule misses the target accuracy in every configuration (ranges are min-max over configurations and both acquisitions gp-ucb & gp-logEI; new Table 2 and App. E):

BO with Wilson's rule: accuracy, num. of queriesC-ICPE: accuracy, num. of queries
Ackley0.14-0.66, 31-980.910-0.935, 15-61
GP max-value0.82-0.84, 8-150.900-0.927, 21-34
Geochem0.35-0.41, 8-160.919-0.925, 21-42

We find that the GP posterior is overconfident on geochem and GP max-value. The rule computes its probability with a GP whose settings, the lengthscales (how fast the function changes) and the scale (how much it varies), are fitted to the first observations and then used as if they were exact. With so few observations these settings can be wrong, and the GP is then too sure about the regions it has not observed. We believe this is why the rule stops too early. On geochem it stops only when it is at least 95% sure, but it is correct on only 35-41% of the tasks. Wilson (2024) also reports this problem when the settings are fitted (88-92% correct against a 95% target).

We also tried to calibrate the threshold used by the stopping rule. A natural fix is to learn the threshold from data, as C-ICPE learns its stopping rule from tasks of the prior. On geochem we stop at the first tt at which ptp_t, the GP's probability that its recommendation is correct, reaches λ\lambda, and choose the smallest λ\lambda that reaches C-ICPE's training target of 0.918 correct on 200 training tasks (new, App. E.5). No λ∈[0,1]\lambda\in[0,1] works: the GP often gives pt=1p_t=1 to a wrong recommendation, so even λ=1\lambda=1 is correct on only 38-50% of 200 held-out tasks.

In general, this points out that there is indeed a benefit in learning when to stop, and that the exploration strategy also matters where stopping cannot explain the gap: on Ackley no black-box baseline reaches the target even with 100 queries, and the C-ICPE-Uniform ablation, which keeps C-ICPE's stopping rule and changes only the queries, fails on binary search and Ackley at ϵ=0.1\epsilon=0.1. On GP max-value, whose prior is itself a GP, the fixed-budget GP-logEI and TPE do reach the target at the budget of C-ICPE-TD3 (Sec. 5.1; Table 20, Table 15 in the submission).

Probabilistic bisection. The revised version adds probabilistic bisection (Horstein, 1963) as an exact-posterior, fixed-confidence reference for binary search (rows PBA-median and PBA-TS of the binary-search tables, App. E.3.1). Recall that this is a noisy environment, where each query returns one corrupted sign per coordinate (Eq. 52). Bisection knows the probability pp with which each sign is flipped, and keeps the exact posterior and stops once its posterior success reaches 1−δ1-\delta, therefore this is a reference that exploits the structure of the problem. We test two exploration approaches: (i) PBA-median, which queries the median of the posterior, while (ii) PBA-TS queries a sample from this posterior (which is the exploration strategy of C-ICPE-TS applied to the exact posterior).

-With median queries, it meets the accuracy target with fewer queries in every configuration (C-ICPE-TS needs 16-48% more). -With Thompson queries, the exploration strategy of C-ICPE-TS, the gap mostly disappears: C-ICPE-TS uses between 8% fewer and 3% more queries for d≤10d\le 10, and up to 21% more at d=20d=20 (23.0 against 19.0).

The revised version reports this in Sec. 5.1 and App. E.3.1

AuthorsOpenReview ↗

Response (2/2)

"Most comparisons are against fixed-budget optimization methods rather than directly comparable fixed-confidence algorithms; [...]"

Most comparisons are against fixed-budget methods because no widely available fixed-confidence method handles these task families, and providing one is the main contribution of the paper. The few that exist for continuous problems each rely on one structure, such as Jedra and Proutiere (2020), our reference on the sphere.

The comparison with fixed-budget methods is: do these methods reach the target accuracy given the same average number of queries of C-ICPE? If so, they would be themselves fixed-confidence learners (Sec. 5, the new paragraph "Fixed budget and fixed confidence"). We have also added probabilistic bisection for binary search (see our answer to reviewer CqnG) and Wilson's stopping rule (Wilson, 2024) to strenghten the comparison

C-ICPE is a general practical method that can cover these tasks, and to our knowledge it is the only fixed-confidence method that does: its purpose is a practical method for task families where deriving dedicated exploration and stopping rules is difficult (Sec. 6).

"[...] the specialized Lazy Track-and-Stop baseline is more sample-efficient on the linear benchmark."

We agree, and the paper says so (Sec. 5.1). LT&S exploits the known linear observation model and a likelihood-ratio stopping rule. It is however interesting to note that, when using the same round-robin exploration strategy, the learned critic/inference models match the performance of LT&S, stopping after 14.8 queries at d=15d=15 with accuracy about 0.96. This seems to indicate that the gap lies mainly in the exploration strategy.

"The geochemical experiment uses GP-fitted surrogate functions rather than direct sequential field measurements." "Furthermore, reliance on task-prior simulators [...] limits applicability" "[...] and known optimal targets limits applicability,"

We agree, and the last two points are shared with the in-context methods C-ICPE builds on:

  • The geochem tasks are surrogates built from real USGS data. Generally, field measurements are costly to obtain, and a field campaign needs dedicated funding, which in turn requires the kind of evidence this study provides.
  • C-ICPE relies on a simulator of the task prior, as do DPT (Lee et al., 2023), ICPE and Deep Adaptive Design, which also learn from simulated tasks before deployment (Sec. 4, App. B); at deployment it only uses its own queries and observations.
  • C-ICPE needs the target of each task in simulation, as DPT and ICPE do: DPT is trained on the optimal action of each task (Lee et al., 2023), and ICPE on its true hypothesis (Russo et al., 2026). The target is used only for training and certification, never at deployment, and reducing this dependence on supervised targets is one of the directions named in Sec. 6.

"[...] while robustness under substantial prior shift remains uncertain."

App. E.4 tests this directly: frozen models trained under a uniform prior are deployed under Beta (α,β)(\alpha, \beta) priors with (α,β)∈{0.5,1,3,5,7}2(\alpha,\beta) \in\lbrace 0.5,1,3,5,7\rbrace^2, and the paper reports where accuracy holds and where it drops:

  • on geochem, accuracy stays at about 0.90-0.93 over the whole grid (App. E.5);
  • on binary search, TS, TTPS and the mixture stay above 0.85, while TD3 drops to 0.370-0.487 at the most skewed priors;
  • on GP max-value, TD3 drops to 0.770 at the most skewed prior, (α,β)=(0.5,7)(\alpha,\beta)=(0.5,7);
  • on Ackley, accuracy degrades to 0.521-0.656 at the most skewed prior.

App. C.10 bounds the accuracy and the stopping time of a frozen learner in terms of the shift between the training and deployment priors, through their density ratio, chi-square divergence or total variation (Props. 11-12).


We thank the reviewer again for the positive evaluation and for these suggestions, which strengthen the comparison with fixed-confidence methods.

Dear Reviewer, Thank you again for your feedback and time spent reviewing our paper. We would like to know whether our response adequately addresses your concerns or if any further clarification is needed, so we can properly address it before the end of the rebuttal period.

Thank you very much again for your time! Best

Reviewer_MxFiOpenReview ↗
Rating
2
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work introduces an algorithm for sequential decision making that is pretrained. Several variants are suggested and evaluated on synthetic and real world experiments accompanied by theoretical contribution.

Critical strengths

Sequential decision making is a relevant field and the idea of pretraining recent and important.

The experimental results show strong performance in accuracy compared to e.g. GP-UCB.

Paper is mainly well written and good to follow.

Disclaimer: Theory (50 pages with 18 Lemmas, several Propositions, several theorems each requiring proofs) beyond my expertise to check within ICLR timeline. While potentially strong, this requires another reviewers evaluation. Once positively evaluated, I'd see my vote changing to acceptance.

Critical weaknesses

Pretraining is complex and success depends on many design choices. I fail to see discussion on the pretraining.

One of the main advantages of pretraining is speed gain upon inference / deployment. I fail to see any numbers / discussion on this.

The other side of the time aspect: pre-training costs time and I fail to see comment on this.

Fig 4: many methods in the legend, only few applicable to each experiment. Feels a bit inconsistent to only see a subset of methods. Better: define classes of experiments for which methods are applicable and group experiments in these classes. This would also be helpful for practioneers using the method.

Related work on amortized BO / design short [1] https://proceedings.mlr.press/v124/swersky20a.html [2] https://arxiv.org/pdf/2501.15458

In related work, it says that Cheng and Huan learn a continous design. [2] also operates on continous domain. Why claim "while learned approaches have been limited to finite recommendation spaces" in abstract?

Theory in main text hard to understand / verify without appendix. However appendix beyond what I can review in ICRL timeline with my expertise: 50 pages with 18 Lemmas, several Propositions, several theorems each requiring proofs.

Further points:

73: learner chooses a query A_t and 74 returns a recommendation. What's the connection?

Variables undefined: 69: h_t Eq 4: bar values undefined. Eq 8: I_\phi

Alg1: 1: while Training is not over -> when is training over? 8: in an algorithm, it's nice to have everything contained. c is not needed any further (at least not in the write-up) and quantities needed to calculated c such as hat{p}, \delta_train... are not variables of the algorithm

Abbrevations (please introduce before first use): ICPE 119: ASHT 167: TS, TTPS, TD3 168: Q_{\phi}

Questions for authors

eq 1: is E[\tau]< \infty needed or duplicate? 92: the learner chooses/samples A_t ~ \pi_t()? not the recommendation x?

6 comments
AuthorsOpenReview ↗

Response (1/2)

We appreciate the reviewer's feedback and the time they dedicated to evaluating our paper. In particular, we appreciate that the reviewer found pretraining for sequential decision making relevant and important, the accuracy gains over methods such as GP-UCB strong, and the paper well written. We have uploaded a revised version of the paper, where the changes are in blue, and each answer points to the changes it makes; this reply cites the revised version.

"Disclaimer: Theory (50 pages with 18 Lemmas, several Propositions, several theorems each requiring proofs) beyond my expertise to check within ICLR timeline." "Once positively evaluated, I'd see my vote changing to acceptance."

We thank the reviewer for the disclaimer. None of the other three reviewers raises a concern about the correctness of the theory. Particularly, Reviewer CqnG expressed no substantive concerns about soundness, Reviewer 7pmX highlights how the dual analysis "allows non-unique minimisers", and Reviewer wooD finds "The Bellman characterization and correctness analysis under a weaker local closedness condition" technically interesting.

To make the theory easier to check, App. C opens with a roadmap of what each subsection up to C.11 proves and what is new relative to the finite case. The main results rest on a short chain, which the revised version now states, in words, at the start of Sec. 3:

  • the optimal recommendation rule (Prop. 3) and stopping written as an action (Lemma 6);
  • the Bellman equation with an attained continuation query (Thm. C.1 and Prop. 4, stated as Thm. 3.1);
  • zero duality gap and correctness of a dual-optimal policy (Thm. C.2 and Thm. C.3).

"Pretraining is complex and success depends on many design choices." "I fail to see discussion on the pretraining."

App. D describes the pretraining, one design choice per subsection with its reason, and gives the full loop in Alg. 3:

  • the history encoder and time pooling (D.1), and the replay of history prefixes (D.2);
  • the regularized inference update (D.3), and the twin critics with masking of stopped transitions (D.4);
  • the actors (D.5) and the cost update (D.6).

App. E chooses the training target δtrain\delta_{train} for the power of the certification test, and with these choices the selected checkpoint of every C-ICPE variant with adaptive queries and critic stopping is certified in all seeds and configurations (Sec. 5.1). Sec. 5 and App. E also ablate the exploration strategy (TS, TTPS, TD3 or uniform queries), the inference model (one Gaussian or a mixture) and the stopping rule (the critic, or the network's own belief in the +CF variants). We agree that the main text said little about this, and the revised version adds to Sec. 4 a paragraph to point the reader to the correct section.

"One of the main advantages of pretraining is speed gain upon inference / deployment." "The other side of the time aspect: pre-training costs time and I fail to see comment on this."

We agree that the paper reports no wall-clock time at deployment, and the revised version adds these times to App. E, after Computational resources (Tab. 5); they show the gain the reviewer expects. On one NVIDIA L40S GPU, in milliseconds per query, where a GP step updates the GP and maximizes the acquisition function, with no stopping test:

C-ICPE: next query and stopGP-UCB stepGP-logEI step
Ackley2.357.057.4
GP max-value1.726.635.8
Geochem1.9-2.451.965.5

The GP steps take 16-34 times longer within each benchmark. C-ICPE trades training time for this: training is paid once per problem (the point is training once, then deploying on new tasks), which takes about 12 hours per run on a V100 or L40S GPU for the synthetic benchmarks and 70 hours on geochem (as the submission reports). After training the trained model runs with forward passes only and no parameter update.

"Fig 4: many methods in the legend, only few applicable to each experiment." "This would also be helpful for practioneers using the method."

We agree, and the revised version adds Table 1 to Sec. 5, which groups the benchmarks by class, point recommendation with A=X\mathcal A=\mathcal X or value estimation with A≠X\mathcal A\neq\mathcal X, and gives each its loss and the methods compared on it.

Regarding the different experiments, we wanted to test C-ICPE on different scenarios:

  • binary search returns one sign per coordinate rather than a function value, so no BO method applies;
  • TS and TTPS need A=X\mathcal A=\mathcal X, while TD3 covers A≠X\mathcal A\neq\mathcal X;
  • Lazy Track-and-Stop needs a linear model, and Posterior-TS, Moment-TS and probabilistic bisection (new in the revised version, on binary search) need the exact posterior.

We believe this also highlights the versatility of C-ICPE.

AuthorsOpenReview ↗

Response (2/2)

"Related work on amortized BO / design short [1] https://proceedings.mlr.press/v124/swersky20a.html [2] https://arxiv.org/pdf/2501.15458" "In related work, it says that Cheng and Huan learn a continous design." "Why claim "while learned approaches have been limited to finite recommendation spaces" in abstract?"

We thank the reviewer for the references, which the revised version now cites in App. B. The sentence of the abstract refers to the problem defined just before it in the abstract, identifying the hypothesis "with as few queries as possible while satisfying a prescribed prior-averaged accuracy level" (lines 013-015 of the revised version; all line numbers below are of this version), that is, fixed-confidence pure exploration.

Learned methods for this problem, such as ICPE (Russo et al., 2026), have so far required a finite recommendation space, since the continuous analysis is more intricate. Even in bandits with finitely many arms, many or infinitely many correct answers are non-trivial to analyse (Garivier and Kaufmann, 2021; Poiani et al., 2025a).

[2] and Cheng and Huan operate on continuous domains, and [1] on strings, but none of them addresses this problem:

  • [1] and [2] run for a fixed budget, with no stopping rule and no accuracy target: [1] maximizes the acquisition function of BO over discrete strings, and [2] pretrains a policy that chooses continuous queries for active learning of a regression function.
  • Cheng and Huan learn when to stop, but against a prescribed user-chosen cost per experiment and an information-gain reward. Their cost plays the role of the dual cost c=1/λc=1/\lambda of ICPE and C-ICPE, but is fixed by the user, not by an accuracy target. Moreover, they do not provide (ϵ,δ)(\epsilon,\delta) guarantees, and return no recommendation, only a posterior.

For [2] and Cheng and Huan, the continuous space they choose from is that of queries, not recommendations (lines 074-076). That is why, to our knowledge, C-ICPE is the first learned method for fixed-confidence pure exploration with a continuous recommendation space. We agree, however, that the sentence is too broad when read on its own, and in the revised version it reads "learned methods that decide when to stop and return a recommendation with a prescribed accuracy have been limited to finite recommendation spaces".

"73: learner chooses a query A_t and 74 returns a recommendation." "What's the connection?"

The query collects information and the recommendation is the answer: the policy samples the query, At∼πt(⋅∣Ht)A_t\sim\pi_t(\cdot\mid H_t) (line 092), and the inference rule returns the recommendation once, at the stopping time, from the whole history, x^τ=Iτ(Hτ)\hat x_\tau=I_\tau(H_\tau) (line 094). The two lie in the same space for localization (X=A\mathcal X=\mathcal A), while in the value estimation problem the query and answer space differ, and the recommendation is a value (lines 074-076).

"Variables undefined: 69: h_t Eq 4: bar values undefined." "Alg1: 1: while Training is not over -> when is training over?" "c is not needed any further (at least not in the write-up) [...]" "Abbrevations (please introduce before first use): ICPE 119: ASHT 167: TS, TTPS, TD3 168: Q_{\phi}"

  • hth_t is a realization of the history HtH_t (line 072), and πˉ,τˉ\bar\pi,\bar\tau are the policy and stopping time on Aˉ=A∪{astop}\bar{\mathcal A}=\mathcal A\cup\lbrace a_{stop}\rbrace (lines 132-133); the revised version now defines them in Sec. 3. IϕI_\phi is the inference network of Eq. (6).
  • cc is the per-query cost in the critic's continuation target ycont=−c+dr^(h′)+(1−d)Vψˉ(h′)y^{cont}=-c+d\hat r(h')+(1-d)V_{\bar\psi}(h') (Sec. 4.2), used by the critic loss of Eq. (10).
  • We agree that Alg. 1 should be self-contained: the revised version lists δtrain\delta_{train}, the step size of the cost update and the networks among its inputs, computes p^\hat p in the line of the cost update, and states when training ends. Generally, training should end once the cost cc and the sample complexity are stable, indicating convergence of the dual problem. In our experiments, we chose a sufficiently large number of episodes so that it reached convergence, and mostly kept it the same across benchmarks, without heavy tuning (App. D).
  • ICPE is expanded at line 046; the revised version expands ASHT at its first use and C-ICPE in the introduction, introduces QψQ_\psi among the inputs of Alg. 1, and points from Alg. 1 to the definitions of TS, TTPS and TD3 in Sec. 4.

"eq 1: is E[\tau]< \infty needed or duplicate?"

This constraint is used to define the class of learners on which the analysis is written, so that x^τ\hat x_\tau is well defined, and lemmas 5 and 6 hold. The revised version now says this after Eq. (1), and App. C.1 states the problem with the same constraint.


We hope that these answers address the reviewer's concerns, in particular on the evaluation of the theory, and in that case we would be very grateful if you would consider increasing your score.

AuthorsOpenReview ↗

Additional result: deployment time with batched trajectories

Dear reviewer, since our last reply, we also measured the inference speed when the trained model is deployed on many independent tasks in parallel. We compared C-ICPE in one forward pass and the GP baselines with one GP per trajectory, updated and maximized in one batched call.

The table below gives the time of one step for the whole batch, in milliseconds:

Batch sizeC-ICPE: next query and stopGP-UCB stepGP-logEI step
Ackley128 / 256 / 5122.6 / 2.7 / 3.2199.8 / 173.1 / 212.7200.2 / 213.9 / 343.6
GP max-value128 / 256 / 5121.9 / 2.0 / 2.1171.1 / 148.8 / 161.7194.0 / 207.3 / 215.3
Geochem128 / 256 / 5122.1-2.7 / 2.5-3.1 / 3.4-3.9172.4 / 219.0 / 336.3222.1 / 238.4 / 365.7

Here the ranges are min-max values over the same C-ICPE variants as tab. 5.

Both methods gain from batching, since their time per trajectory decreases. However, the step time of C-ICPE grows only from 1.7-2.4 ms for one trajectory (tab. 5) to 2.1-3.9 ms for 512, while a GP step grows from 27-66 ms to 162-366 ms. Hence a GP step takes 16-34 times longer than a C-ICPE decision for one trajectory, and 64-108 times longer for 128 to 512 trajectories.

Reviewer_7pmXOpenReview ↗
Rating
2
Confidence
1(−1)
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper extends in-context pure exploration from finite to continuous recommendation spaces. It formulates Bayesian fixed confidence pure exploration with a continuous recommendation variable, characterises the ideal learner through a Lagrangian dual and a stop/continue Bellman equation, and proves that the optimal continuation value is attained on a compact action set. On that basis it builds C-ICPE, which meta-trains an inference network, a stopping critic and an exploration actor over a task prior, then freezes them at deployment. The paper also gives a finite-sample certification test for a trained checkpoint and a bound on relative sample suboptimality in terms of model error. Experiments cover noisy binary search, sphere best-arm identification, Ackley localisation, a multi-lobe problem, GP max-value estimation and a USGS geochemical surrogate, with 3 seeds, 300 test tasks and a 20,000-task certification test per checkpoint.

Critical strengths

Certification is a genuine methodological contribution and it is executed properly. The null is p ≤ 0.9 against a one-sided exact binomial alternative on 20,000 fresh tasks per checkpoint. Results are reported per seed with min and max, and the test discriminates. on GP value estimation every critic-based variant certifies while the critic-free ablation fails in all four configurations. A certification procedure that passes everything would be decoration; this one separates a component the authors claim is necessary from the same network without it.

The continuous formulation closes a real gap. Finite-space pure exploration has sharp allocation rules, and the continuous case has mostly been handled by problem specific or GP based methods without sample-complexity optimisation. Theorem 3.1 shows that the stop/continue structure survives when the recommendation space is continuous; the dual analysis in Theorem C.2 allows non-unique minimisers, and the proof handles attainment through semicontinuity without assuming it.

Critical weaknesses

The tables do not match the text. Table 15 reports accuracy 0.900 [0.877, 0.919] for C-ICPE-TD3 at epslon= 0.2, sigma=0.1, while Table 17 reports p = 4.1e-16 on 20k trials, which implies about 0.917. The two tables are not reconciled anywhere. That is the kind of mismatch that survives when the summary is written before the tables are checked, and it should have been caught before submission.

Empirical comparisons are not equal. The fixed-budget baselines (TPE, GP-UCB, etc..) get no reported tuning budget, while C-ICPE receives 432 GPU hours of meta-training per synthetic benchmark, 420 on geochem. On sphere best-arm identification the learned actor variants need 1.5 to 6 times more queries than Lazy TaS. The round-robin variant ties only because it adopts Lazy TaS's query rule, so the comparison says nothing about the learned exploration rule. On GP value estimation, GP-logeI reaches 0.978 to 0.996 accuracy against 0.900 to 0.927 for C-ICPE-TD3, although the paper's own +GP control closes part of that gap.

The multi-lobe comparison is run at different caps. C-ICPE-TS reaches the target only when the cap is raised to 300. The mixture uses the default 100, and Table 1 lists only the exception. About 20 percent of C-ICPE-TS episodes are still running at t = 100. The means (25.9, 72.8, 90.0) are consistent with the stated 3 to 3.5 times, but the censoring fractions are missing, so the means are hard to interpret.

Proposition 1 is a conditional inequality, if the critic, actor, inference, stopping errors are small enough, the learned learner is close to optimal. None of those errors is estimated. An exact-posterior comparison of Q-values and stop decisions is absent from all six benchmark suites. the theory motivates the architecture but I dont find it explain any reported number.

Questions for authors
  1. On the two-lobe task, the exact posterior is available. Can you compute the ideal Q-function in principle and report the empirical critic error and the rate at which the learned stop decision disagrees with the optimal one?
  2. For the multi-lobe comparison, report censoring fractions or survival curves alongside the mean query counts. The arms use caps of 100 and 300, and the means are not comparable as printed.
  3. Wilson2024 is cited as a GP stopping rule that controls the posterior probability of excessive value regret, which is the fixed-confidence baseline the GP and geochem experiments need. Why is it not run? The same question applies to the direct predecessor, ICPE, on a discretised version of one task.
  4. Table 15, the implied rate sits inside the interval, so the two tables may be using different task samples. Are they? If they use the same sample, please explain the discrepancy and state which number the certification claim relies on. If they use different samples, please report the sample size, the number of seeds and the accuracy with its interval on the sample used for the 20k trial test, so the two results can be compared directly.
4 comments
AuthorsOpenReview ↗

Response (1/3)

We appreciate the reviewer's feedback, and in particular that the reviewer recognized the methodological contribution and that the continuous formulation closes a real gap. We have uploaded a revised version of the paper, where the changes are in blue, and each answer points to the changes it makes; this reply cites the revised version's tables, figures and line numbers.

"The tables do not match the text." "If they use different samples, please report the sample size, the number of seeds and the accuracy with its interval on the sample used for the 20k trial test, so the two results can be compared directly."

We are sorry for the confusion, the tables are already correct: they evaluate the same model on samples that App. E states are different and independent, and their numbers agree.

  • Table 22 (Table 17 in the submission), is about certification. Certification uses, for each of 3 seeds, "n = 20,000 fresh tasks θ\theta ∼\sim ν\nu, one rollout each, independent of training and of the evaluation episodes" (App. E, lines 4438-4441),
  • Table 20 (Table 15 in the submission), comes from the evaluation episodes, 300 tasks with 15 trajectories each per seed (lines 4522-4535).

As requested, for C-ICPE-TD3 at ϵ=0.2\epsilon=0.2, σ=0.1\sigma=0.1, inverting the exact binomial tail of Table 22 at the smallest and largest p-values gives, respectively, accuracies 0.918 [0.914, 0.922] and 0.916 [0.912, 0.920] (95% Clopper-Pearson; both accuracies are inside the interval of Table 20). The revised version now states this in App. E, and every certification table now also gives the accuracy that its p-values imply.

"Wilson2024 is cited as a GP stopping rule that controls the posterior probability of excessive value regret, which is the fixed-confidence baseline the GP and geochem experiments need." "Why is it not run?"

One of the main reasons is that Wilson's stopping rule does not immediately fully adapt to the various settings we have. However, for the sake of completeness, we implemented Wilson's stopping rule, and the revised version adds BO with this probabilistic stopping rule on Ackley, GP max-value and geochem, with GP-UCB and GP-logEI.

The rule roughly works as follows: for each function gg sampled from the GP posterior, Wilson's rule checks if the recommendation x^\hat x has a small value regret, max⁡ag(a)−g(x^)≤ϵ\max_a g(a)-g(\hat x)\le\epsilon. This is the risk loss of geochem, so on geochem we use the rule as published. On Ackley and GP max-value the risk loss is different, and a small value regret does not imply a small loss (new, App. C.12, Prop. 14). So on these two tasks we change the check. On Ackley we check ∥x^−xg⋆∥2≤ϵ\Vert\hat x-x^\star_g\Vert_2\le\epsilon, with xg⋆x^\star_g the optimum of gg. On GP max-value the recommendation x^\hat x is an estimate of the maximum value, and we check ∣x^−max⁡ag(a)∣≤ϵ|\hat x-\max_a g(a)|\le\epsilon.

In general, this stopping rule misses the target accuracy in every configuration (ranges are min-max values over configurations and both acquisitions, gp-ucb and gp-logEI; new Table 2 and App. E):

BO with Wilson's rule: accuracy, num. of queriesC-ICPE: accuracy, num. of queries
Ackley0.14-0.66, 31-980.910-0.935, 15-61
GP max-value0.82-0.84, 8-150.900-0.927, 21-34
Geochem0.35-0.41, 8-160.919-0.925, 21-42

We find that the GP posterior is overconfident on geochem and GP max-value. The rule computes its probability with a GP whose settings, the lengthscales (how fast the function changes) and the scale (how much it varies), are fitted to the first observations and then used as if they were exact. With few observations these settings can be wrong, and the GP is then too sure about the regions it has not observed. We believe this is why the rule stops too early. On geochem it stops only when it is at least 95% sure, but it is correct on only 35-41% of the tasks. Wilson (2024) also reports this problem when the settings are fitted (88-92% correct against a 95% target).

We also tried to calibrate the threshold used by the stopping rule. A natural fix is to learn the threshold from data, as C-ICPE learns its stopping rule from tasks of the prior. On geochem we stop at the first tt at which ptp_t, the GP's probability that its recommendation is correct, reaches λ\lambda, and choose the smallest λ\lambda that reaches C-ICPE's training target of 0.918 correct on 200 training tasks (new, App. E.5). No λ∈[0,1]\lambda\in[0,1] works: the GP often gives pt=1p_t=1 to a wrong recommendation, so even λ=1\lambda=1 is correct on only 38-50% of 200 held-out tasks.

In general, this points out that there is indeed a benefit in learning when to stop, and that the exploration strategy also matters where stopping cannot explain the gap: on Ackley no black-box baseline reaches the target even with 100 queries, and the C-ICPE-Uniform ablation, which keeps C-ICPE's stopping rule and changes only the queries, fails on binary search and Ackley at ϵ=0.1\epsilon=0.1.

AuthorsOpenReview ↗

Response (2/3)

"On GP value estimation, GP-logeI reaches 0.978 to 0.996 accuracy against 0.900 to 0.927 for C-ICPE-TD3, although the paper's own +GP control closes part of that gap."

We agree, as Sec. 5.1 states (lines 469-471): the prior is itself a GP, the model class of the GP baselines, while C-ICPE learns from tasks drawn from the prior without a GP model and meets and certifies the target accuracy in every configuration (App. E.3.4). Refitting a GP to C-ICPE-TD3's queries recovers 0.02-0.03 of a 0.06-0.08 gap (Table 20), so both the queries and the value estimate contribute.

"The round-robin variant ties only because it adopts Lazy TaS's query rule, so the comparison says nothing about the learned exploration rule."

We agree, and the paper says so: "the gap lies in a sub-optimal exploration rule" (lines 466-467). The round-robin variant tests the learned inference and stopping given LT&S's queries, and they match it (14.8 queries at d=15d=15). The two-lobe task, where the ideal learner is known exactly, shows the same (new, App. C.6, Table 3). On average over three seeds, the learned stop decisions lose at most 0.37 queries, while the queries of Thompson sampling account for 66-92% of the excess sample complexity (the expected number of queries beyond the ideal learner). Recall that LT&S exploits the known linear model, while C-ICPE uses a training procedure that is not task specific (Secs. 1 and 6).

"Empirical comparisons are not equal." "The fixed-budget baselines (TPE, GP-UCB, etc..) get no reported tuning budget, while C-ICPE receives 432 GPU hours of meta-training per synthetic benchmark, 420 on geochem."

We understand the concern. However, the comparison is at equal numbers of queries, the quantity C-ICPE minimizes, not at equal compute: at the mean number of queries of C-ICPE, do the baselines reach 1−δ1-\delta? One that does is itself a fixed-confidence learner, so the optimal learner needs at most as many queries (Sec. 5).

Not only that, but also these baselines are limited in the diversity of scenarios they can tackle. Hence, in general, meta-training is what C-ICPE trades for fast deployment and versatility.

This fast deployment is an advantage Reviewer MxFi also points to: a query and its stop decision take 1.7-2.4 ms on one L40S GPU, against 27-66 ms for one GP-logEI or GP-UCB step without a stopping test (new, App. E).

We compare with these fixed-budget methods because no widely available fixed-confidence method handles all these task families: to our knowledge, C-ICPE is the only one that covers all six, and providing it is the paper's main contribution. Still, every benchmark now also has a fixed-confidence reference (new Table 1 of Sec. 5), each tied to one structure, and two of the methods are new: probabilistic bisection on binary search and BO with Wilson's rule (first comment).

Finally, the baseline configuration is reported in App. E.2. The GP baselines also adapt to each task, since they refit their kernel hyperparameters to its observations by marginal likelihood (App. E.2).

"For the multi-lobe comparison, report censoring fractions or survival curves alongside the mean query counts."

The survival curves are already in Fig. 16 (a) (Fig. 15 (a) in the submission): at d=2d=2, the mixture's is close to zero at its cap of 100 steps, and those of C-ICPE-TS and Moment-TS at 300 steps. These two use a cap of 300 steps at d=2d=2, ℓ=0.15\ell=0.15, since with 100 they would miss the target accuracy, and fixed confidence compares learners that meet it (App. E.2, E.3.5).

"The same question applies to the direct predecessor, ICPE, on a discretised version of one task."

It is generally impracticable to directly apply a discretized version of ICPE, simply because the discretization of the recommendation space would grow exponentially in the dimension.

More importantly, on binary search, Ackley, GP max-value, the sphere and geochem, a discretised ICPE would solve a different problem from Eq. (1). ICPE identifies an element of a finite set exactly, while C-ICPE returns a recommendation within ϵ\epsilon in the task's loss (App. C.5), and a grid does not recover that problem: identifying the true answer is stricter than Lθ(x^)≤ϵL_\theta(\hat x)\le\epsilon on a fine grid, where a target near a cell boundary must be told apart from the next cell, while on a coarse grid the returned centre can be farther than ϵ\epsilon from the target.

ICPE's guarantees, moreover, concern exact identification over a finite set, and Eq. (1) needed a new analysis: with a continuous recommendation the posterior probability of the true hypothesis is zero, so C-ICPE builds on the posterior success qt(h,x)q_t(h,x), attains the best query over a continuous action set (Thm. C.1 and Prop. 4), and proves correctness without ICPE's uniqueness assumption (Thm. C.3).

AuthorsOpenReview ↗

Response (3/3)

"Proposition 1 is a conditional inequality, if the critic, actor, inference, stopping errors are small enough, the learned learner is close to optimal." "None of those errors is estimated." "Can you compute the ideal Q-function in principle and report the empirical critic error and the rate at which the learned stop decision disagrees with the optimal one?"

The purpose of Prop. 1 is theoretical, to show how the relative sample complexity error scales in terms of the model errors. These are, in general, difficult to measure. However, we believe it is important to understand from a theoretical perspective how the model accuracy can influence the excess sample complexity, that is, the expected number of queries that a learner uses beyond the ideal learner. Furthermore, in the appendix we prove that the assumption of the upper bound cannot be relaxed (new Lemma 14 and Corollary 1, App. C.6).

We also followed the suggestion of the reviewer, and the revised version computes these quantities on the two-lobe task (d=1d=1), where a tractable dynamic program gives the ideal learner (new, App. C.6): T⋆≈2.4T^\star\approx 2.4 queries at c⋆≈0.073c^\star\approx 0.073. Along fresh trajectories (new Table 4):

  • the stop head errs by 0.045-0.061 in success probability for the mixture and 0.085-0.096 for C-ICPE-TS;
  • the continuation head differs from the values of the dynamic program at the model's own cost and the best recommendation, by 0.069-0.072 for the mixture and 0.038-0.247 for C-ICPE-TS;
  • the learned stop decision differs from that of this dynamic program, given the proposed query, at 8-20% of visited histories for the mixture and at 7-8% for C-ICPE-TS;
  • C-ICPE-TS's learned cost is 0.04-0.06 below c⋆c^\star, and its recommendation loses 0.37-0.43 in success probability, against 0.15-0.16 for the mixture.

Moreover, Corollary 2 (new, App. C.6) shows that the excess sample complexity of a learner is exactly the sum of the losses of its sub-optimal decisions (stopping too early, too late, or choosing a worse query), plus its recommendation error and its accuracy above the target, both multiplied by 1/c⋆1/c^\star. Unlike Prop. 1, it holds for any errors, so it also applies to C-ICPE-TS, whose errors are too large to satisfy the assumption of the upper bound of Prop. 1. The revised version computes each term for the trained (certified) models of three seeds on the two-lobe task (new, Table 3), and the terms add up to the measured excess sample complexity to within 0.02 queries.The result seems positive for the learned stopping of C-ICPE: on average over the three seeds, early and late stops lose at most 0.37 queries, and the recommendation at the stopping time at most 0.02 (the 0.37-0.43 above is in success probability, over all visited histories). Instead, most of the excess sample complexity (66-92%) comes from the choice of queries. For example, C-ICPE-TS uses 7.6 more queries than the ideal learner at ℓ=0.15\ell=0.15, and 7.0 of them come from its choice of queries, by Thompson sampling on a single Gaussian belief, against 1.5 of 1.7 for the mixture. This is consistent with Thm. 4.1, where a single Gaussian belief tends to query between the two lobes.

"the theory motivates the architecture but I dont find it explain any reported number."

The theory defines the problem that C-ICPE is trained to solve, as for other learned methods. For DPT, the exact solution of its pretraining problem performs posterior sampling, under a slight modification of that pretraining (Lee et al., 2023), and Deep Adaptive Design maximizes a lower bound on the information gain of its policy (Foster et al., 2021, Thm. 2); similarly for ICPE. Our theory concerns the two quantities every table reports: accuracy, the constraint of Eq. (1), and sample complexity, its objective. For accuracy, it proves that the problem has an (ϵ,δ)(\epsilon,\delta)-correct dual-optimal learner (Thm. C.3). For sample complexity, it characterizes the ideal learner through its dual and the stop/continue Bellman equation (Thm. 3.1). Lastly, Thm. 4.1 predicts that the queries of a Gaussian belief double when ℓ\ell halves, while those of the exact posterior barely change, as Table 25 (Table 20 in the submission) shows for Moment-TS (8.8 to 16.8 at d=1d=1) and Posterior-TS (3.0). Moreover, we added two new corollaries that explain the errors of C-ICPE (App. C.6). By Corollary 2 gives the decomposition of the excess sample complexity described above, and on the two-lobe task this decomposition shows that most of it comes from the choice of queries.


We hope that these answers address the reviewer's concerns, and in that case we would be very grateful if you would consider increasing your score.

Dear Reviewer, Thank you again for your feedback and time spent reviewing our paper. We would like to know whether our response adequately addresses your concerns or if any further clarification is needed, so we can properly address it before the end of the rebuttal period.

Thank you very much again for your time! Best

Reviewer_CqnGOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes a framework for in-context pure exploration using a meta-training approach. The approach in (Russo et al., 2026) is quite similar. This paper extends it to continuous decision spaces. The optimization objective aims to minimize the cost of observations across tasks under a constraint on the quality of the recommendation made for each task. They relax the constraint and optimize its Lagrangian dual. They identify the optimal Bellman equations for this sequential decision making task and suggest solving it using reinforcement learning (and some simpler methods, e.g. Thomson sampling where possible). The general idea here is to learn an optimal policy and recommendation rule across sampled tasks and generalize to new target tasks (from the same distribution). This kind of in-context idea is different from standard approaches that directly learn from the target task as it makes more observations (e.g. standard GP-UCB). They also analyze their framework theoretically in terms of sample complexity and correctness.

Critical strengths
  • The proposed method is easy to understand and the idea of in-context pure exploration via the proposed meta-training is very interesting.
  • The proposed method is carefully designed and is supported by some theoretical guarantees.
Critical weaknesses
  • Their proposed method C-ICPE is an extension of ICPE from (Russo et al., 2026) to continuous decision spaces. This extension appears to be quite incremental, including both the method itself and the theoretical analysis. The structure of these papers look very similar and I am not convinced that there is enough novelty in this paper over (Russo et al., 2026).

  • I don't find the experimental evaluation very convincing currently. These are the primary issues: (i) all 6 problem settings are synthetic/simulated tasks. (ii) Your method is best on 3 out of 6 tasks: ackley, binary search and geochem. However, for binary search, there is no baseline included beyond variations of your own method. In addition, I don't see a clear discussion about why your method outperforms baselines on these tasks and why it does not do so on the others. It is impossible to understand what aspect of your method makes it improve on the others without this. (iii) The choice of baselines is a bit unclear. For Geochem, your method outperforms the baselines shown in the plot. To my understanding, all baselines shown in Figure 4 for Geochem are under the fixed budget setting. In contrast, your method targets a fixed confidence setting (which allows learned stopping). Then, it may simply be that your method is better because your method allows learned stopping unlike the baselines, and not because your overall learning framework is better.

Questions for authors

Please see the weaknesses above. I would appreciate clarification of these points.

6 comments
AuthorsOpenReview ↗

Response (1/3)

We appreciate the reviewer's feedback and the time they dedicated to evaluating our paper. In particular, we appreciate that the reviewer found the idea of in-context pure exploration via meta-training very interesting, and the method carefully designed and supported by theoretical guarantees. We have uploaded a revised version of the paper, where the changes are in blue, and each answer points to the changes it makes; this reply cites the revised version.

"This extension appears to be quite incremental, including both the method itself and the theoretical analysis."

We understand the concern of the reviewer. However, the continuous case changes the problem and the analysis. ICPE identifies an element of a finite hypothesis set, and Russo et al. (2026) themselves list continuous hypothesis spaces as a limitation whose extension would "require substantial further theoretical development". With a continuous recommendation space, the posterior probability of the true hypothesis, on which ICPE's inference and stop value rest, is zero, so the analysis is built on the posterior success probability qt(h,x)=P(Lθ(x)≤ϵ∣Ht=h)q_t(h,x)=P(L_\theta(x)\le\epsilon\mid H_t=h). The analysis also differs from ICPE's in three places:

  • The best continuation action must be attained over a compact continuous action set, which needs the semicontinuity induction of Lemmas 7-9 and Thm. C.1, and "None of these steps appear in Russo et al. (2026), where attainment is automatic for finite action spaces" (App. C). This part required most of the work, as we outlined precisely every step of the construction and the assumptions under which the theorem holds.
  • ICPE's proof assumes a single dual-optimal policy and bounds the subgradients of the dual by its exact minimizers, while the Hantoute-López characterization both proofs use ranges over near-optimal policies; Thm. C.3 proves this step under strict feasibility and local closedness only (App. C.5, steps 1-2 of the proof), and Example 1 shows that multipliers need not be unique.
  • For certification, we do not use the pooled approach of ICPE, which requires a monotonicity assumption on the training trajectory.

The analysis also contains results with no finite counterpart: Thm. 4.1, the cost of a unimodal belief; Thm. C.6, the separation between value estimation and localization; and Prop. 1, the fixed-confidence approximation bound.

The method changes too, with Gaussian or mixture inference over a continuous space, a stop head regressed on 1{Lθ(x^)≤ϵ}\mathbf 1\lbrace L_\theta(\hat x)\le\epsilon\rbrace (Eq. 9) and continuous actors. The revised version states the novelty more clearly: the contributions now point to a new paragraph, "Relation to ICPE" (App. B), that collects these differences.

"However, for binary search, there is no baseline included beyond variations of your own method."

We thank the reviewer for the suggestion. The revised version adds probabilistic bisection (Horstein, 1963) as an exact-posterior, fixed-confidence reference (rows PBA-median and PBA-TS of the binary-search tables, App. E.3.1). Recall that this is a noisy environment, where each query returns noisy signs (Eq. 52). Bisection knows the probability pp with which each sign is flipped, and keeps the exact posterior and stops once its posterior success reaches 1−δ1-\delta, therefore this is a reference that exploits the structure of the problem. We test two exploration approaches: PBA-median, which queries the median of the posterior, and PBA-TS, which queries a sample from this posterior.

  • With median queries, it meets the accuracy target with fewer queries in every configuration (C-ICPE-TS needs 16-48% more).
  • With Thompson queries, C-ICPE-TS's exploration strategy, the gap mostly disappears: C-ICPE-TS uses between 8% fewer and 3% more queries for d≤10d\le 10, and up to 21% more at d=20d=20 (23.0 against 19.0).

The revised version reports this in Sec. 5.1 and App. E.3.1.

"(i) all 6 problem settings are synthetic/simulated tasks."

We understand the concern of the reviewer, however we respectfully disagree that this weakens the evaluation. Synthetic or simulated task families are common practice in evaluating in-context and pure-exploration methods: DPT (Lee et al., 2023) is evaluated on bandits and simulated navigation, ICPE on bandits, MDPs and search problems, with one semi-synthetic task built from MNIST, and best-arm identification methods such as Lazy Track-and-Stop (Jedra and Proutiere, 2020) on synthetic instances. Synthetic tasks provide evidence about the method and each allows us to investigate the method on different aspects. Moreover, for synthetic tasks we may have optimal/reference methods we can compare to, so that we can understand the difference in performance. Lastly, one of our evaluations, the geochem experiment, is built on real measurements from the USGS dataset. In general, we believe this set of experiments brings important evidence that is necessary for potential real field experiments.

AuthorsOpenReview ↗

Response (2/3)

We answer (ii) and (iii) by the two questions they raise, regrouping their sentences: why the baselines are mostly fixed-budget (this comment), and which part of C-ICPE explains where it does better and where it does not (next comment).


"(iii) The choice of baselines is a bit unclear." "To my understanding, all baselines shown in Figure 4 for Geochem are under the fixed budget setting." "In contrast, your method targets a fixed confidence setting (which allows learned stopping)."

The baselines are mostly fixed-budget because no widely available fixed-confidence method covers all of these task families, and providing one is the main contribution of the paper: to our knowledge, C-ICPE is the only fixed-confidence method that covers all six, with the same training procedure. Most methods for continuous problems run with a fixed budget: as Wilson (2024) notes, "the vast majority of BO runs proceed until a predetermined budget [...] is exhausted", and practitioners face "the continued lack of interpretable stopping rules". Fixed confidence needs such a rule with a guarantee, and deriving one, with its allocation rule, is difficult for the task families C-ICPE targets (Sec. 6); the few that exist for continuous problems each rely on one structure, such as a linear model, an exact posterior or a GP with value regret. The comparison with fixed-budget methods is therefore in queries: one that reaches 1−δ1-\delta at the mean number of queries of the most sample-efficient C-ICPE variant is a fixed-confidence learner that stops at the same time on every task, so the best fixed-confidence learner needs at most as many queries, and fewer when difficulty varies.

Even BO with the stopping rule of Wilson (2024) would not give our guarantee on these tasks: its guarantee holds under the fitted GP and for the value regret of the returned point, and it needs function values. A new proposition (App. C.12, Prop. 14) shows that value regret certifies neither localization, the loss of Ackley, nor value identification, the loss of GP max-value: for any M>0M>0, there are tasks on which a rule certified for value regret stops before any query, while every correct learner for these risk losses needs at least MM queries on average. On binary search, whose queries return signs, the rule does not apply.

The revised version of the draft nonetheless runs it on Ackley, GP max-value and geochem, adapted to our criterion on Ackley and GP max-value (App. E.2). We also refer to our response to reviewer wooD for more details.

Wilson' stopping rule misses the target accuracy in every configuration (ranges are min-max values over configurations, with GP-UCB or GP-logEI; new Table 2 in the paper):

BO with Wilson's rule: accuracy, queriesC-ICPE: accuracy, queries
Ackley0.14-0.66, 31-980.910-0.935, 15-61
GP max-value0.82-0.84, 8-150.900-0.927, 21-34
Geochem0.35-0.41, 8-160.919-0.925, 21-42

Lastly, where the structure allows a fixed-confidence reference, the revised version compares with it (new Table 1 of Sec. 5): Lazy Track-and-Stop on the sphere, exact-posterior rules on multi-lobe, and probabilistic bisection on binary search (previous comment).

AuthorsOpenReview ↗

Response (3/3)

"(ii) Your method is best on 3 out of 6 tasks: ackley, binary search and geochem." "In addition, I don't see a clear discussion about why your method outperforms baselines on these tasks and why it does not do so on the others." "It is impossible to understand what aspect of your method makes it improve on the others without this." "For Geochem, your method outperforms the baselines shown in the plot." "Then, it may simply be that your method is better because your method allows learned stopping unlike the baselines, and not because your overall learning framework is better."

We understand the concern of the reviewer, and we agree that the submission discussed this too briefly; the revised version adds a summary near the end of Sec. 5.1, and a longer discussion in App. B.1, "Learned stopping and exploration". The reviewer is right that part of the gain comes from learning when to stop, which a fixed-confidence learner does and a fixed-budget baseline cannot do. A fixed-budget method spends the same number of queries on every task, while C-ICPE decides on each task when to stop, so it can stop early on easy tasks and continue on hard ones. A fixed-budget method that reaches the target is itself a fixed-confidence learner, so the optimal learner needs at most as many queries, and fewer when the difficulty varies across tasks (Sec. 5).

However, learned stopping is part of our framework, not separate from it. Moreover, allowing a method to stop is not enough if the stopping rule is not good enough. For example, with the same inference network and actor, the +CF variants replace the learned stopping rule with a threshold on the network's own belief, and they fail the certification test on four benchmarks and need 2.2-2.5 times more queries on geochem (Sec. 5.1). Similarly, adding Wilson's stopping rule to BO does not bring it to the target on any of the three benchmarks (previous comment).

Where to explore also matters when stopping cannot explain the gap. On Ackley no black-box baseline reaches the target even with 100 queries (C-ICPE's max horizon), and C-ICPE-Uniform, which keeps C-ICPE's stopping rule but queries uniformly, fails on Ackley and binary search at ϵ=0.1\epsilon=0.1. On geochem, instead, GP-logEI reaches 0.98 accuracy with 100 queries, and we believe that there the learned stopping rule, which the +CF variants lack, accounts for much of the gain, although the fixed-budget comparison alone does not separate the two (Sec. 5.1). Calibrating BO's stopping rule does not close the gap either: with a threshold on its GP's success probability chosen on training tasks, BO stops after 11.6-29.1 queries but is correct on only 38-50% of the tasks (Table 2).

C-ICPE does not win where another method exploits a structure that C-ICPE has to learn from samples: Lazy Track-and-Stop uses the linear model of the sphere, bisection the known flip probability of binary search, Posterior-TS the exact posterior of multi-lobe, and on GP max-value the prior is itself a GP, the model class of the GP baselines. It is however interesting to note that, with the same exploration strategy, C-ICPE comes close on the sphere and on binary search: with round-robin queries it matches Lazy Track-and-Stop (14.8 queries at d=15d=15), and with Thompson queries it needs about as many queries as bisection for d≤10d\le 10. This seems to indicate that the gap lies mainly in the exploration strategy.

In general, the gains come from what the fixed-confidence formulation asks C-ICPE to learn: when to stop and, on the harder tasks, the belief or actor that guides where to query. We believe this also highlights the purpose of C-ICPE, a practical method with one training procedure for all six task families, which meets the target on all of them (Sec. 5.1).


We hope that these answers address the reviewer's concerns, and in that case we would be very grateful if you would consider increasing your score.

Comments on the paper

3 comments
AuthorsOpenReview ↗

General response (1/3)

We thank the reviewers for their careful reading and their constructive feedback. We have uploaded a revised version of the paper, which uses the extra page of the main text and shows new and changed text in blue. Tables and figures below are numbered as in the revised version. In this part we summarize the main novelty and give the list of changes in the revised version; parts (2/3) and (3/3) then address the points raised by several reviewers.


Main novelty

C-ICPE addresses Bayesian fixed-confidence pure exploration with a continuous recommendation space: the learner minimizes the expected number of queries subject to the prior-averaged guarantee P(Lθ(x^τ)≤ϵ)≥1−δP(L_\theta(\hat x_\tau)\leq\epsilon)\geq 1-\delta (Eq. 1). Here the posterior probability of the true hypothesis, on which the finite case of ICPE (Russo et al., 2026) is built, is zero, so the analysis is built on the posterior success probability qt(h,x)=P(Lθ(x)≤ϵ∣Ht=h)q_t(h,x)=P(L_\theta(x)\le\epsilon\mid H_t=h). On this basis we prove that:

  • the stop/continue Bellman equation holds with an attained continuation query over a compact continuous action set (Thm. 3.1, Thm. C.1);
  • a dual-optimal policy is (ϵ,δ)(\epsilon,\delta)-correct without the uniqueness assumption of ICPE (Thm. C.3).

The method learns its belief, stopping and a continuous recommendation jointly, and queries from the learned belief or with a learned actor.

The main goal is devising a practical method. In fact, deriving optimal strategies is notoriously difficult, and depends on the underlying problem. C-ICPE tries to abstract away this difficulty by training a sequential neural architecture, with a general training procedure.


Changes in the revised version

Main text (one extra page):

  1. Sec. 1: a pointer from the contributions to "Relation to ICPE" in App. B, and the existing fixed-confidence methods described as App. B describes them (CqnG, 7pmX, wooD).
  2. Sec. 5: the paragraph "Fixed budget and fixed confidence" and Table 1, with each benchmark's loss, fixed-confidence reference and fixed-budget baselines (CqnG, wooD, 7pmX, MxFi).
  3. Sec. 5.1: added probabilistic bisection (also in Fig. 3), BO with Wilson's stopping rule, a table of the methods (Table 2), more discussion on the experiments, see also the extended discussion int he appendix (CqnG, 7pmX, wooD).
  4. Abstract, Secs. 1-4: the precise claim on finite recommendation spaces, the chain of the main results, notation and abbreviations, the role of E[τ]<∞\mathbb{E}[\tau]<\infty, the training design, and the inputs of Alg. 1 (MxFi).

Appendix:

  1. App. C.12, new: a proposition on value regret against our risk losses (7pmX, wooD).
  2. App. B: "Relation to ICPE", what each fixed-confidence method for continuous problems covers, the references of MxFi, and a new discussion (App. B.1), to which the shorter Sec. 6 now points.
  3. App. E: probabilistic bisection, Wilson's stopping rule and a calibrated GP threshold on geochem, with a figure of accuracy against sample complexity, the corrected sentence on +GP, the samples behind the accuracy and certification tables, with the accuracy each p-value implies, the baselines' configuration (GP-UCB's kernel corrected from Matérn to RBF), and deployment times (7pmX, MxFi, CqnG).
  4. App. C.5: how the proofs differs from ICPE's (CqnG). App. C.6: why the assumption of Prop. 1's upper bound cannot be relaxed (new Lemma 14 and Corollary 1), an exact decomposition of the queries that a learner uses beyond the optimal one (new Corollary 2), and the errors of Prop. 1 on the two-lobe task (7pmX).
  5. App. D: when training ends; App. C.1: the problem stated with the constraint E[τ]<∞\mathbb{E}[\tau]<\infty (MxFi).

The points raised by several reviewers, in parts (2/3) and (3/3), are G1, why the baselines are mostly fixed-budget; G2, what is new relative to ICPE; G3, what each benchmark isolates; G4, real data, simulators and prior shift; G5, training and deployment cost.

AuthorsOpenReview ↗

General response (2/3)

G1: why the baselines are mostly fixed-budget (CqnG, 7pmX, wooD)

There is no widely available fixed-confidence method that handles active testing problem in this generality in continuous decision spaces: the few that exist rely on the underlying structure of the problem (e.g., a linear model, a GP, or an exact posterior). Providing such a method is the main contribution of the paper: to our knowledge, C-ICPE is the only fixed-confidence method that covers all six task families, with the same training procedure.

In our comparisons, we mainly compare in terms of accuracy/sample-complexity: at the same sample complexity, does a fixed-budget method reach 1−δ1-\delta accuracy? One that does is a fixed-confidence learner by definition. Moreover, C-ICPE trades training time for a faster inference speed (see next comment).

Two ablations separate the parts within C-ICPE:

  • C-ICPE-Uniform changes only the exploration, and fails on binary search and Ackley at ϵ=0.1\epsilon=0.1;
  • the +CF variants change only the stopping rule, and fail the certification test on four benchmarks.

On Ackley, no baseline reaches the target within 100 queries, C-ICPE's max horizon, with a fixed budget or with the stopping rule below (at most 0.60 of the tasks at d=5d=5), so stopping cannot explain the gap.

Where the structure allows a fixed-confidence reference, the revised version of the paper compares with it (new Table 1 of Sec. 5): Lazy Track-and-Stop on the sphere, exact-posterior rules on multi-lobe and, new, probabilistic bisection on binary search (G3). Also new is BO with the stopping rule of Wilson (2024) on Ackley, GP max-value and geochem, with GP-UCB or GP-logEI queries on a GP fitted to the observations of each task (App. E.2):

  • Wilson's rule estimates, from samples of the GP posterior, the probability that the recommendation is correct, and stops once a Clopper-Pearson interval shows that it is at least 0.95, so that, like C-ICPE, it stops early on easy tasks;
  • as published, a sample counts as correct when the recommendation's value regret is at most ϵ\epsilon, which is the loss of geochem;
  • the losses of Ackley and GP max-value can define problems of different complexity (new proposition, App. C.12), so there we adapted the rule: a sample counts as correct when its optimum is within ϵ\epsilon of the recommendation (Ackley), or its maximum within ϵ\epsilon of the recommended value (GP max-value).

In general this rule misses the target accuracy in every configuration (ranges are min-max values over configurations and both acquisitions, gp-ucb and gp-logEI; new Table 2 and App. E):

BO with Wilson's rule: accuracy, queriesC-ICPE: accuracy, queries
Ackley0.14-0.66, 31-980.910-0.935, 15-61
GP max-value0.82-0.84, 8-150.900-0.927, 21-34
Geochem0.35-0.41, 8-160.919-0.925, 21-42

On geochem and GP max-value, the fitted GP overstates the success of its own stops:

  • the rule stops only once its test shows a posterior success of at least 0.95, so under its own model its stops would be correct with probability at least 0.9;
  • nearly every episode ends this way, yet only 35-41% (geochem) and 82-84% (GP max-value) of the tasks are solved;
  • Wilson's own runs on GP sample paths with fitted hyperparameters fall short too (88-92% against a 95% target).

On Ackley, instead, the cap of 100 queries ends 11-96% of the episodes. C-ICPE's accuracy is certified on fresh tasks from the prior (Prop. 2).


G2: what is new relative to ICPE (CqnG, 7pmX)

C-ICPE departs from ICPE in three places of the analysis, and adds results with no counterpart in ICPE. ICPE assumes finite hypothesis and action sets, and its success event is the identification of an almost surely unique hypothesis. Here:

  • attaining the best continuation query needs Lemmas 7-9 and Thm. C.1, and "None of these steps appear in Russo et al. (2026)" (App. C);
  • Thm. C.3 needs only strict feasibility and local closedness, where ICPE assumes a singleton set of dual-optimal policies;
  • certification tests each frozen checkpoint, where ICPE's test is pooled.

Thm. 4.1, Thm. C.6 and Prop. 1 have no counterpart in ICPE.

The revised version collects these differences in a new paragraph of App. B, "Relation to ICPE", to which the contributions now point.


G3, G4 and G5 follow in part (3/3).

AuthorsOpenReview ↗

General response (3/3)

G3: what each benchmark isolates (CqnG, 7pmX, MxFi, wooD)

The benchmarks are designed so that each one isolates one aspect of the continuous fixed-confidence problem (App. E.1 and E.5), with a fixed-confidence reference wherever its structure provides one (Table 1):

  • binary search returns one noisy sign per coordinate, up to d=20d=20, so it tests learning from feedback that is not a function value, where no BO method applies. Probabilistic bisection on the exact posterior needs fewer queries with median queries (C-ICPE-TS needs 16-48% more), while with the same Thompson queries, a comparison that isolates the learned inference and stopping, C-ICPE-TS needs about as many for d≤10d\le 10 (new, App. E.3.1);
  • the sphere has a directional loss and a linear observation model, and Lazy Track-and-Stop is a dedicated algorithm for it. Running C-ICPE with the round-robin queries of Lazy Track-and-Stop isolates the learned inference and stopping, and with these queries C-ICPE matches it (14.8 against 15.0 queries at d=15d=15), while its other actor variants typically need more;
  • Ackley has random parameters and a multimodal surface with many local optima, so it tests exploration that is not purely local. No baseline reaches the target within 100 queries, while C-ICPE meets and certifies it, and with uniform queries C-ICPE drops to 0.293 at d=5d=5;
  • multi-lobe has a multimodal posterior that is known exactly, so it tests the inference model. On the two-lobe task, Thm. 4.1 shows that the queries of a single Gaussian belief grow as 1/ℓ1/\ell, while those of the exact posterior stay bounded, and at d=2d=2, ℓ=0.15\ell=0.15, C-ICPE-TS, with a single Gaussian, needs 72.8 queries, against 25.9 for the mixture and 17.4 for Posterior-TS;
  • GP max-value queries locations but recommends a value, so it tests the TD3 actor, which is needed when A≠X\mathcal A\neq\mathcal X. Here GP-logEI and TPE are more accurate at matched budget, while C-ICPE meets the target accuracy without a GP model, although the prior is itself a GP;
  • geochem uses GP priors fitted to USGS copper data and evaluates on held-out regions with a different spatial structure, so it tests realistic surfaces and a shift to regions not seen in training. C-ICPE meets the target accuracy, while the fixed-budget baselines at matched budget and BO with Wilson's rule or a calibrated threshold do not (Fig. 4 and new Table 2).

Hence a method that exploits a structure C-ICPE must learn can need fewer queries. However, on the sphere and on binary search for d≤10d\le 10, the learned inference and stopping come close to these references with the same exploration strategy, so the gap there lies in the exploration strategy. C-ICPE also meets and certifies the target on all six benchmarks. The revised version summarizes this near the end of Sec. 5.1.


G4: real data, simulators and prior shift (CqnG, wooD)

Synthetic or simulated task families are common practice in evaluating in-context and pure-exploration methods, as for DPT (Lee et al., 2023), ICPE and best-arm identification, and C-ICPE, like DPT and ICPE, learns from simulated tasks (Sec. 4). Regarding real data:

  • geochem fits one GP prior to the USGS copper measurements of each of 17 regions and holds out 5 regions (App. E.5); we agree that it is a surrogate;
  • direct field measurements cost hundreds to thousands of dollars per sample (App. E.5), and a funded campaign first requires the kind of evidence provided in this paper;
  • the surrogates also allow each checkpoint to be certified on 20,000 fresh tasks with known ground truth.

Under shifted Beta deployment priors (App. E.4 and E.5; bounds in App. C.10):

  • accuracy stays at about 0.90-0.93 on geochem;
  • on binary search, TS, TTPS and the mixture stay above 0.85, while TD3 drops to 0.370-0.487 at the most skewed priors;
  • on Ackley it degrades to 0.521-0.656 at the most skewed prior.

G5: training and deployment cost (7pmX, MxFi)

C-ICPE trades training time for fast forward inference at deployment, since it is trained once and then runs on any new task with forward passes only, without parameter updates. On one NVIDIA L40S GPU, its next query and stop decision take 1.7-2.4 ms together, while one step of GP-logEI or GP-UCB takes 27-66 ms, 16-34 times longer within each benchmark. When both methods process 128 to 512 trajectories in one batch, a GP step takes 64-108 times longer than a C-ICPE step. We also refer to our answer to reviewer MxFi.


We thank the reviewers and the area chair again for their time and their constructive feedback, which improved the paper. We hope that these changes and our answers address the points raised, and we are happy to answer any further question during the discussion.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.