acceptodds
Under review as a conference paper at ICLR 2027

Tail-Influence Sampling for CVaR Policy Evaluation

Abstract

Policies with similar mean returns can differ sharply in rare failures, yet estimating lower-tail conditional value-at-risk (CVaR) accurately can require many costly rollouts. When different conditional components of a stochastic workflow can be queried separately, we ask how to allocate a fixed evaluation budget to estimate a fixed policy’s CVaR most accurately. We derive a tail influence for each queryable conditional law that aggregates how its uncertainty affects CVaR across every Bellman reuse. Its variance yields the fixed-design efficiency bound and the oracle Neyman allocation. Tail-Influence Sampling (TIS) estimates these influence scales from a pilot model and reallocates fresh queries toward kernels that matter most for the tail; a visitation-anchored variant protects against pilot underallocation. Under fixed dimension and a positive quantile margin, TIS attains oracle asymptotic variance and first-order MSE including pilot cost, while the anchored variant is within a factor two of the oracle. We also characterize an exact-grid regime in which tail- and mean-optimal allocations coincide. On CliffWalking, TIS reduces MSE by 41% versus learned occupancy and 76% versus complete rollouts at the same charged transition budget. In frozen language-model review workflows, anchored TIS beats an equally regularized mean-influence blend in 23 of 24 MMLU-Pro settings and reaches lower MSE than rollouts on six-call FinQA reviews.

Primary area:
reinforcement learning
Keywords:
distributional reinforcement learning, policy evaluation, conditional value-at-risk (CVaR)

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_qUfsOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the problem on how to efficiently allocate limited sampling budget to estimate a given policy's CVaR. The authors proposed Tail-Influence Sampling (TIS) framework under the assumption of direct access to samples from individual state–action pairs. The proposed framework uses the pilot model to identify transition pairs that influence CVaR most significantly and allocates more queries to them accordingly. Theoretically, TIS achieves asymptotically optimal allocation variance under fixed dimensional and positive quantile-margin assumptions. The proposed methods is evaluated on tabular settings and frozen LLM experiments, which generally shows improvement than benchmark methods.

Critical strengths
  1. The design and derivation of tail-influence score in Section 3 is quite interesting and provided a principled way to capture how uncertainty in each transition kernel affects CVaR.
  2. The theoretical analysis of the proposed framework established oracle asymptotic variance and first order MSE, which I belive provide valuable insight on similar problems.
Critical weaknesses
  1. I think the biggest weakness is the proposed framework requires a finite, known state-action space with direct sampling access. Such setting is quite restrictive, and it's unclear if the proposed framework can be generalized to more complex and practical settings.
  2. Theoretical guarantees rely on positive quantile margin assumption and are asymptotic. More discussion on the finite sample sensitivity to margin requirements and exploration floor would be needed to gain a better understanding of the proposed framework.
Questions for authors
  1. Section 2: The problem formulation assumes a finite, known state–action space, with all dimensions fixed in the asymptotic analysis. This setting seems restrictive. Could the authors provide a concrete practical scenario where these assumptions hold and discuss the main challenges in extending the approach to continuous state or action spaces?
  2. Line 175: The results in this section assumes a positive quantile margin to keep the CVaR cutoff locally unchanged. I was wondering the sensitivity of the results on finite-sample accuracy would depend on this assumption.
  3. Line 396: The experiment settings sampled from a well-calibrated conditional distributions. While this provides a clean benchmark for comparisons, it’s unclear whether the improvement of the proposed framework could persist in more realistic workflows with richer outputs, history-dependent behavior, or distribution shift. Could the authors provide further discussion on this?
  4. Table 2: It’s interesting that on held-out FinQA, combining mean-based sampling with visitation-based sampling performs better than anchored TIS in 15 of 16 comparisons. I was wondering whether the improvements require the proposed tail-specific score or can be achieved with the simpler mean-based alternative, and how to quantify the additional benefit from tail-specific score?
1 comment

Thank you for your review, and for the questions.

Q1, a practical scenario. Consider an inventory simulator that can be set to a chosen state, in which the period and the stock level form a finite state. A query sets the state and returns one period's profit and the next stock level, without revealing the demand or disruption probabilities. The new Q6 uses this simulator with a language model as the ordering policy, so that each query is one model call, and TIS has .20–.45 of the MSE of complete rollouts evaluated with the same estimator at α=.01\alpha = .01 (Appendix I). The access can also be weaker, since TIS keeps its advantage when it may restart only from states reached in its own pilot runs (Appendix F.5). Our experiments use finite states and actions. Continuous ones require a discretization, whose error would need a bound like that of the return grid in Appendix E.1. The main challenges are that a finer grid means more kernels, each reused less often, and that our asymptotics hold the number of kernels fixed. Section 7 lists it as future work.

Q2, the margin and the floor at finite budgets. The margin enters through Equation 34, which bounds the probability of a wrong cutoff by c3exp⁡(−c4 nmin⁡mα2)c_3 \exp(-c_4\, n_{\min} m_\alpha^2), where nmin⁡n_{\min} is the smallest sample size and mαm_\alpha is the margin; Section 4 now states this. Two results show the margin's effect in practice. In the two-call workflows, the two questions that dominate the error have margins of .001 and .006. The oracle allocation gives a heavily reached kernel few samples there and has 3.8–5.9 times the anchor's MSE even at 1,600 and 6,400 queries per kernel. The anchor stays within 3% of its first-order prediction, because its visitation share keeps such kernels sampled (Appendix G.2). At α=.01\alpha = .01 on inventory, several cases have margins near zero (down to 2×10−72\times10^{-7}), yet the anchor outperforms the mean-based blend in all 18 (Appendix F.4). A small margin was therefore harmful only when a kernel that many runs reach was left with few samples.

The floor N−1/4N^{-1/4} keeps nmin⁡n_{\min} growing, which is how the theory controls this term. On inventory the allocation is robust to it: over pilot fractions from 3.7% to 28.3% and floors N−1/6N^{-1/6}, N−1/4N^{-1/4} and N−1/3N^{-1/3}, the learned allocation's leading variance has a median of 1.02–1.12 times the oracle's (300 pilots per setting; Appendix E.2.2). In the six-call review workflows the floor does matter for plain TIS, since raising it from .072 to .40 brings its MSE from 1.38 and 1.77 times uniform's down to .47 and .55 (Appendix E.2.4). The anchor's visitation share has the same effect. A finite-budget MSE guarantee remains open, and Section 7 says so.

Q3, richer and history-dependent workflows. The gain needs a state that can be reset, as Sections 1 and 7 now state. With history-dependent behaviour it does not persist: in a live review loop where every call sees the whole conversation, restarts from stored conversations were on par with complete runs in two studies (MSE ratios .89 and .91) and less accurate in the third (1.31; Appendix H). In Q6, where a language model drives a resettable simulator, the allocation yields a clear gain (Appendix I): in a live run in which Qwen3-4B sampled 26.8 million orders, TIS has .24 (α=.01\alpha = .01) and .46 (α=.05\alpha = .05) of the MSE of complete rollouts evaluated with the same estimator. Each order is a single token, which makes the ground truth exact; extending the method to richer outputs and to distribution shift is future work (Section 7).

Q4, the added value of the tail score. We measure it against the same design with a mean score. The two designs share the pilot, the floor, the mixture and the estimator, and differ only in the score.

Settinganchor / mean-score design
inventory, raw profit, α=.1/.05/.01\alpha = .1 / .05 / .01.84 / .78 / .65 (significantly lower in 16, 17, 18 of 18; 0 higher)
inventory, raw profit, α=.001/.0001\alpha = .001 / .0001.52 / .40 (significantly lower in all 18)
language model as policy, α=.05/.01\alpha = .05 / .01.60 / .42 (Qwen3-4B), .69 / .54 (Phi-4-mini)
CliffWalking, 100–400 queries per kernel.83–.87
MMLU-Pro, Brier utilitylower in 11 of 18 cells, higher in none
held-out FinQAmean score better in 15 of 16 cells
live conversation loops.99–1.01 (no difference)

The gain grows for rarer tails, and the mean score is better where the two coincide, as on FinQA. A rule fixed before evaluation, based on the pilot's tail–mean distance, picks the better or tied design in 141 of 148 cells (Q5).

Reviewer_pyhQOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper considers the static cvar policy evaluation problem, i.e., how to improve sample efficiency of estimating the average tail performance of a fixed policy. The proposed method (TIS) assumes a generative model of the environment that allows sampling for each individual state-action transition function. It computes a tail influence to characterize how estimation error in each individual transition contributes to the final cvar estimation, and uses the score to estimate an approximately optimal sampling.

Critical strengths
  1. The perspective of reweighting the data point to improve the efficiency of cvar policy evaluation is interesting. Unlike conventional visitation-based or mean-influence sampling, the proposed method explicitly accounts for the contribution of each transition kernel to cvar estimation uncertainty.

  2. The paper provides theoretical guarantees for the proposed sampling strategy.

Critical weaknesses
  1. Writing. I feel the manuscript is heavily LLM-assisted and is difficult to follow. In Section 5, the authors present experimental results without introducing the experimental environments, making it difficult to understand the settings and interpret the results. Furthermore, the related work is only discussed near the end of the manuscript (Section 6), and the discussion does not sufficiently clarify how the proposed method differs from existing approaches or establish its novelty.

  2. Strong assumptions. The proposed method assumes access to a generative model that allows direct sampling from arbitrary state-action transition kernels. This is a strong assumption that may not hold in many practical applications, potentially limiting the applicability of the method. On the other hand, if the transition probabilities are fully known, static CVaR policy evaluation can be performed through dynamic programming (see Time-Consistent Decisions and Temporal Decomposition of Coherent Risk Functionals, https://www.jstor.org/stable/24736258). The authors should clarify the distinction between sampling access and full knowledge of the transition model, and explain under what circumstances the proposed sampling-based approach is preferable to dynamic programming.

  3. Experimental comparisons. The comparison with complete trajectory rollouts is not fully controlled, as the proposed method benefits from both direct access to transition kernels and model-based distributional Bellman evaluation. Consequently, the reported improvements may result from the combined effects of the evaluation framework and the proposed tail-influence sampling strategy, rather than the sampling strategy alone. Although the authors include several allocation baselines, an additional ablation study separating the contributions of model-based evaluation and adaptive sampling would help establish the specific benefits of the proposed method.

Questions for authors

N/A

6 comments

Thank you for the review. Your three points led to the largest changes in the paper.

Writing, related work and novelty. Table 1 now lists every benchmark task up front, stating what one query returns, the number of kernels and the horizon. Related work has moved to Section 2, and each of its paragraphs presents prior work and says how our setting differs. Neyman allocation and the adaptive allocation of simulation effort are classical. Our contribution is the score for a tail quantity defined by a Bellman recursion: one sample of a reused kernel affects every stage that reuses it, and how much it matters depends on the rest of the model and on the tail cutoff, both unknown. Theorem 3 shows that this score can be learned from a pilot with no loss in the limit. Section 6 now opens with a box of the main findings, and each question is followed by its answer. In the introduction and in Sections 3 to 5 we clarified the passages that introduce the main ideas; no equation or theorem changed.

Sampling access versus a known model. You are right that, with known transition probabilities, static CVaR follows from dynamic programming (Pflug and Pichler, 2016). Section 2 now says so, and Section 3 states the access we assume. Our evaluator never sees the probabilities: each kernel is a black box, such as one step of a simulator or one model call, and each sample has a cost. Our estimator applies the same recursion to the fitted probabilities, and the question we study is where to sample. With known probabilities dynamic programming is exact and preferable; our method is for unknown probabilities and costly samples. We also tested a form of access weaker than a generative model, in which TIS may restart only from states it reached in its own pilot runs. It then has .73 of the MSE of rollouts under our estimator on CliffWalking (CI .61–.88), and anchored TIS has .93 on inventory (CI .88–.98; Appendix F.5). In our language-model studies we read the model's probabilities only to compute the ground truth, and the evaluator is given samples.

Estimator versus sampling strategy. We agree, and the ablation you asked for shows that both effects contribute. Applying our estimator to the complete rollouts' own transitions, which we call fitted-model rollouts, reduces their MSE to about a third on CliffWalking and to .11–.40 on inventory. With the estimator held fixed, the allocation still helps on the control and inventory tasks:

CliffWalking, queries per kernel100200400
fitted-model / complete rollouts.32.31.34
TIS / fitted-model rollouts.79.83.68
TIS / learned mean influence.89*.82.79

*not significant (∣z∣<2|z| < 2).

Inventory, raw profitα=.1\alpha = .1α=.05\alpha = .05α=.01\alpha = .01
fitted-model / complete rollouts.40.28.11
TIS / fitted-model rollouts.86.75.60
learned mean / fitted-model rollouts1.071.051.04

The inventory table is at 1,200 queries per kernel. At α=.1\alpha = .1 the gain appears from 600 queries per kernel, with TIS / fitted-model ratios of 1.05, .98, .86, .86 and .79 at 150, 300, 600, 1,200 and 2,400 queries.

Access alone does not explain the gain, since querying each kernel equally has 66–87 times the MSE of fitted-model rollouts on CliffWalking.

On the FinQA review workflows, the same ablation shows that fitted-model rollouts match or outperform the anchor (Appendix G.6), so the gains over rollouts we reported there come from the estimator. Q4 now attributes them to the estimator, and the abstract cites the inventory results. This control was run on FinQA and was not repeated on MMLU-Pro. With a language model as the policy of the inventory simulator, by contrast, the allocation does help: TIS has .20–.45 of the MSE of fitted-model rollouts at α=.01\alpha = .01 (Q6, Appendix I).

The theory holds unchanged, because the theorems compare allocations under one estimator, independently of the comparison with complete rollouts.

Thanks for the reply.

I remain unconvinced by the motivation behind the proposed method. Given the assumed access to a generative model, one can directly estimate the transition probabilities through sampling and subsequently perform static CVaR policy evaluation via its temporal decomposition in the augmented state space (s,α)(s,\alpha) (Pflug and Pichler, 2016), without explicitly estimating the entire return distribution. It remains unclear what advantages the proposed method offers over this more direct approach.

Thank you for following up. The approach you describe is the estimator we already use. Our contribution is the step that comes before it: deciding how many samples each state-action pair receives.

We estimate each transition law from samples and compute the static CVaR of the fitted model (Section 3). Whether that CVaR is computed with the temporal decomposition of Pflug and Pichler (2016) or with our recursion, the result is the same number, because both are exact for the fitted model (our return grids contain every reachable return). The return distribution is only an intermediate step of the computation; nothing extra is estimated.

What the direct approach does not specify: "estimate the transition probabilities through sampling" requires choosing how many samples to draw from each state-action pair, and when samples are costly this choice determines the error. With shares wgw_g of the budget, the estimate's asymptotic MSE is V(w)=∑gσg2/wgV(w)=\sum_g \sigma_g^2/w_g, where σg\sigma_g is the tail influence of pair gg (Theorem 1). Theorem 2 shows that, for given shares, no regular estimator has a smaller asymptotic variance, whichever way the CVaR is computed. The only way to reduce the error is therefore to choose better shares. This is not a standard variance-based allocation, because σg\sigma_g depends on the whole model and on the tail cutoff, both unknown. TIS learns the σg\sigma_g from a pilot and chooses the shares that minimize V(w)V(w), at no first-order cost (Theorem 3).

The direct approach with equal samples per pair is our "uniform" baseline, and with samples taken from complete runs it is our "fitted-model rollouts" baseline. Both use the same estimator as TIS.

  • Against uniform: TIS has about two thirds of its MSE on inventory at 1,200 queries per pair (Table 4) and .16–.22 in the controlled example of Q1. On CliffWalking, uniform has 66–87 times the MSE of fitted-model rollouts, whereas TIS has .68–.83.
  • Against fitted-model rollouts: TIS has .60–.86 of its MSE on inventory, and .20–.45 at α=.01\alpha = .01 when a language model is the policy.

So TIS does not compete with the temporal decomposition. It tells the evaluator where to sample, and the decomposition can then be applied to those samples. The paragraph "Estimating return distributions" in Section 2 of the revised PDF now says this.

Reviewer_SxKhOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies how to estimate the lower-tail CVaR of a fixed policy's return in a Markovian workflow where each state-action transition can be sampled directly. Instead of full rollouts, it samples transitions and computes the return distribution with a categorical distributional Bellman recursion. The question is how to split a fixed query budget across transitions. The authors derive a per-transition tail influence, the effect of one sample on the CVaR estimate summed over every step where the transition is reused, and allocate queries by Neyman allocation on this score. Tail-Influence Sampling learns the scores from a uniform pilot, and an anchored variant mixes them with visitation shares to guard against pilots that miss rare outcomes. The theory shows that the learned allocation attains the oracle asymptotic error, that the anchor is within a factor two of it, and that for rare failures the tail and mean allocations coincide. Experiments cover tabular benchmarks and language-model review workflows on MMLU-Pro and FinQA, replayed from frozen output distributions.

Critical strengths

Carefully scoped and executed theory, with an honest treatment of its own limits, backed by a rigorous evaluation. The theoretical results carry exactly the assumptions they need, and the authors are open about where those results stop applying, including the case where a tail-specific score cannot help (Theorem 3, Proposition 2, pp. 6 and 9). The evaluation is equally transparent: it reports the method's own failures in the main text and fixes the held-out protocols before running them (Section 5).

Critical weaknesses

1. The setup is not realistic. The language-model workflows are built to be Markovian and are replayed from saved output distributions (Appendix E.2.6), so they do not show the method on a realistic agent, although the paper pitches it for auditing agent loops (Section 7). Even inside this favourable setup, the tail score beats the simpler mean-based score only when the lower tail has several levels of severity. The rare-failure regime that motivates the paper (Section 1) is exactly where the authors' own result says the tail score is not needed (Proposition 2). On the held-out FinQA task, the blend without the tail score has significantly lower error in 15 of 16 settings (Table 2).

2. The increment over prior work is modest, and the positioning omits its closest relatives. The authors describe the allocation rule as Neyman allocation, and the new part as the Bellman influence and the proof that it can be learned from a pilot (p. 9, Table 18). Directing simulation effort toward the samples that most change a tail-risk estimate is older than this work. Nested sequential simulation allocates inner samples one at a time to the scenarios where an additional sample most changes an estimate of a large-loss probability [1]. Active model estimation directs data collection in an MDP toward the state-action pairs where the model is hardest to estimate, though for model accuracy rather than for a functional of the return [2]. Neither line is discussed in the paper. Nor is rare-event simulation, such as adaptive multilevel splitting [3] or adaptive importance sampling, both for testing a fixed policy in a simulator [4] and for rare language-model outputs [5]. Rare-event simulation is a standard approach to the paper's motivating case of rare severe failures in resettable simulators (p. 1). The claims as stated by the authors hold, but when this work is taken into account, the step beyond established allocation methods is smaller, which is a modest increment for a main-track paper.

[1] https://doi.org/10.1287/mnsc.1110.1330 Efficient Risk Estimation via Nested Sequential Simulation (Broadie, Du, Moallemi, Management Science 2011) [2] https://arxiv.org/abs/2003.03297 Active Model Estimation in Markov Decision Processes (Tarbouriech et al., UAI 2020) [3] https://doi.org/10.1051/proc:071909 Adaptive Particle Techniques and Rare Event Estimation (Cérou, Guyader, ESAIM: Proceedings 2007) [4] https://arxiv.org/abs/1811.00145 Scalable End-to-End Autonomous Vehicle Testing via Rare-event Simulation (O'Kelly et al., NeurIPS 2018) [5] https://arxiv.org/abs/2410.13211 Estimating the Probabilities of Rare Outputs in Language Models (Wu, Hilton, ICLR 2025)

Questions for authors

Both questions are clarification requests.

1. Oracle allocation at H = 2. In Table 9, at H = 2, oracle+floor has several times the MSE of the anchored and learned-occupancy designs for Qwen3-4B and Qwen3-32B. What causes this, and does it limit how well the first-order variance guides the choice of allocation at practical budgets?

2. Which score is used by the MMLU-Pro headline. The abstract says the anchor beats the mean-influence blend in 23 of 24 MMLU-Pro settings. This count uses the confident-error utility, not the primary Brier utility (Table 2). Appendix E.2.6 says the utility variations were added post hoc. Was the confident-error utility fixed before the runs?

3 comments

Thank you for a careful review and for the references.

Realism. We agree, and the new studies support your point. We ran the review loop live with Qwen3-4B, so that every call sees the whole conversation and the evaluator can only restart conversations stored in its own pilot runs. The anchor's restarts were on par with complete runs in two studies (MSE ratios .89 and .91) and less accurate in the third (1.31; Appendix H). Sections 1 and 7 therefore require a state that can be reset and that determines what happens next, and agent loops are outside the scope.

Within that scope we added a study with a language model in the loop (Q6, Appendix I). The model makes the ordering decisions of the inventory simulator, each query is one model call, and the simulator can be set to any state. With orders sampled from the models' exact probabilities, TIS has .20 (Qwen3-4B) and .45 (Phi-4-mini) of the MSE of complete rollouts under the same estimator at α=.01\alpha = .01, and .40 and .62 at α=.05\alpha = .05. In a live run in which Qwen3-4B sampled all 26.8 million orders, the ratios are .24 (95% CI .19–.31) and .46 (.35–.60).

In the frozen workflows, what we store is the model's exact output distribution for each prompt, so sampling it is equivalent to calling the model. What they simplify is the state, which is Markov by construction. On FinQA the gains over complete rollouts come from the estimator, and the paper now says so (Q4, Appendix G.6).

Whether the tail score is needed. Proposition 2 requires all returns below the maximum, taken together, to be rarer than α\alpha. The worst α\alpha-fraction of runs then contains every failure, and CVaR moves with the mean. When failures differ in severity and do not all fit in the tail, the two scores differ. We now say this after the proposition, and we tested it:

Anchor / same design with a mean scoreMSE ratio
inventory, raw profit, α=.1/.05/.01\alpha = .1 / .05 / .01.84 / .78 / .65 (significantly lower in 16, 17, 18 of 18; 0 higher)
inventory, raw profit, α=.001/.0001\alpha = .001 / .0001.52 / .40 (significantly lower in all 18)
language model as policy, α=.05/.01\alpha = .05 / .01.60 / .42 (Qwen3-4B), .69 / .54 (Phi-4-mini)
held-out FinQAmean score better in 15 of 16 cells

FinQA is the case your review describes, since the two scores coincide there (median tail–mean distance .000). You are right that a mean score suffices when all failures fit inside the tail; the tail score helps when they do not, as in inventory. A rule fixed before evaluation, based on the pilot's tail–mean distance, picks the better or tied design in 141 of 148 cells.

Novelty. Section 2 now covers all five references. Nested sequential simulation (Broadie et al., 2011) is the closest single-stage relative. Active model estimation (Tarbouriech et al., 2020) targets model accuracy, and an allocation in its spirit has over 100 times TIS's MSE on CliffWalking. Importance sampling needs likelihood ratios, which a black-box kernel does not give. Splitting needs only restarts, so we ran it as a baseline, including adaptive multilevel splitting, with its tuning parameter set to the best value and, in one version, the true value function. At 1,200 queries per kernel, TIS has lower MSE at every tail level from .1 to .0001 (general response; Appendix F.6). Our setting adds reuse and a tail defined by a Bellman recursion: one sample of a kernel affects every stage that reuses it, through the rest of the model and a cutoff, both unknown. Theorem 3 shows that learning the tail score from a pilot attains the oracle's asymptotic MSE, pilot cost included.

Q1, the oracle at H=2H = 2. The oracle's allocation is correct to first order. Its MSE is higher because, in the question dominating each model's error, one kernel is reached in 86% (Qwen3-4B) or 97% (Qwen3-32B) of runs but has almost no tail influence, so it gets only 58 or 30 samples. With margins of .001 and .006, the noise in that kernel moves the estimated cutoff. Evaluating the same samples at the true cutoff brings the oracle's MSE from .19 and .11 of uniform's down to .018 and .016, below the anchor's .030 and .028 (Appendix G.2). So the answer to the second part of your question is yes: the first-order variance is misleading for a design that leaves a heavily reached kernel with few samples, even at 6,400 queries per kernel, although it stays accurate for designs that sample every reached kernel, as the anchor does.

Q2, the confident-error utility. No. It was added after the Brier results as a sensitivity analysis, and the abstract no longer cites it. For MMLU-Pro, Q5 now reports the Brier analysis (11 of 18 cells in the anchor's favour, none against), and Table 11 labels the confident-error row as post hoc.

Thank you for the detailed response.

Realism. Thank you for running the live conversation loop and for stating the scope plainly in Sections 1 and 7. This is the evidence I was missing, and it addresses my first concern within that scope.

The tail score. Your clarification after Proposition 2 is helpful, and I had read the proposition perhaps too broadly. The inventory results answer my point.

Related work. Thank you for adding the references and for running splitting as a baseline. I noticed that splitting runs evaluated with your estimator reach about the oracle allocation’s MSE (Appendix F.6), which is consistent with my view that the step beyond established allocation methods is modest.

Q1. Thank you for the clear explanation and the check at the true cutoff, and for answering the second part directly.

Q2. Thank you for the direct answer and for removing the result from the abstract.

These changes improve the paper, but my view on the size of the contribution stands, so I am keeping my rating at 2.

Thank you for reading the revision.

One clarification on Appendix F.6, where splitting's samples are evaluated with our estimator. The numbers below are at 1,200 queries per kernel.

  • This combination relies on tuning that an evaluator cannot do. Its tilt is set, at each tail level, to the value with the lowest error, and it is scaled by the true standard deviation of the return. Tuned in this way, it has 9–15% lower MSE than TIS in most settings (more at 300 queries per kernel, less at 2,400).
  • Splitting with its own estimator has higher MSE than TIS at every tail level (TIS has .09–.92 of its MSE).
  • TIS learns its shares from its own pilot, with no parameter chosen from the results. It has 1.13–1.26 times the MSE of the oracle allocation, which comes from Theorems 1 and 2. No splitting variant has significantly lower MSE than the oracle.

We respect your assessment of the size of the contribution, and thank you for a constructive review.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response

We thank the three reviewers for their careful reading. Their questions led us to run new studies that test the method directly. New and rewritten text is in blue in the revised PDF. No equation, theorem or proof has changed.

What the new studies show. All numbers below are ratios of mean squared errors (MSE) at equal numbers of queries, so a value below 1 favours TIS.

First, the allocation yields a gain on its own, which is the effect Reviewer pyhQ asked us to isolate. When both methods use the same estimator, TIS has .86, .75, .60, .37 and .26 of the MSE of complete rollouts at tail levels .1, .05, .01, .001 and .0001 (inventory, 1,200 queries per kernel). The gain grows as the tail becomes rarer, whereas mean-based allocation gives no gain at α≥.01\alpha \ge .01. The rest of the gap to complete rollouts comes from the estimator, and on FinQA all of it does.

Second, on the realism concerns of Reviewers SxKh and qUfs: in a live run with a language model as the policy of a resettable simulator (26.8 million orders sampled from Qwen3-4B), TIS has .24 (α=.01\alpha = .01) and .46 (α=.05\alpha = .05) of the MSE of complete rollouts under the same estimator (new Q6). For an agent whose every call sees its whole conversation, restarting stored conversations was on par with complete runs in two live studies (.89 and .91) and less accurate in the third (1.31). The paper now states this scope: the state must be resettable and must determine what happens next.

Third, we compared TIS with splitting, the rare-event method Reviewer SxKh pointed to, which needs only restarts. We ran it on the 18 inventory cases at equal numbers of simulated transitions and gave it every advantage: its tuning parameter is set to the best value for each tail level, and one version ranks its runs with the true value function.

TIS's MSE relative toα=.1\alpha = .1.05.01.001.0001
adaptive multilevel splitting.60.50.32.16.09
fixed-time splitting.86.85.78.62.56
fixed-time, true value function as level.92.87.91.85.84

The table reports results at 1,200 queries per kernel. At 300 queries, splitting with the true value function is comparable to TIS (ratios .94–1.07) and has lower MSE at α=.0001\alpha = .0001. Splitting's runs can also be evaluated with our estimator. This combination has about the oracle allocation's MSE, never significantly less, and TIS has 1.10–1.17 times its MSE with the true value function and 1.13–1.15 at α≥.01\alpha \ge .01 without it (Appendix F.6).

In summary. TIS is a computable, learnable allocation for estimating CVaR when kernels are reused across steps, with the oracle's first-order error, pilot cost included (Theorem 3). With the estimator held fixed, the allocation alone lowers the error, more so for rarer tails; at 1,200 queries per kernel it has lower error than splitting, and it carries over to a live language-model policy. The method applies wherever the state can be reset, and the paper now states this scope more clearly.

Main changes in the PDF.

  • Related work is now Section 2, before the method, and covers the references suggested by Reviewer SxKh.
  • Section 6 has a table of tasks, a box with the main findings, a table that separates the estimator from the allocation, and the new Q6.
  • The appendix is organized by question; the new studies are in F.2 to F.6, G.2, G.6, H and I.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.