- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies the problem on how to efficiently allocate limited sampling budget to estimate a given policy's CVaR. The authors proposed Tail-Influence Sampling (TIS) framework under the assumption of direct access to samples from individual state–action pairs. The proposed framework uses the pilot model to identify transition pairs that influence CVaR most significantly and allocates more queries to them accordingly. Theoretically, TIS achieves asymptotically optimal allocation variance under fixed dimensional and positive quantile-margin assumptions. The proposed methods is evaluated on tabular settings and frozen LLM experiments, which generally shows improvement than benchmark methods.
Critical strengths
- The design and derivation of tail-influence score in Section 3 is quite interesting and provided a principled way to capture how uncertainty in each transition kernel affects CVaR.
- The theoretical analysis of the proposed framework established oracle asymptotic variance and first order MSE, which I belive provide valuable insight on similar problems.
Critical weaknesses
- I think the biggest weakness is the proposed framework requires a finite, known state-action space with direct sampling access. Such setting is quite restrictive, and it's unclear if the proposed framework can be generalized to more complex and practical settings.
- Theoretical guarantees rely on positive quantile margin assumption and are asymptotic. More discussion on the finite sample sensitivity to margin requirements and exploration floor would be needed to gain a better understanding of the proposed framework.
Questions for authors
- Section 2: The problem formulation assumes a finite, known state–action space, with all dimensions fixed in the asymptotic analysis. This setting seems restrictive. Could the authors provide a concrete practical scenario where these assumptions hold and discuss the main challenges in extending the approach to continuous state or action spaces?
- Line 175: The results in this section assumes a positive quantile margin to keep the CVaR cutoff locally unchanged. I was wondering the sensitivity of the results on finite-sample accuracy would depend on this assumption.
- Line 396: The experiment settings sampled from a well-calibrated conditional distributions. While this provides a clean benchmark for comparisons, it’s unclear whether the improvement of the proposed framework could persist in more realistic workflows with richer outputs, history-dependent behavior, or distribution shift. Could the authors provide further discussion on this?
- Table 2: It’s interesting that on held-out FinQA, combining mean-based sampling with visitation-based sampling performs better than anchored TIS in 15 of 16 comparisons. I was wondering whether the improvements require the proposed tail-specific score or can be achieved with the simpler mean-based alternative, and how to quantify the additional benefit from tail-specific score?
Thank you for your review, and for the questions.
Q1, a practical scenario. Consider an inventory simulator that can be set to a chosen state, in which the period and the stock level form a finite state. A query sets the state and returns one period's profit and the next stock level, without revealing the demand or disruption probabilities. The new Q6 uses this simulator with a language model as the ordering policy, so that each query is one model call, and TIS has .20–.45 of the MSE of complete rollouts evaluated with the same estimator at α=.01 (Appendix I). The access can also be weaker, since TIS keeps its advantage when it may restart only from states reached in its own pilot runs (Appendix F.5). Our experiments use finite states and actions. Continuous ones require a discretization, whose error would need a bound like that of the return grid in Appendix E.1. The main challenges are that a finer grid means more kernels, each reused less often, and that our asymptotics hold the number of kernels fixed. Section 7 lists it as future work.
Q2, the margin and the floor at finite budgets. The margin enters through Equation 34, which bounds the probability of a wrong cutoff by c3exp(−c4nminmα2), where nmin is the smallest sample size and mα is the margin; Section 4 now states this. Two results show the margin's effect in practice. In the two-call workflows, the two questions that dominate the error have margins of .001 and .006. The oracle allocation gives a heavily reached kernel few samples there and has 3.8–5.9 times the anchor's MSE even at 1,600 and 6,400 queries per kernel. The anchor stays within 3% of its first-order prediction, because its visitation share keeps such kernels sampled (Appendix G.2). At α=.01 on inventory, several cases have margins near zero (down to 2×10−7), yet the anchor outperforms the mean-based blend in all 18 (Appendix F.4). A small margin was therefore harmful only when a kernel that many runs reach was left with few samples.
The floor N−1/4 keeps nmin growing, which is how the theory controls this term. On inventory the allocation is robust to it: over pilot fractions from 3.7% to 28.3% and floors N−1/6, N−1/4 and N−1/3, the learned allocation's leading variance has a median of 1.02–1.12 times the oracle's (300 pilots per setting; Appendix E.2.2). In the six-call review workflows the floor does matter for plain TIS, since raising it from .072 to .40 brings its MSE from 1.38 and 1.77 times uniform's down to .47 and .55 (Appendix E.2.4). The anchor's visitation share has the same effect. A finite-budget MSE guarantee remains open, and Section 7 says so.
Q3, richer and history-dependent workflows. The gain needs a state that can be reset, as Sections 1 and 7 now state. With history-dependent behaviour it does not persist: in a live review loop where every call sees the whole conversation, restarts from stored conversations were on par with complete runs in two studies (MSE ratios .89 and .91) and less accurate in the third (1.31; Appendix H). In Q6, where a language model drives a resettable simulator, the allocation yields a clear gain (Appendix I): in a live run in which Qwen3-4B sampled 26.8 million orders, TIS has .24 (α=.01) and .46 (α=.05) of the MSE of complete rollouts evaluated with the same estimator. Each order is a single token, which makes the ground truth exact; extending the method to richer outputs and to distribution shift is future work (Section 7).
Q4, the added value of the tail score. We measure it against the same design with a mean score. The two designs share the pilot, the floor, the mixture and the estimator, and differ only in the score.
The gain grows for rarer tails, and the mean score is better where the two coincide, as on FinQA. A rule fixed before evaluation, based on the pilot's tail–mean distance, picks the better or tied design in 141 of 148 cells (Q5).