- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper proposes DAPRO, a dynamic budget allocation method for multi-round LLM evaluation. This method adjusts the probability of continuing to run based on interaction history, and combines inverse probability weighting and statistical calibration to estimate the prediction bound of event occurrence time and overall event rate under a limited budget. Experiments cover scenarios such as toxic generation, jailbreaking, and instruction following.
Critical strengths
The problem has practical significance, combining adaptive budget allocation with statistical validity, and providing theoretical analysis in terms of coverage and expected budget.
Critical weaknesses
(1) The only substantive external baseline is Static, while Uncalibrated and DAPRO w/o CRC serve as a basic control and an ablation, respectively. These comparisons cannot substitute for comparisons with alternative budget allocation strategies. Consequently, the current results do not distinguish whether the gains arise from using interaction history, dynamic stopping, or the proposed optimization procedure.
(2) The main experiments emphasize coverage deviations and estimator variance, but these metrics are insufficient to establish that the predictive bounds are more informative in practice. Moreover, the experiments rely on 4,000 additional training trajectories to construct the predictor, while excluding their acquisition cost from the main calibration budget.
(3) The limited experimental presentation in the main text substantially weakens readability and persuasiveness. The main text presents results primarily through a few box plots, while several important diagnostics, robustness analyses, and experimental details are deferred to the appendix. The metrics and model coverage also vary across tasks, and the absence of a compact summary table makes it difficult to assess the consistency and scope of the improvements. The authors should integrate experiments that directly support the main claims into the main text in a more compact format, rather than leaving essential evidence scattered throughout the appendix.
Questions for authors
(1) Can the authors include comparisons with more recent related methods for budget allocation and budget-constrained evaluation, beyond the Static baseline?
(2) Does DAPRO still offer an overall cost advantage when accounting for training trajectory collection, predictor training, policy learning, and online scoring? Please report the total cost of the complete evaluation pipeline.
(3) Can the authors move the key ablation studies, robustness analyses, and results under different budgets from the appendix into the main text, and provide a unified summary table? The current presentation makes it difficult to assess the consistency of the improvements and the scope of the method’s effectiveness.
3 comments
Thanks for the response and additional experiments. The evaluation still relies heavily on Static, while alternative comparisons remain limited and mainly in the appendix. I also find the manuscript difficult to follow as a human reviewer, making its contribution hard to assess. I will keep my original score.
Dear Reviewer yJjv,
Thank you for reading our response and for your reply.
On the comparisons: to our knowledge, Static (Davidov et al., 2026) is the only existing method that constructs bounds on the time-to-event with coverage guarantees in this setting, which is why it is our primary baseline. We therefore designed additional comparisons specifically for this setting. Table 2 in the main text compares DAPRO with a Uniform+CRC policy, a random-score variant and a uniform-reference objective. The revision also includes a sweep over the weight of our optimization objective and compares DAPRO with a simple monotone rule and Active Statistical Inference. In the final version, we will move the monotone-rule and ASI comparisons into the main text as well.
On assessing the contribution: Section 1.1 summarizes our four main contributions: a tighter coverage guarantee that does not require conditional independence between censoring and event times, a budget-validity guarantee, population-level metric estimation, and a comprehensive empirical evaluation. Table 1 in the main text summarizes the comparison with Static across all evaluated configurations. DAPRO lowers variance in 74 of 80 configurations, with mean variance reductions of 58.5% for lower bounds, 27.2% for upper bounds, 52.6% for event-rate estimation and 66.4% for RMTTE estimation, while keeping bound sizes broadly comparable (a mean change of at most 0.4 turns).
On readability: Section 3.3 explains both theorems in plain language. To make the paper easier to follow, the final version will add an overview figure of the DAPRO pipeline.
Thank you again for your time.
Dear Reviewer yJjv,
Thank you again for your careful review. As the discussion period ends on November 18, we wanted to check whether our response addressed your concerns. We implemented each of your three requests:
Do these additions address your main concerns? If any issue remains unresolved, we would appreciate your feedback and the opportunity to clarify it before the discussion closes.
Thank you again for your time and consideration.