- Rating
- 2
- Confidence
- 4
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
The paper studies when tree-structured sampling reduces policy-gradient estimator variance at matched sampling cost. It defines a weighted estimator with path-predictable branching and ancestor-based leave-branch-out baselines, derives an exact variance decomposition through cut refinement that captures prefix scores, baseline-anchor offsets, and sibling noise, and provides conditional branching/re-anchoring comparisons and a late-branching lower bound. Experiments on MATH and HotpotQA examine noise location and random, entropy-based, and oracle placement; separate HotpotQA training experiments compare existing tree samplers and baseline estimators. The oracle is effective in fixed-model variance diagnostics, but this placement rule is not used in the training experiments.
Critical strengths
-
The full increment analysis for dependent trees has substantive content. Equations (40) and (54) track how revealing a subtree changes earlier prefix-gradient contributions and affects other branches through an ancestor baseline, while handling unequal branch masses and path-predictable branching. Joint moments of random continuation scores and rewards, multilevel dependence, and cross-branch baselines do not follow automatically from multiplication by a fixed score matrix. This is a technical extension beyond shared-context advantage analysis that deserves credit.
-
The baseline tradeoff and restricted lower bound provide useful insight. The decomposition makes explicit that a local baseline can reduce the anchor offset while increasing estimation noise from a finite number of siblings. Under a front-loaded noise profile and its stated sampling conditions, Proposition I.1 gives a global lower bound showing that sufficiently late branching cannot outperform independent sampling at the same cost. Table 1 also shows that placement information can materially change fixed-model variance. These results support the paper's value as a diagnostic framework for tree estimators.
Critical weaknesses
The experiments do not connect the oracle variance advantage to achievable placement benefits or training gains. Appendix J.2 uses additional sibling continuations 2–3 to estimate candidate-specific , selects a location using Eq. (55), and evaluates it with the original suffix and sibling 1. Separating the continuations used for selection and evaluation is a useful control, but the probing continuations are excluded from cost. Thus the relative variances of 0.86 on MATH and 0.63 on HotpotQA in Table 1 demonstrate placement potential with additional information. They do not establish an efficiency advantage including information-acquisition cost, or show how these quantities can be obtained cheaply at sampling time. The authors explicitly disclose the oracle setup; the issue is the contribution that this experiment can substantiate.
The training comparisons in Section 5.3/Table 2 include existing random and entropy-based sampling rules and estimator variants such as the shipped and root/local-anchor estimators, but do not train with the theory-guided placement rule above. They therefore do not test the chain from theoretically identified lower-variance locations to better learning curves or final performance. The existing root/local comparisons deserve credit, but they change baseline selection and cannot substitute for a placement-rule comparison; the differing training configurations also do not clearly isolate the effects of the new dependence terms.
More fundamentally, reducing the trace of the raw gradient covariance does not guarantee better optimizer updates or final task performance: covariance directions, objective curvature, optimizer transformations, and policy changes during training all affect this relationship. The paper analyzes estimator variance at a fixed policy, without a conditional result or controlled experiment sufficient to connect it to training gains. The training results mainly show that the tested random/entropy rules do not consistently help; they do not validate a training benefit from the oracle's lower variance. The theoretical contribution should be assessed through the estimator insights actually established, without additionally crediting an unshown training benefit.
Questions for authors
see weakness
2 comments
Thank you for the clarification. I appreciate the variance analysis and the diagnostic insights it provides. My main concern is whether the framework can use preliminary diagnostics to assess, before training, whether a given heuristic branching strategy is worth adopting. Such guidance would help practitioners make informed sampling decisions before committing to a full training run, making the framework’s practical value clearer. The current submission and rebuttal have not yet demonstrated this capability, and it also remains unclear how the framework could identify suitable branching points in practice. These questions leave my main concern unresolved, so I will maintain my original rating.
Thank you for your careful review and detailed comments. We address the weaknesses pointed out below
W1: sufficiency of placement potential as a contribution
Paragraph 1: “relative variances … in Table 1 demonstrate placement potential without additional information”, “authors explicitly disclose the oracle setup; the issue is the contribution that this experiment can substantiate.”
You are completely right in that the experiments in section 5.2 are meant to demonstrate placement potential only. However, we would argue that measuring placement potential is in fact a valuable contribution in itself, because it shows that existing heuristic-guided branching methods do not reach that potential. Furthermore, this constitutes a diagnostic framework that can show exactly for what type of tasks there is no benefit to branching.
See also point 1 of our general response.
W2: connection between reduced variance and training benefits
Paragraphs 2 and 3: “comparisons in Section 5.3 … do not train with the theory-guided placement rule”, “reducing the trace of the raw gradient covariance does not guarantee better optimizer updates or final task performance”
This is a fair point, and while we mainly wanted to show that in the absence of variance reduction, there is no training benefit, we agree that our case would be stronger if we can also show that where there is variance reduction, there is training benefit. We have taken your suggestion to train with the theory-guided placement rule, and aim to have the results ready by the camera ready version.
See also point 2 of our general response.