- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies why calibrated uncertainty alone cannot determine which intervention will improve a model’s answer. It develops a boundary-based characterization of intervention effects and proposes FAVE, a factorized action-value estimator evaluated using randomized one-action logs across five interventions.
Critical strengths
Clear motivating distinction between uncertainty and intervention value.
Useful matched comparisons and explicit reporting of negative results.
Critical weaknesses
The distinction between needing intervention and benefiting from a particular intervention is useful, and I appreciate the controlled comparisons. However, I have several major concerns:
First, the theoretical results do not yet establish a sufficiently substantive contribution beyond their definitions. Theorem 1 follows directly by substituting the definition of the score displacement into the post-intervention margin. Theorem 2 constructs observationally identical environments with different optimal actions, so its lower bound follows from assigning probability at most to one action. Theorem 3 explicitly invokes classical maximal-invariant factorization. These results organize the problem, but they do not establish when practical uncertainty-based methods lose information or when FAVE can recover it. Can you identify a nontrivial prediction or algorithmic consequence of this theory that would not follow from directly learning conditional action values?
Second, the maximal-invariant characterization depends on a narrower transformation group than the broader decision-preservation language suggests. Section 3.2 removes global translations, although positive scaling also preserves answer rankings. Anchored margins retain scale information and therefore characterize translation-invariant effects rather than precisely the information necessary to preserve decisions. The paper does declare its groups, but the choice determines both the retained information and the minimal dimension. Why is translation the appropriate observation ambiguity for the answer-score interface, and which intervention decisions require the numerical margin information that would disappear under independent positive rescaling?
Third, the practical advantage over a strong simple policy is not established in the main utility regime. At the primary token price, best fixed achieves .1447 utility versus FAVE’s .1381, while using 16.5 rather than 20.2 extra tokens per item. FAVE also underperforms pooled prediction at 10% logging coverage and has no clear advantage at 25%. The high-harm experiment is useful, but it establishes a benefit in a particular cost regime rather than a general advantage under sparse feedback.
Questions for authors
What concrete algorithmic insight follows from the boundary geometry and maximal-invariant results beyond directly estimating conditional action values?
Under what sample-size and harm-cost conditions does FAVE reliably outperform fixed retrieval, which achieves higher utility and lower token use at the primary token price?