- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper investigates how optimizing repeated LLM queries for classification accuracy can distort estimated treatment effects, potentially reversing their signs. It develops theoretical counterexamples, characterizes the effects of adaptive stopping, and proposes correction procedures under known response distributions. Semi-synthetic experiments using measured LLM responses illustrate the phenomenon.
Critical strengths
The paper raises a relevant concern about using LLM-generated labels for downstream causal inference. The exact stopping-rule constructions and characterization of attainable effect distortions are technically interesting.
Critical weaknesses
-
My first main concern is limited conceptual novelty. The central observation that higher classification accuracy does not guarantee less bias in prevalence or treatment effect estimates is already well established. The elaborate stopping-rule constructions do not clearly establish a broadly new methodological contribution.
-
Also, the main results rely on carefully constructed response distributions, oracle posterior decoding, and treatment-dependent stopping. The repeated sign reversals require very small treatment effects and narrow query-price intervals, limiting their practical relevance.
-
Theory and experiments seem somewhat disconnected. The theoretical framework assumes oracle posterior decoding, whereas the empirical implementation uses a pooled decoder with arm-specific stopping. This weakens the connection between the theoretical findings and empirical evidence.
-
My second main concern lies in limited relevance to observational causal inference. Although motivated by downstream causal inference, the analysis focuses primarily on randomized treatment contrasts or synthetically constructed treatment arms. It remains unclear how the findings translate into observational studies requiring confounding adjustment, or how adaptive LLM labeling affects commonly used causal estimators. A theoretical analysis or realistic observational-data application would help establish the broader implications for causal inference.
-
My third concern is questionable practical contribution. A common stopping rule largely eliminates the observed reversals with negligible accuracy loss. The proposed correction requires known response distributions and thus can substantially increase variance or become unreliable under misspecification. The practical advantage over simpler alternatives remains unclear. Further, sign reversals occur mainly under specially calibrated treatment distributions and selected ambiguous records, but not under natural designs or among records satisfying the main theoretical better-than-chance assumption.
Questions for authors
see my comments in the weakness section