- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The authors develop MaSI to estmate a population mean using labeled text and LLM-generated covariates. The method learns semantic features, reweights generated samples, and applies a residual correction. Simulations and text experiments report shorter confidence intervals.
Critical strengths
The main strength is the treatment of dependence among texts generated from the same source; the negative controls and variance sensitivity analysis make it easy to assess how much of the reported precision gain survives that dependence.
Critical weaknesses
-
Accounting for the observed positive covariance increases interval-width ratios from about 0.93 to 0.96–0.98 (Table F.20). This really reduces the practical gains..
-
The interval guarantee assumes that errors in the learned weights become negligible, but this is not established for the estimator used. The simulation also gives the generator true population parameters; please explain what comparable information supports the gains in real data.
Questions for authors
Please report intervals that account for source covariance and ratio estimation. What assumption connecting the generator to the target population establishes the required calibration rate and supports the remaining precision gain?
6 comments
Response to Reviewer HcmK [2/4]
The theory considers covariance tending to under the corresponding conditions, with Main and Block becoming closer. Main presents the inference guarantee, precision criterion, and variance budget under the structure studied; when the cross term is zero or very small, omitting its estimation may also reduce finite-sample variance-estimation noise. With the same point estimate and critical value, including positive covariance makes Block wider than Main, while negative covariance makes it narrower. The positive cross term in MHS explains the widening there; the paired simulation below also includes cases with narrower Block intervals.
The true density ratio in MHS is unknown, so replacement with true weights cannot isolate the effect of weight learning. We therefore add the following paired simulation with a known true density ratio to distinguish weight estimation from covariance treatment.
(2) Paired weight and covariance comparisons under matched marginal distributions.
The simulated text vectors contain dimensions of score-related content, dimensions of style factors unrelated to the outcome, and measurement noise. The outcome is a nonlinear function of content, including an interaction between two content factors and independent label noise. Pure style factors have large marginal variances, so many high-variance directions in the raw vectors reflect expression factors unrelated to the score. This tests the role of extracting score-related content under strong style interference.
Each source generates two groups of observations: the first preserves content and changes style for representation learning; the second introduces same-source variation along outcome-related directions to construct the synthetic-group mean prediction while matching the real-data population marginal covariate distribution. The true density ratio is therefore . We use five-fold cross-fitting and paired repetitions at each of . Within each repetition:
-
Fit the content representation and outcome predictor once using the same real sample, generated data, and fold split.
-
Fix these fits and use either the density ratio estimated on the training folds or the known true density ratio.
-
Compute the MaSI point estimate under each weighting scheme; different weights may give different point estimates. Within each scheme, construct Main and Block, which retains source cross-covariance, around that scheme's same point estimate.
Each cell below gives empirical coverage / mean interval-width ratio relative to Classical, at nominal coverage.
| Density ratio | Main | Block | |
|---|---|---|---|
| Estimated density ratio | |||
| Oracle density ratio | |||
| Estimated density ratio | |||
| Oracle density ratio | |||
| Estimated density ratio | |||
| Oracle density ratio |
At all three sample sizes, replacing the weights with the true density ratio leaves interval widths essentially unchanged, and the reported Block widths do not exceed Main widths. For example, at , both weighting schemes give a Block width ratio of about , approximately shorter than Classical, with coverage of and , respectively. Estimated weights already yield coverage close to nominal and about interval shortening; Oracle replacement gives no clear width improvement. This provides empirical sensitivity evidence under matched marginal distributions: shortening in this simulation does not require advance knowledge of the true weights and persists after including source cross-covariance. Table directly evaluates the remaining gains in MHS.
Response to Reviewer HcmK [3/4]
3. Known population parameters in the simulation can guide favorable generation. When these parameters are unknown in real applications, how can generation be guided and the corresponding quality of generated texts be assessed?
Real applications use the task definition and source context to specify semantic generation rules, then use real labels and source-pairing diagnostics to assess statistical usefulness, without first recovering the population parameters in the simulation. Good data must serve each channel's purpose: the style channel changes expression while keeping rating cues relatively stable, and the evaluation channel provides predictive local semantic variation. Fluency and textual diversity alone do not establish these properties.
(1) Task-specific semantic rules and the scope of generation.
In MHS, hostility, attack targets, author stance, irony, and quotation attribution may affect real ratings. Even without their numerical coefficients, the task definition allows the permitted changes to be specified. For , instructions change sentence structure and wording while preserving semantic cues that determine the rating. For example, when a comment quotes offensive language to refute it, the rewrite must preserve quotation attribution and the author's opposing stance. For , instructions allow local semantic variation around hostility or ambiguity cues in the source while preserving targets, stance, facts, and context, and avoiding unsupported new attack targets or facts. These rules are specified in advance; the generator receives no real labels, numerical ratings, or target-population parameters.
(2) Style distinguishability and prediction drift.
Table , Panel A jointly diagnoses expression changes and the stability of outcome-related content: classification AUC between originals and style rewrites is in the raw embeddings and decreases to about -- in the learned content coordinates; style-class identification accuracy in the remaining expression coordinates is , above chance at . This supports retention of expression differences with reduced distinguishability between originals and rewrites in the content coordinates. The absolute mean prediction drift is --, further assessing how rewriting affects outcome prediction.
(3) Real-label prediction, source-group permutation, and calibration diagnostics.
We assess task-related information in using real labels rather than relying solely on LLM judgments of text plausibility. Appendix , Table , Panel B keeps the synthetic text pool fixed, permutes only the correspondence between complete synthetic groups and real sources, and compares prediction on held-out real labels. Predictive is higher under correct pairing in every configuration, with differences of --, showing that the predictive cues relate to the true outcomes of their specific sources. Prediction-mean differences and weight-stability diagnostics further assess whether these texts support stable calibration. These checks use real labels and source relationships to assess the practical value of semantic changes and provide empirical evidence for generation quality.
Response to Reviewer HcmK [1/4]
The theoretical goal of this paper is to connect learning and inference through statistical error conditions on the final fitted quantities, providing a common framework for different representation and weight-learning rules and directly analyzing the final MaSI statistic with estimated weights. Weight error enters the statistic through a centered sampling term and a population prediction-mean bias: the former is controlled by L2 consistency of the weights under cross-fitting, whereas the latter requires βw,n=op(n−1/2). Appendix D further gives sufficient conditions for controlling model approximation bias, a first-order parameter effect, and a second-order remainder. Thus, under all the inference conditions in this paper, learning rules meeting these error requirements have the corresponding first-order expansion and interval guarantee; the density-ratio function class, regularization, and solver can be chosen for the data. For the implementation used in this paper, Response 2 reports empirical coverage and interval width using estimated weights and including source cross-covariance, and evaluates the finite-sample effect of weight estimation through paired replacement with true weights in the simulation.
(1) Centered sampling error and weight consistency.
Fix the representation and outcome predictor learned in each fold. Using estimated weights and the true density ratio gives the exact decomposition
μ(w)−μ(w∘)=∑kπk(Qk,n−Q)(φk−φk∘)+Rw,n,
where φk=wkhk and φk∘=wk∘hk, with all functions evaluated in the content coordinates of the corresponding fold; Qk,n averages first within each source and then across held-out sources.
The first term is centered sampling error. Cross-fitting separates held-out sources from the training fits while allowing dependence among texts from the same source. Appendix D proves that this term, multiplied by n, is Op(εw,n), where εw,n is the fold-weighted root-mean-square L2 error of the density ratio. Weight consistency therefore suffices to control this component.
(2) Calibration rate for the population prediction mean.
Appendix D defines the prediction-direction calibration error βw,n and proves, under the local regularity conditions of Proposition D.2, that
∣Rw,n∣≤βw,n≤aβ,n+χw,nrγ,n+CwMhrγ,n2.
Here, aβ,n measures model approximation bias in the prediction direction, rγ,n measures parameter estimation error, and χw,n measures the first-order sensitivity of the prediction mean to parameter changes. The theorem requires this upper bound to be op(n−1/2). Ordinary weight consistency and the prediction-mean calibration accuracy required for inference are thus distinct requirements, and cross-fitting alone does not eliminate population calibration error. Together with the remaining conditions on representation learning and outcome fitting, these controls yield the fixed-center first-order expansion in Theorem 3; the moment and covariance conditions in Appendix E then give the corresponding interval guarantee. The theory applies when the final weights satisfy these learning and calibration conditions.
After accounting for source cross-covariance, MHS intervals remain 3.5 shorter than Classical on average. A paired simulation with matched marginal distributions fixes the samples, representations, and predictors, replaces estimated weights with true weights, and computes Main and Block under each. It shows similar width gains under the two weighting schemes, with gains retained after including the cross term in this simulation setting.
(1) Covariance correction and remaining gains in MHS.
Table F.20 reports Main and Block for the same MaSI point estimate: both retain dependence among generated texts from the same source, and Block additionally includes cross-covariance between the real residual and the synthetic-group mean prediction. At nominal 90 coverage, the average width ratio across five sample sizes changes from 0.933 to 0.965, and average empirical coverage changes from 89.7 to 90.4. Appendix F.4, Table F.8 shows that the estimated cross term relative to additive variance decreases from 0.113 at n=100 to 0.055 at n=800; in Table F.20, the mean width increase of Block relative to Main also decreases from about 4.9 to 2.7.