- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies the identification of joint potential outcome distributions using a specified pattern of shared latent factors. Specifically, the paper gives rank conditions for model saturation and identification under factor independence and equal shared factor loadings. The paper then develops maximum likelihood estimation and evaluates counterfactual rankings on synthetic data and randomized recommendation data.
Critical strengths
- The paper studies the identification of joint dependence across potential outcomes, an important problem beyond identifying individual outcome distributions.
- The identification argument is clear and constructive, and the rank conditions and examples help explain when the proposed representation is sufficient.
- The paper provides finite-sample experiments, uncertainty estimates, and sensitivity analysis.
Critical weaknesses
- The choice of the factor sharing matrix M is not well justified. Identification assumes that M is specified, but observed marginal distributions do not generally determine the correct structure.
- The ranking conclusion is sensitive to small changes in the factor loadings. Appendix F shows that slightly changing lambda1 can change which treatment is most likely to be best, and fewer than half of the tested settings satisfy the variance conditions.
- The synthetic experiments estimate only the location parameters while treating the other NIG parameters as known, whereas the real application estimates all parameters. Therefore, the experiments provide limited evidence for the full estimation problem.
- The real application lacks sufficient model checks. Viewing time is nonnegative and contains zeros, while the NIG model allows negative values, and each treatment includes many different videos.
Questions for authors
- How should M be selected, and how do the conclusions change under alternative compatible structures?
- How should the sensitivity results be interpreted when the variance conditions fail or the fitted marginal distributions change?
- What are the estimation errors and interval coverage when all distribution parameters are estimated?
- What video sampling rule defines each treatment, and how well does the fitted distribution describe the observed viewing times?
3 comments
Comment: 3. The synthetic experiments estimate only the location parameters while treating the other NIG parameters as known, whereas the real application estimates all parameters. Therefore, the experiments provide limited evidence for the full estimation problem.
Questions 3. What are the estimation errors and interval coverage when all distribution parameters are estimated?
Our Response: We conducted a synthetic experiment using the same three-treatment, eight-parameter model as in the real-data analysis, estimating all location, scale, and shape parameters, with the same group sizes as the real data (3,939, 8,069, and 5,028), and report it in Appendix E (Table 7). Over 100 simulated datasets, the absolute bias of PoB was at most 0.0051, and the RMSEs were 0.033, 0.032, and 0.007 for urban, TV, and animation, respectively. The intervals covered the true PoB in 90%, 90%, and 94% of the datasets, with average widths (0.120, 0.120, and 0.029) close to those in the real-data analysis (0.115, 0.115, and 0.032). Coverage for urban and TV was slightly below the nominal level, which may partly reflect the small number of bootstrap samples (100, owing to time constraints); we will increase both the number of bootstrap samples and the number of simulated datasets in the final version. This supplements the existing experiments in Appendix E by assessing the accuracy of the same estimation procedure used in the real-data analysis.
Comment: 4. The real application lacks sufficient model checks. Viewing time is nonnegative and contains zeros, while the NIG model allows negative values, and each treatment includes many different videos.
Questions 4. What video sampling rule defines each treatment, and how well does the fitted distribution describe the observed viewing times?
Our Response: In KuaiRand’s random exposure, a video is sampled uniformly from 7,583 candidates and displayed. Our analysis is restricted to videos with the specified tag sets, and each treatment is defined as randomly presenting a video from the candidates in that category (66, 152, and 89 videos for urban, TV, and animation, respectively). Because the video is drawn at random within each category, each potential outcome includes variation across the videos in that category; the treatment is the randomized presentation of a video from the category. We have stated this sampling rule in Appendix F. We use NIG as a continuous approximation to represent the skewness and tails of viewing times. To assess this approximation, we added distributional checks: the observed proportion of zero viewing times was 0.24–0.34%, the fitted probability of negative values was at most 0.11%, and the maximum difference from the empirical CDF was 0.015–0.021. These results indicate a close overall fit to the marginal distributions, with negligible probability assigned to negative values. We have added these results and group-specific QQ plots to Appendix F (Figure 5).
Thank you for the clarifications and additional experimental results. My concerns have been addressed, and I maintain my original rating.
We thank the reviewer for their time and valuable feedback. We have uploaded a revised manuscript addressing the reviewers’ feedback, with major revisions highlighted. Below, we reproduce your comments and respond to each in turn.
Our Response: M is the matrix that specifies the factor-sharing structure, which is the core of our assumptions, as discussed in Examples 1–3. M cannot be determined from the marginal distributions alone and must be specified from domain knowledge. In our application, each tag is assigned an additive response component, and M follows the sharing relationships among the tag sets {39}, {39,43} and {20,43} (Section 6 and Appendix F). In practice, we recommend comparing the results across several plausible factor-sharing structures. The conclusions may differ depending on the assumptions, as the comparison with rank preservation (RP) in Section 6 and the sensitivity analysis in Appendix F illustrate; the extent of this dependence varies across datasets.
Our Response: We acknowledge that the results do not support a single conclusion about which treatment is best. Each grid point should be interpreted conditional on its assumed loadings. At grid points that fail the sample-variance check in (73), the sample variances conflict with conditions that Assumption 1 implies under those loadings, so these settings deserve less weight; however, passing the check does not confirm a setting.
Some patterns persist across the explored grid. First, across all 81 grid points, the PoB of television drama never exceeds 0.60, in contrast to 0.998 under RP. Second, the PoB of animation remains between 0.33 and 0.40, so animation is the best option for roughly one-third of users in every setting. Third, 32 of the 36 settings in which urban short drama has the largest PoB fail the check; among the 38 settings that pass, urban short drama has the largest PoB at only 4. Regarding the fitted marginals, the identification theory keeps the marginal distributions fixed, whereas our implementation refits the NIG model at each setting, so the results at each grid point reflect both the assumed loadings and the refitted marginals. In the revision, we have added these results to Appendix F and summarized the main findings in Section 6.