- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper introduces orthogonal embedding projections to correct counterfactual predictions as usual corrections might yield results that are no distribution. It supports its findings by theoretical statements and some numerical results.
Critical strengths
None
Critical weaknesses
While I am not too familiar with this area of research, there is a wide range of severe issue with the generalo scientific representation of the contribution:
-
Lack of introduction / related work / research questions: the paper fails to clearly state its contribution and takeaways; there is little to no connection to the related work or the general problem setting -- related work is discussed in the appendix despite the contribution not reaching the page limit. Overall the paper rather reads as a collection of different theoretical statements and some results than to follow a coherent research question.
-
Stating multiple digits of precision in the result section instead of capturing the overall tendency:
projecting [...] reduces RFF correction from 0.005434 to 0.001227 for action 0 and from 0.007752 to 0.001856 for action 1. Its held-out CRPS is 16.953 and 17.181, respectively. CDF projection gives 15.830 and 16.184
- Supplementary / AI guided search: the provided code strongly suggests that generative AI tools were used to automatically run experiments until some numerically favourable results were attained. I don't have any issues with using AI for code generation or running and also giving ideas for experiments but that for example the following lines are included in a supplementary markdown file along with the code suggest that the generated output was, in contrast to what is claimed in the "AI Use Statement", in fact not checked again.
(from protocol.md of the suppl.) This is an exploratory extension requested by the author after the original Gaussian and STAR comparisons showed no CDF advantage over positive part. The original results remain unchanged. Every attempted setting is retained.
Questions for authors
-
What is a "stratum" / "strata"? I am not aware of this terminology in any ML context and it is never defined
-
In Theorem 2, what does "The argument applies fold by fold to a fixed-fold cross-fit." mean?