- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper proposes to investigate how divergence measures as commonly applied with default implementation detect synthetically created shifts in the embeddings on public datasets. Namely trace preserving covariance changes. It shows that tests miss these synthetic shifts or require large sample sizes. MMDAug provides better power and a battery of tests can be used to identify specific tests.
Critical strengths
A number of different embeddings spaces are tested. The papers description of the methodology is reproducible.
Critical weaknesses
The first critique is that the entire paper reads like an modern LLM-AI wrote it. I find it hard to parse with strangely succient but unclear phrasing. For example what does "matched in transport" mean? The authors should work to expand the writing for clarity, using consistent and simple word choices for describing the methdology.
The second critique is that it is not clear why rotation in embedding space is a meaningful shift. Also some rotations may end up with data alignment. Likewise what physical meaning would 'isotropic contraction' have in the embddding space? Because this is not a physically meaningful change (i.e., there is no change to the actual input) it cannot be called groudn truth. When rotation is only for some principal component dimensions it is also not clear that the data distribution itself may be nearly iostropic in those directions so any rotations will create the same distribution.
Questions for authors
"real model against its own training data once the location component is removed" what does removing the location component mean?
what are 'coverage losses constructed to leave the mean unchanged'
why would near-duplicate samples exist ('controlled fraction of training-set copies')?
Why would rotation be present in per-genre symbolic subsets?
Its not clear why in the reproducibility statement that the earlier version of the harness was discussed, wouldn't only the corrected version be included in the results?
2 comments
I think the authors for the update. While the paper is certainly clearer, its significance and contribution in actual generative evaluation is wholly lacking. Section 7 proposes a way to reference calibration which to me seems like a form of dataset augmentation to enable kernel selection. However, it is only test on synthetic transformations and still does not discuss whether this for of reference-calibrated kernel selection actually improves power in practice compared to kernel learning/selection with train/test splits, or standard kernel size selections.
Dear Reviewer,
You wrote that the paper was hard to parse and that its phrasing was unclear. We have revised the manuscript to improve its clarity, and we have adjusted its structure so that it better reflects our contribution and the method that follows from it. The original experiments and their findings are unchanged. Terms such as "matched in transport" are replaced by plain descriptions. Rotations and contractions are presented as known transformations, not as ground truth (Section 3.3), and Section 4.3 reports how much each one changes the distribution. Appendices A.3, A.13 and A.24 answer your questions about centering, cluster removal and injected copies.
Thank you for your questions and suggestions, which have made this a better paper. If you find the revision clearer, we would be grateful if you would consider raising your score towards acceptance.
The authors