- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This manuscript introduces a new approach for overcoming covariate shift due training models on synthetically generated data instead of real data. Specifically, the authors assume access to a collection of synthetic generators, each of which samples different regions of the task space. The goal is to selectively combine their outputs so as to produce a useful finite sample training set. Each generator is assessed by the probability that it produces samples near the observed training data and the authors show these can be combined so as to reduce prediction error. This approach is theoretically justified on Nadarya-Watson kernel predictors via a risk bound that connects these probabilities to its prediction error. They then empirically demonstrate this approach on, among others experiments, a 23-design common-support family task and show that task matching removes the fixed-label gap. They then apply their approach to 8 synthetic seven-joint arm-like variants using kernel ridge regression to demonstrate its performance beyond NW kernel estimators.
Critical strengths
-
This paper studies an important problem which is overcoming the distribution shift incurred by the sim-to-real gap. Their approach to optimally allocate generators to improve training coverage appears to be a novel contribution.
-
The authors derive many theoretical results to support their arguments.
Critical weaknesses
-
This paper is very difficult to read and its exposition has made it close to impossible to determine its contributions. For example, many relatively simple concepts are given long and complex names. Neighborhood probabilities are termed "Finite-scale local-mass fields" and “Rigidity of local summaries without finite-scale rigidity” means the statistical summary stays the same when the neighborhood probabilities change. There are also a mass of theoretical results and its not clear how they all relate to each other or the problem. There is a risk bound on NW estimators, a counter example about histograms, another bound on rejection sampling, and so on. Many of these theoretical results also do not fully define their notation, which makes them difficult to assess.
-
This paper needs to more clearly connect its theoretical contributions to past work on kernel regression. For example, the upper bound in theorem 1 appears to be an corollary of Lemma 2 in Pathak et. al. (2022), which the authors cite. If you take the bound Eqn. 10 in the proof of Lemma 2 and substitute it into the last equation of the proof, I believe you would nearly recover the upper bound in Theorem 1 just using different notation. The only difference seems to be that Theorem 1 (Eqn. 4) assumes (which isn't noted until the table at the end). However, in that case the bound in Lemma 2 would already be uniform over , so adding on the would not change the bound. I was not able to verify whether the other theoretical results are also similarly connected to prior work.
Questions for authors
-
Could the authors please revise this work so that the problem and logical progression through the theoretical results are more clear? Please also make sure that all notation is defined where it used, rather than a large table at the end.
-
What is the practical benefit of theorem 1? I'm not sure how exactly the risk certificate should be used for either improving the source mixing or understanding this problem more abstractly.
-
This approach appears to be intended as a tool for guiding synthetic data acquisition more generally than for just NW estimators. How well does this approach apply for other predictors beyond kernel methods? Could the authors provide more empirical evidence to demonstrate this that
-
Do the experiments show that this approach improves predictive performance? Figure 4 seems to be showing that the distribution can be corrected, but does this result in meaningful predictive gains?