- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
Based on my understanding, the paper aims to address the following problem: Given the evaluations of candidate images from several source models, can we predict a new model's preferences over the remaining candidate images by querying only a small number of its responses, and thereby select the version with the highest utility?
I believe the paper adopts a well-motivated optimization objective, and the key experiments are sufficiently comprehensive and fair. The mathematical derivations are also relatively rigorous. However, I am not fully convinced that the reported improvements are stable and consistent across different settings. Furthermore, I believe that the main contribution lies in the task-specific integration and design of existing techniques, while the novelty of the core technical components themselves is relatively limited.
Critical strengths
(1) The paper highlights an important distinction: accurately predicting the numerical values of all responses is not necessarily equivalent to correctly selecting the optimal candidate version.
(2) The baseline used in this paper, SVD + empirical Bayes (SVD+EB), is well chosen and appropriate for the task.
(3) The paper reports extensive experimental results, including content-level, model-level, and model-family-level bootstrap analyses, providing a relatively comprehensive evaluation of the results.
Critical weaknesses
(1) The evidence is still insufficient to demonstrate whether the proposed method can consistently benefit observers from previously unseen model families.
(2) The main contribution lies in the task-specific integration and design of existing techniques, while the novelty of the core technical components themselves is relatively limited.
(3) The applicability of the proposed method is relatively limited, and the cost of constructing and maintaining the source response library requires further justification.
Questions for authors
(1) Under what conditions is the episodic approach more effective?
(2) Why does dense-fit outperform the proposed method in the main experiments?
2 comments
Keep my score
Thanks for the authors responce. I agree that some explaination is helpful, like the comparison between dense fit and episodic training. However, my main concern is if the proposed method can improve the performance for unseen model??
Please calrify if the these 5 comfirmantion observers belong to model families excluded from source-library construction, training, and model selection??
Thanks
Response to r59w
We sincerely thank Reviewer r59W for the thoughtful review and constructive questions. We address the two related questions together below.
Both variants learn a decision subspace within the same spectral dictionary. Dense fit uses complete source-response coordinates, while episodic training optimizes decisions after few-query calibration.
Why dense fit is stronger on CIFAR-10. Training with complete source coordinates provides a more direct decision-fitting signal, consistent with dense fit’s lower development regret. Both variants nevertheless outperform SVD+EB (0.1282) in the main comparison, supporting the value of decision-oriented subspaces.
When episodic training helps. Episodic training matches the few-query adaptation used at deployment. It achieves lower mean regret in both evaluated frozen-transfer settings, improving over dense fit by 0.00129 across five confirmation observers and 0.00026 on CIFAR-100. Thus, dense fit’s development advantage does not extend to these transfer means, supporting episodic training’s additional transfer value.