- Rating
- 4(+2)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
Fastdv values a sample by the change in a regularized density ratio utility after deletion. This gives leave one out scores without repeatedly training a downstream model. With the support set, kernel anchors and hyper-parameters fixed, all deletions reuse a shared matrix inverse through exact rank one updates. The real-world value is that clients can combine their local statistics to obtain a global ranking without pooling raw data.
Critical strengths
(1) It is a very good point that the deletion formula accounts for the change from N to N -1 without changing the coefficient lambda. With the kernel map and support statistic fixed, the same matrix inverse can be then reused across different deletions.
(2) Also it is a plus that raw client data can stay local. Count weighted statistics give the centralized scores with the same kernel map, support statistic and regularization.
(3) Some experiment designs are solid. For example, both removal directions are tested, and the runtime comparison includes anchor construction. In the Products removal test in Fig. 5, 200 anchors gives accuracy similar to 5k anchors. It also takes less valuation time.
Critical weaknesses
(1) In the Fig.4, authors compare methods with different utilities and pools clean/noisy samples. However, the correlation of ~0.8 with model based loo may partly come from separating those groups. The result actually right now does not well show the separate benefit of exact deletion.
(2) In the Table 4, the improvement may include better client allocation. There is a confounding factor there.
(3) The label free score avoids target labels, but sigma/lambda use support set validation. I would suggest that for the comparison to be clear, it would be good to give the validation objective and label access for Fastdv and the baselines.
Questions for authors
please see weakness part.
4 comments
Dear Reviewer 4sbo,
Thank you for your valuable insights and the time spent reviewing our work, especially during this busy period. We greatly appreciate it.
As we near the final day of the discussion period, we wanted to check in and see if our responses have addressed your concerns. We welcome your further queries and suggestions and are ready to provide any additional information or clarification you might need.
We understand that reviewing is a time-consuming task and greatly appreciate your commitment to advancing academic discourse. Our team is dedicated to improving our work based on your feedback, believing it will enhance our contribution to the field.
Thank you once again for your guidance and support.
Best regards,
The Authors
I thank authors for their response which address most of my concerns. Therefore, I have raised my score to accept accordingly.
We are very pleased to hear that our rebuttal clarified your concerns. Thank you for your generous time and invaluable support throughout this process.
We sincerely thank the reviewer for the careful assessment of our work, particularly the recognition of our exact deletion formulation, efficient shared-inverse computation, and distributed valuation without sharing raw client data. Below, we address your concerns point by point.
W1: Thanks for your suggestion. Following your suggestion and that of Reviewer eFU3, we extend the correlation analysis in Fig. 4. Specifically, we increase the sample size from 200 to 1,000 and separately evaluate clean and noisy samples to examine whether the overall correlation is primarily driven by group separation.
As shown in Table R1, FastDV consistently achieves the highest Spearman correlations, reaching 0.80 and 0.69 for clean and noisy samples, respectively. These results indicate that FastDV's higher overall correlation is not solely driven by clean/noisy group separation, but also reflects meaningful within-group ranking agreement.
Moreover, we provide additional visualizations in Appendix A.5.3 and highlighted them in blue to offer a more intuitive illustration.
W2: We emphasize that adaptive cross-client data allocation is a unique advantage of FastDV. By leveraging the partition composability of additive statistics, FastDV recovers centralized-equivalent sample values across clients without pooling raw data (Corollary 3.2). This enables global ranking and selection across distributed clients, which is one of our core contributions and was intentionally highlighted in the original experiments.
To provide a more controlled comparison, following your suggestion and that of Reviewer eFU3, we additionally introduce FastDV_local, which uses the same globally computed FastDV scores but removes the lowest-valued 40% of samples within each client, following the identical selection protocol as the baselines. The results are reported in Tables R4 and R5. Notably, FastDV_local outperforms Kairos in seven of the eight settings, demonstrating competitive valuation effectiveness even under identical per-client selection budgets. These results indicate that FastDV's effectiveness is not solely attributable to better client allocation, while global selection provides additional benefits. We will incorporate these additional results into the revised manuscript.
W3: We thank the reviewer for this helpful suggestion. Regarding hyperparameter selection, FastDV builds upon uLSIF, which provides an established selection strategy based on the least-squares density-ratio estimation loss. Accordingly, the kernel bandwidth σ and regularization coefficient λ can be selected using only target and support features, without accessing labels from either set. The selected hyperparameters remain fixed throughout valuation. Thus, FastDV follows the same label-access restrictions as the baselines, with neither labels used before the completion of valuation.
To ensure a transparent comparison, we summarize the label-access protocols in Table R6. All methods perform valuation without accessing target or support labels. Only after valuation and sample selection are the selected target samples labeled for downstream training, while support labels are used for downstream validation. These clarifications will be incorporated into the revised manuscript.