- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper looks at the challenges of buying data for supervised learning. Specifically, how to value data before purchasing it, how to stop sellers from lying about their costs or label quality, and how to pick the best mix of datasets. The authors propose GRADES, a system that estimates a surrogate value using only the data features, selects winners using a distorted greedy algorithm, and pays them using a critical-bid system combined with a quality score. Using a linear Gaussian model with OLS, the paper shows that this surrogate value matches the expected drop in test error. This proves that the system keeps sellers honest about their costs, ensures truthful label delivery (under a uninformative prior), and offers a solid welfare guarantee. The authors also extend this framework to other statistical models and test it on synthetic and real datasets, showing it competes well on welfare and runs much faster than the DAVED baseline.
Critical strengths
-
Sellers share their features and costs first, and only hand over labels if they are selected. Because the feature-based surrogate accurately predicts how much the test error will drop (in the OLS setting), the buyer can pick winners without seeing the labels upfront. This turns a purely theoretical concept into a practical tool.
-
The payment structure is very well thought out. By recentering the bonus, the system keeps sellers honest about both their costs and their labels. Tackling fake costs and fake labels at the same time is a major plus, since ignoring either would break the market.
-
The math backing up the submodularity bounds is solid and nicely covers multi-record sellers and general task weights. The authors also provide matching hardness results, which clearly explain what the system can guarantee and why finding a perfect solution is so hard. This perfectly justifies their approximation approach.
Critical weaknesses
-
The proof for "cost-truthfulness" relies on the assumption that sellers only see their labels after they are selected, meaning they can't factor the quality bonus into their bids. But in real-world data markets, sellers already hold labeled datasets. If they already know their labels, they can predict their bonus and game the system. Since the DSIC guarantee is a core part of the paper, the authors really need to clarify what kind of real-world procurement settings actually fit this timing assumption.
-
The experiments don't clearly show when the surrogate value actually leads to worthwhile purchases. The theory says the surrogate matches prediction improvement, but this assumes full-rank initial data and OLS. The experiments, however, use ridge regression with no initial data. As a result, higher design welfare didn't actually translate to lower test error. In fact, on datasets like RSNA, Fitzpatrick17K, and DrugLib, buying data actually resulted in worse test errors than buying nothing. The paper needs to explain when we can actually trust this surrogate to improve real-world predictions.
-
The comparison protocol feels inconsistent, and the speed tests don't tell the whole story. The main text says all baselines cap the number of selected sellers at k, but the appendix mentions that ranked methods don't have this cap, making it hard to interpret the welfare and MSE results fairly. Also, the speedup claim leaves out the time it takes to compute the critical payments, while the DAVED baseline was run with a fixed 2,000 updates. Without comparing the full runtime of the mechanisms at matching accuracy levels, the speedup claim isn't completely convincing.
Questions for authors
-
Do sellers already know their labels when they submit bids? A footnote on page 4 assumes they are realized after allocation, but the problem formulation implies sellers are holding existing datasets. If they do know the labels, does the cost-truthfulness proof still work? If they don't, how does this match the real-world data markets mentioned in the introduction?
-
Why does buying data sometimes hurt the model? As noted in the appendix, the ridge regression setup doesn't match the theoretical assumptions. On RSNA, Fitzpatrick, and DrugLib, the mean post-acquisition MSE is actually worse than the no-purchase baseline. What is driving these results, and do the experiments actually prove that this surrogate method improves real-world predictions?
-
Which limits were actually applied to the baselines? Section 4.1 mentions a strict cap of k sellers, but Appendix Table 2 says ranked methods had no count cap. Please clarify exactly what protocol was used for Table 1 and the figures.