acceptodds
Withdrawn from ICLR 2027

Truthful Data Procurement via Experimental Design

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Building a model for a specific task, whether reading medical images or training a warehouse robot from video, often requires buying data from multiple sellers. How should a buyer pay for data she has not yet seen? While the problem can be viewed as a reverse auction, the nature of data introduces three idiosyncratic difficulties. A buyer must evaluate data without fully accessing it, since once she has seen it she has no reason to pay (Arrow's inspection paradox). A seller who has already been paid can misreport labels. And even if sellers were honest, finding the welfare-maximizing set of datasets is computationally intractable. We study data procurement for supervised learning, where a buyer purchases datasets from strategic sellers with private costs, and we design a mechanism, the Greedy Auction for Data Elicitation via Surrogate (GRADES), which addresses those three difficulties. In the OLS V-optimal design setting of lu2024daved, GRADES is cost-truthful and individually rational in expectation over the label noise, makes truthful label delivery a dominant strategy, and achieves a bi-criteria welfare guarantee. We further extend GRADES to generalized linear models and to Bayesian decision problems with informational substitutes. Finally, we show empirically that it matches the welfare of an existing non-strategic procurement method.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
data markets, mechanism design, information elicitation, incentive compatibility

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_mdcROpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper looks at the challenges of buying data for supervised learning. Specifically, how to value data before purchasing it, how to stop sellers from lying about their costs or label quality, and how to pick the best mix of datasets. The authors propose GRADES, a system that estimates a surrogate value using only the data features, selects winners using a distorted greedy algorithm, and pays them using a critical-bid system combined with a quality score. Using a linear Gaussian model with OLS, the paper shows that this surrogate value matches the expected drop in test error. This proves that the system keeps sellers honest about their costs, ensures truthful label delivery (under a uninformative prior), and offers a solid welfare guarantee. The authors also extend this framework to other statistical models and test it on synthetic and real datasets, showing it competes well on welfare and runs much faster than the DAVED baseline.

Critical strengths
  1. Sellers share their features and costs first, and only hand over labels if they are selected. Because the feature-based surrogate accurately predicts how much the test error will drop (in the OLS setting), the buyer can pick winners without seeing the labels upfront. This turns a purely theoretical concept into a practical tool.

  2. The payment structure is very well thought out. By recentering the bonus, the system keeps sellers honest about both their costs and their labels. Tackling fake costs and fake labels at the same time is a major plus, since ignoring either would break the market.

  3. The math backing up the submodularity bounds is solid and nicely covers multi-record sellers and general task weights. The authors also provide matching hardness results, which clearly explain what the system can guarantee and why finding a perfect solution is so hard. This perfectly justifies their approximation approach.

Critical weaknesses
  1. The proof for "cost-truthfulness" relies on the assumption that sellers only see their labels after they are selected, meaning they can't factor the quality bonus into their bids. But in real-world data markets, sellers already hold labeled datasets. If they already know their labels, they can predict their bonus and game the system. Since the DSIC guarantee is a core part of the paper, the authors really need to clarify what kind of real-world procurement settings actually fit this timing assumption.

  2. The experiments don't clearly show when the surrogate value actually leads to worthwhile purchases. The theory says the surrogate matches prediction improvement, but this assumes full-rank initial data and OLS. The experiments, however, use ridge regression with no initial data. As a result, higher design welfare didn't actually translate to lower test error. In fact, on datasets like RSNA, Fitzpatrick17K, and DrugLib, buying data actually resulted in worse test errors than buying nothing. The paper needs to explain when we can actually trust this surrogate to improve real-world predictions.

  3. The comparison protocol feels inconsistent, and the speed tests don't tell the whole story. The main text says all baselines cap the number of selected sellers at k, but the appendix mentions that ranked methods don't have this cap, making it hard to interpret the welfare and MSE results fairly. Also, the speedup claim leaves out the time it takes to compute the critical payments, while the DAVED baseline was run with a fixed 2,000 updates. Without comparing the full runtime of the mechanisms at matching accuracy levels, the speedup claim isn't completely convincing.

Questions for authors
  1. Do sellers already know their labels when they submit bids? A footnote on page 4 assumes they are realized after allocation, but the problem formulation implies sellers are holding existing datasets. If they do know the labels, does the cost-truthfulness proof still work? If they don't, how does this match the real-world data markets mentioned in the introduction?

  2. Why does buying data sometimes hurt the model? As noted in the appendix, the ridge regression setup doesn't match the theoretical assumptions. On RSNA, Fitzpatrick, and DrugLib, the mean post-acquisition MSE is actually worse than the no-purchase baseline. What is driving these results, and do the experiments actually prove that this surrogate method improves real-world predictions?

  3. Which limits were actually applied to the baselines? Section 4.1 mentions a strict cap of k sellers, but Appendix Table 2 says ranked methods had no count cap. Please clarify exactly what protocol was used for Table 1 and the figures.

Reviewer_xf4aOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper builds upon previous work in data acquisition (Lu et al. 2024) of using buyer features to select from pool of seller data given a capacity k to maximize seller welfare. The paper extends the mechanism to a three-stage game for cost bids and label reporting. They propose GRADES, which combines distorted greedy allocation for standard auction setting with label quality paymets and show truthfulness and individual rationality in expectation over labels and bi-criteria welfare guarantee. Experiments show improvment in welfare and selection runtime over baselines on datasets from Lu 2024

Critical strengths
  • Extend original setting of Lu 2024 to label delivery by formulating reporting mechanism for cost truthfulness
  • Rigorous theoretical analysis provide bounds for V-optimal surrogate and hardness approximation.
  • Simple, deterministic allocation with multiplicative guarantee compared to Lu 2024's continuous relaxation with additive guarantee
Critical weaknesses
  • Work consists of composing existing work of Harshaw 2019 (distorted greedy algorithm), Lu 2024 (problem and experimental setting), Deng 2025 (greedy auction conversion)
  • It seems that the proposed method requires additional assumptions that diminish some of the original problem motivation for data acquisition such as requiring labeled D_0 with full rank X_0, and test labels at payment. X_test must also have full rank so m>d, unlike the original DAVED setting where the buyer has only single query d>>m=1.
  • Empirical settings do not seem to match theory. No experiments simulate cost misreporting or label manipulation. Payment reruns are excluded from runtime comparisons. The original setting prioritizes buyer MSE while in this formulation maximizes sociall welfare.
Questions for authors
  • How does the proposed method perform in the more difficult setting when D_0 is small, feature dimension is large, and there are many sellers. The original paper tested up to d=1,000 and N=100K but it appears GRADES is only tested at N=300 and d<=16? A larger scale experiment should be conducted for the efficiency claims at larger capacity e.g. k>100 and d > 100.
  • Why is the multi-DAVED baseline optimized for 2,000 steps? Did you try matching k steps for comparison to GRADES?
  • Why are payments reruns excluded from GRADES runtime?
  • Why is MSE worse than buying nothing compared (no-acquisition baseline)?
  • Can you add incentive experiments to validate the theoretical guarantees empirically?
Reviewer_1he5OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes GRADES, a procurement mechanism combining experimental-design-based data valuation, greedy allocation, critical-bid payments, and label-quality rewards. It provides incentive and welfare guarantees under an OLS setting and evaluates the method on synthetic and real datasets.

Critical strengths

The paper addresses a relevant problem in data markets: acquiring useful data from strategic sellers without fully observing it beforehand. The attempt to jointly consider cost reporting, label incentives, and computational efficiency is reasonable.

Critical weaknesses
  1. Limited methodological novelty and unclear research contribution.

The proposed mechanism largely combines established techniques: experimental-design-based acquisition from Lu et al. (2024), distorted greedy optimization from Harshaw et al. (2019), truthful procurement mechanisms from Deng et al. (2025), and classical information-elicitation methods.

The paper does not convincingly establish what fundamentally new mechanism-design insight emerges from combining these components. Although it provides specialized technical results, the broader theoretical contribution seems incremental relative to the existing literature.

The assumptions further limit its significance. The mechanism assumes truthful disclosure of features, an uninformative common prior, and access to evaluation signals supporting truthful label delivery. These assumptions are restrictive for actual data markets, where features may themselves contain valuable or sensitive information and sellers may have different beliefs.

  1. Weak connection between theoretical guarantees and empirical evidence.

The theoretical results are established for OLS with a full-rank initial labeled dataset, whereas the empirical evaluation uses ridge regression without initial labeled data. The paper acknowledges that its central valuation identity is not established for these experiments. Moreover, the experiments use distortion parameter δ = 1 without verifying the submodularity condition required by the main approximation theorem.

The empirical results are also less convincing than the presentation suggests. The main improvements are measured using a surrogate design-welfare objective rather than actual prediction quality. On several real datasets, acquiring data through GRADES produces higher test MSE than acquiring no data at all. The reported runtime advantage excludes critical-payment computation, so it does not establish an equivalent end-to-end mechanism speedup.

Finally, the experiments simulate sellers with constructed costs and do not test strategic cost misreporting or label manipulation. Consequently, the core incentive claims receive little empirical validation.

The writing and organization also need substantial improvement. There are frequent grammatical and formatting issues, and the central contribution is obscured by lengthy discussions of established results.

Questions for authors
  1. What is the substantive methodological novelty beyond combining Lu et al. (2024), Harshaw et al. (2019), and Deng et al. (2025)? Which theoretical result would not follow from existing frameworks?

  2. How applicable are the incentive and welfare guarantees when features are private, sellers have heterogeneous beliefs, and the buyer lacks reliable evaluation labels? In particular, what remains of the proposed mechanism without these assumptions?

  3. Why should the empirical results support the theoretical claims when the experiments use a different learning setup and do not verify the required submodularity condition? Can the authors provide evidence of improved realized prediction welfare and evaluate the complete mechanism, including payments and strategic behavior?

Lu et al. (2024). Data Acquisition via Experimental Design for Data Markets.

Harshaw et al. (2019). Submodular Maximization beyond Non-negativity: Guarantees, Fast Algorithms, and Applications.

Deng et al. (2025). Procurement Auctions via Approximately Optimal Submodular Optimization.

Hu et al. (2026). Buying Data of Unknown Quality: Fisher Information Procurement Auctions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.