- Rating
- 3(+1)
- Confidence
- 3(+1)
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes CARVEPrep, a model-aware data preparation framework that jointly decides whether to retain, repair, delete, or downweight each record. Candidate preparations are evaluated through downstream model feedback, and the resulting losses are further used to update record priorities across rounds. The method is evaluated on tabular data, sensor time series, and instruction tuning, with extensive comparisons and ablations.
Critical strengths
-
The paper provides a compelling formulation of data preparation as model-aware action allocation rather than simply maximizing data correctness.
-
The experimental evaluation is extensive, covering many baselines, ablations, multiple data types, scale studies, and held-out acceptance tests.
Critical weaknesses
-
The action-value regressors are trained using self-injected error types that closely match the synthetic corruption used in the main evaluation, so robustness to natural or mismatched errors remains unclear.
-
CARVEPrep evaluates many candidate preparations through repeated downstream post-training, but the paper does not provide a compute-matched comparison with the baselines. It is therefore unclear how much of the gain comes from the proposed allocation strategy versus additional search budget.
Questions for authors
-
How does CARVEPrep perform when the test-time corruption types differ from those used in the self-injected perturbation trials?
-
Can the authors provide a comparison under a matched candidate-evaluation or post-training compute budget?
-
For the instruction-tuning experiment, can the full learned action-value mechanism be evaluated instead of the evidence-based allocation rule?
5 comments
Response to Reviewer DEGn (2 of 2)
To W2 and Q2, The gain holds without the additional search budget. We evaluated CARVEPrep with in-context adoption, removing short post-training from candidate comparison (Appendix Table 14). It reaches 0.779, above Jellyfish's 0.741. CARVEPrep takes 97 seconds per run, compared with 366 for Jellyfish on the same hardware (Table D2). These times include preparation and final training. Over five data seeds, the gains are 7.8 points over NoFix ([4.7, 11.6]) and 3.8 over Jellyfish ([1.6, 6.4]), at about a quarter of Jellyfish's cost. We ran a second control with the same adoption rule and the same number of short runs across candidate sets, as detailed below. Appendix C.8 of the submission specifies the fixed search and equal short-training counts per finalist.
Under the same adoption rule and short-training budget, combining CARVEPrep candidates with the fourteen baseline tables reaches 0.816, above either candidate set alone (Appendix Table 3). The means are 0.803 for CARVEPrep candidates and 0.814 for the baseline tables plus selection candidates. CARVEPrep's candidates give higher accuracy than the baseline tables on five of six tables on the submitted data seeds. Over five seeds they do so on four of six tables, while the baseline tables under the rule reach a mean of 0.784.
Table D2 (new runs, Section 4.3, Appendix Tables 9 and 14). Five-seed six-table means. Time in seconds per run covers preparation and one final training on shared hardware.
| Method | Seconds | Short post-training runs | Accuracy |
|---|---|---|---|
| NoFix | 24 | 0 | 0.701 |
| RHO-LOSS | 20 | 0 | 0.727 |
| Ref. likelihood | 19 | 0 | 0.727 |
| Raha+Baran | 239 | 0 | 0.708 |
| Jellyfish | 366 | 0 | 0.741 |
| CARVEPrep, in-context adoption | 97 | 0 | 0.779 |
| CARVEPrep-ER (external repairs, in-context adoption, no record cap) | 111 plus the external methods (605) | 0 | 0.795 |
| CARVEPrep | 822 | 18 to 24 | 0.771 |
To Q3, We ran the full learned action-value mechanism on Alpaca over three seeds. It reduces held-out loss from 1.397 for NoFix to 1.336, and deletion by reference loss gives 1.339 in the same runs. The new experiment injects text errors in the trials, fits per-action value regressors on the text evidence and uses the same allocation rule as for tables (Appendix Tables 7 and 17, Appendix C.5). The submission used the evidence rule on text for a practical reason. In-context loss does not separate a clean pool from a corrupted one there, so every perturbation trial needs a short LoRA training run, and a run with learned values takes about 10,000 seconds against about 1,500 for the evidence rule (Appendix C.5).
We welcome further discussion if any concerns remain.
Dear Reviewer DEGn,
We are writing to ask whether our rebuttal has addressed your concerns. We would be happy to discuss any remaining issues.
Response to Reviewer DEGn (1 of 2)
Thank you for the focused review and for finding the model-aware formulation compelling. Below are the new runs for mismatched errors, matched compute and the learned values on text.
To W1 and Q1, In our design the trials only determine which candidates are proposed, and a model-based test on the reference set determines whether a candidate is adopted. Based on your question, we added three comparisons using new feature errors, pair-flip label noise or both (Appendix Table 13, Table D1 below). CARVEPrep ranks first among the non-oracle methods run in all three settings. The mean accuracies are 0.846 for new feature errors with pair-flip labels, 0.875 for new feature errors alone and 0.849 for pair-flip labels alone, against the corresponding strongest baselines at 0.800, 0.867 and 0.811. These are changes of error family, evaluated on the submitted data seeds with three training runs per entry. Separately, changing the perturbation trials while keeping the submitted training-table errors leaves accuracy within 0.2 points of the default-trial rerun.
Table D1 (new runs, Section 4.3, Appendix Table 13). Six-table means of test accuracy over three training runs, rerun on the submitted data seeds. Strongest baseline is the best of the baselines run in that setting. Picket cannot run on Adult and Breast Cancer under added feature errors alone, since its prepared labels exceed the ten classes TabPFN-v2 supports. Flights has no numeric column for the three error types of that setting, so Flights keeps its clean table there.
The first three settings differ from the trials in kind, since the added feature errors are heavy-tailed outliers, unit inconsistencies and cross-column dependency breaks that ignore any declared dependency (Appendix C.8). In the trials, dependency violations rewrite only values that a declared functional dependency determines. Pair-flip noise sends each class to one fixed class, whereas the trials redraw labels.
CARVEPrep adopts a preparation only after it passes the gain margin and paired bootstrap test on the reference set, whose records carry no injected errors. Records with rejected labels are deleted by the allocation rule without using the learned values, so their deletion does not depend on the trials.
On the natural dirty versions of Beers, Flights, Hospitals and Tax, CARVEPrep retains the original table and its mean accuracy of 0.854, while eleven of the fourteen baselines fall below it. This is within 0.35 points of clean training at 0.857. These runs evaluate natural feature errors with intact labels and no injected corruption (Appendix C.9).