- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies multi-domain data allocation under limited budgets when the task objective is a nonlinear utility of multiple metrics. It identifies a limitation of conventional single-domain scalar-gain valuation, which evaluates each domain independently and therefore cannot capture how candidate value changes with the evolving joint allocation. To address this, the authors propose Compose-Then-Scalarize (CTS), which reuses fixed single-domain pilot runs to fit signed metric-level response curves, composes these responses into a predicted joint metric state, and then applies the nonlinear task utility to obtain allocation-dependent candidate values without additional pilot training. The paper further analyzes when CTS reduces to conventional scalar valuation and how nonlinear utility can induce preference reversals. Experiments on autonomous driving and FMoW-WILDS show that CTS consistently achieves the highest mean task score across evaluated budgets, while matched ablations and controlled retraining further support the benefit of joint-state valuation and its improved prediction of background-dependent candidate preferences.
Critical strengths
The paper’s main strength is its clear identification of a key limitation in conventional scalar data valuation: under nonlinear multi-metric utilities, the marginal value of a domain can depend on the evolving joint allocation. CTS addresses this in a simple and principled way by composing metric-level responses before applying the task utility, while reusing the same single-domain pilot evidence and avoiding additional pilot training.
A second strength is the careful empirical validation. The authors include matched comparisons with M-STC and Static-vector variants, which help isolate the benefit of joint-state valuation from other modeling choices. The controlled retraining experiments further support the proposed mechanism by showing that CTS better captures background-dependent candidate preferences.
Critical weaknesses
A key weakness is the limited independence of the autonomous-driving evaluation protocol. Pilot feedback is obtained from the same navtest split used for final performance reporting, which may introduce optimistic bias. Although the paper includes a reassuring held-out feedback-split experiment for navtrain, an analogous independent evaluation for OpenScene would further strengthen the evidence.
A second concern is the additive response assumption underlying CTS. The method composes independently fitted domain-level metric responses into a joint state, but this does not explicitly model cross-domain interactions arising during shared model training. Since such interactions may affect the true marginal value of additional data, a more direct validation of the predicted joint metric state under mixed-domain allocations would improve confidence in the method’s generality.
Questions for authors
Could the authors clarify whether an independent validation/test protocol can also be applied to OpenScene? Since pilot feedback and final evaluation currently reuse navtest, it would be helpful to know whether CTS maintains its advantage when response fitting and allocation are based on feedback completely disjoint from the final test split.
Could the authors also provide a more direct evaluation of the additive joint-state approximation used by CTS? In particular, comparing predicted and actual metric states, utility values, or candidate rankings under several mixed-domain allocations would help assess how accurately the surrogate captures cross-domain interactions during training.
6 comments
Q1. Independent pilot feedback on OpenScene
Question summary. The original driving protocol uses navtest for both pilot feedback and final reporting. Can the independent-feedback check already provided for navtrain be extended to OpenScene, and does CTS retain its advantage?
Response. Thank you for carefully examining the evaluation protocol and noting the existing navtrain check. We have now completed the corresponding OpenScene experiment.
Yes: with pilot feedback from held-out training logs, CTS exceeds STC (MOSAIC) at every OpenScene budget by 0.99–1.64 EPDMS points, compared with 1.02–1.61 points under the original protocol.
Experimental purpose and protocol. We test whether CTS’s advantage persists when response fitting and allocation use feedback disjoint from the final test split.
- Validation split. We randomly hold out 10% of the OpenScene trainval logs. Only logs containing no base clips are eligible, preserving the base dataset and checkpoint.
- Training and candidate pool. Held-out clips are excluded from training and removed from the candidate pool. Pilots whose ranked prefixes change are retrained.
- Pilot feedback and allocation. We evaluate the base and pilot models on Validation, refit the response curves, and rerun allocation and final retraining separately for STC and CTS.
- Final evaluation. navtest is used only to evaluate the final retrained models.
OpenScene results. The table reports mean final EPDMS on navtest. CTS scores are bold; parentheses show improvements over STC under the same feedback protocol.
| Pilot feedback | Method | B = 250 | 500 | 1000 | 2000 | 4000 |
|---|---|---|---|---|---|---|
| navtest, original | STC | 77.38 | 79.38 | 81.68 | 82.78 | 84.25 |
| CTS | 78.76 (+1.38) | 80.99 (+1.61) | 82.78 (+1.10) | 84.04 (+1.26) | 85.27 (+1.02) | |
| Validation, disjoint from navtest | STC | 77.11 | 79.14 | 81.46 | 82.59 | 84.08 |
| CTS | 78.46 (+1.35) | 80.78 (+1.64) | 82.53 (+1.07) | 83.88 (+1.29) | 85.07 (+0.99) |
Effect on scores and selections. Score decrease is original-protocol EPDMS minus Validation-feedback EPDMS. Selection overlap is the percentage of additional selected clips shared by the two protocols for the same method, seed, and budget.
| Measurement | B = 250 | 500 | 1000 | 2000 | 4000 |
|---|---|---|---|---|---|
| STC score decrease, EPDMS points | 0.27 | 0.24 | 0.22 | 0.19 | 0.17 |
| CTS score decrease, EPDMS points | 0.30 | 0.21 | 0.25 | 0.16 | 0.20 |
| STC selection overlap (%) | 89.5 | 91.0 | 92.7 | 94.1 | 95.0 |
| CTS selection overlap (%) | 91.6 | 93.2 | 94.8 | 95.5 | 96.3 |
The comparative advantage is preserved. Changing the feedback protocol lowers scores slightly for both methods, while changing CTS’s advantage by at most 0.03 EPDMS points. CTS remains ahead at every budget.
The selected data remain stable. CTS retains 91.6–96.3% of its selected clips, with higher overlap than STC at every budget.
The result is consistent with the existing navtrain check. That check (Appendix A.4, Table 10) reports a CTS advantage of 0.72–1.09 points under independent feedback, compared with 0.71–1.02 under the original protocol. The margin changes by at most 0.08 points, while scores decrease by 0.26–0.38 points.
With independent feedback, CTS also exceeds the previously reported results of the six other baselines in Table 1 at every budget. Its margins over the best of those baselines are 1.44–2.66 points on OpenScene and 0.95–1.63 points on navtrain.
Conclusion and reporting choice. CTS’s advantage persists on both driving settings when pilot feedback is disjoint from final evaluation. FMoW-WILDS already uses OOD-validation feedback and reserves the OOD test split for final reporting, providing complementary evidence under a separated protocol.
We retain the original-protocol driving results in the main Table 1 for direct comparison with MOSAIC under its reported protocol. The independent-feedback results are reported in Section 5.5 and Appendix E.5.
Locations. Section 5.5, “Pilot feedback disjoint from the test split”; Appendix E.5, Tables 24–25. The original navtrain check appears in Appendix A.4, Table 10.
Q2. Accuracy of additive joint-state prediction
Question summary. CTS combines independently fitted single-domain responses into a joint state. How closely do its predicted metric states, utility gains, and candidate preferences match actual mixed-domain retraining?
Response. We now directly evaluate all three levels: the metric state, the utility gain, and the candidate preference. CTS achieves a mean state error of 0.41 percentage points per metric and predicts both allocation gains and candidate preferences more accurately than STC.
Experimental setup and reuse. We reuse the OpenScene models already retrained for the original conditional-preference experiment in Appendix D.3.
Each comparison contains two models sharing the same 1000-clip base dataset and 400-clip background, but receiving different fixed 400-clip candidate batches. The background mixes the two remaining cities at fractions 0, 0.25, 0.5, 0.75, and 1. Each model received 800 additional clips and was retrained for 40 epochs.
Six city pairs, five background mixtures, and three seeds yield 90 paired comparisons and 180 model evaluations. Holding the candidate batches fixed while changing the background tests whether predictions track their changing relative value.
All fitted curves use only single-domain pilots; the reused mixed-domain outcomes are used solely for validation.
Three levels of validation.
- Metric state: State MAE measures absolute prediction error across the nine component metrics and all model evaluations.
- Utility gain: Gain MAE measures error in the predicted improvement over the same-seed base model, using observed official EPDMS gains as the reference.
- Candidate preference: Correct preferences count how often the predicted difference identifies the better candidate. Gap MAE measures error in that difference; correlation and calibration assess how closely predicted differences track observed ones.
Calibration fits observed gap = intercept + slope × predicted gap, with ideal slope 1 and intercept 0.
| Level | Measurement | STC | CTS |
|---|---|---|---|
| Metric state | State MAE ↓, pp | Not predicted | 0.41 |
| Utility gain | Gain MAE ↓, EPDMS points | 0.58 | 0.46 |
| Candidate preference | Correct preferences ↑ | 49/90 | 72/90 |
| Gap MAE ↓, EPDMS points | 0.61 | 0.23 | |
| Predicted–observed gap correlation ↑, Pearson’s r | 0.34 | 0.78 | |
| Calibration slope, ideal = 1 | 0.29 | 0.82 | |
| Calibration intercept, ideal = 0 | −0.08 | 0.03 |
What the results show. The low state error directly supports the usefulness of the additive joint-state approximation. Its decision value is demonstrated by more accurate gain predictions and 72/90 correct preferences, compared with 49/90 for STC. Across all six city pairs, CTS predicts 10–15 of 15 preferences correctly, versus 7–9 for STC, and achieves lower Gap MAE for every pair. It also captures the observed Singapore–Boston preference reversal as the background changes.
Section 5.4 compares CTS (72/90) with the matched control M-STC (47/90). Here we compare the same CTS predictions with STC (MOSAIC), which fits scalar gains directly (49/90).
Interactions and further improvement. CTS uses an additive response approximation, analogous to additive source-contribution models in prior work (Mahmood et al., 2025; Dimlioglu et al., 2026). It already advances beyond STC by allowing complementary or conflicting metric changes to affect candidate values at the predicted joint state. For example, improving one metric can increase the value of another domain’s improvement to a different metric, while positive and negative responses can offset each other before utility is computed. It obtains this capability from the same single-domain pilots, without additional pilot training.
Single-domain pilots alone cannot identify cross-domain training interactions that change the responses themselves. Learning these interactions would require additional mixed-domain pilots and extra training cost. CTS could also benefit from such response models: they would supply richer joint-state predictions to the same utility function.
Conclusion. Direct comparison with actual mixed-domain retraining supports the approximation at all three levels under the tested mixtures: CTS predicts component states with low average error and improves both gain and preference prediction over STC using the same single-domain evidence.
Locations. Section 5.5, “Additive composition under joint retraining,” and Table 3(a); Appendix E.1 and Table 18. Appendix F discusses richer response models using additional mixed-domain pilots.
Thank you again for these thoughtful questions and for your careful engagement with the paper. We hope the clarifications and direct experimental evidence address your concerns, and we welcome further discussion.
Response to Reviewer r3Fk: Overview
Thank you for your thoughtful review and careful assessment. We appreciate your recognition of the key limitation identified in independent scalar-gain valuation and CTS’s simple and principled approach to addressing it. We also thank you for your positive assessment of the presentation and empirical validation, particularly the matched ablations and controlled retraining that support the practical benefits of joint-state valuation.
Both questions were very helpful, and both concerns are addressable through direct empirical checks. We report an OpenScene experiment separating pilot feedback from final evaluation, together with a direct assessment of joint-state prediction using existing mixed-domain retraining models. Section 5.5 summarizes the findings, with protocols and full results in Appendices E.1 and E.5. Below, STC denotes MOSAIC’s scalar-gain valuation (Dimlioglu et al., 2026).
In the revised manuscript, the corresponding headings for Q1 are marked in green. The joint-state analysis for Q2 is marked in purple, as this concern is shared with the other reviewers.