acceptodds
Under review as a conference paper at ICLR 2027

Compose Then Scalarize: Joint-State Valuation for Data Allocation

Abstract

Allocating a limited data budget across domains requires estimating the value of candidate additions from limited training evidence. Independent scalar-gain valuation based on single-domain pilots predicts each domain’s task-score gain independently of allocations to other domains. When the task score aggregates multiple metrics through a nonlinear utility, however, a candidate's marginal value can change with the joint allocation. We propose Compose-Then-Scalarize (CTS), a joint-state valuation method that reuses fixed single-domain pilot evidence to reassess candidate additions as the joint allocation changes. CTS fits signed metric-response curves, additively composes their predictions across domains into a joint metric state, and applies the utility to obtain state-dependent candidate values without additional pilot training. Experiments on autonomous driving and satellite image classification show that CTS achieves the highest mean task score at every evaluated allocation size. Compared with the strongest of the seven baselines at matched budgets, CTS improves the composite driving score by 0.71–1.61 points and worst-region accuracy by 0.93–3.21 percentage points. Matched ablations support the benefit of joint-state valuation, while controlled retraining shows that CTS more accurately predicts background-dependent candidate preferences than a matched control that values candidates at their own-domain states. Code is available at https://anonymous.4open.science/r/ICLR-CTS.

Primary area:
optimization
Keywords:
data allocation, data valuation, nonlinear utility, data mixture optimization, data-efficient learning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_r3FkOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies multi-domain data allocation under limited budgets when the task objective is a nonlinear utility of multiple metrics. It identifies a limitation of conventional single-domain scalar-gain valuation, which evaluates each domain independently and therefore cannot capture how candidate value changes with the evolving joint allocation. To address this, the authors propose Compose-Then-Scalarize (CTS), which reuses fixed single-domain pilot runs to fit signed metric-level response curves, composes these responses into a predicted joint metric state, and then applies the nonlinear task utility to obtain allocation-dependent candidate values without additional pilot training. The paper further analyzes when CTS reduces to conventional scalar valuation and how nonlinear utility can induce preference reversals. Experiments on autonomous driving and FMoW-WILDS show that CTS consistently achieves the highest mean task score across evaluated budgets, while matched ablations and controlled retraining further support the benefit of joint-state valuation and its improved prediction of background-dependent candidate preferences.

Critical strengths

The paper’s main strength is its clear identification of a key limitation in conventional scalar data valuation: under nonlinear multi-metric utilities, the marginal value of a domain can depend on the evolving joint allocation. CTS addresses this in a simple and principled way by composing metric-level responses before applying the task utility, while reusing the same single-domain pilot evidence and avoiding additional pilot training.

A second strength is the careful empirical validation. The authors include matched comparisons with M-STC and Static-vector variants, which help isolate the benefit of joint-state valuation from other modeling choices. The controlled retraining experiments further support the proposed mechanism by showing that CTS better captures background-dependent candidate preferences.

Critical weaknesses

A key weakness is the limited independence of the autonomous-driving evaluation protocol. Pilot feedback is obtained from the same navtest split used for final performance reporting, which may introduce optimistic bias. Although the paper includes a reassuring held-out feedback-split experiment for navtrain, an analogous independent evaluation for OpenScene would further strengthen the evidence.

A second concern is the additive response assumption underlying CTS. The method composes independently fitted domain-level metric responses into a joint state, but this does not explicitly model cross-domain interactions arising during shared model training. Since such interactions may affect the true marginal value of additional data, a more direct validation of the predicted joint metric state under mixed-domain allocations would improve confidence in the method’s generality.

Questions for authors

Could the authors clarify whether an independent validation/test protocol can also be applied to OpenScene? Since pilot feedback and final evaluation currently reuse navtest, it would be helpful to know whether CTS maintains its advantage when response fitting and allocation are based on feedback completely disjoint from the final test split.

Could the authors also provide a more direct evaluation of the additive joint-state approximation used by CTS? In particular, comparing predicted and actual metric states, utility values, or candidate rankings under several mixed-domain allocations would help assess how accurately the surrogate captures cross-domain interactions during training.

6 comments
AuthorsOpenReview ↗

Response to Reviewer r3Fk: Overview

Thank you for your thoughtful review and careful assessment. We appreciate your recognition of the key limitation identified in independent scalar-gain valuation and CTS’s simple and principled approach to addressing it. We also thank you for your positive assessment of the presentation and empirical validation, particularly the matched ablations and controlled retraining that support the practical benefits of joint-state valuation.

Both questions were very helpful, and both concerns are addressable through direct empirical checks. We report an OpenScene experiment separating pilot feedback from final evaluation, together with a direct assessment of joint-state prediction using existing mixed-domain retraining models. Section 5.5 summarizes the findings, with protocols and full results in Appendices E.1 and E.5. Below, STC denotes MOSAIC’s scalar-gain valuation (Dimlioglu et al., 2026).


In the revised manuscript, the corresponding headings for Q1 are marked in green. The joint-state analysis for Q2 is marked in purple, as this concern is shared with the other reviewers.

AuthorsOpenReview ↗

Q1. Independent pilot feedback on OpenScene

Question summary. The original driving protocol uses navtest for both pilot feedback and final reporting. Can the independent-feedback check already provided for navtrain be extended to OpenScene, and does CTS retain its advantage?

Response. Thank you for carefully examining the evaluation protocol and noting the existing navtrain check. We have now completed the corresponding OpenScene experiment.

Yes: with pilot feedback from held-out training logs, CTS exceeds STC (MOSAIC) at every OpenScene budget by 0.99–1.64 EPDMS points, compared with 1.02–1.61 points under the original protocol.


Experimental purpose and protocol. We test whether CTS’s advantage persists when response fitting and allocation use feedback disjoint from the final test split.

  • Validation split. We randomly hold out 10% of the OpenScene trainval logs. Only logs containing no base clips are eligible, preserving the base dataset and checkpoint.
  • Training and candidate pool. Held-out clips are excluded from training and removed from the candidate pool. Pilots whose ranked prefixes change are retrained.
  • Pilot feedback and allocation. We evaluate the base and pilot models on Validation, refit the response curves, and rerun allocation and final retraining separately for STC and CTS.
  • Final evaluation. navtest is used only to evaluate the final retrained models.

OpenScene results. The table reports mean final EPDMS on navtest. CTS scores are bold; parentheses show improvements over STC under the same feedback protocol.

Pilot feedbackMethodB = 250500100020004000
navtest, originalSTC77.3879.3881.6882.7884.25
CTS78.76 (+1.38)80.99 (+1.61)82.78 (+1.10)84.04 (+1.26)85.27 (+1.02)
Validation, disjoint from navtestSTC77.1179.1481.4682.5984.08
CTS78.46 (+1.35)80.78 (+1.64)82.53 (+1.07)83.88 (+1.29)85.07 (+0.99)

Effect on scores and selections. Score decrease is original-protocol EPDMS minus Validation-feedback EPDMS. Selection overlap is the percentage of additional selected clips shared by the two protocols for the same method, seed, and budget.

MeasurementB = 250500100020004000
STC score decrease, EPDMS points0.270.240.220.190.17
CTS score decrease, EPDMS points0.300.210.250.160.20
STC selection overlap (%)89.591.092.794.195.0
CTS selection overlap (%)91.693.294.895.596.3

The comparative advantage is preserved. Changing the feedback protocol lowers scores slightly for both methods, while changing CTS’s advantage by at most 0.03 EPDMS points. CTS remains ahead at every budget.

The selected data remain stable. CTS retains 91.6–96.3% of its selected clips, with higher overlap than STC at every budget.

The result is consistent with the existing navtrain check. That check (Appendix A.4, Table 10) reports a CTS advantage of 0.72–1.09 points under independent feedback, compared with 0.71–1.02 under the original protocol. The margin changes by at most 0.08 points, while scores decrease by 0.26–0.38 points.

With independent feedback, CTS also exceeds the previously reported results of the six other baselines in Table 1 at every budget. Its margins over the best of those baselines are 1.44–2.66 points on OpenScene and 0.95–1.63 points on navtrain.


Conclusion and reporting choice. CTS’s advantage persists on both driving settings when pilot feedback is disjoint from final evaluation. FMoW-WILDS already uses OOD-validation feedback and reserves the OOD test split for final reporting, providing complementary evidence under a separated protocol.

We retain the original-protocol driving results in the main Table 1 for direct comparison with MOSAIC under its reported protocol. The independent-feedback results are reported in Section 5.5 and Appendix E.5.

Locations. Section 5.5, “Pilot feedback disjoint from the test split”; Appendix E.5, Tables 24–25. The original navtrain check appears in Appendix A.4, Table 10.

AuthorsOpenReview ↗

Q2. Accuracy of additive joint-state prediction

Question summary. CTS combines independently fitted single-domain responses into a joint state. How closely do its predicted metric states, utility gains, and candidate preferences match actual mixed-domain retraining?

Response. We now directly evaluate all three levels: the metric state, the utility gain, and the candidate preference. CTS achieves a mean state error of 0.41 percentage points per metric and predicts both allocation gains and candidate preferences more accurately than STC.


Experimental setup and reuse. We reuse the OpenScene models already retrained for the original conditional-preference experiment in Appendix D.3.

Each comparison contains two models sharing the same 1000-clip base dataset and 400-clip background, but receiving different fixed 400-clip candidate batches. The background mixes the two remaining cities at fractions 0, 0.25, 0.5, 0.75, and 1. Each model received 800 additional clips and was retrained for 40 epochs.

Six city pairs, five background mixtures, and three seeds yield 90 paired comparisons and 180 model evaluations. Holding the candidate batches fixed while changing the background tests whether predictions track their changing relative value.

All fitted curves use only single-domain pilots; the reused mixed-domain outcomes are used solely for validation.

Three levels of validation.

  1. Metric state: State MAE measures absolute prediction error across the nine component metrics and all model evaluations.
  2. Utility gain: Gain MAE measures error in the predicted improvement over the same-seed base model, using observed official EPDMS gains as the reference.
  3. Candidate preference: Correct preferences count how often the predicted difference identifies the better candidate. Gap MAE measures error in that difference; correlation and calibration assess how closely predicted differences track observed ones.

Calibration fits observed gap = intercept + slope × predicted gap, with ideal slope 1 and intercept 0.

LevelMeasurementSTCCTS
Metric stateState MAE ↓, ppNot predicted0.41
Utility gainGain MAE ↓, EPDMS points0.580.46
Candidate preferenceCorrect preferences ↑49/9072/90
Gap MAE ↓, EPDMS points0.610.23
Predicted–observed gap correlation ↑, Pearson’s r0.340.78
Calibration slope, ideal = 10.290.82
Calibration intercept, ideal = 0−0.080.03

What the results show. The low state error directly supports the usefulness of the additive joint-state approximation. Its decision value is demonstrated by more accurate gain predictions and 72/90 correct preferences, compared with 49/90 for STC. Across all six city pairs, CTS predicts 10–15 of 15 preferences correctly, versus 7–9 for STC, and achieves lower Gap MAE for every pair. It also captures the observed Singapore–Boston preference reversal as the background changes.

Section 5.4 compares CTS (72/90) with the matched control M-STC (47/90). Here we compare the same CTS predictions with STC (MOSAIC), which fits scalar gains directly (49/90).

Interactions and further improvement. CTS uses an additive response approximation, analogous to additive source-contribution models in prior work (Mahmood et al., 2025; Dimlioglu et al., 2026). It already advances beyond STC by allowing complementary or conflicting metric changes to affect candidate values at the predicted joint state. For example, improving one metric can increase the value of another domain’s improvement to a different metric, while positive and negative responses can offset each other before utility is computed. It obtains this capability from the same single-domain pilots, without additional pilot training.

Single-domain pilots alone cannot identify cross-domain training interactions that change the responses themselves. Learning these interactions would require additional mixed-domain pilots and extra training cost. CTS could also benefit from such response models: they would supply richer joint-state predictions to the same utility function.


Conclusion. Direct comparison with actual mixed-domain retraining supports the approximation at all three levels under the tested mixtures: CTS predicts component states with low average error and improves both gain and preference prediction over STC using the same single-domain evidence.

Locations. Section 5.5, “Additive composition under joint retraining,” and Table 3(a); Appendix E.1 and Table 18. Appendix F discusses richer response models using additional mixed-domain pilots.

Thank you again for these thoughtful questions and for your careful engagement with the paper. We hope the clarifications and direct experimental evidence address your concerns, and we welcome further discussion.

Reviewer_bfhqOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies how to allocate a limited additional training-data budget across multiple domains using a fixed set of single-domain pilot experiments. Existing approaches fit each domain’s scalar task-score gain and sum these gains to estimate the value of a joint allocation. The proposed Compose-Then-Scalarize (CTS) method instead fits changes in individual performance metrics, adds the predicted changes across domains, and then applies a nonlinear scoring function. This allows candidate additions to be reassessed as the allocation changes, without further pilot training. Experiments on autonomous driving and satellite image classification show improvements over the evaluated allocation baselines.

Critical strengths

The evaluation covers autonomous driving and satellite image classification, with multiple budgets, matched ablations, and controlled retraining experiments that examine whether candidate preferences change with the background allocation.

Critical weaknesses
  1. Presentation and readability. The manuscript could benefit from more intuitive explanations and simpler terminology. Compound phrases such as “scalar-gain valuation,” “allocation-relevant metric-response differences,” and “fixed single-domain pilot evidence” make the central idea harder to follow than necessary. Introducing these concepts through a concrete example would help readers understand the motivation and distinguish the proposed method from existing approaches.

  2. Novelty and underlying assumptions. The main methodological change is composing predicted metric responses before applying a nonlinear utility, which appears relatively incremental, and the theoretical contribution could be clarified further. More importantly, I would appreciate a clearer explanation of when a small number of single-domain pilot runs can reliably predict the benefits of additional samples or larger data budgets. Samples within a domain may vary substantially in informativeness, redundancy, and difficulty, and fixing their ordering does not by itself establish a predictable performance curve. For complex tasks, sample dependence and interactions across data sources may also affect the additive metric-response approximation. Clarifying these assumptions and examining sensitivity to sample ordering, temporal redundancy, and within-domain heterogeneity would strengthen confidence in the method’s applicability.

Questions for authors

Could the authors clarify the methodological and theoretical advances beyond changing the order of metric composition and nonlinear aggregation?

Under what conditions can a small number of single-domain pilot runs reliably predict performance gains at larger data budgets? How is this predictive accuracy validated?

How sensitive are the fitted response curves and resulting allocations to within-domain sample ordering, temporal redundancy, and differences in sample informativeness?

When is the additive metric-response approximation reasonable, and how does CTS perform when data sources exhibit strong complementary or conflicting effects during training?

12 comments
AuthorsOpenReview ↗

W1. Presentation and readability

Thank you for your careful reading and constructive feedback. We particularly appreciate your recognition of the evaluation across autonomous driving and satellite image classification, including multiple budgets, matched ablations, and controlled retraining experiments examining how candidate preferences change with the background allocation.

Your questions helped us identify where clearer explanations and direct validation would be most useful. The concerns about pilot extrapolation, sample ordering, and additive composition are addressable through empirical checks, which we now provide in Section 5.5 and Appendix E. These include new ranking experiments and analyses reusing existing pilots and controlled-retraining models. Below, STC denotes MOSAIC’s scalar-gain valuation (Dimlioglu et al., 2026).

Question summary. Some compound terms make the central idea harder to follow. A concrete example would help explain the motivation and the difference from existing approaches.

Response. Thank you for identifying the specific phrases. The central idea is that the usefulness of the next data addition depends on which metrics the current allocation has already improved. CTS retains individual metric responses from the pilots so that it can update candidate values as the allocation changes.


PhrasePlain meaningLocation
Scalar-gain valuationPredict the overall score improvement from adding a given amount of data from each domain alone, then add these predicted improvements across domains.Section 3.2, Eqs. (2)–(4)
Allocation-relevant metric-response differencesTwo domains can produce similar overall score improvements while affecting individual metrics differently. Those differences can change which domain is more useful after other data have been added.Section 3.3, Figure 1
Fixed single-domain pilot evidenceA small set of training runs, each adding data from one domain to the base dataset. Their observations and fitted curves are reused throughout allocation.Section 3.1; Appendix A.4

A concrete example. Consider two metrics with utility u[x,y]=xyu[x,y]=xy, starting from (1.00,1.00)(1.00,1.00). Three domains each offer one equally sized block:

  • A improves the first metric by 0.120.12.
  • B improves the second metric by 0.100.10.
  • C improves the first metric by 0.400.40.

The budget allows two blocks. Initially, their utility gains are 0.120.12, 0.100.10, and 0.400.40, so both methods select C first. The predicted joint state becomes (1.40,1.00)(1.40,1.00), with one block remaining.

To isolate the valuation rule, we compare CTS with M-STC, the matched control using the same metric responses and utility. M-STC values A and B without incorporating C’s response; CTS uses the updated joint state.

Remaining candidateM-STC: gain without C’s responseCTS: gain at (1.40,1.00)(1.40,1.00)
A: first metric +0.12+0.121.12×1.00−1.00=0.121.12\times1.00-1.00=0.121.52×1.00−1.40=0.121.52\times1.00-1.40=0.12
B: second metric +0.10+0.101.00×1.10−1.00=0.101.00\times1.10-1.00=0.101.40×1.10−1.40=0.141.40\times1.10-1.40=0.14
Second choiceAB
Final allocationC + AC + B
Utility of the final joint state1.52×1.00=1.521.52\times1.00=1.521.40×1.10=1.541.40\times1.10=\mathbf{1.54}

After C improves the first metric, improving the second becomes more valuable: B’s gain rises from 0.100.10 to 0.140.14, while A’s remains 0.120.12. The response estimates and utility are unchanged; the valuation state changes the preferred addition.

Figure 1 provides the empirical motivation. Boston and Las Vegas have similar mean EPDMS gains of 0.890.89 and 0.920.92 points, but their extended-comfort decreases differ substantially, at 0.400.40 and 0.920.92 percentage points, and their traffic-light compliance responses have opposite signs. Figure 2(b) tests the decision consequence: the observed preference between fixed Singapore and Boston batches reverses as the shared background changes, and CTS correctly predicts all 15 comparisons.


Locations. Section 3.3 and Figure 1; Section 4.3 and Appendix C; Section 5.4 and Figure 2(b).

AuthorsOpenReview ↗

Q1. Methodological and theoretical advances

Question summary. Composing metric responses before applying the utility appears to be a relatively small modification. What problem does it address, what is new in the method, and what does the analysis establish?

Response. Our contribution is to identify a structural limitation of independent scalar-gain valuation, introduce joint-state valuation using the same pilot evidence, and establish its decision benefits through theory and matched experiments.


1. Identifying the structural limitation. For any fixed domain-wise scalar-gain curves gig_i, the separable value model

VSTC(b)=V0+∑igi(bi)V_{\mathrm{STC}}(\mathbf b)=V_0+\sum_i g_i(b_i)

assigns an additional example from domain ii the marginal value

gi(bi+1)−gi(bi).g_i(b_i+1)-g_i(b_i).

At a fixed bib_i, this value cannot change when allocations to other domains change. This holds for any choice of the single-domain fitted curves, so better fitting cannot fix this background independence.

The limitation matters when different metric responses become more or less valuable as the background changes. Figure 1 shows that similar scalar gains can conceal different component responses; the controlled retraining in Figure 2 demonstrates an actual background-dependent preference reversal.

2. Addressing it through joint-state valuation. CTS combines two design choices:

  • Retain signed metric responses from each pilot. Keeping improvements and degradations preserves how each domain affects the individual metrics.
  • Compose these responses and revalue candidates at the joint state. After each allocation step, CTS updates the predicted metric state and recomputes each candidate’s marginal utility there.

Thus, fixed response estimates support changing candidate values as allocation proceeds. Retaining vectors supplies the information; updating the valuation state makes that information relevant to the current decision.

CTS uses the same single-domain pilot runs as STC, with no additional model training during allocation. On the measured navtrain budgets of 100, 200, and 400 clips, response fitting and allocation together take 26.18, 42.45, and 75.13 ms, compared with complete-workflow wall times of 126.44–147.33 hours.

3. Establishing when it matters and demonstrating its benefits. To isolate the valuation rule, M-STC shares CTS’s fitted responses and utility but values each increment at its own-domain state. Section 4.3 and Appendix C establish three results:

  • Affine equivalence. With an affine utility, CTS and M-STC give identical predicted values and marginal gains, and identical greedy allocations under the same tie-breaking rule.
  • Marginal adjustment. An exact finite-difference expression quantifies how incorporating other domains changes candidate value. For a smooth utility, this adjustment reflects curvature coupling the candidate’s metric increment with the other domains’ accumulated responses. The analysis also bounds its magnitude in terms of utility curvature and the sizes of these responses.
  • Preference reversal. CTS reverses M-STC’s preference exactly when the background-induced adjustment favoring the initially less preferred candidate exceeds the original preference margin. In the W1 example, B gains an additional 0.04 of value, exceeding A’s original 0.02 advantage.

For the driving utility, the adjustment reflects utility curvature; for worst-region accuracy, it reflects changes in the bottleneck region.

Matched empirical validation. The original results in Section 5.3, Table 2 show that CTS achieves higher mean final performance than both M-STC and Static-vector at every evaluated budget on both tasks. These methods share fitted responses, utility, greedy framework, and retraining protocol, and independently allocate and retrain. M-STC isolates the benefit of incorporating other domains into candidate valuation; Static-vector shows the benefit of updating the valuation state. The original controlled-retraining results in Section 5.4, Figure 2, and Appendix D.3 further show that CTS predicts background-dependent candidate preferences more accurately than M-STC. Together, these results establish the practical value of joint-state valuation beyond retaining component responses alone.

Relation to prior work. MOSAIC fits and sums scalar-gain curves from single-domain pilots (Dimlioglu et al., 2026). RegMix and Data Mixing Laws learn predictors from runs on different mixtures (Liu et al., 2025; Ye et al., 2025). Our contribution concerns how fixed single-domain evidence can support background-dependent valuation, with theoretical characterization and controlled empirical validation.


Locations. Sections 3.2 and 4.2–4.3; Appendix C; Section 5.3 and Table 2; Section 5.4 and Appendix D.3; Appendix D.7 and Table 17.

AuthorsOpenReview ↗

Q2 (1/2). Held-out prediction

Question summary. A small number of pilots may not reliably predict the benefit of adding more data. Fixing the ordering defines the selected subsets, but does not itself establish a predictable response curve.

Response. We test held-out prediction accuracy at larger pilot counts. CTS predicts larger pilots more accurately than STC across all three tested settings.


Modeling basis. Learning-curve and scaling-law modeling provide established precedents for predicting performance from smaller training runs (Hestness et al., 2017; Kaplan et al., 2020; Rosenfeld et al., 2020; Viering and Loog, 2023).

Our extrapolation assumes a fixed ranking and training protocol, a response family that captures the trend over the queried range, and pilot observations informative enough to estimate its parameters despite training variability. We assess these conditions through held-out prediction and richer-pilot allocation experiments.

CTS and STC (MOSAIC’s scalar-gain valuation) use the same exponential response family and pilot runs, following MOSAIC’s sparse-pilot setup (Dimlioglu et al., 2026). CTS’s improvements therefore require no additional pilot training relative to STC.

Protocol and reuse. We reuse the navtrain pilots at counts {50, 100, 200, 400} reported in Appendix D.5, covering four cities and three seeds. Each pilot starts from the same seed’s base checkpoint and trains for 10 epochs on the 460-clip base dataset plus the corresponding ranked prefix.

Each test fits two nonzero counts together with the base observation and predicts a larger count excluded from fitting:

  • Pilot Gain MAE measures error in predicting the observed official EPDMS improvement.
  • Correct domain orderings count correctly ranked domain pairs at the held-out count: six pairs per seed, or 18 comparisons.
  • Response MAE measures CTS’s prediction error across the nine component responses, in percentage points.

Held-out prediction results.

Fitting counts → held-out countRatioGain MAE: STC ↓Gain MAE: CTS ↓Correct orderings: STC ↑Correct orderings: CTS ↑Response MAE: CTS, pp ↓
{100, 200} → 4002×0.340.1914/1817/180.12
{50, 100} → 2002×0.380.2213/1816/180.16
{50, 100} → 4004×0.790.4411/1815/180.29

Ratio = held-out count / largest fitting count. Pilot Gain MAE averages four cities and three seeds, in EPDMS points.

CTS improves both gain prediction and domain ordering in all three settings, with lower gain error in every city. At 4× extrapolation, it reduces Pilot Gain MAE from 0.79 to 0.44 points and correctly orders 15/18 pairs, compared with 11/18 for STC.


Locations. Section 5.5 and Table 3(b); Appendix E.2 and Table 19; Appendix D.5 provides the reused pilots.

Reviewer_6mdYOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper focus on how to estimate the value of multi-domain training data under a limited data budget, and proposes Compose-Then-Scalarize (CTS). Existing methods fit one scalar gain curve per domain and sum the predictions. A domain's predicted marginal gain then depends only on its own allocated count, so the value of the same candidate does not respond to allocations made elsewhere, and several metrics are flattened into one number. CTS changes the order of operations: it fits per-metric response curves from fixed single-domain pilots, composes them across domains into a predicted joint metric state, and applies the nonlinear utility only afterwards. Candidate increments are valued at that state. The paper sets out the mechanics of CTS, and analyzes when independent scalar-gain valuation hides allocation-relevant metric responses and when joint-state valuation changes a candidate preference. Experiments cover autonomous driving and satellite image classification. CTS achieves the best mean score at every budget, and ablations confirm the role of the valuation rule.

Critical strengths
  1. The paper identifies an important limitation of existing scalar-gain valuation methods: they ignore how the value of a candidate domain changes with the current allocation. CTS addresses this issue by composing metric-level responses before applying the nonlinear utility function. The method is clearly motivated and supported by a theoretical analysis of when candidate preferences may change. The matched ablations against M-STC and Static-vector further help isolate the benefit of joint-state valuation.

  2. The experimental results are convincing. CTS achieves the highest mean task score across all evaluated budgets on both autonomous driving and satellite image classification. The comparisons cover seven baselines and multiple allocation budgets. More importantly, the controlled candidate-preference experiment directly supports the central motivation of the paper. CTS correctly predicts 72 out of 90 candidate preferences, compared with 47 for M-STC. This provides useful evidence that joint-state valuation can better capture background-dependent data value.

Critical weaknesses
  1. The method assumes that samples within each domain have already been ranked. As described in Appendix A.4, driving clips are ranked by EPDMS computed using the base model, while FMoW-WILDS images are ranked by true-class confidence. Both ranking strategies rely on ground-truth information. However, in practical data acquisition scenarios, candidate samples may not have ground-truth labels available. The paper does not investigate whether CTS remains effective with label-free ranking strategies. This leaves its applicability to unlabeled candidate pools unclear.

  2. CTS estimates the joint metric state by summing responses learned from single-domain pilots. However, the effect of adding data from one domain may change when it is jointly trained with data from other domains. Such cross-domain interactions are not explicitly captured by the additive approximation. Although the candidate-preference experiments demonstrate the effectiveness of CTS, they do not directly evaluate the accuracy of the predicted joint metric states.

Questions for authors
  1. How would CTS perform when ground-truth labels are unavailable for within-domain sample ranking? Could the authors evaluate a label-free ranking strategy, such as prediction entropy or feature-based selection, to examine whether the improvements remain consistent?

  2. CTS constructs the joint metric state by adding response predictions from individual domains. However, joint training may introduce interactions between domains that are not captured by this approximation. Could the authors compare the predicted joint metric states with those obtained from actual retraining? This would help clarify how sensitive CTS is to violations of the additive assumption.

6 comments
AuthorsOpenReview ↗

Response to Reviewer 6mdY: Overview

Thank you for your thoughtful review and for recognizing the importance of the problem, the clear motivation and presentation, and the theoretical analysis of CTS. We also appreciate your assessment that the experiments are convincing, particularly your recognition of the matched ablations and the controlled retraining evidence for background-dependent data value.

The two questions were both very helpful. Both concerns are addressable through direct empirical checks. We repeated pilot training, allocation, and retraining under three label-free rankings, and reused the controlled-retraining models to assess joint-state prediction. Below, STC denotes MOSAIC’s scalar-gain valuation (Dimlioglu et al., 2026).

AuthorsOpenReview ↗

Q1. Label-free within-domain ranking

Question summary. The default rankings use ground-truth information. Does CTS retain its advantage when candidates are ranked without labels, for example using predictive entropy or feature-based selection?

Response. Yes. We tested predictive entropy, feature-based k-center, and random within-domain ordering. CTS outperforms STC (MOSAIC) at every budget under all three orderings on both tasks. Entropy and k-center also keep CTS close to its default-ranking scores and above the six other baselines.


Experimental setup. On navtrain and FMoW-WILDS (3 seeds, 5 budgets each), we replace the default ranking with:

  • Predictive entropy: descending entropy of the base model’s trajectory-candidate probabilities for driving or class probabilities for FMoW-WILDS.
  • Feature-based k-center: greedily selecting the candidate with the largest distance to its nearest neighbor in the base dataset or selected prefix, using base-model features. This follows the feature-coverage principle (Sener & Savarese, 2018).
  • Random ordering: a seed-specific random permutation within each domain.

Entropy and k-center rank the unlabeled candidate pool using only the base model’s predictions or features. For each ordering and seed, we retrain single-domain pilots on the base dataset augmented with the corresponding ranked prefixes, then refit the response curves. STC and CTS share the base model, domain partition, ordering, pilot runs, and training budgets. Each method independently allocates data and retrains its final model.

Results. Scores are means over three seeds. CTS scores are bold, with improvements over STC under the same ordering shown in parentheses. pp denotes percentage points.

(a) navtrain: final EPDMS (gains in EPDMS points).

OrderingMethodB = 1002004008001600
EntropySTC85.9386.6687.8988.7189.84
CTS86.96 (+1.03)87.72 (+1.06)88.86 (+0.97)89.48 (+0.77)90.59 (+0.75)
k-centerSTC86.0286.7787.9888.9289.91
CTS87.04 (+1.02)87.88 (+1.11)89.00 (+1.02)89.64 (+0.72)90.68 (+0.77)
RandomSTC85.1085.7486.9887.8088.88
CTS85.89 (+0.79)86.71 (+0.97)87.84 (+0.86)88.44 (+0.64)89.56 (+0.68)
Best reported result among six other baselines85.2986.2287.3188.5289.68

(b) FMoW-WILDS: OOD-test worst-region accuracy (%; gains in pp).

OrderingMethodB = 200040006000800010000
EntropySTC21.3223.6923.0424.5924.48
CTS23.44 (+2.12)25.64 (+1.95)27.32 (+4.28)28.43 (+3.84)29.39 (+4.91)
k-centerSTC21.4623.8623.2724.9024.70
CTS23.73 (+2.27)25.83 (+1.97)27.63 (+4.36)28.74 (+3.84)29.55 (+4.85)
RandomSTC21.1523.4022.7724.4924.26
CTS23.07 (+1.92)24.68 (+1.28)26.51 (+3.74)27.67 (+3.18)28.53 (+4.27)
Best reported result among six other baselines23.1224.7124.6727.3227.41

The advantage over STC is preserved. CTS exceeds STC in all 30 label-free comparisons, by 0.64–1.11 EPDMS points on navtrain and 1.28–4.91 pp on FMoW-WILDS. These gains are comparable to those under the default ranking (Table 1): 0.71–1.02 points and 2.05–4.97 pp, respectively.

Small performance loss with label-free ranking. With entropy or k-center, CTS is only 0.15–0.35 EPDMS points and 0.25–0.61 pp below its default-ranking score. It exceeds the best reported result among the six other baselines at every budget on both tasks.

Even with no ranking signal, CTS still helps. Under random within-domain ordering, CTS exceeds both STC and the Random baseline, which samples uniformly from the entire candidate pool, at every budget. Overall, CTS exceeds the best reported result among the six other baselines in 26 of the 30 comparisons. The remaining four occur under random ordering and are close: CTS is lower by only 0.08–0.12 EPDMS points on navtrain and 0.03–0.05 pp on FMoW-WILDS.


Conclusion. CTS retains its advantage over STC under all three tested label-free rankings. Random ordering lowers CTS scores by up to 1.39 EPDMS points and 1.47 pp relative to the default. Thus, a good within-domain ranking and joint-state valuation are complementary.

Revision locations and color. Section 5.5, “Within-domain ranking,” and Table 3(c) summarize the findings; Appendix E.3 and Table 21 provide the protocol and full results. The corresponding headings are marked in purple, as this concern is shared with Reviewer bfhq.

AuthorsOpenReview ↗

Q2. Accuracy of additive joint-state prediction

Question summary. CTS adds metric responses learned from single-domain pilots, while joint training may introduce interactions between domains. How accurately does the predicted joint metric state match actual mixed-domain retraining?

Response. We now directly validate additive composition at three levels: the metric state, the utility gain, and the candidate preference. CTS achieves a mean state error of 0.41 percentage points per metric and predicts both allocation gains and candidate preferences more accurately than STC.


Experimental setup and reuse. We reuse the OpenScene models already retrained for the original conditional-preference experiment in Appendix D.3. This adds a direct prediction-accuracy analysis to the existing mixed-domain retraining results.

Each comparison contains two models sharing the same 1000-clip base dataset and 400-clip background, but receiving different fixed 400-clip candidate batches. The background mixes the two remaining cities at five fractions: 0, 0.25, 0.5, 0.75, and 1. Each model received 800 additional clips and was retrained for 40 epochs. Six city pairs, five background mixtures, and three seeds yield 90 paired comparisons and 180 model evaluations.

All fitted curves use only single-domain pilot evidence. The reused mixed-domain outcomes are used solely for validation. Because the reference comes from actual mixed-domain retraining, this directly tests prediction accuracy even though CTS does not explicitly model cross-domain training interactions.

Three levels of validation.

  1. Metric state: State MAE is the mean absolute prediction error across the nine component metrics and all model evaluations.
  2. Utility gain: Gain MAE measures error in the predicted improvement over the base model, using observed official EPDMS gains as the reference. CTS computes its predicted gains through the utility applied to mean component metrics.
  3. Candidate preference: Correct preferences count how often the predicted score difference identifies the better candidate. Gap MAE measures error in that difference, and Pearson’s r measures its correlation with the observed difference.
LevelMeasureSTCCTS
Metric stateState MAE ↓, percentage pointsNot predicted0.41
Utility gainGain MAE ↓, EPDMS points0.580.46
Candidate preferenceCorrect candidate preferences ↑49/9072/90
Gap MAE ↓, EPDMS points0.610.23
Predicted–observed gap correlation ↑, Pearson’s r0.340.78

What the results show. The low state error provides direct empirical support for the additive joint-state approximation. Its decision value is supported by more accurate gain predictions and 72/90 correct candidate preferences, compared with 49/90 for STC. Across all six city pairs, CTS predicts 10–15 of 15 preferences correctly, versus 7–9 for STC, and achieves lower Gap MAE for every pair. It also captures the observed Singapore–Boston preference reversal as the background changes. Together, these results show that, under the tested mixtures, the retained metric responses support useful joint-state predictions and improve valuation over independently fitted scalar gains.

Section 5.4 compares CTS (72/90) with the matched control M-STC (47/90). Here we compare the same CTS predictions with STC (MOSAIC), which fits scalar gains directly (49/90).

Interactions and further improvement. CTS uses an additive response approximation, analogous to additive source-contribution models in prior work (Mahmood et al., 2025; Dimlioglu et al., 2026). It already advances beyond STC by allowing complementary or conflicting metric changes to affect candidate values at the predicted joint state. It obtains this capability from the same single-domain pilots, without additional pilot training.

Single-domain pilots alone cannot identify cross-domain training interactions that change the responses themselves. Learning these interactions would require additional mixed-domain pilots and extra training cost. CTS could also benefit from such response models: they would supply richer joint-state predictions to the same utility function.


Revision locations and color. Section 5.5, “Additive composition under joint retraining,” and Table 3(a) summarize the findings; Appendix E.1 and Table 18 provide the protocol and full results. The corresponding headings are marked in purple, as this concern is shared by all three reviewers. Appendix F discusses the extension using additional mixed-domain pilots.

Thank you again for these helpful questions. We hope the new evidence addresses the concerns raised in your review, and we welcome further discussion.

Comments on the paper

4 comments
AuthorsOpenReview ↗

General Response to All Reviewers

We thank Reviewers 6mdY, bfhq, and r3Fk for their time, careful reading, and valuable feedback. We appreciate 6mdY and r3Fk’s recognition that our work identifies an important limitation of independent scalar-gain valuation. Both reviewers rated the presentation as “clear and easy to follow,” and r3Fk described CTS as a “simple and principled” approach. 6mdY also recognized the theoretical analysis of when candidate preferences may change. We appreciate all three reviewers’ recognition of the evaluation across two tasks, matched ablations, and controlled retraining (6mdY, bfhq, r3Fk).

Addressing the reviewers’ concerns

The reviewers’ concerns are addressable through targeted validation and clarification. The main empirical concerns converge on one question: do single-domain pilots support accurate value predictions and robust allocation decisions? The table below summarizes five additional analyses in the question order of Appendix E.

Throughout these comparisons, STC denotes MOSAIC’s scalar-gain valuation (Dimlioglu et al., 2026), the closest prior method and our main point of comparison. Paired results below are reported as CTS versus STC; pp denotes percentage points.

Concern and validationRaised byKey resultManuscript location
Joint-state prediction. Compare predictions from single-domain pilots with measured outcomes under mixed-domain retraining on OpenScene.6mdY, bfhq, r3FkState MAE: 0.41 pp. Gain MAE: 0.46 vs. 0.58 EPDMS points. Correct candidate preferences: 72/90 vs. 49/90.Section 5.5, Table 3(a); Appendix E.1, Table 18.
Pilot extrapolation. Predict held-out pilots at 2× and 4× the largest fitting count, and repeat allocation with a richer pilot set.bfhqLower Pilot Gain MAE in all three held-out settings; at 4×, 0.44 vs. 0.79 EPDMS points. Richer pilots improve CTS’s final EPDMS by only 0.05–0.08 points.Section 5.5, Table 3(b); Appendix E.2, Tables 19–20.
Within-domain ranking. Rebuild pilots and repeat allocation under entropy, k-center, and random orderings on both tasks.6mdY, bfhqCTS exceeds STC in all 30 comparisons, by 0.64–1.11 EPDMS points on navtrain and 1.28–4.91 pp on FMoW-WILDS.Section 5.5, Table 3(c); Appendix E.3, Table 21.
Heterogeneity and temporal redundancy. Measure sample characteristics along the ranking and test orderings with different temporal proximity.bfhqk-center lowers CTS’s budget-averaged temporal-proximity rate (R30R_{30}) from 52.6% to 42.9%, while CTS exceeds STC by 0.72–1.11 EPDMS points.Section 5.5, Table 3(d); Appendix E.4, Tables 22–23.
Test-split reuse. Repeat fitting, allocation, and retraining using feedback from held-out training logs, disjoint from the final test split.r3FkCTS exceeds STC at every budget by 0.72–1.09 EPDMS points on navtrain and 0.99–1.64 points on OpenScene; the CTS–STC gap changes by at most 0.08 points from the original protocol.Section 5.5, “Pilot feedback disjoint from the test split”; Appendix E.5, Tables 24–25.

All five analyses support the original conclusions. The method, the original experiments, and their reported results remain unchanged. We also address Reviewer bfhq’s comments on presentation and the methodological and theoretical contribution in our individual response.

Changes in the revised PDF

We have uploaded a revised PDF; all section, figure, and table numbers refer to it.

  • Section 5.5, “Validity and Robustness of Pilot-Based Valuation,” and Table 3 summarize the central findings in the main text.
  • Appendix E, “Details of the Validity and Robustness Analyses,” provides the protocols, metric definitions, full results, and interpretation. Its opening lists the five questions and identifies the reviewers who raised them.
  • Appendix F, “Limitations and Future Work,” updates the discussion of cross-domain response interactions and possible extensions.

The revised PDF uses four colors:

ColorMeaning
Dark redGeneral revisions
PurpleConcerns shared by several reviewers
BlueReviewer bfhq
GreenReviewer r3Fk

Both of 6mdY’s questions are shared concerns and appear in purple. Appendix E subsection titles also list reviewer IDs.

Self-contained presentation

We have made the added main-text discussion, each subsection of Appendix E, and the individual responses as self-contained as possible. Each presents the purpose, protocol, metrics, results, and interpretation needed to assess the point without repeatedly consulting earlier sections. Some repetition is intentional to facilitate review; we will consolidate it in the final version.

Thank you again for your time and constructive feedback. Your comments have helped us strengthen the validation and improve the presentation. We welcome further questions during the discussion.

AuthorsOpenReview ↗

General Response Update: Measured Cross-Domain Interaction

We thank all reviewers again for their time and feedback. We have uploaded a revised PDF with one additional experiment, motivated by the shared question on additive composition (6mdY Q2, bfhq Q4, r3Fk Q2).

Measured cross-domain interaction (Appendix E.6, Table 26). On navtrain, we trained two-domain pilots for all 6 city pairs and 3 seeds and compared each with the corresponding single-domain pilots. Training on two cities together retains about 86% of the summed single-domain response (overlap coefficient κ=0.14\kappa=0.14; doubling the data within one city gives 0.18). The interactions omitted by additive composition are therefore mostly sub-additive and largest for TTC and EC. They affect any additive valuation, including STC’s sum of scalar gains, and are consistent with the validation in Appendix E.1.

The main text, the original experiments, and all previously reported results are unchanged. The new subsection is marked in purple, and Appendices E.1 and F now refer to it. We have posted a short follow-up under each review with the details relevant to that reviewer’s question.

Thank you again. We welcome further discussion.

AuthorsOpenReview ↗

Thank You to All Reviewers

We thank all reviewers for their time and thoughtful feedback during the review and discussion. Your questions led to new validation experiments and a clearer presentation, and the paper has improved substantially as a result. We are grateful for your engagement.

AuthorsOpenReview ↗

Closing Summary of the Discussion

Dear Area Chair and Senior Area Chair,

Thank you for your time and for handling our submission. As the discussion concludes, we would like to summarize how the reviewers' concerns were addressed, where the discussion stands, and what the paper contributes.

How the concerns were addressed

The reviewers recognized the importance of the limitation we identify (6mdY, r3Fk), the simple and principled approach (r3Fk), the theoretical analysis (6mdY), and the evaluation across two tasks with matched ablations and controlled retraining (6mdY, bfhq, r3Fk). Their questions focused on three issues: whether single-domain pilots provide a reliable basis for valuation, whether the evaluation is independent of pilot feedback, and whether the contribution and its presentation are clear. We addressed them with new experiments, summarized in Section 5.5 and detailed in Appendix E, and with targeted revisions.

  1. Joint-state prediction (6mdY, bfhq, r3Fk). To test the additive joint-state approximation directly, we compared CTS's predictions with actual mixed-domain retraining. CTS predicts the metric state with a mean error of 0.41 percentage points and identifies the better candidate in 72 of 90 comparisons, versus 49 for STC (MOSAIC). We also directly measured cross-domain interactions using two-domain pilots. These analyses support the usefulness of the additive approximation under the tested mixtures and clarify its limitations (Appendices E.1 and E.6).
  2. Sparse pilots and within-domain ranking (6mdY, bfhq). We predicted held-out pilots beyond the fitting range and repeated allocation under three label-free orderings. CTS achieves lower held-out gain error than STC in all three extrapolation settings (0.44 vs. 0.79 EPDMS points at 4×) and outperforms STC in all 30 label-free comparisons across two tasks and five budgets.
  3. Independent pilot feedback (r3Fk). To separate pilot feedback from final evaluation, we reran fitting, allocation, and retraining with feedback from held-out training logs. CTS retains its advantage over STC at every budget on both navtrain and OpenScene.
  4. Methodological contribution (bfhq). We clarified that independent scalar-gain valuation cannot allow a candidate's value to change with allocations to other domains for any choice of fitted curves, and that our analysis characterizes when joint-state valuation changes the preferred candidate (Sections 3.2 and 4.3; Appendix C).

Every question thus received a direct response through additional experiments, analyses, clarifications, or manuscript revisions. The method and original results are unchanged, and the main text remains within the page limit.

Where the discussion stands

  • Reviewer r3Fk confirmed that our responses addressed their concerns and raised their rating from 2 to 3.
  • Reviewer 6mdY, whose initial rating was weak acceptance, has not yet replied. Both of their questions were shared with other reviewers: r3Fk confirmed that our joint-state analysis addressed the same concern, and bfhq, who also raised the ranking question, acknowledged the additional experiments.
  • Reviewer bfhq found that the additional experiments provide useful insights, and the only remaining concern raised in their reply was the readability of compound terminology. Although 6mdY and r3Fk rated the presentation as clear and easy to follow, we took this concern seriously: the revised paper rewrites the cited phrases and similar wording throughout, explains key terms at first use, and adds a terminology list in Appendix A, with the changes marked in blue.

Contribution

The paper shows that, when the task objective combines multiple metrics nonlinearly, the value of additional data must account for what other domains have already contributed to the task's metrics. Independent per-domain scalar gains cannot express this dependence, irrespective of how accurately the individual curves are fitted. We formalize this as joint-state valuation and implement it in CTS, which retains each domain's metric responses, composes them into a predicted joint metric state, and values every candidate at that state through the task utility. This makes the same single-domain pilot evidence useful as the joint allocation changes, without additional pilot training for allocation. Theoretical analysis, matched ablations, and actual retraining connect this capability to better allocation decisions and final performance across two tasks. Because composite and worst-case objectives are common in practice, as in our driving and satellite tasks, the principle extends beyond these benchmarks to data allocation under nonlinear multi-metric objectives. Collectively, these results establish CTS as a practical approach to multi-metric data allocation.

Thank you again for your careful consideration. We hope the evidence and revisions from this discussion support a favorable assessment of the paper.

Best regards, The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.