acceptodds
Under review as a conference paper at ICLR 2027

STELLA-Flow: Dynamic Data Recipe Valuation for Data-Efficient Model Iteration

Abstract

As large language models mature, development is shifting from a model-centric to a data-centric paradigm. Model capabilities depend not only on data scale, but also on data quality, diversity, and composition. However, existing studies largely focus on individual-sample valuation or aggregate metric optimization, leaving the dynamic value of complete data recipes under evolving model feedback insufficiently characterized. We propose STELLA-Flow, a dynamic framework that evaluates candidate data through intrinsic quality, conditional model gain, and business value. It first assesses data quality, then uses a small number of real model updates to predict which existing errors candidate data recipes would correct and which new errors they would introduce, and finally incorporates scenario importance and error costs to select data for the current model. Across model scales, STELLA-Flow outperforms random selection and static quality filtering. Under a unified advertising content moderation benchmark, a 2B model updated with STELLA-Flow-selected data outperforms a 32B model updated with randomly selected data, GLM-5.3, and DeepSeek-V4-Pro. These results demonstrate that STELLA-Flow provides a unified method for dynamic data valuation and selection across model scales, enabling limited training budgets to continuously shift toward higher-utility, lower-risk data recipes as model capabilities evolve. This improves the data efficiency of iterative model development and offers a new path for smaller models to achieve competitive business performance at lower cost.

Primary area:
foundation or frontier models, including LLMs
Keywords:
training data valuation, data recipe selection, data-centric AI, model behavior prediction, dynamic data selection, iterative model improvement, multimodal content moderation

Reviews

Among:
Shape:
1234this paper, Bottom 25%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.75 over 4 reviews, above 25% of the others.
Reviewer_EyYkOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

STELLA-Flow combines intrinsic data-quality admission, model-conditional response prediction, and business-aware utility to select training-data recipes. Its response predictor uses low-rank decomposition and regression to predict case-level margin changes from recipe composition. Predicted repairs and regressions are then evaluated against a no-update reference through EXECUTE, PROBE, and REJECT decisions. Experiments concern advertising content moderation across several model sizes, with a separate 44-recipe response-prediction study.

Critical strengths

The framework evaluates harmful regressions alongside repaired errors and explicitly includes a no-update option. IMO, this addresses a consequential limitation of ranking-only selection: the best available recipe can still worsen the current model.

Critical weaknesses
  • The main performance comparisons use the evaluation set for selection and reporting, preventing an unbiased assessment of generalization. Section 5.2 and Appendix A.4 explicitly state that Table 2 reports each method’s run with the highest realized business value on the same 1,000-item evaluation set used to report performance. Tables 3 and 5 likewise compare selected checkpoints without establishing a separate final holdout. Keeping evaluation images out of training does not resolve this selection bias: their outcomes still influence which result is reported. No uncertainty estimates establish whether the reported differences are reliable, particularly the small advantage over DeepSeek-V4-Pro in Table 5. This directly undermines the headline claims of superiority over alternative selection methods and larger models.

  • The experiments do not establish the effectiveness or total cost of the proposed dynamic update policy. The 44 recipes are trained independently from one checkpoint, rather than through a prospective sequence of model replacements and recalibrations. All realized utilities are negative, so a predictor that always predicts negative utility would also achieve 100% sign accuracy. This study therefore cannot establish whether the policy recognizes beneficial updates or successfully balances execution, probing, and rejection. The utility-error radius of 378, compared with the best realized utility of −27, further makes gate-level calibration important, yet empirical interval coverage and authorization error rates are not reported. Finally, the selected-run training costs do not provide an end-to-end accounting of the real updates required to learn the predictor, additional probes, and recalibration after model replacement. These omissions leave the central claims of dynamic improvement and data-efficient iteration unsubstantiated.

Questions for authors
  • Can the authors freeze recipe and checkpoint selection using a validation set, then report all methods on an untouched test set with uncertainty estimates and matched selection budgets?

  • Can the authors evaluate prospective update sequences containing both beneficial and harmful recipes, comparing STELLA-Flow with no-update, ranking-only, and static-selection policies?

  • What are the empirical interval coverage, harmful EXECUTE rate, and beneficial-update rejection rate? Does the structured response predictor outperform direct utility regression and constant-response baselines?

  • What is the total compute cost, including initial response-generating updates, probes, predictor fitting, and rebuilding the response model after execution? At matched total budgets, does STELLA-Flow still outperform the baselines? These are important evaluation dimensions.

Reviewer_UVpTOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

In this paper, it proposes STELLA-Flow to solve dynamic data selection problem in model training. Existing method only check static data quality, but this paper think the data recipe value should depend on current model state. For solving this, the framework use three steps: ADMIT, PREDICT and AUTHORIZE. Experiment result show that 2B model updated with this method can beat 32B model on advertising task. This prove that the framework is very effective and data-efficient.

Critical strengths
  1. Most of existing method only evaluate data base on static quality or single sample score. But this paper point out that data value is depend on current model state. The three-layer framework (admit, predict, authorize) provide a very logic and useful pipeline for continuous model update.
  2. Experiment result show that 2B model update with some selected example can beat 32B model and other big baseline. This prove data composition sometimes is more important than parameter size, and I think this is very useful find for our community
Critical weaknesses
  1. This paper claim it achieve data-efficient model iteration. But if we look Section 4.2, to build case-level response matrix, author need independently train some recipe from current checkpoint. In 44-recipe study, this mean they actually train 44 model just for fit predictor. Train dozens of LLM just to build data-selection predictor is very expensive. Paper do not report this huge meta-training cost. So I think the efficient claim is very misleading.
  2. Core math formula of STELLA-Flow heavy rely on binary label to calculate false positive and false negative. This work very perfect for their advertising content moderation task. But modern LLM mainly focus generative task (like math reasoning, coding, and etc.). We can not easy calculate simple decision margin for free-form text generation. Paper do not give solution how to apply this framework to general generative task, so making this method very limit.
  3. For prove SOTA performance, paper use in-house advertising content moderation benchmark. It compare 2B model with big model like GLM-5.3 and DeepSeek-V4-Pro by use their own business value (BV) metric. Because dataset and BV metric is not public, so it is impossible for other researcher reproduce this result. ICLR paper should evaluate method on standard public dataset to prove general effective.
Questions for authors
  1. How much compute (in GPU hours) did you actually spend to train the n recipes to build the STELLA predictor
  2. Your method heavily relies on binary classification. How can we apply STELLA-Flow to generative tasks?
  3. can you provide experiment results on standard public LLM benchmarks
Reviewer_ySV4OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies data selection for model updates, focusing on how a training recipe repairs existing errors and introduces new ones. The authors propose STELLA-Flow, which combines data-quality checks, model-specific response prediction, and business-weighted utility. Using independent fine-tuning runs from the same checkpoint, STELLA learns a low-rank model that maps a recipe’s scenario proportions to prediction changes on fixed evaluation cases. These changes are converted into utility under scenario weights and error costs. The framework then uses predicted utility and uncertainty to decide whether to adopt an update, train a model copy to gather more evidence, or reject the recipe. Experiments on advertising content moderation report higher business value than random and manual selection across several Qwen and InternVL models. Component ablations show that optimizing business value can come at the expense of F1. A separate study of 44 recipes evaluates response prediction and finds that every update increases errors relative to the starting checkpoint, illustrating why the best candidate may still be worse than retaining the current model. The proposed feedback loop is not evaluated over a sustained sequence of model replacements.

Critical strengths
  1. Evaluating data through both error corrections and regressions is well motivated. An update may improve overall performance while introducing errors that are more costly for the application. Predicting these behavioral changes before applying business weights provides a useful way to assess data under different application priorities.
  2. Separating recipe ranking from the decision to update is practically useful. The best available recipe may still be worse than keeping the current model. Including an explicit no-update option and testing uncertain recipes before adoption is a sensible approach to model iteration, where preserving existing capabilities also matters.
Critical weaknesses
  1. The methodological advance over existing data-to-performance models is insufficiently established. RegMix [1] and Data Mixing Laws [2] already predict performance from data mixtures, while Datamodels [3] predicts outputs on individual evaluation examples. STELLA adds low-rank response modeling and an update gate, but the current comparisons do not establish their benefit over relevant approaches or simpler predictors trained on the same recipe outcomes.
  2. The main performance comparisons use the evaluation set for run selection. Table 2 reports each method’s highest-BV run on the same 1,000-item set used to report performance. Although this is disclosed, the results do not establish whether the gains persist on an independent test set. This also limits the conclusions drawn from comparing the selected small model with larger reference models.
  3. The claimed dynamic benefit is not demonstrated across changing model states. The reported studies evaluate recipes trained independently from starting checkpoints. They do not show whether additional probes improve selection or whether the method continues to identify useful recipes after the model changes. The proposed feedback loop therefore remains insufficiently validated as a method for iterative model development.
  4. The data-efficiency claim lacks complete cost accounting. STELLA requires real fine-tuning runs to fit and update its predictor, but it is unclear whether the reported costs include all pilot and probe runs. Its selected 2B run is also more expensive than Human Strategy, MIWV, and NAIT. Without accounting for the full search process, the practical advantage over simpler selection methods or direct recipe search remains unclear.
Questions for authors
  1. Does the low-rank response model improve selection over simpler predictors? Using the same observed recipe updates, could you compare STELLA with directly predicting utility from scenario proportions, and with predicting each case’s response without the low-rank step? Reporting prediction errors and the actual utility of the selected recipes would clarify what the proposed structure adds beyond simpler data-to-performance models.
  2. Do the reported gains hold on an independent test set? Table 2 selects each method’s best run using the same set on which performance is reported. Could you evaluate them on a separate set that was not used to fit the predictor or select recipes and checkpoints? A comparison on one backbone, including the starting checkpoint and a competitive selection baseline, would help establish whether the gains generalize.
  3. Does updating the predictor help when the model changes? On one small model, could you run two or three successive update rounds and compare recalibrating the predictor after each model change with keeping the initial predictor fixed? Reporting model performance and the number of additional trial runs per round would show whether the feedback loop improves later recipe choices.
  4. Does STELLA retain an advantage when the full selection cost is included? Could you report total GPU time for the initial recipe trials, additional probes, predictor fitting, selection-related evaluation, and final update? A comparison with one cheaper baseline under the same total budget would help establish whether the method offers a practical efficiency advantage.
Reviewer_qaFDOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper presents STELLA-Flow, a data selection method for iterative model training. Instead of scoring individual data samples or tuning mixture ratios against validation loss, the method evaluates candidate data recipes by predicting their impact on model behavior. The pipeline has three stages. First, an admission filter removes corrupted, duplicated, or overlapping data. Second, a response predictor estimates how a candidate recipe would shift decision margins across individual evaluation cases. This is done by running a small batch of actual updates, applying singular value decomposition to extract common response modes, and fitting a regression model from recipe composition to mode weights. Third, an authorization step maps predicted margin changes to business costs and errors, placing recipes into execution, probing, or rejection based on an empirical uncertainty interval. The authors test this on an advertising content moderation task using compact vision-language models and analyze prediction accuracy on a set of 44 recipe updates.

Critical strengths

Treating update authorization separately from candidate ranking is a practical idea. In production maintenance, candidate data pools are often imperfect, and even the top-ranked recipe can degrade existing model capabilities. Explicitly comparing against the baseline of leaving the model untouched gives the paper a realistic setup. The paper clearly shows how standard evaluation metrics can diverge from business priorities. Training on quality data selected purely for model gain pushes up the overall F1 score while reducing weighted business performance, whereas factoring in scenario weights achieves the intended business goal.

Critical weaknesses

The evaluation methodology has severe issues with test set selection and inconsistent reporting. The authors state that checkpoints in Table 2 and Table 3 were chosen based on peak business value on the 1,000-item evaluation set. Selecting winning runs on the test set without a held-out validation split introduces substantial selection bias. More troubling is an uneven comparison across tables: the random baseline for the 0.8B model in Table 3 matches the five-run mean from Table 1, whereas the proposed method is reported using its single best checkpoint. Comparing a maximum against an average inflates the apparent improvement. It is also unclear whether the evaluation items used to fit the response predictor overlap with the test set. The core claim of dynamic iterative selection is unsubstantiated. Although the title and diagrams emphasize dynamic feedback loops and sequential updates, the empirical evaluation contains zero multi-step experiments. Every experiment starts from the initial checkpoint and runs a single update. Furthermore, the 44-recipe study used to validate response prediction is degenerate. The authors highlight perfect sign accuracy, but every single recipe in that set resulted in negative utility. A trivial model predicting negative gains would also be 100% accurate. With an uncertainty radius over ten times larger than the best observed outcome, the decision gate cannot reliably distinguish good updates from bad ones.

Questions for authors
  1. Did the case-level evaluation set used to train the response predictor overlap with the 1,000-item test set used to evaluate final performance? How many trial recipes were trained to fit the predictor, and was that compute included in the reported cost?
  2. Why does Table 3 compare the best single checkpoint of the proposed method against the five-run average of the random baseline? Can you provide mean and standard deviation across all runs on a held-out test set?
  3. What are MIWV and NAIT, and how do they select data recipes?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.