acceptodds
Under review as a conference paper at ICLR 2027

Closing the Curation Loop: Semantic Compression and Learning-Guided Expansion for MLLMs Mid-Training

Abstract

Efficient mid-training of multimodal large language models (MLLMs) requires both reducing repetitive supervision and adapting training data to the model's changing learning needs. We propose a closed-loop data curation framework for image–text mid-training that combines coverage-preserving semantic compression with learning-guided expansion. First, we jointly assess image–text pair specificity and local semantic diversity in an MLLM-derived representation space, and use cluster-level budget allocation to construct an initial image–text training set with broad semantic coverage and reduced redundancy. We then evaluate the model's prediction loss across semantic regions using a fixed probe set, with reference-relative loss gaps serving as an empirical signal of learning potential to guide subsequent data acquisition. By alternating between model diagnosis, data acquisition, and continued mid-training, the framework adapts the training data mixture to the model's learning state while preserving semantic coverage. Under the same sample budget, our framework achieves better overall performance than existing data-selection methods across multiple widely used multimodal image–text understanding benchmarks. Ablation studies further support the complementary roles of semantic compression and learning-guided expansion.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
MLLMs, Adaptive Data Selection, Data-Efficient Learning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_1wpdOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a closed-loop data curation framework for MLLM mid-training, combining coverage-preserving semantic compression with learning-guided data expansion. The semantic compression stage constructs an initial training set by jointly considering image-text pair specificity, local semantic diversity, and cluster-level coverage. The subsequent expansion stage dynamically adjusts training data selection based on model feedback, using prediction loss and reference-relative loss gaps to identify semantic regions that may benefit from additional training. By iteratively performing model diagnosis, data acquisition, and continued mid-training, the framework aims to adapt the training data distribution to the model's evolving learning needs. Experiments demonstrate improvements over existing data selection methods under the same training sample budget.

Critical strengths

The paper addresses an important problem in efficient MLLM training: how to reduce redundant training data while adapting data selection to the model's evolving learning needs. The proposed framework combines semantic coverage with model-dependent feedback in a systematic manner. The overall methodology is clearly presented, and the integration of static semantic compression and dynamic data acquisition provides a reasonable approach to improving training data efficiency.

Critical weaknesses
  1. The proposed framework introduces considerable computational overhead beyond standard mid-training, including multimodal embedding extraction over millions of samples, semantic clustering and nearest-neighbor retrieval, training two LoRA-based reference models, and repeatedly evaluating a 500K-sample probe set during training. Despite this complexity, the overall performance improvement is relatively modest: the proposed closed-loop method achieves an average score of 64.7, compared with 64.0 for semantic compression alone and 63.2 for random sampling. Without accounting for the additional data curation costs, it remains unclear whether the proposed framework offers a meaningful advantage over simpler data selection strategies or allocating the same computational resources to additional training.

  2. The proposed closed-loop framework repeatedly evaluates the model on a fixed probe set and uses the resulting losses to guide subsequent data acquisition. Although the probe samples are excluded from direct training, repeatedly optimizing the training data distribution based on the same probe set may lead to adaptive overfitting to its particular semantic composition. Furthermore, the reference-relative loss gap is not sufficiently justified as an indicator of future learning potential. The reference models are adapted on different folds of the probe set, while the target model has already undergone mid-training on other samples from the same semantic clusters. Consequently, it is unclear whether the observed loss gaps genuinely reflect the marginal benefit of additional training data, rather than differences in model adaptation, training distributions, or reference-model capacity.

Questions for authors
  1. How do the authors prevent adaptive overfitting to the fixed probe set during closed-loop data selection? Although the probe samples and their near-duplicates are excluded from target-model training, the same probe set is repeatedly used to determine the semantic distribution of subsequently acquired training data. This creates a risk that the data selection strategy progressively adapts to the specific characteristics of the fixed probe set rather than the model's general learning needs.

  2. Why should the reference-relative loss gap be considered a reliable measure of future learning potential? The reference models are adapted using two-fold cross-fitting on the probe set, whereas the target model has already been trained on other samples belonging to the same semantic clusters. Therefore, the reference-relative loss gap does not necessarily represent the additional improvement achievable through further mid-training.

4 comments
AuthorsOpenReview ↗

Response to Reviewer 1wpd

We sincerely appreciate your positive comments for our work! We set out below our responses to each of the questions.

Q1: How do the authors prevent adaptive overfitting to the fixed probe set during closed-loop data selection?


Response: The probe-set design considers two risks: data leakage and potential overfitting to a fixed probe set. To address the reviewer’s concern about overfitting, we summarize the safeguards already described in the paper:

  • We sample evenly across semantic clusters to prevent frequent semantic regions from dominating the probe distribution.
  • The probe set guides selection proportions at the cluster level; samples within each cluster are still selected by semantic score. This reduces adaptation to individual probe samples.
  • During data expansion, the budget is split equally between diagnosis-guided and coverage-preserving selection. This allows the method to respond to learning needs while maintaining semantic coverage, limiting overfitting to the feedback signal.

The ablation in Figure 5 shows that combining diagnosis-guided selection (setting C) with coverage-preserving selection (setting B) yields better downstream performance than either strategy alone.

Q2: Why should the reference-relative loss gap be considered a reliable measure of future learning potential?


Response: Firstly, a high loss for the target model may reflect either samples that the model has not yet learned and could benefit from further training, or samples that are difficult to learn effectively under the current training conditions, such as noisy samples. Inspired by [1, 2], we introduce the reference-relative loss gap as an additional signal to help distinguish these cases. Two design choices support the reliability of this signal: the reference model is adapted from the base checkpoint and then kept fixed, reducing the influence of the mid-training data distribution on the baseline; and two-fold cross-fitting ensures that each probe’s reference loss comes from a model that was not trained directly on that probe, reducing direct-fitting bias. Figure 5 and Appendix Table 7 provide empirical evidence for the effectiveness of this design: using only the current loss (setting D) or only the reference gap (setting E) performs worse than combining both (setting F). Setting F outperforms D and E across General, Text-rich, Robustness, and MME Total.

Q3: Does the proposed framework still offer an advantage over using the same computational resources for additional training?


Response: Based on our computational cost measurements, our method requires 65.9 GPU·h on a single A800 GPU for data selection and model diagnosis (see iVKz-Q1 for the detailed cost breakdown). This is approximately equivalent to the GPU time required to train on 2M additional samples. To test whether using this compute for more training data yields comparable gains, we added a random-data baseline: starting from the 5M-sample Compression setting, we randomly add 2M samples to obtain a 7M-sample mid-training setting, while keeping the subsequent SFT setup unchanged. The results are shown below.

MME TotalGneral Avg.Robustness Avg.Text-rich Avg.Overall
5M1885.860.4271.8063.2063.96
5M + 2M1848.060.0371.0262.8463.48
Closed loop1821.560.5573.7564.0564.73

Adding 2M random samples reduces Overall performance from 63.96 to 63.48 relative to Compression alone, suggesting that the additional data may contain noisy or less useful samples. Under an approximately equal GPU budget, Closed loop also achieves better overall performance than the 7M setting. These results indicate that how additional compute is used matters more than simply increasing the training sample count in this setting. We thank the reviewer for suggesting this valuable comparison and will add the results and discussion to the experimental section.

We will continue to monitor the follow-up questions and discussions of the reviewer.

Reference

[1] Sören Mindermann et al. “Prioritized Training on Points That Are Learnable, Worth Learning, and Not Yet Learnt.” ICML, 2022.

[2] Sang Michael Xie et al. “DoReMi: Optimizing Data Mixtures Speeds Up Language Model Pretraining.” NeurIPS, 2023.

Dear Reviewer 1wpd,

We greatly appreciate your careful review and constructive criticism. In particular, your concerns about the fixed probe set and the reliability of the reference-relative loss gap prompted us to provide further analysis and controlled experiments. We hope these additions have helped clarify our approach, and we would be glad to hear if there are still any unresolved issues worth discussing. Should you consider our responses sufficient to address your main concerns, we would be grateful if you could consider raising your score.

Many thanks for your continued consideration of our work.

The authors.

After reviewing the authors' responses, I have decided to maintain my original rating.

Regarding Q1 and Q2, the responses largely reiterate the explanations already presented in the paper, without providing new evidence that directly addresses my concerns.

Regarding Q3, a meaningful compute-matched comparison should at least consider simpler data selection strategies rather than random sampling. Moreover, it is unreasonable to conclude that having additional computational resources would actually cause a method's results to deteriorate. Therefore, I do not find this additional experiment convincing evidence of a favorable compute–performance trade-off.

We thank the reviewer for the further feedback. Your comments have helped us identify which concerns in our initial response require more direct experimental evidence. We address each point below.

Q1: Does repeatedly using a fixed probe set lead to adaptive overfitting?


Response: The design mechanisms described in our initial response primarily aim to prevent direct data leakage and limit excessive concentration of data selection driven by probe feedback. To examine whether the improvements guided by such feedback generalize to new samples, we evaluated the same 3M, 4M, and 5M checkpoints on both the original probe set and an independent held-out probe set. We constructed the held-out probe set using samples from the same semantic clusters that were not used for diagnosis, reference model adaptation, or target model training.

Expansion roundOriginal probe mean ΔLossIndependent probe mean ΔLossRetained reduction
3M → 4M0.05480.053697.8%
4M → 5M0.02230.021998.2%

Across the two expansion rounds, the independent probes retained 97.8% and 98.2% of the mean loss reductions observed on the original probes, respectively. These results provide more direct evidence that the improvements observed on the fixed probes are not limited to the repeatedly used probe samples, but largely transfer to unseen samples. They also indicate that we did not observe substantial overfitting to the original probe samples in the two evaluated expansion rounds.

Q2: Can the reference-relative loss gap indicate subsequent learning gains?


Response: In our rebuttal to iVKz, we compared high/low gap clusters at similar initial mean prediction difficulty (UcU_c). Each group received the same number of additional samples, selected using the same within cluster semantic ranking rule. The mean loss reductions illustrate a 25.8% larger reduction in the high gap group. This comparison provides that the gap is associated with subsequent loss reduction. We therefore treat the gap as an empirical signal of learning potential.

Q3: Does the closed-loop method offer a favorable compute–performance trade-off compared with simpler data selection strategies?


Response: We acknowledge that the experiment involving additional randomly selected data does not isolate a single variable and is therefore insufficient to establish the compute–performance trade-off of the closed-loop method. In our response to reviewer iVKz, we also compared our method with Herding, a simpler selection baseline. Herding incurs an end-to-end cost of 611.6 GPU·h, approximately 1.6% higher than the 602.1 GPU·h by our closed-loop method. Their Overall scores are 63.33 and 64.73, respectively. Since the measured computational costs are comparable, these results provide a useful reference for evaluating the compute–performance trade-off.

Meanwhile, to more directly compare two different uses of the same computational budget, we are conducting an additional experiment starting from the mid-training checkpoint of the original 5M Compression setting. We continue training with additional data selected using the same selection algorithm and principles as the Semantic Compression stage described in the paper. We keep the SFT and evaluation settings unchanged and use the end-to-end GPU cost of the closed-loop method as the target computational budget. Due to time and computational resource limits, this experiment may not be completed before the rebuttal deadline. We will include the corresponding experimental results and conclusions in the paper's appendix.

We thank the reviewer again for pointing out the limitations of these comparisons. We appreciate the opportunity to clarify the scope of the existing results and provide additional evidence addressing each concern.

Reviewer_HKRNOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper propose a method to select midtraining data for MLLMs and a multi-round training that iteratively cover the reference gap of the current midtraining checkpoint. The method clusters image-text data using joint embedding produced by off-the-shelf models, and additionally leverage image-to-text and text-to-image embedding to calculate semantic values to rank the samples within the cluster. After the initial round of training, the checkpoint is compared against reference models to identify the reference gap. Then the next round of training adjusts the weights of each semantic cluster accordingly. Overall, this midtraining recipe demonstrates strong improvement on downstream benchmarks compared to baselines.

Critical strengths
  • The paper is well written and is easy to follow.
  • The method is well motivated and the motivation is clearly explained in section 2.1
  • The paper shows that the benefits of midtraining remain after the SFT stage, which is important for validation the effectiveness of the midtraining data selection.
  • The idea of dynamically adjusting the training data distribution is interesting. This is different from a static curriculum where the data mixture is defined in advance. Splitting into multiple training rounds mitigates the data loading infrastructure modifications.
Critical weaknesses
  • Why is the starting checkpoint for the reference model the base checkpoint? Why is the current midtrain checkpoint not used instead? Intuitively, it makes more sense to use the current midtrain checkpoint because it is the starting checkpoint of the next round of training. On a related topic, why are the reference model fix instead of co-evolving with the midtrain checkpoint?
  • Why is LoRA used for the reference model? The paper lack ablations comparing LoRA with full fine-tuning. For example, is it possible that some concepts are hard to learn by LoRA but can be learned by full fine-tuning?
  • Does the same image-text clustering method work for VQA, document layouts, and other forms of datasets? This is important as midtraining datasets are moving towards a larger and more diverse composition than just captions.
Questions for authors

N/A

4 comments
AuthorsOpenReview ↗

Response to Reviewer HKRN

Thanks a lot for your recognition of the contribution of our work and for finding the motivation behind it interesting. We set out our answers to each of your questions.

Q1: Why is the starting checkpoint for the reference model the base checkpoint? and Why is the current midtrain checkpoint not used instead?


Response: We would like to clarify the motivation for our reference-model design. While the target model captures the evolving learning state during training, the reference model provides a stable comparison baseline for identifying data that may still be valuable for further training. We aim to keep this baseline as independent as possible of the target model’s mid-training data mixture. Initializing the reference model from the 3M checkpoint is a reasonable alternative, and we have added a corresponding comparison experiment. Likewise, keeping the reference model fixed preserves a consistent baseline without making the feedback static: after each target-model update, cluster-level data allocations are recomputed to reflect its changing learning state. We have also compared this design with updating the reference model across rounds, as shown below:

MME TotalGneral Avg.Robustness Avg.Text-rich Avg.Overall
3M mid-training+ fixed1838.159.3972.8462.8763.62
3M mid-training+update1812.959.6271.2661.8463.09
Base + fixed(Ours)1821.560.5573.7564.0564.73

The results show that initialization from the base checkpoint with a fixed reference achieves the best aggregate performance. Initializing from the 3M checkpoint while keeping the reference fixed yields a slightly higher MME Total score (1838.1 vs. 1821.5). Overall, these findings support our original design choice. We will add these results and the discussion to the experimental section as a further ablation of the reference-model design.

Q2: Why is LoRA used for the reference model?


Response: Thank you for your question. The reference model is intended to provide an informative signal for subsequent data allocation. LoRA provides adaptation capacity through constrained parameter updates while reducing the computational cost of reference-model training and data selection. We agree that whether stronger adaptation yields a more effective reference-relative loss gap is an valuable question. To address it, we have added the following comparison between LoRA and full fine-tuning for reference-model adaptation:

MME TotalGneral Avg.Robustness Avg.Text-rich Avg.Overall
full1867.159.0071.8559.8162.42
LoRA(Ours)1821.560.5573.7564.0564.73

The results show that full fine-tuning achieves a higher MME score, while LoRA(Ours) retains a higher Overall average across the eight benchmarks (64.73 vs. 62.42). We analyze that, because each reference model is trained on a limited number of probe samples, updating all model parameters may lead to overfitting and make the reference loss a less stable basis for data selection. Given its lower cost(20% less training time and 50% less GPU memory),we consider LoRA the more suitable choice in this setting.

Q3: Does the same image-text clustering method work for VQA, document layouts, and other forms of datasets?


Response: This is a constructive question! We believe that our image–text clustering and semantic compression methods can be extended to other data formats, and the representations need to be adapted to each format. The key to clustering is obtaining multimodal vector representations. For VQA, images provide visual information, while questions and answers provide linguistic information. For documents and web pages with interleaved text and images, page images, OCR text, and layout elements such as headings can be incorporated. Semantic coverage and cluster-level budget allocation can then follow the same procedure, while learning diagnosis can use the supervised token loss appropriate to each task. This offers a feasible path for extension, but its effectiveness on heterogeneous data remains to be tested. Due to time constraints, we regret that we cannot provide results here and leave this evaluation for future work.

We hope that the above answers will solve your doubts and will continue to pay attention to your follow-up questions.

Dear Reviewer HKRN,

Thank you for your encouraging assessment and insightful suggestions. Your questions motivated us to conduct further experiments on reference model design, updating strategies, and adaptation methods, which have helped strengthen our work. We would welcome any additional questions you may have. If you find that our new results and clarifications have resolved your concerns, we would sincerely appreciate your reconsideration of the score.

Thank you again for your time and consideration.

The authors.

Thank you for the rebuttal. I will maintain my score.

Thank you for your response. We would be pleased to provide further clarification if you have any additional questions.

Reviewer_iVKzOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a closed-loop data curation framework for MLLM mid-training under a fixed sample budget. It constructs an initial subset by ranking samples using a combination of image–text pair specificity and local semantic diversity, with per-cluster selection budgets determined by cluster size. In subsequent rounds, half of the additional data budget is allocated to clusters with high average probe losses, with relative allocations determined by reference-model loss gaps, while the remainder preserves semantic coverage. Experiments report improved average performance over the evaluated data-selection baselines.

Critical strengths
  • The paper considers both redundancy in the initial dataset and changing data needs during training. Combining these considerations is a reasonable direction for improving mid-training data selection.

  • The ablations provide some evidence that coverage preservation and model-guided selection are complementary, lending empirical support to their combination.

Critical weaknesses
  • The paper does not sufficiently justify the LoRA-adapted reference or establish that its loss gap predicts subsequent learning gains. Cluster budgets use model-dependent signals, whereas individual samples are selected using static semantic scores, without showing that higher-scoring samples address the diagnosed gaps. This leaves the central selection mechanism insufficiently justified.

  • The average gain over the strongest baseline is not accompanied by compute accounting for embedding extraction, neighbor retrieval, reference adaptation, and repeated diagnosis. Comparisons with simpler coreset methods, such as herding, are also missing. Consequently, it remains unclear whether the framework’s gains justify its added complexity.

Questions for authors
  • The framework requires embedding extraction, neighbor retrieval, LoRA reference adaptation, and repeated probe evaluation. Comparing its preprocessing and selection costs with those of other methods, in terms of runtime and GPU hours, would clarify the computational overhead not captured by equal sample budgets.

  • The paper does not sufficiently justify using a LoRA-adapted model as the reference for estimating learning potential. Comparisons with simpler references, such as the pre-mid-training checkpoint, would help validate this choice. Furthermore, it remains unclear whether clusters with larger reference-relative loss gaps benefit more from additional training data.

  • The loss-based diagnosis identifies clusters that may benefit from additional training, but it remains unclear whether the semantic score identifies the samples most useful for further training. Comparisons with random selection and selection based on per-sample loss differences between the current and reference models would help justify the use of the semantic score.

  • The paper lacks comparisons with standard coreset selection methods, such as herding [1], which would help assess whether the proposed semantic compression offers an advantage over simpler alternatives.

[1] Max Welling. Herding dynamical weights to learn. ICML, 2009.

6 comments
AuthorsOpenReview ↗

Response to Reviewer iVKz

Thank you to the reviewers for suggestions that help improve the quality of our work,we make the following responses:

Q1: The framework requires embedding extraction, neighbor retrieval, LoRA reference adaptation, and repeated probe evaluation.


Response: We have added a breakdown of preprocessing and data-selection costs for all methods, as shown in the table below. Our method and several embedding-based baselines use the same embedding scheme. All GPU costs are reported in the same unit(GPU hours on a single A800 GPU), while CPU costs are reported separately in core hours.

The results show that our method has relatively low CPU overhead. Its GPU cost comes mainly from the shared embedding extraction and the reference adaptation and probe evaluation added by the closed-loop stage. Compared with one-shot selection methods, closed-loop feedback incurs additional compute, but its total GPU cost remains lower than that of baselines requiring additional NLL computation(e.g., COINCIDE, D2-Pruning, and DataTailor). Considering downstream performance, our method is not the least expensive, but offers a reasonable balance between computational cost and performance gains. We agree that computational cost is an important dimension of data-selection evaluation that the original manuscript did not sufficiently address. We will add the table to Appendix C and summarize its implications in the main text.

MethodCPU Cost (Core·h)GPU Cost (GPU·h)
PRISM0.21Embedding: 122.2
SemDeDup0.22Embedding: 122.2
COINCIDE0.08Embedding (122.2) + NLL computation (130.0) = 252.2
D2-Pruning0.08Embedding (122.2) + NLL computation (130.0) = 252.2
DataTailor0.08Embedding (122.2) + NLL computation (130.0) = 252.2
Length1.130
CLIP-Score0MetaCLIP scoring: 40.6
Perplexity0.08NLL computation: 130.0
Compression (Ours)0.005Embedding (122.2) + K-means/kNN (0.9) = 123.1
Closed-loop(Ours)0.037Embedding (122.2) + K-means/kNN (0.9) + LoRA adaptation (25.6) + reference-loss computation (16.8) + probe-loss evaluation (22.6) = 188.1

Q2: The question of whether a LoRA-adapted reference model is appropriate and whether larger reference-relative loss gaps indicate greater gains from additional training.


Response: We would like to clarify that the reference model is designed to assess whether further training on a sample may be beneficial, whereas the pre-mid-training checkpoint primarily reflects the model’s initial learning state on that sample. We also use LoRA to construct the reference model at relatively low computational cost(discussion of comparison with full fine-tuning in Reviewer HKRN –Q2). To examine this design choice, we added a control experiment that directly uses the unadapted base checkpoint as the reference. The results show that this alternative yields lower overall performance than our current design.

Reference settingMME TotalGeneral Avg.Robustness Avg.Text-rich Avg.Overall
Base checkpoint1820.459.4271.5159.8362.54
Base initialization + LoRA adaptation (Ours)1821.560.5573.7564.0564.73

Regarding whether larger reference-relative loss gaps guide data acquisition toward greater learning gains, we measured the changes in loss across semantic clusters before and after training on the newly acquired data, as shown below.

Cluster groupNumber of clustersMean gapMean ΔLoss
Top 10%1,4870.75170.04279
Bottom 10%1,4870.3370.03065

The results show that additional data acquired under the guidance of larger reference-relative loss gaps are associated with greater mean loss reductions in the corresponding clusters, supporting the reference-relative loss gap as an empirical signal of learning potential.

Note: Due to the character limit, we will address the remaining questions in the following response.

Q3: The question of whether semantic scores identify the most useful samples within diagnosed clusters, compared with random or per-sample loss-gap selection.


Response: To evaluate the effectiveness of the semantic score during closed-loop expansion, we conducted an additional ablation on the within cluster selection criterion. Keeping the loss-based diagnosis procedure and cluster-level sampling proportions fixed, we compared (i) random sampling, (ii) selection based solely on per-sample loss gaps, and (iii) selection using our semantic score.

Selection criterionMME TotalGeneral Avg.Robustness Avg.Text-rich Avg.Overall
Random sampling1802.859.6770.8862.963.28
Per-sample loss gap1794.759.5371.2560.9462.81
Semantic score (ours)1821.560.5573.7564.0564.73

Selection using the semantic score achieves the highest Overall score, outperforming random sampling and per-sample loss-gap selection by 1.45 and 1.92 points, respectively. These results support the effectiveness of our semantic score for within-cluster sample selection during subsequent training.

Q4: The paper lacks comparisons with standard coreset selection methods, such as herding[1]


Response: We thank the reviewer for suggesting the additional comparisons ! We used herding to select 5M samples from the 10M candidate pool for one single compression step, keeping the embedding extraction procedure and subsequent MT/SFT settings unchanged. The results are shown below.

MethodMME TotalGeneral Avg.Robustness Avg.Text-rich Avg.Overall
Herding1818.859.5871.2762.8963.33
Compression (ours)1885.860.4571.8163.1863.96
Closed-loop (ours)1821.560.5573.7564.0564.73

The Overall results show that both our semantic compression method and the closed-loop framework outperform this baseline. We also measured the computational costs of data selection and model training. The selection cost of semantic compression alone is 0.9 GPU·h, a 98.6% reduction compared with herding’s 65.8 GPU·h. The selection and diagnosis cost of the closed-loop framework is 65.9 GPU·h, comparable to herding. We will add the method description, results, and discussion to the experimental section of the revised manuscript, together with the corresponding reference.

MethodEmbedding (GPU·h)MT (GPU·h)SFT (GPU·h)Selection (GPU·h)Total (GPU·h)
Herding122.2204.6219.065.8611.6
Compression(ours)122.2186.7219.00.9528.8
Closed-loop (ours)122.2195.0219.065.9602.1

We follow your valuable advice, and add more experimental results. We sincerely hope that you could reconsider the overall performance and contribution of our articles. We continue to pay attention to any follow-up questions and actively discuss them.

Reference

[1] Max Welling. Herding dynamical weights to learn. ICML, 2009.

Thank you for the detailed response and the additional experiments. However, two main concerns remain. First, the top/bottom-gap analysis does not validate the signal. High-gap clusters receive more data by design, so a larger loss reduction there is expected regardless of whether the gap is informative. A fair test would give both groups the same amount of data. Second, the random sampling result matches ablation A on every metric, even though ablation A is fully random acquisition without diagnosis or cluster proportions, whereas the response describes random sampling within clusters with the diagnosis and cluster proportions held fixed. Please clarify whether this experiment was actually conducted under the described setting. Finally, please clarify whether herding was run with the same cluster-level budget allocation. If not, herding should be given the same cluster-level allocation so that only the selection criterion differs.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Reviewers' Feedback and Authors' Response

We sincerely thank all reviewers for their constructive feedback and thoughtful engagement with our work. Their comments have been invaluable in helping us strengthen our manuscript through additional experiments, clearer methodological explanations, and more comprehensive evaluations. For clarity, we refer to the reviewers as R1 (iVKz), R2 (HKRN), and R3 (1wpd) throughout this summary.

Strengths: We appreciate the reviewers' recognition of the importance of addressing both data redundancy and evolving learning needs in MLLM mid-training (R1, R3). R1 acknowledged the motivation for combining these considerations and the complementary roles of semantic coverage and model-guided selection. R2 highlighted the clear motivation of our method, the interesting idea of dynamically adjusting training data distributions beyond static curricula, and the persistence of mid-training improvements after SFT. R3 recognized the importance of the problem, the systematic integration of semantic compression and model-dependent feedback, and the overall methodological clarity.

In response to the reviewers' concerns, we have substantially strengthened our experimental validation. We introduced controlled experiments to investigate the relationship between reference-relative loss gaps and subsequent learning gains (R1, R3) and whether semantic scoring provides an advantage over alternative within-cluster selection strategies (R1). We further investigated different reference-model designs (R1, R2), added comparisons with the Herding baseline (R1), and provided detailed computational cost accounting and comparisons against additional training under an approximately matched compute budget (R1, R3). We also clarified the safeguards against adaptive overfitting to the fixed probe set (R3) and discussed the framework's applicability to more diverse multimodal data (R2).

In the revised manuscript, we have added the following experiments and analyses to address the reviewers' concerns:

  • Section 3.2&3.4: Additional baseline comparisons and within cluster selection ablations.
  • Appendix C.4: Reference Model Design Ablations.
  • Appendix C.5: Validation of Reference-Relative Loss Gaps.
  • Appendix C.6–C.7: Computational Cost Analysis and Additional Training Comparisons.

Through these revisions, we have provided stronger empirical evidence for our key design choices and a more transparent assessment of the framework's computational trade-offs. We believe the revised paper now presents a more rigorous and comprehensive study of adaptive data curation for efficient MLLM mid-training.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.