- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper proposes MM-DataLoop, a framework for iteratively improving multimodal post-training data through five operation families and an Observe → Diagnose → Plan → Execute → Verify → Reflect cycle. A global rule memory preserves experience from successful and unsuccessful experiments. Importantly, each candidate is trained from the same base checkpoint with a fresh LoRA adapter; the accumulated state consists of data recipes, experiment history, and rules, rather than progressively updated model weights. The paper reports a 31-round study on Qwen3.5-0.8B, with the best configuration increasing the five-benchmark average from 57.72% to 62.95%. It also documents recipe-dependent findings concerning filtering, synthetic supervision, and data composition. The contribution is primarily an integrated experimental workflow and an empirical case study of data-side iteration.
Critical strengths
- Restarting each round from the same base model helps distinguish data-recipe changes from cumulative weight updates. The explicit acceptance rules, integrity checks, and experiment records provide a useful structure for auditing the search.
- Removing different groups identified by the same groundedness filter produces opposite effects, illustrating why a filtering criterion should not be treated as a universally reliable measure of training utility.
- The paper acknowledges that many round-to-round comparisons are not single-variable causal tests and that repeated evaluation of a checkpoint does not measure training or data-generation variability.
Critical weaknesses
- My main concern is the distinction between finding a better data recipe and demonstrating that MM-DataLoop is a better way to find that recipe. The improvement over the zero-shot model supports the usefulness of the resulting training data, but does not isolate the benefit of diagnosis, operation composition, reflection, or global memory.
- The same benchmarks guide planning and evaluate the resulting candidates. Without an untouched final evaluation, the reported gains may partly reflect adaptation to the benchmarks driving the search.
- Same-checkpoint reevaluations do not capture training or generation variability. Moreover, changing dataset sizes under fixed epochs changes training exposure, making filtering effects difficult to isolate without matched-size and matched-compute controls.
Questions for authors
- What does the closed-loop framework contribute beyond ordinary data-recipe search?
- How is generalization evaluated independently of the benchmarks driving the loop?