- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes CoFlow, a gradient-flow-compatible data selection framework designed to improve the stability and efficiency of LLM supervised fine-tuning. The authors argue that existing data selection methods primarily evaluate examples independently, overlooking potential gradient conflicts when selected examples are jointly optimized. CoFlow consists of three stages: (1) Flow Probing, which extracts compact gradient signatures from candidate examples; (2) Cone-Compatible Selection, which groups examples according to gradient compatibility and selects representative samples while preserving diversity; and (3) Smooth Flow Scheduling, which organizes minibatches to reduce abrupt changes in gradient directions during training. Experiments across mathematical reasoning, general instruction following, medical QA, and code generation demonstrate promising results. CoFlow improves average downstream performance by 10.6% relative to the strongest reported selection baseline while using only 4.1% of the candidate data.
Critical strengths
-
Clear motivation: The paper identifies an important limitation of conventional data selection methods, which often focus on individual sample importance while overlooking interactions among selected examples during optimization.
-
Practical framework: CoFlow combines gradient-compatible selection with minibatch scheduling, considering not only which examples to select but also how to organize them during training. The framework is intuitive and potentially useful for improving training stability.
-
Promising empirical results: The method demonstrates strong performance across four instruction-tuning domains and several representative baselines. The ablation studies, gradient conflict analyses, and computational cost comparisons provide useful evidence of its effectiveness.
Critical weaknesses
-
W1. Limited novelty and insufficient comparison with related work: The central motivation of CoFlow, namely that independently selected examples may exhibit gradient conflicts or redundancy when optimized together, has been explored in prior gradient-based data selection studies, such as G2IS (ACL 2025), TAGCOS (NAACL 2025), FisherSFT(ICML 2025) and SPICE (ICLR 2026). However, these closely related works are neither cited nor included in the experimental comparisons. A more comprehensive discussion and comparison with these related methods would strengthen the paper.
- Nevertheless, Smooth Flow Scheduling is an interesting component of the framework, and investigating whether it can consistently improve subsets selected by other methods would help demonstrate its independent contribution and generalizability.
-
W2. Heuristic assumptions about gradient compatibility: CoFlow assumes that selecting gradient-compatible examples and minimizing minibatch direction changes can improve optimization stability and downstream performance. However, lower gradient conflict does not necessarily imply higher information value or better generalization, as conflicting examples may still provide complementary training signals. Although CoFlow preserves different gradient directions through multiple cones, it remains unclear whether its compatibility-based selection may exclude informative but conflicting examples. Furthermore, the method relies on gradient signatures computed during the selection stage, whose compatibility relationships may change as model parameters evolve during training.
-
W3. Experimental evaluation and generalizability: Although the paper provides evaluations across multiple domains, a 14B scale check, and computational cost analyses, the experimental validation remains insufficient in several important aspects, as detailed in Questions Q3–Q6.
Questions for authors
-
Q1. Could the authors further clarify the technical differences between CoFlow and existing gradient-based data selection methods that consider sample interactions, such as G2IS, TAGCOS, and SPICE? Additional comparisons under matched data budgets would help better establish the novelty and effectiveness of the proposed approach.
-
Q2. Since gradient-conflicting samples may still contain valuable information, could the authors investigate whether CoFlow excludes informative but conflicting examples? Additionally, how stable are the initially computed gradient compatibility relationships throughout the fine-tuning process?
-
Q3. Although Appendix E.4 provides budget sensitivity analyses for CoFlow, could the authors extend the evaluation to matched data selection budgets (e.g., 1%, 4.1%, 10%, and 20%) across different methods and downstream tasks? Such comparisons would help establish the data efficiency advantages of CoFlow more convincingly.
-
Q4. Could the authors investigate whether Smooth Flow Scheduling can generalize to subsets selected by other methods? Such experiments would help isolate its independent contribution and assess its broader applicability.
-
Q5. Could the authors provide further cross-model transfer experiments, such as using gradient signatures obtained from one model family to select data for another? Additionally, how sensitive are the selected subsets and downstream performance to different projection seeds, candidate pool samplings, and cone construction orders?
-
Q6. Given the additional computational overhead of gradient probing, could the authors further analyze the end-to-end efficiency of CoFlow on larger models and candidate pools? In particular, under what conditions do the computational savings from reduced fine-tuning outweigh the additional data selection costs?
If the authors can address my concerns and questions, I would be willing to raise my score.
9 comments
Response to Reviewer ph2w (2/7)
Thanks for your questions. Thanks also for highlighting the four-domain results and the evidence from ablations and conflict diagnostics.
Although Appendix E.4 provides budget sensitivity analyses for CoFlow, could the authors extend the evaluation to matched data selection budgets (e.g., 1%, 4.1%, 10%, and 20%) across different methods and downstream tasks?
R3. At the same 10% budget the example four-domain means are 36.58% for CoFlow and 34.85% for SPICE. LESS is at 32.78%. We use the same pool and fine-tuning setup within each domain. We also compare the same methods at 1% and 4.1% and 20%. The paper will show these rows alongside the results at each method’s usual budget.
Could the authors investigate whether Smooth Flow Scheduling can generalize to subsets selected by other methods?
R4. The LESS subset stays fixed in this example. The score is 31.71% in the original order and 32.46% with CoFlow scheduling. Rotation drops from 0.53 to 0.40. We also keep the CoFlow batches fixed and shuffle their order. That gives 32.02% and rotation of 0.49. The last comparison tests the order by itself.
Response to Reviewer ph2w (3/7)
Thanks for your questions. Thanks also for highlighting the practical combination of gradient-compatible selection and minibatch scheduling.
Could the authors provide further cross-model transfer experiments, such as using gradient signatures obtained from one model family to select data for another? Additionally, how sensitive are the selected subsets and downstream performance to different projection seeds, candidate pool samplings, and cone construction orders?
In particular, under what conditions do the computational savings from reduced fine-tuning outweigh the additional data selection costs?
R5. We keep the LLaMA3-8B target fixed and change the selector. At the 4.1% budget the example math scores are 30.72% from the Qwen2.5-7B probe and 31.10% from the target’s own probe. Random selection gives 27.85%. The Qwen 14B result in the paper tests scale within the same family. Selection pays off once it saves more training time than preparation and probing cost.
The setup, probe, and selection costs pay off once they are smaller than the training time saved. For runs sharing the candidate pool and probe checkpoint, recover preparation and probing over those runs. One-factor seed sweeps distinguish variability in projection, pool sampling, cone insertion, and final training. We will name the source and target checkpoints and state the amortization assumptions with the timing results.
Response to Reviewer ph2w (1/7)
Thanks for your careful review and the specific questions. Thanks also for highlighting the practical combination of gradient-compatible selection and minibatch scheduling.
R1. CoFlow picks a set of examples and decides the order of the training batches. The examples cover more than one gradient direction. With influence and diversity held fixed the score rises from 31.9 to 35.2 as coherence increases. On the same selected examples scheduling raises the average from 34.82 to 36.25.
CoFlow also controls successive batch directions through Rot(Bt,Bt+1)=1−μt⊤μt+1, where μt is the normalized sum of batch signatures. We will compare the named selectors at the same selection budget with a common candidate pool and fine-tuning protocol.
Revision. We will state the subset and ordered schedule together at the start of the method, and compare the named selectors in related work.
R2. An example can disagree with one cone and still fit another. We keep multiple cones so useful directions can stay in the selected set. We also check the same example pairs later in training to see how long their initial agreement lasts.
Keeping useful directions. We keep more than one cone and allocate more examples to larger cones. An example outside one cone can fit another or start a new one. Max-min selection keeps some spread within each cone. This gives us several directions across the dataset and examples with nearby directions in a batch.
Each subset contains 410 examples from the same influence-score band. We match average influence and semantic diversity and use the same training setup. The low-coherence subset scores 31.9. Medium scores 33.6 and high scores 35.2. We will count selected examples by influence band and cone size to see how many useful examples from small cones survive.
Does the probe still agree later? The probe averages gradients at three checkpoints before projection and normalization. Training updates the whole model. We will use the same pairs at later checkpoints to compare the early probe with full-model gradients. We will show the agreement by domain and stage.
Report inclusion rate by individual influence band and cone size, using the pre-selection pool for the denominator. Comparing retained and excluded examples in the same band reveals whether a small but useful mode loses coverage. For checkpoint drift, compare cosine signs for identical pairs against full-model sketches and record batch-direction agreement separately.