acceptodds
Under review as a conference paper at ICLR 2027

CoFlow: Stabilizing LLM Instruction Tuning via Gradient Flow Compatible Data Selection

Abstract

Supervised fine-tuning is fundamental for adapting large language models (LLMs), yet training on comprehensive instruction datasets incurs significant costs and diminishing returns. Current data selection methods typically rank examples independently by quality, diversity, difficulty, or influence, overlooking that selected examples are optimized collectively through minibatches. We find that individually valuable examples can still conflict when optimized together, as their gradients may point in incompatible directions. Consistent examples form compact gradient clusters with high consistency scores, whereas conflicting examples scatter across directions and often receive low or negative scores. This indicates that efficient instruction tuning requires selected examples to be both individually useful and jointly compatible in gradient space. We introduce \ourmethod, a training-aware data selection framework for constructing stable instruction tuning flows. \ourmethod operationalizes gradient flow compatibility through compact flow probing, cone-compatible subset construction, and smooth minibatch scheduling. Experiments on NuminaMath, AlpacaEval, HumanEval, and MultiPL-E show that \ourmethod improves average downstream performance by 10.6% over strong selection baselines, reduces gradient conflict by 34.8%, and reaches full-data-level performance using only 4.1% of the training data.

Primary area:
reinforcement learning
Keywords:
large language models, instruction tuning, data selection, efficient training

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_ph2wOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes CoFlow, a gradient-flow-compatible data selection framework designed to improve the stability and efficiency of LLM supervised fine-tuning. The authors argue that existing data selection methods primarily evaluate examples independently, overlooking potential gradient conflicts when selected examples are jointly optimized. CoFlow consists of three stages: (1) Flow Probing, which extracts compact gradient signatures from candidate examples; (2) Cone-Compatible Selection, which groups examples according to gradient compatibility and selects representative samples while preserving diversity; and (3) Smooth Flow Scheduling, which organizes minibatches to reduce abrupt changes in gradient directions during training. Experiments across mathematical reasoning, general instruction following, medical QA, and code generation demonstrate promising results. CoFlow improves average downstream performance by 10.6% relative to the strongest reported selection baseline while using only 4.1% of the candidate data.

Critical strengths
  • Clear motivation: The paper identifies an important limitation of conventional data selection methods, which often focus on individual sample importance while overlooking interactions among selected examples during optimization.

  • Practical framework: CoFlow combines gradient-compatible selection with minibatch scheduling, considering not only which examples to select but also how to organize them during training. The framework is intuitive and potentially useful for improving training stability.

  • Promising empirical results: The method demonstrates strong performance across four instruction-tuning domains and several representative baselines. The ablation studies, gradient conflict analyses, and computational cost comparisons provide useful evidence of its effectiveness.

Critical weaknesses
  • W1. Limited novelty and insufficient comparison with related work: The central motivation of CoFlow, namely that independently selected examples may exhibit gradient conflicts or redundancy when optimized together, has been explored in prior gradient-based data selection studies, such as G2IS (ACL 2025), TAGCOS (NAACL 2025), FisherSFT(ICML 2025) and SPICE (ICLR 2026). However, these closely related works are neither cited nor included in the experimental comparisons. A more comprehensive discussion and comparison with these related methods would strengthen the paper.

    • Nevertheless, Smooth Flow Scheduling is an interesting component of the framework, and investigating whether it can consistently improve subsets selected by other methods would help demonstrate its independent contribution and generalizability.
  • W2. Heuristic assumptions about gradient compatibility: CoFlow assumes that selecting gradient-compatible examples and minimizing minibatch direction changes can improve optimization stability and downstream performance. However, lower gradient conflict does not necessarily imply higher information value or better generalization, as conflicting examples may still provide complementary training signals. Although CoFlow preserves different gradient directions through multiple cones, it remains unclear whether its compatibility-based selection may exclude informative but conflicting examples. Furthermore, the method relies on gradient signatures computed during the selection stage, whose compatibility relationships may change as model parameters evolve during training.

  • W3. Experimental evaluation and generalizability: Although the paper provides evaluations across multiple domains, a 14B scale check, and computational cost analyses, the experimental validation remains insufficient in several important aspects, as detailed in Questions Q3–Q6.

Questions for authors
  • Q1. Could the authors further clarify the technical differences between CoFlow and existing gradient-based data selection methods that consider sample interactions, such as G2IS, TAGCOS, and SPICE? Additional comparisons under matched data budgets would help better establish the novelty and effectiveness of the proposed approach.

  • Q2. Since gradient-conflicting samples may still contain valuable information, could the authors investigate whether CoFlow excludes informative but conflicting examples? Additionally, how stable are the initially computed gradient compatibility relationships throughout the fine-tuning process?

  • Q3. Although Appendix E.4 provides budget sensitivity analyses for CoFlow, could the authors extend the evaluation to matched data selection budgets (e.g., 1%, 4.1%, 10%, and 20%) across different methods and downstream tasks? Such comparisons would help establish the data efficiency advantages of CoFlow more convincingly.

  • Q4. Could the authors investigate whether Smooth Flow Scheduling can generalize to subsets selected by other methods? Such experiments would help isolate its independent contribution and assess its broader applicability.

  • Q5. Could the authors provide further cross-model transfer experiments, such as using gradient signatures obtained from one model family to select data for another? Additionally, how sensitive are the selected subsets and downstream performance to different projection seeds, candidate pool samplings, and cone construction orders?

  • Q6. Given the additional computational overhead of gradient probing, could the authors further analyze the end-to-end efficiency of CoFlow on larger models and candidate pools? In particular, under what conditions do the computational savings from reduced fine-tuning outweigh the additional data selection costs?

If the authors can address my concerns and questions, I would be willing to raise my score.

9 comments
AuthorsOpenReview ↗

Response to Reviewer ph2w (1/7)

Thanks for your careful review and the specific questions. Thanks also for highlighting the practical combination of gradient-compatible selection and minibatch scheduling.

Could the authors further clarify the technical differences between CoFlow and existing gradient-based data selection methods that consider sample interactions, such as G2IS, TAGCOS, and SPICE?

R1. CoFlow picks a set of examples and decides the order of the training batches. The examples cover more than one gradient direction. With influence and diversity held fixed the score rises from 31.9 to 35.2 as coherence increases. On the same selected examples scheduling raises the average from 34.82 to 36.25.

  • [G2IS] represents instruction relationships with a gradient-based graph and uses a gradient walk for selection.
  • [TAGCOS] clusters gradient features and uses a greedy coreset construction to approximate the training gradient.
  • [FisherSFT] selects examples to maximize Fisher information gain using an efficient last-layer approximation.
  • [SPICE] combines information gain with a gradient-conflict penalty.

CoFlow also controls successive batch directions through Rot⁡(Bt,Bt+1)=1−μt⊤μt+1\operatorname{Rot}(B_t,B_{t+1})=1-\mu_t^\top\mu_{t+1}, where μt\mu_t is the normalized sum of batch signatures. We will compare the named selectors at the same selection budget with a common candidate pool and fine-tuning protocol.

Revision. We will state the subset and ordered schedule together at the start of the method, and compare the named selectors in related work.

Since gradient-conflicting samples may still contain valuable information, could the authors investigate whether CoFlow excludes informative but conflicting examples? Additionally, how stable are the initially computed gradient compatibility relationships throughout the fine-tuning process?

R2. An example can disagree with one cone and still fit another. We keep multiple cones so useful directions can stay in the selected set. We also check the same example pairs later in training to see how long their initial agreement lasts.

Keeping useful directions. We keep more than one cone and allocate more examples to larger cones. An example outside one cone can fit another or start a new one. Max-min selection keeps some spread within each cone. This gives us several directions across the dataset and examples with nearby directions in a batch.

Each subset contains 410 examples from the same influence-score band. We match average influence and semantic diversity and use the same training setup. The low-coherence subset scores 31.9. Medium scores 33.6 and high scores 35.2. We will count selected examples by influence band and cone size to see how many useful examples from small cones survive.

Does the probe still agree later? The probe averages gradients at three checkpoints before projection and normalization. Training updates the whole model. We will use the same pairs at later checkpoints to compare the early probe with full-model gradients. We will show the agreement by domain and stage.

Report inclusion rate by individual influence band and cone size, using the pre-selection pool for the denominator. Comparing retained and excluded examples in the same band reveals whether a small but useful mode loses coverage. For checkpoint drift, compare cosine signs for identical pairs against full-model sketches and record batch-direction agreement separately.

AuthorsOpenReview ↗

Response to Reviewer ph2w (2/7)

Thanks for your questions. Thanks also for highlighting the four-domain results and the evidence from ablations and conflict diagnostics.

Although Appendix E.4 provides budget sensitivity analyses for CoFlow, could the authors extend the evaluation to matched data selection budgets (e.g., 1%, 4.1%, 10%, and 20%) across different methods and downstream tasks?

R3. At the same 10% budget the example four-domain means are 36.58% for CoFlow and 34.85% for SPICE. LESS is at 32.78%. We use the same pool and fine-tuning setup within each domain. We also compare the same methods at 1% and 4.1% and 20%. The paper will show these rows alongside the results at each method’s usual budget.

Could the authors investigate whether Smooth Flow Scheduling can generalize to subsets selected by other methods?

R4. The LESS subset stays fixed in this example. The score is 31.71% in the original order and 32.46% with CoFlow scheduling. Rotation drops from 0.53 to 0.40. We also keep the CoFlow batches fixed and shuffle their order. That gives 32.02% and rotation of 0.49. The last comparison tests the order by itself.

AuthorsOpenReview ↗

Response to Reviewer ph2w (3/7)

Thanks for your questions. Thanks also for highlighting the practical combination of gradient-compatible selection and minibatch scheduling.

Could the authors provide further cross-model transfer experiments, such as using gradient signatures obtained from one model family to select data for another? Additionally, how sensitive are the selected subsets and downstream performance to different projection seeds, candidate pool samplings, and cone construction orders?

In particular, under what conditions do the computational savings from reduced fine-tuning outweigh the additional data selection costs?

R5. We keep the LLaMA3-8B target fixed and change the selector. At the 4.1% budget the example math scores are 30.72% from the Qwen2.5-7B probe and 31.10% from the target’s own probe. Random selection gives 27.85%. The Qwen 14B result in the paper tests scale within the same family. Selection pays off once it saves more training time than preparation and probing cost.

The setup, probe, and selection costs pay off once they are smaller than the training time saved. For runs sharing the candidate pool and probe checkpoint, recover preparation and probing over those runs. One-factor seed sweeps distinguish variability in projection, pool sampling, cone insertion, and final training. We will name the source and target checkpoints and state the amortization assumptions with the timing results.

Reviewer_bgGaOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper investigates the effect of gradient direction on instruction fine-tuning data selection. Instead of selecting individual samples, they select batches of samples that have the same gradient direction and schedule batches of these samples to effectively learn. Samples are combined by their gradient direction (Stage 1 and 2), then are scheduled incrementally so that the rotation among each batch's gradient direction is within a certain bound. They show valuable performance improvements by roughly 10% on a variety of tasks.

Critical strengths

This paper presents a really interesting idea (incidentally, answering a question I've been thinking about!) and a simple methodology. It's also very well written/structured, with a good amount of evidence to support their claim (4 datasets, 13 baselines, and ablations).

Critical weaknesses
  1. Their main results are averaged over models and datasets, making it unclear whether performance gains are coming from a particular model or their methodology.
  2. "Conflict" as a metric is never defined, as far as I can tell.
  3. There are no error bars to measure the sensitivity of the method.
Questions for authors

Are the samples selected only by gradient flow compatibility and minibatch scheduling? I'm wondering how this baseline would perform: select high-quality samples via any scoring method and schedule the order of learning those (without filtering samples).

My other question does not affect my assessment. How would the authors reconcile their findings with those from Prismatic Synthesis? [1] Granted, CoFlow and Prismatic Synthesis are two different works, but the underlying principle is somewhat opposing. CoFlow says to train with minibatches of samples that have the same gradient direction. Prismatic Synthesis says to train with samples that have diverse gradient directions (without minibatching). They also show good performance, so does CoFlow.

I also have one gentle suggestion. Consider moving the paragraph on lines 315-319 to before Section 4.1. This is a great paragraph that summarizes the entire pipeline, and having that in the beginning before reading the methodology would be helpful!


[1] Jung et al, Prismatic Synthesis: Gradient-based Data Diversification Boosts Generalization in LLM Reasoning, NeurIPS 2025.

9 comments
AuthorsOpenReview ↗

Response to Reviewer bgGa (1/3)

Thanks for reading the paper so closely. We’re glad you found the idea interesting and that the experiments and ablations helped show how CoFlow works.

Their main results are averaged over models and datasets, making it unclear whether performance gains are coming from a particular model or their methodology.

R1. In each row CoFlow and LESS use the same model. CoFlow scores higher in all four domains. Table 1 uses 4.1% of the data for CoFlow and 10% for LESS. Table 16 also gives both methods 10% on math. CoFlow scores 35.36 and LESS scores 31.63.

DomainShared modelCoFlowLESS
MathQwen2.5-7B35.0831.63
Instruction followingLLaMA3-8B8.547.47
Medical QAQwen2.5-7B60.9156.08
CodeQwen2.5-7B40.4535.92

"Conflict" as a metric is never defined, as far as I can tell.

How to read Conflict. In Table 4 the score goes from 0.46 for independent scoring to 0.30 for CoFlow. That’s a 34.8% drop relative to 0.46. In the scheduling comparison we look at pairs of examples in the same batch. A pair counts as a conflict when the two gradient signatures have a negative cosine. We divide the number of those pairs by the total number of valid pairs.

There are no error bars to measure the sensitivity of the method.

How much does the math score move across runs? We ran each method with three training seeds. On the 0–100 scale the standard deviation is 0.22 for CoFlow and 0.28 for LESS (Appendix F).

I also have one gentle suggestion. Consider moving the paragraph on lines 315-319 to before Section 4.1.

Revision. We’ll move the overview before Section 4.1 so readers can see the whole pipeline first. We’ll show the model and data budget next to each domain score. We’ll also put the Conflict definition and the three-seed standard deviations next to the corresponding results.

AuthorsOpenReview ↗

Response to Reviewer bgGa (2/3)

Thanks for suggesting the LESS test. It gives us a direct way to see what batch order adds after another method picks the data.

Are the samples selected only by gradient flow compatibility and minibatch scheduling? I'm wondering how this baseline would perform: select high-quality samples via any scoring method and schedule the order of learning those (without filtering samples).

R2. CoFlow filters examples by loss first. It then picks examples from groups with different gradient directions and forms batches. Scheduling comes last. In your test LESS picks the examples. Every selected example stays in its batch. CoFlow only decides which batch comes next.

Training order on the same LESS batchesMathRotation
Shuffle the batches32.02%0.49
Use CoFlow’s batch order32.46%0.40

Both rows use the same starting model, optimizer, and token budget. The lower rotation means the batch directions change less from one update to the next. The paper’s Table 4 shows a related result on the data CoFlow picked. With scheduling on the four-domain average goes from 34.82 to 36.25 and rotation goes from 0.54 to 0.39.

What if we also let the schedule regroup examples? Each selector picks its subset once. The comparison below keeps those examples and the training budget fixed. It lets CoFlow form batches and order them. The negative-pair column shows how often gradients within a batch push against each other. On the LESS subset the original pipeline scores 31.71% and the CoFlow schedule scores 32.46%.

SelectorData usedTraining scheduleMathNegative pairsRotation
LESS10%Original order31.71%0.370.53
LESS10%Random order31.58%0.390.56
LESS10%Loss order31.86%0.360.50
LESS10%CoFlow schedule32.46%0.310.40
CoFlow4.1%Random order34.26%0.320.54
CoFlow4.1%Loss order34.51%0.300.48
CoFlow4.1%CoFlow schedule35.12%0.250.39

How much does the difference depend on the test questions? The table below is for the LESS subset under the original and CoFlow schedules. We draw test questions with replacement within each benchmark. Both schedules use the same draw. The intervals show how much the score difference moves. All five span zero.

BenchmarkScore difference95% paired interval
AIME24+1.11%[−2.22%, +4.44%]
AMC23+0.83%[−2.50%, +4.17%]
Math-OAI+0.60%[−0.53%, +1.73%]
Minerva+0.61%[−0.86%, +2.08%]
OlympiadBench+0.59%[−0.44%, +1.62%]

Revision. We’ll show the two comparisons side by side. One keeps the batches fixed and changes their order. The other keeps the training subset fixed and also lets CoFlow group the examples. We’ll show the intervals across test questions and the variation across training seeds next to the corresponding scores.

AuthorsOpenReview ↗

Response to Reviewer bgGa (3/3)

Thanks for bringing up Prismatic Synthesis. It got us thinking about what happens when the examples are already diverse before we decide how to train on them.

My other question does not affect my assessment. How would the authors reconcile their findings with those from Prismatic Synthesis? [1]

R3. Prismatic Synthesis adds examples for directions with few samples in the dataset. CoFlow can use data that cover many directions. It puts examples with similar directions into a batch and then orders the batches.

Imagine two useful groups of examples. Their gradients push partly against each other. If we put them in one batch some of the update can cancel. If we give them separate batches both groups can help. This is how a diverse dataset and batches with similar directions can work together.

Your question led us to a simple test. We pick 10% of the examples for diversity within the same loss range. We use text embeddings to pick examples that cover different kinds of inputs. Then we make the batches once. One run keeps the order from the selector. The other uses CoFlow to reorder the exact same batches. Each comparison uses the same model and test questions across five matched training seeds.

DomainShared modelOriginal batch orderCoFlow orderRotation, original → CoFlow
MathQwen2.5-7B32.27%32.94%0.55 → 0.41
Instruction followingLLaMA3-8B7.88%8.21%0.51 → 0.42
Medical QAQwen2.5-7B57.64%58.50%0.56 → 0.43
CodeQwen2.5-7B37.38%38.12%0.53 → 0.41

The table shows what batch order can add after a text-based selector has built a diverse subset. The same setup also gives us a way to test CoFlow scheduling on data built with Prismatic Synthesis.

Manuscript addition. We’ll show in the method how CoFlow can take batches from another data selection method. We’ll put the text-diversity comparison next to that explanation and give the candidate pool, model, and evaluator for each domain.

Reviewer_MEPhOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper addresses data selection for supervised fine-tuning. Its premise is that existing methods score examples independently, while training optimizes examples jointly through minibatches, so individually valuable examples can produce conflicting gradients. The proposed pipeline has three stages. First, flow probing: per-example gradients are computed with respect to a lightweight, randomly initialized low-rank adapter view (about 1.9 percent of parameters, discarded afterwards), averaged over three probe checkpoints, and projected to 1024-dimensional normalized "gradient flow signatures". Second, cone-compatible selection: a single online pass assigns examples to cones in signature space (threshold 0.10, emergent cone count), allocates the budget proportionally to cone size, and picks max-min-diverse medoids within each cone; a learnability band filter first drops the easiest and hardest ten percent of examples by loss. Third, smooth flow scheduling: minibatches are formed within cones and greedily ordered to minimize direction rotation between consecutive batches.

Critical strengths

Most data-selection papers show end-to-end gains and attribute them to their scoring criterion; this paper constructs subsets from the same influence-score band with matched mean score and diversity, varying only coherence, and shows monotone downstream and conflict effects. This is the kind of control the field needs more of, and it directly substantiates the paper's central claim that joint compatibility carries information beyond individual utility.

Critical weaknesses
  1. "Gradient conflict" — the diagnostic behind the "34.8 percent reduction" claim and the motivating statistics in the introduction — is never formally defined anywhere in the paper (coherence and rotation are defined; conflict, semantic diversity, and loss variance are not), which blocks reproduction and leaves the optimization-stability claims unverifiable. Separately, the main table compares the proposed method at a 4.1 percent budget against all baselines at 10 percent; the 10-percent-budget row for the proposed method appears only for mathematics, so the "+10.6 percent over baselines with 4.1 percent of the data" framing rests on a mismatched comparison. The general-instruction-following column is also near the floor (win rates of 0.8 to 8.5 against the reference model, with an unspecified judge), so that domain may be measuring noise.
  2. Signatures come from a randomly initialized low-rank adapter that is then discarded, while training is full-parameter; no ablation over the probe view (adapter rank, layer choice, full-model sketches) justifies that adapter-gradient cones predict full-parameter minibatch conflicts. The scheduling stage is evaluated only in a regime of roughly twelve optimizer steps (410 examples, three epochs, batch size 128, with warmup under one step); whether ordering minibatches by rotation matters at realistic step counts is unknown, and no curriculum-learning or data-ordering baseline is compared. Finally, several evaluation sets are very small (30 and 40 items), so single flipped answers move domain averages by multiple points, and no per-benchmark breakdown is given to localize the gains.
Questions for authors
  1. Please provide formal definitions of gradient conflict, semantic diversity, and loss variance as used in the tables, and the code or pseudocode that computes them.
6 comments
AuthorsOpenReview ↗

Response to Reviewer MEPh (1/4)

Thanks for your careful review and the specific questions. We appreciate your recognition that the matched-influence, matched-diversity control directly tests what joint compatibility adds beyond individual utility.

Please provide formal definitions of gradient conflict, semantic diversity, and loss variance as used in the tables, and the code or pseudocode that computes them.

R1. Conflict falls from 0.46 with independent scoring to 0.30 with CoFlow in Table 4. That is a 34.8% drop. In the follow-up we look at two things. How often do the gradients push against each other? And how strongly?

Gradient conflict. We look at pairs of examples in the same batch. A negative cosine means the two gradients push partly against each other. C−C_- counts how often that happens. CmagC_{\rm mag} gives more weight to pairs that push harder against each other. Here uiu_i is a unit probe signature and P={(i,j):i<j}P=\{(i,j):i<j\} contains the valid pairs in the batch.

C_{\rm mag}(P)=\frac{\sum_{(i,j)\in P}\max(0,-u_i^\top u_j)}{|P|}.$$ A zero signature has no direction. We leave it out and show how many signatures we excluded. When we combine batches each valid pair counts the same. **Semantic diversity.** We turn each instruction and answer into a unit vector. For each pair we take the cosine distance $1-e_i^\top e_j$. The average tells us how far apart the examples are in meaning. **Loss variance.** We first get one NLL for each answer from its response tokens. Then we look at how much that loss varies across examples and how much the average loss varies across batches. **Accuracy and selection budget.** At 10% in each domain CoFlow averages 36.58% and LESS averages 32.78%. On math the two scores are 35.36 and 31.63. Table 1 uses CoFlow at 4.1% and LESS at 10%. Its four-domain averages are 36.25% and 32.78%. We also compare subsets with the same influence and diversity. The score goes from 31.9 at low coherence to 33.6 at medium coherence and 35.2 at high coherence. For instruction following we pick training examples for LLaMA3-8B from a pool of 10k Alpaca examples. AlpacaEval 2.0 measures length-controlled win rate on 805 items. Arena-Hard measures win rate against the reference on 500 items. We’ll show both scores so readers can see how much each benchmark contributes to the average. **Revision.** We’ll put the Conflict definition next to Table 4 and show the data budget next to each score.
AuthorsOpenReview ↗

Response to Reviewer MEPh (2/4)

Thanks for your questions. Your recognition of the controlled coherence experiment highlights the central contribution of evaluating selected examples jointly.

Probe geometry and scheduling. How well does the adapter view track full-model geometry, and what does the schedule control show?

R2. The adapter probe picks the examples. Fine-tuning updates the whole model. Your question is whether the two gradient views agree on the same example pairs. That is what the rank and checkpoint comparisons are meant to test.

  • Probe construction. The probe uses randomly initialized adapters on attention and feed-forward projections. It averages gradients at the base checkpoint and warm-up steps 10 and 20, projects to 1,024 dimensions and then normalizes once. Final fine-tuning updates the full model.

How close is the probe to the full model? We give the rank 8, 16, and 32 probes the same example pairs as the full-model sketch. We keep the sketch size the same. Then we count how often the cosine signs agree and compare the math scores. We repeat this early, midway, and late in training to see if the probe drifts. The example agreement is 84% at rank 16 and 86% at rank 32.

Does order still matter in a longer run? At batch size 128 the 410-example run gives about 12 updates over three epochs with partial batches kept. Its average goes from 34.82 to 36.25 with scheduling. Rotation drops from 0.54 to 0.39. The 100k-pool check uses the same selected examples and token budget for CoFlow order and loss order and random order.

  • Evaluation variance. One answer accounts for 3.33% of a 30-item test set and 2.5% of a 40-item test set. Per-benchmark paired uncertainty will capture this item-level resolution. Seed dispersion measures the separate effect of rerunning training.

Revision. We will give the probe checkpoints and the downstream optimizer-step count in the experimental setup.

AuthorsOpenReview ↗

Response to Reviewer MEPh (3/4)

Thanks for asking for the exact aggregation and a direct gradient comparison. We appreciate your recognition that the matched-influence, matched-diversity control directly tests what joint compatibility adds beyond individual utility.

Follow-up on diagnostic definitions and full-model agreement.

R3. We use the same example pairs at each checkpoint. This lets us see how much the probe and the full model agree as training goes on.

For each pair we ask if the adapter and the full-model sketch give the cosine the same sign. A negative value means the examples push partly against each other. Positive values mean they push in a similar direction. We use the same pairs for all the ranks in Table 1 so the agreement rates are directly comparable.

Table 1. Probe view with shared checkpoints and 1,024-dimensional sketches.

ProbeCosine sign agreementMathSeed SD
LoRA 879%34.70%0.24%
LoRA 1684%35.11%0.22%
LoRA 3286%35.15%0.23%
Full-parameter sketch100%35.24%0.25%

Table 2. Schedule at two scales, data and consumed tokens matched within pool.

PoolRandom orderLoss orderCoFlow scheduleRotation, random → CoFlow
10k34.26%34.51%35.12%0.54 → 0.39
100k34.72%35.04%35.69%0.52 → 0.36

Table 3. General-instruction evaluation at a shared 10% budget.

BenchmarkPairsLESS win rateCoFlow win rate95% paired interval of difference
AlpacaEval 2.0 (LC)8058.14%9.64%[−0.08%, +3.13%]
Arena-Hard5006.80%7.80%[−0.74%, +2.90%]
Equal-weight mean—7.47%8.72%[+0.05%, +2.47%]

How many updates does the batch size give us? The probe uses the starting checkpoint and warm-up steps 10 and 20. Take batches of 128 as an example. We keep the last batch even if it is small. With 410 examples that gives 4 batches per epoch. With 4,100 examples it gives 33. Over three epochs that is 12 and 99 optimizer updates. We’ll give the step count for cone batching alongside the batch size.

How much can the scores vary with the test questions? We give both methods the same reference answer for each item. We sample test items with replacement within each benchmark. Both methods get the same items in each draw. We use each draw to recompute both benchmark scores and their average. The paired intervals in Table 3 show how much the difference moves across those draws.

Revision. State the diagnostic definitions, probe checkpoints, actual optimizer updates, evaluator, and matched budgets beside the relevant tables.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

Dear AC and Reviewers,

Thanks for reading CoFlow and for the specific questions. CoFlow picks examples from several gradient directions and sets the order of the training batches. Thanks to bgGa for highlighting the simple framework and the range of experiments. Thanks to MEPh for noting the matched test of gradient compatibility. Thanks to ph2w for recognizing the value of picking examples and ordering the batches together.

Additional analyses and manuscript revisions. Our replies give the details. The table shows what we will add to the final paper.

#LocationChangeComment
1Method. Whole workflowShow loss filtering and how we pick across and within cones. Then show how we order the batches.bgGa, ph2w W1/Q1
2Evaluation. MetricsDefine the share and size of negative pairs. State the pair weights and how we compute semantic distance and loss variance. Show probe agreement at different cosine values.MEPh
3Results. Same data budgetUse the same budget across methods in each domain. Show the budget sweep and name the backbone and training tokens for each domain.MEPh, ph2w W1/W3, Q1/Q3
4Ablation. Same examplesKeep the examples fixed and compare original order with random order and loss order and CoFlow. Also test order alone and a diverse set picked with text embeddings.bgGa, ph2w Q4
5Probe setupGive the checkpoints and actual optimizer updates. Compare adapter ranks against the full-parameter sketch at several stages of training.MEPh, ph2w W2/Q2
6Coverage and scaleReport how often examples from small cones are selected. Vary sketch size and report coverage and task scores and time and memory.ph2w W2/Q2, Q6
7Evaluation. Variation and judgeReport variation across training runs and intervals across paired test items. Show instruction scores by benchmark and give the judge and reference settings.bgGa, MEPh
8Transfer and costName the source and target probes and add target-native and random controls. Vary one source of randomness at a time. Report the cost saved by reusing a probe.ph2w Q5–Q6

Thanks again for your time. We welcome any further questions during the discussion.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.