acceptodds
Under review as a conference paper at ICLR 2027

Latent Flow Transformer: Flow Matching the Hidden Dynamics of Language Models

Abstract

Transformers, the standard implementation for large language models (LLMs), typically consist of tens to hundreds of discrete layers. While depth improves performance, it is an inefficient use of capacity next to the continuous formulations that diffusion and flow-based models use for image generation. We propose the Latent Flow Transformer (LFT), which replaces a contiguous block of layers with a single transport operator trained by flow matching, a post-training procedure that leaves the surrounding architecture unchanged. To identify replaceable spans, we introduce the Recoupling Ratio, an optimal-transport measure of how far a span's endpoint latents deviate from an optimal pairing. We further introduce Flow Walking (FW), which addresses the failure of existing flow-based methods to preserve latent pair coupling. On Pythia-1.4B, an LFT trained with FW replaces 12 of 24 layers with a single operator and retains 92% of the teacher's downstream accuracy (57.5% vs. 62.8%) at 63% of its parameters and 72% of its per-token FLOPs, whereas skipping the same span attains 53.8%. FW matches a regression transport layer at six replaced layers, surpasses it at twelve, and is less sensitive than standard flow matching to the latent coupling of the replaced span, bridging the gap between autoregressive and flow-based generation paradigms.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
language modelling, flow matching, layer compression, layer skipping, simulation free flow matching

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_qVUDOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes the Latent Flow Transformer (LFT), a structural compression framework for Large Language Models (LLMs) that replaces a contiguous block of Transformer layers with a single continuous transport operator trained via Flow Matching. To systematically identify which layers are suitable for replacement, the authors introduce the Recoupling Ratio, an Optimal Transport-based metric that quantifies the misalignment between latent pairs across layers, revealing that middle layers are highly compressible. Furthermore, to address the "trajectory crossing" problem inherent in standard Flow Matching when applied to deterministic paired data, they propose Flow Walking (FW), a numerical integration algorithm that preserves latent pair coupling. Experiments on Pythia-1.4B show that LFT-FW can replace half of the model's layers (12 out of 24) with a single operator, retaining 92% of the teacher's downstream accuracy while reducing parameters to 63% and FLOPs to 72%.

Critical strengths
  1. Highly Novel and Interdisciplinary Perspective: Bridging continuous-time dynamical systems and Flow Matching (typically dominant in image/video generation) with the discrete layer-wise forward pass of LLMs is a highly creative approach. Framing Transformer layers as discretized ODEs and compressing them into a single learned transport operator offers a fresh theoretical and practical perspective on structural model compression.

  2. Insightful Diagnostics and Algorithmic Solutions: The introduction of the Recoupling Ratio provides a theoretically grounded, interpretable metric to diagnose layer redundancy, successfully validating the hypothesis that middle layers act as compressible "routing" spaces. Additionally, identifying the trajectory crossing issue in paired latent data and solving it via the Flow Walking (FW) algorithm shows a deep understanding of the limitations of standard Flow Matching when applied to deterministic sequence modeling.

Critical weaknesses
  1. Lack of Generalization: The entire empirical validation is restricted to the Pythia model family (410M and 1.4B), which uses an older architecture (GPT-NeoX) and was trained on "The Pile". Modern LLMs employ significantly different architectural components (e.g., RoPE, SwiGLU, Grouped-Query Attention, RMSNorm) and are trained on vastly superior, deduplicated corpora. The latent dynamics and redundancy distributions in Pythia may not reflect those in modern LLMs.

  2. "Insufficient and Weak Baselines": The paper only compares LFT against naive Layer-Skipping and a basic Regression baseline (a single Transformer layer trained with MSE loss). It completely ignores the rich landscape of state-of-the-art LLM compression, structured pruning, and distillation techniques. For instance, the paper cites ShortGPT (Men et al., 2024) regarding layer redundancy but fails to include it as an empirical baseline. Other relevant methods like LLM-Pruner, Wanda, or block-wise knowledge distillation methods are completely absent.

Questions for authors
  1. Does the Recoupling Ratio reliably identify compressible spans in modern architectures (e.g., Llama-3, Mistral, Qwen) that utilize RoPE and SwiGLU? Or at least, can you provide any evidence or theoretical justification that Flow Walking will scale effectively to 7B+ parameter models without suffering from optimization collapse or requiring prohibitive training FLOPs?

  2. Why were advanced structured pruning methods (e.g., ShortGPT, LLM-Pruner) and modern knowledge distillation techniques excluded from the baselines? How does LFT-FW compare to these SOTA pruning methods in terms of the trade-off between compression ratio, downstream accuracy retention, and the total training compute required?

3 comments
AuthorsOpenReview ↗

Generalization to Qwen2.5 and Comparisons with Pruning Baselines

We thank the reviewer for taking the time and effort to review the paper. Please see below for our response to the weaknesses and the questions.

W1: Generalization to modern model architectures

We observe that the advantage of LFT persists as we extend evaluations on Qwen2.5 and compare it against additional baselines in layer pruning. Note that Qwen2.5 is a modern architecture, using RoPE, SwiGLU, Grouped-Query Attention, and RMSNorm as mentioned by the reviewer.

We evaluate our proposed framework and algorithm on Qwen2.5 1.5B and 7B models. We follow the same procedure as in the paper to select the layers. For the 1.5B model, the span from layer 5 - 19 is selected; for the 7B model, the span from layer 7 - 21 is selected. Furthermore, we report comparisons against two additional layer pruning baselines - ShortGPT and SLEB.

Qwen2.5-1.5B

RowOperator+recovery tokensParamPPL/acc
Teacher-1.544B8.52/71.00%
FW (layer5-19, k3)20M+20M952M13.01/62.08%
Regression (layer5-19)20M+20M952M13.23/61.72%
Skip (layer5-19)0+40M889M14.16/59.50%
ShortGPT (12 layers)0+40M982M14.48/59.46%
SLEB (12 layers)0+40M982M12.84/60.64%

Qwen2.5-7B

RowOperator+recovery tokensParamPPL/acc
Teacher-7.616B6.85/76.34%
FW (layer7-21, k=3)20M+20M4.68B10.15/65.20%
Regression (layer7-21)20M+20M4.68B10.35/64.16%
Skip (layer7-21)0+40M4.35B11.37/63.14%
ShortGPT (12 layers)0+40M4.82B10.73/64.78%
SLEB (12 layers)0+40M4.82B10.25/65.10%

We will include these results in the revised manuscript.

W2: Additional pruning baselines

As mentioned in the table at W1, we will also include these additional comparisons along with pruning baselines such as ShortGPT and SLEB. Empirical validation shows that LFT also outperforms these two baselines.

Q1: Recoupling Ratio and scalability

We observed that the layer selection criterion based on Recoupling Ratio holds when evaluated on Qwen2.5-1.5B (28 transformer layers). We trained the same methods on spans of the same width with different RR:

SpanMean RFW k3 PPLSFM k3 PPLregression PPLskip PPL
[5,19) (selected)0.0017.821.018.169.6
[14,28)0.0317356017623,492
[0,14)0.9719,6524,2932,7023.2 × 10⁶

As to evaluation on 7B+ models, we tested the method on Qwen2.5-7B and observe the same findings as that at 1.5B model (see the Table above in W1).

Experiments at scales beyond 7B remain future work. However, we surmise that our method still holds as modern LLM layers are known to be highly redundant [1,2].

[1] W. Sun et al., The Curse of Depth in Large Language Models [2] A. Gromov et al., The Unreasonable Ineffectiveness of the Deeper Layers

Q2: Compression, accuracy, and training-compute trade-offs

As shown in the tables at W1, we include comparisons with pruning methods such as ShortGPT and SLEB. We observe that FW outperforms these methods while requiring higher decoding latency due to a number of inference evaluations (see the table below). The training cost for ShortGPT and SLEB at the repairing stage is lower than that for FW or regression as they do not require training of the compressed layers.

Parameters (%)PrefillDecode per tokenGreedy 128-token requestKV at end
Teacher (24 layers)100%74.1 ms12.60 ms1675 ms239.6 MiB
LFT-FW k = 3 (replaces L6–L18)63.0%55.8 ms10.41 ms1378 ms179.7 MiB
LFT-FW k = 1 (replaces L6–L18)63.0%42.1 ms8.04 ms1064 ms139.8 MiB
Regression r = 1 (replaces L6–L18)63.0%38.6 ms7.37 ms974 ms129.8 MiB
ShortGPT, 11 layers removed60.8%38.4 ms6.92 ms918 ms129.8 MiB
Reviewer_qVUDOpenReview ↗

Qwen2.5 Exp Improved Generalization Evidence , but Flow Walking’s Compute–Accuracy Tradeoff Remains Unclear

Thank you for the additional experiments. The Qwen2.5-1.5B and 7B results address my concern about generalization beyond Pythia, and the ShortGPT and SLEB comparisons strengthen the empirical evaluation.

My main remaining concern is the benefit of Flow Walking at comparable inference cost. On Pythia-1.4B, three-step FW and regression have the same parameter count, but after KL repair, FW improves accuracy from 57.1% to 57.5% while increasing per-token FLOPs by about 20% and measured decoding latency by about 41%. Single-step FW achieves accuracy comparable to regression. On Qwen2.5-7B, FW is competitive with SLEB (65.20% vs. 65.10% accuracy), but the rebuttal does not report inference latency or total training compute for that comparison. A useful control would be to distill a shared-weight Transformer block applied repeatedly, using the same endpoint MSE and matching parameter count and inference compute. This would help isolate the benefit of the flow formulation from the benefit of additional block evaluations.

The new span results also support the Recoupling Ratio as a selection heuristic. However, the selected span is easier to compress for all tested methods, including regression and direct skipping. A comparison against a simpler criterion such as endpoint cosine similarity would clarify what Recoupling Ratio adds.

The rebuttal resolves enough of my original concerns that I would raise Soundness from 1 to 2. I maintain my weak reject recommendation because the current evidence leaves the incremental benefit of Flow Walking at comparable compute unclear.

AuthorsOpenReview ↗

Scope of the paper, qual-compute shared-weight regression, and why Recoupling Ratio rather than endpoint cosine

Thank you for the follow-up.

We would like to emphasize that our paper studies latent representation matching in language models with fixed input–output pairs. In standard flow matching (SFM), trajectories from different pairs can meet at the same time and prescribe conflicting velocities. We introduce Flow Walking (FW) and show that it reduces the impact of such crossings relative to SFM. We apply it to layer compression, using the Recoupling Ratio to select a span and FW to train its replacement operator; there, LFT-FW outperforms LFT-SFM.

On shared-weight regression. We agree this is an important baseline. In our response to reviewer nozo, we report a regression baseline trained end-to-end through six applications of the same block, matching the six block evaluations of three-step FW. This recursive regression can be seen as a special case of FW without time conditioning.

On Pythia-1.4B, Regression (r=6) slightly improves over LFT (k=3) at equal inference compute. In contrast, on Qwen2.5-1.5B, more inference compute degrades the recursive regression baseline. We surmise that this difference may partly arise from the blocks’ residual architectures. Pythia applies attention and the FFN in parallel to the same input, whereas Qwen applies them sequentially, with the FFN receiving the attention-updated state. Repeated composition of a Qwen block therefore introduces additional multiplicative interactions between the attention and FFN Jacobians, potentially making optimization more sensitive to initialization and repeated changes in the hidden-state distribution. FW takes smaller steps toward the block’s predicted output, which may reduce the accumulation of large hidden-state changes across repeated evaluations.

ModelRegression r=1Regression r=6FW k=3
Pythia-1.4B (layer6-18)21.98/53.18%18.13/54.82%20.73/54.30%
Qwen-1.5B (layer5-19)18.07/58.96%18.44/58.72%17.82/59.16%

Comparing span-selection criteria is beyond the scope of the paper: Comparing span-selection criteria requires a meta-metric that scores the criteria themselves, e.g. the rank correlation between a criterion's score and post-compression quality over all candidate spans, under a fixed operator, training budget and evaluation task. Defining and validating such a meta-metric is a research question of its own. Our paper studies latent representation matching in language models with fixed input–output pairs. When we apply it to layer compression, the problem we must solve is choosing a span whose fixed pairs a flow can learn. For this we propose the Recoupling Ratio as the span-selection criterion, and we validate the choice by training the replacement operator on the selected span and on spans of the same width with higher Recoupling Ratio (Fig. 3 of the paper; the Qwen2.5-1.5B span table above).

Under this problem formulation, endpoint cosine is not a suitable criterion. It scores each pair in isolation, whereas the difficulty of learning a flow with fixed pairs comes from interactions between pairs: crossing trajectories prescribe conflicting velocities at the same point and time. A per-pair score cannot detect crossings, nor tell a large but consistent displacement from a conflicting one.

Despite both RR and end-point cosine are cosines that compare an observed quantity with a reference (see Table below), they meant differently for our problem setting. Endpoint cosine compares each latent with its own endpoint and ignores vector norms and all other samples. The Recoupling Ratio compares the given pairing II with the optimal-transport pairing MM of the whole batch; as both are permutation matrices, cos⁡(vec⁡(M),vec⁡(I))=Tr⁡(M)/N\cos(\operatorname{vec}(M), \operatorname{vec}(I)) = \operatorname{Tr}(M)/N, the fraction of pairs a globally optimal assignment keeps. The two can disagree both ways. If two latents on the same ray exchange endpoints, endpoint cosine is 11, yet the trajectories meet halfway with opposite velocities and OT reassigns both pairs. If every latent is shifted by a common vector, yi=xi+by_i = x_i + b, endpoint cosine can be low, yet nothing crosses and the pairing is already optimal (Recoupling Ratio 00). Because the Recoupling Ratio measures the crossing condition that makes paired flow matching hard, we consider it more appropriate for our setting.

CriterionDefinitionWhat it measuresSees other samplesDetects crossings
Endpoint cosine1N∑icos⁡(xi,yi)\frac{1}{N}\sum_i \cos(x_i, y_i)Average angular alignment of each latent with its own endpointNoNo
Recoupling Ratio1−E[cos⁡(vec⁡(M),vec⁡(I))]=1−E[Tr⁡(M)/N]1-\mathbb{E}[\cos(\operatorname{vec}(M), \operatorname{vec}(I))] = 1-\mathbb{E}[\operatorname{Tr}(M)/N]Expected fraction of the given pairs that the globally optimal assignment changesYesYes
Reviewer_nozoOpenReview ↗
Rating
3
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes replacing a contiguous sequence of Transformer layers in a pre-trained large language model (LLM) with a time-conditioned "latent flow" layer, which maps hidden states from the input of the sequence to its output. During training, the authors employ a "Flow Walking" (FW) approach: they perform kk integration steps starting from randomly sampled time points and conduct backpropagation based solely on the mean squared error (MSE) at the terminal point, thereby avoiding trajectory crossings that would otherwise arise from fixed point-to-point mappings. To identify which layers are suitable for replacement, the authors define a "Recoupling Ratio" (RR)—derived from the "Optimal Transport" (OT) plan between the inputs and outputs of a layer sequence—which quantifies the extent to which the OT coupling deviates from an identity mapping. In their experiments, the authors first perform a layer-selection scan on the Pythia-410M model, targeting four distinct segments, each consisting of 6 Transformer layers. They then validate the compression efficacy using the Pythia-1.4B model, applying distillation on The Pile dataset and evaluating the model’s perplexity on The Pile, as well as its performance across five zero-shot tasks. Key experimental results demonstrate that by replacing layers 6 through 18 of the Pythia-1.4B model—reducing the parameter count to 63% and the computational cost (FLOPs) per token to 72% of the original—the model retains 92% of the teacher model’s accuracy on downstream tasks (57.5% versus 62.8%).

Critical strengths
  1. Flow Walking fixes a real problem, and it produces the strongest result in the paper. When a block of layers maps nearby inputs to outputs that are far apart, the straight-line training targets of standard flow matching conflict with each other. The model then learns an average direction that is wrong for every token involved. Flow Walking avoids this by checking only where each trajectory ends, not the direction at every point along the way. On the span where the conflict is worst (410M layers 0-6), standard flow matching fails (PPL 347.95), while Flow Walking reaches PPL 20.76, also better than the regression baseline at 32.49, though with three network calls against one.

  2. The Recoupling Ratio is a cheap check that runs before any training. It asks: if the inputs and outputs of a block were re-paired to minimize total distance, what fraction of tokens would end up paired with a different output than the one the teacher actually produces? The span with the highest ratio (0-6) is exactly where standard flow matching collapses, and on the zero-ratio spans (6-12, 6-18) it does not.

Critical weaknesses

The paper's results use a flow layer that is run three times for each token. The baseline it is compared against, a single layer trained to copy the removed block's output directly (regression), is run only once. The flow layer is therefore doing about three times as much work at inference (Table 1, p8). Running the same layer more times can improve results by itself, so a better score here may come from the extra runs rather than from flow training. The paper does not try the fair version, in which the baseline layer is also run three times.

The compute savings may also be overstated. The paper says it uses an update rule (Eq. 4, p6) that runs the network twice per step, but the compute numbers in Table 1 appear to count one run per step. If two runs are counted, my estimate is that the compressed model uses about 91% of the original model's compute, not the 72% stated in the abstract.

Questions for authors
  1. Please add a baseline that gets the same compute as the flow layer run three times, for example a shared-weight regression layer trained to be applied three times. Please also report measured inference time, not only estimated compute. If the flow layer still wins under equal compute, my main concern would be resolved.
  2. The paper describes two ways to take each step: one that runs the network once (Eq. 3) and one that runs it twice (Eq. 4). Which one produced the results in Tables 1 and 4? If it is the twice-per-step version, please redo the compute numbers, including the 72% claim in the abstract. If it is the once-per-step version, please correct p6, which says Eq. 4 is used.
  3. The paper argues that reflow cannot be used because it changes the original input-output pairs (p2), and uses this to motivate Flow Walking. Since the method already fine-tunes the whole model afterwards (the repair stage), could reflow followed by one-step compression and repair also work here, as in Layer Compression of Deep Networks with Straight Flows (Gong et al., AAAI 2024)? A short discussion would help clarify what Flow Walking adds.
2 comments
AuthorsOpenReview ↗

Equal-Compute Comparisons, Corrected Compute Accounting, and Flow Walking

We thank the reviewer for taking the time and effort to review the paper. Please see below for our response to the weaknesses and the questions.

W1: Equal-compute recursive regression baseline

The recursive regression baseline is a special case of FW when the time conditioning is removed.

We further conducted a fair comparison where we repeat the regression block r times and train through the full composition. Note that for LFT inference, we follow Eq 4 in the paper (two calls per step). Thus a fair number of recursive regression layers is r=2×kr=2\times k.

Results are shown below. We observed that in Pythia-1.4B, Regression (r=6) slightly improves over LFT (k=3) at equal inference compute. Interestingly, in contrast to results on Pythia-1.4B, on Qwen2.5-1.5B, increasing the inference compute for the recursive regression baseline yields degraded performance. We surmise that this difference may partly arise from the blocks’ residual architectures. Pythia applies attention and the FFN in parallel to the same input, whereas Qwen applies them sequentially, with the FFN receiving the attention-updated state. Repeated composition of a Qwen block therefore introduces additional multiplicative interactions between the attention and FFN Jacobians, potentially making optimization more sensitive to initialization and repeated changes in the hidden-state distribution. FW takes smaller steps toward the block’s predicted output, which may reduce the accumulation of large hidden-state changes across repeated evaluations.

ModelRegression r=1Regression r=6FW k=3
Pythia-1.4B (layer6-18)21.98/53.18%18.13/54.82%20.73/54.30%
Qwen-1.5B (layer5-19)18.07/58.96%18.44/58.72%17.82/59.16%

We will limit our claims and update the results and discussion in the revised manuscript.

W2: Inference compute accounting

Yes, the inference update rule reported is implemented with Eq 4 throughout the paper. Thank you for spotting this embarrassing mistake. We will correct it in the updated manuscript.

Q1: Equal-compute comparison and measured inference time

As per the reply to W2, we report an additional experiment on the shared-weight regression layer at 6 calls to the shared layer for a fair comparison with FW (k=3). We observe the same performance improvement under this setting.

As to the inference time of the methods, please see below.

Parameters (%)PrefillDecode per tokenGreedy 128-token requestKV at end
Teacher (24 layers)100%74.1 ms12.60 ms1675 ms239.6 MiB
LFT-FW k = 3 (replaces L6–L18)63.0%55.8 ms10.41 ms1378 ms179.7 MiB
LFT-FW k = 1 (replaces L6–L18)63.0%42.1 ms8.04 ms1064 ms139.8 MiB
Regression r = 1 (replaces L6–L18)63.0%38.6 ms7.37 ms974 ms129.8 MiB
ShortGPT, 11 layers removed60.8%38.4 ms6.92 ms918 ms129.8 MiB

We will include these in a separate table in the revised manuscript.

Q2: Inference update rule and compute correction

We follow the standard implementation in taking the midpoint when solving the ODE, i.e. Eq. 4. As per our reply to W2, we will correct the compute numbers accordingly.

Q3: Reflow and the motivation for Flow Walking

The mentioned paper employs a standard flow matching objective to learn the transport of latent distributions between layers. This is the same as the SFM setting reported in our paper. In the case of learning latents in transformer language models, the input-output pairing relationship needs to be preserved, while SFM encourages distribution-level matching, it suffers from crossing trajectories in paired data (as discussed in Sec 4.). The proposed FW algorithm incorporates the numerical integration with discrete time points to allow the degrees of freedom that disambiguate the crossing trajectories.

Thank you for the response and the additional experiments. The equal-compute results show that FW does not clearly outperform recursive regression at matched inference compute, so the compute-savings claims should be revised and the FLOPs corrected for Eq. 4. I still find the analysis of trajectory crossing in paired latent matching, and the clear improvement of FW over standard flow matching, a useful contribution. I keep my score.

Reviewer_m4yvOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes Latent Flow Transformer (LFT), which compresses a pretrained language model by replacing consecutive Transformer layers with a shared, time-conditioned operator applied over several integration steps. It introduces the Recoupling Ratio to identify suitable replacement spans and Flow Walking to train the operator through endpoint reconstruction, preserving the teacher’s input–output correspondence.

Critical strengths
  • The method presents an interesting approach to reducing parameter count by replacing multiple Transformer layers with a shared transport operator. This aligns with the motivation behind looped architectures: increasing effective computational depth while reusing parameters.

  • The main methodology is generally explained clearly, making the central ideas accessible. However, the presentation of several figures could be improved.

Critical weaknesses
  • Several figures are difficult to interpret because of their small size and crowded layout. For example, the text in Figure 6 is densely packed, while Figure 2 is too small to inspect the architectural details comfortably. Larger labels, clearer annotations, and improved spacing would help.

  • The paper does not explain how KV caching is supported during autoregressive decoding, particularly how cached keys and values are maintained across different flow steps. Since KV caching is important for practical deployment, the authors should clarify the implementation and report decoding latency, throughput, and cache memory usage. Parameter and FLOP reductions alone do not establish practical inference gains.

  • The experiments do not systematically examine performance across different sequence lengths. The reported training setup uses a maximum sequence length of 2,048 tokens, but it remains unclear how approximation quality and inference efficiency vary with context length. Evaluating multiple lengths within the supported context window would strengthen the practical claims.

  • Evaluation is limited to Pythia-410M and Pythia-1.4B. These experiments provide initial evidence, but do not establish scalability to substantially larger models or transferability to other architectures. Experiments on a larger model or a different model family would strengthen the conclusions.

  • The generality of the Recoupling Ratio as a layer-selection criterion remains insufficiently established. Both evaluated models have 24 layers, leaving its behavior across different network depths unexplored. It would be useful to test whether the metric consistently identifies suitable replacement spans across architectures and depths, and whether its estimates are stable across sampled token sets.

  • The method involves several interacting configuration choices, including the replacement boundaries (m,n), the training integration count (k_{\text{train}}), and the inference integration count (k_{\text{infer}}). Although the paper includes some ablations, stronger evidence on sensitivity and transferable default settings would help assess the tuning burden when applying the method to new models.

Minor comments:

  • The discussion should clearly acknowledge the closely related contemporaneous contributions of flow matching by Lipman et al., rectified flow by Liu et al., and stochastic interpolants by Albergo and Vanden-Eijnden. Although all three are cited in Section 2, the bibliography needs correction. In particular, the flow-matching paper is by Yaron Lipman, Ricky T. Q. Chen, Heli Ben-Hamu, Maximilian Nickel, and Matt Le. The rectified-flow paper is by Xingchao Liu, Chengyue Gong, and Qiang Liu, titled “Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flow,” with arXiv identifier 2209.03003.

To the authors: I used an AI tool to help analyze the manuscript and improve my understanding. I formulated my own human-based concerns and used the tool to polish their wording. I carefully reviewed the revised text to ensure that it accurately reflected my original concerns.

Questions for authors

see weakness

1 comment
AuthorsOpenReview ↗

Clarifications on Inference Efficiency, Model Generalization, and Layer Selection

We thank the reviewer for taking the time and effort to review the paper. Please see below for our response to the weaknesses and the questions.

W1: Figure readability

We will enlarge labels and rearrange the layouts of Figure 2 and Figure 6 to make these more reader-friendly.

W2: KV caching and practical inference performance

Regarding KV caching for autoregressive generation, each retained layer keeps its own cache, and the shared flow layer keeps one cache per solver stage, because the same weights see different inputs at each stage. The time embedding is cached beforehand to reduce overheads. We report below a measurement on Pythia-1.4B with the setting: prefill 512 tokens, generate 128 tokens, B = 1, and on one A6000.

# called layers per tokenPrefillDecode per token128-token requestKV(slots)
Teacher2474.1 ms12.60 ms1675 ms239.6 MiB(24)
ShortGPT (removed 11 layers)1338.4 ms6.92 ms918 ms129.8 MiB(13)
Regression (removed L6-18)1338.6 ms7.37 ms974 ms129.8 MiB(13)
FW (k=1, removed L6-18)1442.1 ms8.04 ms1064 ms139.8 MiB(14)
FW (k=3, removed L6-18)1855.8 ms10.41 ms1378 ms179.7 MiB(18)

We will include the practical benefits and limitations of our proposed method in the revised manuscript.

W3: Sequence-length evaluation

We evaluated the quality reduction in NLL with respect to the teacher at each respective sequence length for the Pythia-1.4B checkpoint.

12851210242048
teacher-relative ΔNLL of repaired LFT0.4860.4810.4730.471

We observe that quality loss is stable across the supported window. Further testing beyond the 2048 sequence length at longer contexts requires a substantial adaptation of the base models (Pythia) through long-context post-training which is beyond the scope of this work. We will include a discussion on this in our revised manuscript.

W4: Larger models and other architectures

We observe that the advantage of LFT persists as we extend evaluations on Qwen2.5 and compare it against additional baselines in layer pruning.

We evaluate our proposed framework and algorithm on Qwen2.5 1.5B and 7B models. We follow the same procedure as described in the paper to select the layers. For the 1.5B model, the span from layer 5 - 19 is selected; for the 7B model, the span from layer 7 - 21 is selected. Furthermore, we report comparisons against two additional layer pruning baselines - ShortGPT and SLEB.

Qwen2.5-1.5B

RowOperator+recovery tokensParamPPL/acc
Teacher-1.544B8.52/71.00%
FW (layer5-19, k3)20M+20M952M13.01/62.08%
Regression (layer5-19)20M+20M952M13.23/61.72%
Skip (layer5-19)0+40M889M14.16/59.50%
ShortGPT (12 layers)0+40M982M14.48/59.46%
SLEB (12 layers)0+40M982M12.84/60.64%

Qwen2.5-7B

RowOperator+recovery tokensParamPPL/acc
Teacher-7.616B6.85/76.34%
FW (layer7-21, k=3)20M+20M4.68B10.15/65.20%
Regression (layer7-21)20M+20M4.68B10.35/64.16%
Skip (layer7-21)0+40M4.35B11.37/63.14%
ShortGPT (12 layers)0+40M4.82B10.73/64.78%
SLEB (12 layers)0+40M4.82B10.25/65.10%

W5: Recoupling Ratio across architectures and depths

We observed that the layer selection criterion holds when evaluated on Qwen2.5-1.5B (28 transformer layers). We trained the same methods on spans of the same width with different RR:

SpanMean RFW k3 PPLSFM k3 PPLregression PPLskip PPL
[5,19) (selected)0.0017.821.018.169.6
[14,28)0.0317356017623,492
[0,14)0.9719,6524,2932,7023.2 × 10⁶

W6: Configuration sensitivity and transferable defaults

Our proposed method involves principled ways to identify and distill transformer layers into an LFT layer. It starts with identifying layers through selecting layer spans with low recoupling ratio, followed by fine-tuning with FW, and finishes with repair through language modelling fine-tuning.

Figure 7 of the paper shows how performance varies as kinferk_{\text{infer}} changes under a fixed ktraink_{\text{train}} while the result tables above show the applicability of our method across architectures.

We will clarify these as we update our manuscript.

MC1: Bibliography corrections

We thank the reviewer for pointing out these embarrassing mistakes and we will correct these in our updated manuscript.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.