- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper proposes the Latent Flow Transformer (LFT), a structural compression framework for Large Language Models (LLMs) that replaces a contiguous block of Transformer layers with a single continuous transport operator trained via Flow Matching. To systematically identify which layers are suitable for replacement, the authors introduce the Recoupling Ratio, an Optimal Transport-based metric that quantifies the misalignment between latent pairs across layers, revealing that middle layers are highly compressible. Furthermore, to address the "trajectory crossing" problem inherent in standard Flow Matching when applied to deterministic paired data, they propose Flow Walking (FW), a numerical integration algorithm that preserves latent pair coupling. Experiments on Pythia-1.4B show that LFT-FW can replace half of the model's layers (12 out of 24) with a single operator, retaining 92% of the teacher's downstream accuracy while reducing parameters to 63% and FLOPs to 72%.
Critical strengths
-
Highly Novel and Interdisciplinary Perspective: Bridging continuous-time dynamical systems and Flow Matching (typically dominant in image/video generation) with the discrete layer-wise forward pass of LLMs is a highly creative approach. Framing Transformer layers as discretized ODEs and compressing them into a single learned transport operator offers a fresh theoretical and practical perspective on structural model compression.
-
Insightful Diagnostics and Algorithmic Solutions: The introduction of the Recoupling Ratio provides a theoretically grounded, interpretable metric to diagnose layer redundancy, successfully validating the hypothesis that middle layers act as compressible "routing" spaces. Additionally, identifying the trajectory crossing issue in paired latent data and solving it via the Flow Walking (FW) algorithm shows a deep understanding of the limitations of standard Flow Matching when applied to deterministic sequence modeling.
Critical weaknesses
-
Lack of Generalization: The entire empirical validation is restricted to the Pythia model family (410M and 1.4B), which uses an older architecture (GPT-NeoX) and was trained on "The Pile". Modern LLMs employ significantly different architectural components (e.g., RoPE, SwiGLU, Grouped-Query Attention, RMSNorm) and are trained on vastly superior, deduplicated corpora. The latent dynamics and redundancy distributions in Pythia may not reflect those in modern LLMs.
-
"Insufficient and Weak Baselines": The paper only compares LFT against naive Layer-Skipping and a basic Regression baseline (a single Transformer layer trained with MSE loss). It completely ignores the rich landscape of state-of-the-art LLM compression, structured pruning, and distillation techniques. For instance, the paper cites ShortGPT (Men et al., 2024) regarding layer redundancy but fails to include it as an empirical baseline. Other relevant methods like LLM-Pruner, Wanda, or block-wise knowledge distillation methods are completely absent.
Questions for authors
-
Does the Recoupling Ratio reliably identify compressible spans in modern architectures (e.g., Llama-3, Mistral, Qwen) that utilize RoPE and SwiGLU? Or at least, can you provide any evidence or theoretical justification that Flow Walking will scale effectively to 7B+ parameter models without suffering from optimization collapse or requiring prohibitive training FLOPs?
-
Why were advanced structured pruning methods (e.g., ShortGPT, LLM-Pruner) and modern knowledge distillation techniques excluded from the baselines? How does LFT-FW compare to these SOTA pruning methods in terms of the trade-off between compression ratio, downstream accuracy retention, and the total training compute required?
3 comments
Qwen2.5 Exp Improved Generalization Evidence , but Flow Walking’s Compute–Accuracy Tradeoff Remains Unclear
Thank you for the additional experiments. The Qwen2.5-1.5B and 7B results address my concern about generalization beyond Pythia, and the ShortGPT and SLEB comparisons strengthen the empirical evaluation.
My main remaining concern is the benefit of Flow Walking at comparable inference cost. On Pythia-1.4B, three-step FW and regression have the same parameter count, but after KL repair, FW improves accuracy from 57.1% to 57.5% while increasing per-token FLOPs by about 20% and measured decoding latency by about 41%. Single-step FW achieves accuracy comparable to regression. On Qwen2.5-7B, FW is competitive with SLEB (65.20% vs. 65.10% accuracy), but the rebuttal does not report inference latency or total training compute for that comparison. A useful control would be to distill a shared-weight Transformer block applied repeatedly, using the same endpoint MSE and matching parameter count and inference compute. This would help isolate the benefit of the flow formulation from the benefit of additional block evaluations.
The new span results also support the Recoupling Ratio as a selection heuristic. However, the selected span is easier to compress for all tested methods, including regression and direct skipping. A comparison against a simpler criterion such as endpoint cosine similarity would clarify what Recoupling Ratio adds.
The rebuttal resolves enough of my original concerns that I would raise Soundness from 1 to 2. I maintain my weak reject recommendation because the current evidence leaves the incremental benefit of Flow Walking at comparable compute unclear.
Scope of the paper, qual-compute shared-weight regression, and why Recoupling Ratio rather than endpoint cosine
Thank you for the follow-up.
We would like to emphasize that our paper studies latent representation matching in language models with fixed input–output pairs. In standard flow matching (SFM), trajectories from different pairs can meet at the same time and prescribe conflicting velocities. We introduce Flow Walking (FW) and show that it reduces the impact of such crossings relative to SFM. We apply it to layer compression, using the Recoupling Ratio to select a span and FW to train its replacement operator; there, LFT-FW outperforms LFT-SFM.
On shared-weight regression. We agree this is an important baseline. In our response to reviewer nozo, we report a regression baseline trained end-to-end through six applications of the same block, matching the six block evaluations of three-step FW. This recursive regression can be seen as a special case of FW without time conditioning.
On Pythia-1.4B, Regression (r=6) slightly improves over LFT (k=3) at equal inference compute. In contrast, on Qwen2.5-1.5B, more inference compute degrades the recursive regression baseline. We surmise that this difference may partly arise from the blocks’ residual architectures. Pythia applies attention and the FFN in parallel to the same input, whereas Qwen applies them sequentially, with the FFN receiving the attention-updated state. Repeated composition of a Qwen block therefore introduces additional multiplicative interactions between the attention and FFN Jacobians, potentially making optimization more sensitive to initialization and repeated changes in the hidden-state distribution. FW takes smaller steps toward the block’s predicted output, which may reduce the accumulation of large hidden-state changes across repeated evaluations.
| Model | Regression r=1 | Regression r=6 | FW k=3 |
|---|---|---|---|
| Pythia-1.4B (layer6-18) | 21.98/53.18% | 18.13/54.82% | 20.73/54.30% |
| Qwen-1.5B (layer5-19) | 18.07/58.96% | 18.44/58.72% | 17.82/59.16% |
Comparing span-selection criteria is beyond the scope of the paper: Comparing span-selection criteria requires a meta-metric that scores the criteria themselves, e.g. the rank correlation between a criterion's score and post-compression quality over all candidate spans, under a fixed operator, training budget and evaluation task. Defining and validating such a meta-metric is a research question of its own. Our paper studies latent representation matching in language models with fixed input–output pairs. When we apply it to layer compression, the problem we must solve is choosing a span whose fixed pairs a flow can learn. For this we propose the Recoupling Ratio as the span-selection criterion, and we validate the choice by training the replacement operator on the selected span and on spans of the same width with higher Recoupling Ratio (Fig. 3 of the paper; the Qwen2.5-1.5B span table above).
Under this problem formulation, endpoint cosine is not a suitable criterion. It scores each pair in isolation, whereas the difficulty of learning a flow with fixed pairs comes from interactions between pairs: crossing trajectories prescribe conflicting velocities at the same point and time. A per-pair score cannot detect crossings, nor tell a large but consistent displacement from a conflicting one.
Despite both RR and end-point cosine are cosines that compare an observed quantity with a reference (see Table below), they meant differently for our problem setting. Endpoint cosine compares each latent with its own endpoint and ignores vector norms and all other samples. The Recoupling Ratio compares the given pairing with the optimal-transport pairing of the whole batch; as both are permutation matrices, , the fraction of pairs a globally optimal assignment keeps. The two can disagree both ways. If two latents on the same ray exchange endpoints, endpoint cosine is , yet the trajectories meet halfway with opposite velocities and OT reassigns both pairs. If every latent is shifted by a common vector, , endpoint cosine can be low, yet nothing crosses and the pairing is already optimal (Recoupling Ratio ). Because the Recoupling Ratio measures the crossing condition that makes paired flow matching hard, we consider it more appropriate for our setting.
| Criterion | Definition | What it measures | Sees other samples | Detects crossings |
|---|---|---|---|---|
| Endpoint cosine | Average angular alignment of each latent with its own endpoint | No | No | |
| Recoupling Ratio | Expected fraction of the given pairs that the globally optimal assignment changes | Yes | Yes |
Generalization to Qwen2.5 and Comparisons with Pruning Baselines
We thank the reviewer for taking the time and effort to review the paper. Please see below for our response to the weaknesses and the questions.
W1: Generalization to modern model architectures
We observe that the advantage of LFT persists as we extend evaluations on Qwen2.5 and compare it against additional baselines in layer pruning. Note that Qwen2.5 is a modern architecture, using RoPE, SwiGLU, Grouped-Query Attention, and RMSNorm as mentioned by the reviewer.
We evaluate our proposed framework and algorithm on Qwen2.5 1.5B and 7B models. We follow the same procedure as in the paper to select the layers. For the 1.5B model, the span from layer 5 - 19 is selected; for the 7B model, the span from layer 7 - 21 is selected. Furthermore, we report comparisons against two additional layer pruning baselines - ShortGPT and SLEB.
Qwen2.5-1.5B
Qwen2.5-7B
We will include these results in the revised manuscript.
W2: Additional pruning baselines
As mentioned in the table at W1, we will also include these additional comparisons along with pruning baselines such as ShortGPT and SLEB. Empirical validation shows that LFT also outperforms these two baselines.
Q1: Recoupling Ratio and scalability
We observed that the layer selection criterion based on Recoupling Ratio holds when evaluated on Qwen2.5-1.5B (28 transformer layers). We trained the same methods on spans of the same width with different RR:
As to evaluation on 7B+ models, we tested the method on Qwen2.5-7B and observe the same findings as that at 1.5B model (see the Table above in W1).
Experiments at scales beyond 7B remain future work. However, we surmise that our method still holds as modern LLM layers are known to be highly redundant [1,2].
[1] W. Sun et al., The Curse of Depth in Large Language Models [2] A. Gromov et al., The Unreasonable Ineffectiveness of the Deeper Layers
Q2: Compression, accuracy, and training-compute trade-offs
As shown in the tables at W1, we include comparisons with pruning methods such as ShortGPT and SLEB. We observe that FW outperforms these methods while requiring higher decoding latency due to a number of inference evaluations (see the table below). The training cost for ShortGPT and SLEB at the repairing stage is lower than that for FW or regression as they do not require training of the compressed layers.