- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes a very interesting idea: instead of compressing the weights, keep the model intact and "compress" the inputs. In addition to this approach, the authors also introduce Compressed CP-Conv (CCP-Conv) in order to prevent producing dense outputs. They test their models on different tasks and models, showing significant speed up and computational efficiency gains.
Critical strengths
The idea is interesting and may prove useful for speeding up expensive workloads, such as training and inference.
Keeping the model intact allows preserving the expressivity of the kernels of the models, in contrast with methods that compress the model. Moreover compressing the inputs and compressing the models could be used in a complementary way.
For training in particular, the cost of decomposing the inputs can be done once and then amortized over different epochs or even on different models trained on the same data. This is a valuable property as one can for instance "compress" large video or scientific datasets once, and use the obtained representation as often as needed.
Critical weaknesses
- One of the important weakness, in my view, is the lack of an end to end comparison for data that is not natively in CP form. The authors provide training and inference cost comparison that excludes the cost of the compression of the inputs, which is reported separately in the appendix and that appears high.
For example, a naive rough extrapolation for UCF101 (101 classes):
Cost per sample from your table 12: 10.944
Appendix 1.3 dataset size: 10,160/3,160 training/test samples
Table 3 training time: 200 for ccp, 4200 for dense
This leads to ~145774s, ~40 hours, approximately 34 times more costly than dense training (~1.2 hours).
-
The comparisons are evaluated on settings where the absolute performance is weak (for example, UCF101 101 cls has 9.5% accuracy. For FP-035, 3 cls, the accuracy of CCP is 38% relatively to 33% chance.) This makes the claim of "comparable to dense" weaker and less informative. In particualr, in the one setting setting with good accuracy (R3D-18, 84% of accuracy), the speedup shrinks to about 5% (not considering input compression cost) and requires a rank of R = 300 (whose time is not reported in Table 12).
-
The native CP dataset comparison lacks natural baselines. A simple model like a mlp on the factors themselves would have been a relevant baseline.
Questions for authors
Could you provide a end to end wall-clock comparison of the different methods? And provide a discussion on the advantages, drawbacks and practical recommendations.
One of the motivations of this approach is to keep the model weights intact to preserve its expressivity. Could you also provide more insights on the possible drawback, ie. the impact of input fidelity loss by training on compressed inputs? In particular, practical suggestion on how to choose the rank and the impact of large decomposition error?
In Appendix A, you state that "The matched dense baseline deliberately uses the same unstrided convolution and output-compression structure to isolate this difference. More aggressively optimized dense pipelines using striding or pooling could reduce the absolute runtime gap". Could you provide comparisons with the actual optimized pipeline to evaluate if the advantage persists and to evaluate if the gap does not come from this choice?
6 comments
Author Response (2/3): Predictive Performance and FP Baselines
Weakness 2: Predictive performance and realistic deep architectures
We agree that demonstrating computational speedups on tasks with relatively low classification accuracy is less informative than demonstrating comparable accuracy on more challenging applications.
Accordingly, we conducted additional experiments to strengthen the predictive evaluation.
First, we repeated our R3D-18 experiments on the complete UCF101 dataset using the standard stem stride of (1, 2, 2) for both CCP-Conv and dense convolution using optimized PyTorch code.
Both models were trained for 30 epochs under matched configurations and evaluated across three random seeds.
The results are:
| Metric | CCP-Conv | Dense Conv |
|---|---|---|
| Test accuracy (%) | 84.63 ± 0.24 | 85.20 ± 0.79 |
| Training time per epoch (s) | 1,860 ± 128 | 2,488 ± 43 |
| Total training time (hours) | 15.50 | 20.73 |
Under this stride-matched comparison, CCP-Conv achieves comparable classification accuracy (84.63% versus 85.20%), while providing approximately 1.34× faster training, corresponding to a 25.2% reduction in training time.
This demonstrates that the computational advantage persists when comparing models with standard convolution strides, rather than relying on the original unstrided dense baseline.
We acknowledge that the training-time advantage is smaller for this deeper architecture than for the shallow classification experiments. Nevertheless, it remains meaningful while preserving similar predictive performance.
We also conducted additional experiments on high-dimensional native CP data, as described below.
Weakness 3: Stronger factor-native baselines
We agree that the original Fokker–Planck experiments lacked sufficiently strong factor-native baselines.
Accordingly, we expanded the experiments to include linear and RBF Ridge classifiers, log-variance Ridge features, and MLPs trained directly on the CP factors.
Furthermore, we increased the FP-05 dataset from 1,000 to 20,000 samples and adopted a longer training budget.
We also introduced two additional seven-dimensional datasets designed to test more challenging dependencies:
- FP-HO: A dataset involving high-order interactions.
- FP-Product: A dataset involving nonlinear products of underlying parameters.
The five-seed test results (mean ± standard deviation, in %) are presented below:
| Method | FP-05 | FP-HO | FP-Product |
|---|---|---|---|
| CCP-Conv | 93.47 ± 3.45 | 100.00 ± 0.01 | 93.80 ± 1.28 |
| Factor MLP | 99.83 ± 0.09 | 85.17 ± 1.34 | 62.63 ± 0.75 |
| Linear Ridge | 71.30 ± 0.91 | 75.12 ± 0.36 | 33.92 ± 0.72 |
| RBF Ridge | 70.86 ± 1.23 | 79.27 ± 0.47 | 47.10 ± 0.58 |
| Log-variance Ridge | 99.89 ± 0.04 | 33.30 ± 0.86 | 37.51 ± 0.73 |
We acknowledge that simple statistical features outperform CCP-Conv on the original FP-05 problem.
However, CCP-Conv outperforms all evaluated factor-native baselines on FP-HO and FP-Product, demonstrating that our approach can achieve meaningful predictive performance on more complex high-dimensional problems.
We will include these additional benchmarks and explicitly discuss the limitations of the original FP experiments.
Q1: End-to-end wall-clock comparisons and practical recommendations
We agree that the original distinction between preprocessing costs and model-training times was insufficient for understanding the practical advantages of our approach.
We have addressed this question in Weakness 1, where we provide additional preprocessing benchmarks, estimated end-to-end costs, and amortization calculations.
We will also clarify the three application scenarios where we expect our method to be most relevant: native CP data, memory-constrained training, and repeated training on decomposed datasets.
Author Response (3/3): Input Fidelity and Optimized Dense Pipelines
Q2: Input fidelity loss and CP rank selection
We agree that preserving the expressivity of the convolutional kernel does not eliminate approximation errors introduced by compressing dense inputs into CP form.
To investigate the effect of these approximation errors, we conducted rank ablations on UCF101.
In our original R3D-18 rank-ablation experiment, increasing the CP rank improved classification accuracy:
| CP rank | Test accuracy (%) |
|---|---|
| 100 | 77.0 ± 1.0 |
| 220 | 83.0 ± 0.4 |
| 300 | 84.8 ± 0.5 |
These results illustrate the trade-off between input compression and predictive performance. Increasing the rank generally preserves more information but also increases factor storage, computational cost, and preprocessing requirements.
For example, in our UCF101 rank-150 preprocessing experiment, the mean relative reconstruction error was approximately 5.8 ± 2.76%.
In practice, we recommend selecting the CP rank by examining the relative reconstruction error. For each candidate rank, a representative subset of samples can be decomposed, and the lowest rank satisfying a predefined reconstruction-error tolerance can be selected. As a practical starting point, we propose a tolerance of 4-6%, although the appropriate threshold depends on the dataset and downstream task.
Our rank-ablation experiments suggest that classification accuracy can eventually saturate as the CP rank increases. However, the corresponding reconstruction-error threshold is task-dependent, and sometimes relying on validation accuracy as well can be useful.
We have added a detailed discussion of how the CP rank was selected for each experiment to the appendix of the revised paper.
Q3: Comparison with optimized dense pipelines
We agree that the original matched dense baseline was primarily designed to isolate the computational difference between dense convolution and CCP-Conv, rather than represent the fastest possible dense architecture.
Accordingly, we conducted additional comparisons against more optimized dense pipelines, including spatiotemporal resizing, temporal subsampling, patch pooling, and truncated SVD.
For UCF101-10, the revised five-seed results are:
| Method | Training time (s) | Test accuracy (%) |
|---|---|---|
| CCP-Conv | 6.64 | 32.72 ± 2.89 |
| Dense Conv | 155.86 | 32.06 ± 2.84 |
| Spatiotemporal resize | 28.72 | 32.30 ± 2.14 |
| Temporal subsampling | 14.60 | 32.24 ± 3.00 |
| Patch pooling | 28.48 | 33.31 ± 2.69 |
| Truncated SVD | 157.76 | 31.76 ± 1.42 |
All methods were evaluated using a consistent PyTorch training implementation, with preprocessing costs excluded from the reported training times.
The results show that more aggressively reduced dense representations can substantially decrease training costs while maintaining similar predictive accuracy.
Nevertheless, CCP-Conv remains approximately 2.20× faster than temporal subsampling, the fastest alternative in this comparison, while achieving comparable classification accuracy.
Furthermore, as discussed in Weakness 2, our additional R3D-18 experiment uses the same standard stride (1, 2, 2) for CCP-Conv and the dense comparator. Under these matched conditions, CCP-Conv retains a 25.2% training-time reduction while achieving similar classification accuracy.
We acknowledge that the advantage of CCP-Conv depends on the architecture and application, and is smaller for some optimized dense pipelines than for the original matched dense baseline.
We will include these comparisons in the revised manuscript and clarify the distinction between operator-level efficiency, training efficiency, and complete end-to-end computational cost.
We thank the reviewer again for their observations, which helped us strengthen the experimental evaluation and clarify the practical advantages and limitations of our approach.
We hope that the additional benchmarks, stronger baselines, rank ablations, and end-to-end cost analysis adequately address the concerns raised in the review.
Author Response (1/3): CP Decomposition and End-to-End Efficiency
We thank the reviewer for the constructive review and for recognizing the role of input compression, unrestricted kernels, and preprocessing amortization.
Due to the character limit, our response is divided into three parts: (1) CP decomposition and end-to-end costs; (2) predictive performance and factor-native baselines; (3) input fidelity, rank selection, and optimized dense pipelines.
Weakness 1: CP decomposition cost and end-to-end efficiency
We agree with the reviewer that the cost of decomposing dense inputs into CP form must be considered when evaluating the end-to-end computational efficiency of our approach.
We would like to emphasize that our approach is particularly relevant in three situations:
Native CP data: Applications where the data already exist in CP form, eliminating the decomposition cost entirely. This is particularly relevant to high-dimensional scientific applications, such as our Fokker–Planck experiments.
Memory-constrained training: Applications where the dense representation is too large to be processed efficiently on the available hardware. For example, our high-resolution Ego4D experiment involves videos of size
1920 × 2560 × 128, corresponding to approximately 2.34 GiB per video in float32. With a batch size of 20, the dense inputs alone would require approximately 46.9 GiB, before accounting for convolutional outputs, gradients, and intermediate activations. The combined memory requirements make dense training infeasible under our experimental hardware constraints, whereas CCP-Conv allows training directly on the compressed factors.Repeated training: Applications where the same decomposed dataset is reused across multiple training runs, allowing the one-time decomposition cost to be amortized.
In the first two scenarios, CP-Conv enables computations that would otherwise be infeasible with conventional dense convolution under the available computational resources, whereas the third scenario offers a potential computational advantage through the amortization of preprocessing costs.
To investigate the third scenario, we conducted additional experiments measuring the effect of parallelizing CP decomposition at rank 300, corresponding to the rank used in our R3D-18 experiments.
We benchmarked decomposition with different numbers of parallel CPU workers and evaluated its amortization against the revised stride-matched R3D-18 training times on the complete UCF101 dataset (13,320 videos).
In this experiment, dense R3D-18 training requires 20.73 hours, compared with 15.50 hours for CCP-Conv, corresponding to a saving of approximately 5.23 hours per training run.
The preprocessing costs and corresponding break-even points are:
Break-even values represent repeated complete R3D-18 training runs using cached CP factors.
These results demonstrate that parallelization substantially reduces the preprocessing wall-clock time. With eight workers, the estimated cost of decomposing the complete UCF101 dataset decreases from approximately 41.6 hours to 8 hours.
More importantly, given the approximately 5.23 hours saved per R3D-18 training run, the preprocessing cost can be amortized after only two complete training runs when using eight workers.
We acknowledge that CP decomposition remains computationally expensive and can cost more than the training time saved, particularly when training only once on a dataset that initially exists in dense form.
Nevertheless, decomposition costs do not affect applications with native CP representations, and memory savings can enable training configurations that are infeasible with dense inputs on the available hardware. For deeper architectures such as R3D-18, the substantial savings in training time also make amortization significantly more favorable.
Finally, we emphasize that these break-even estimates are illustrative rather than an exhaustive optimization of the preprocessing pipeline. They combine measured decomposition throughput with training benchmarks and depend on the execution environment and parallelization strategy. Alternative CP decomposition algorithms, optimized implementations, and improved parallelization strategies could further reduce preprocessing costs and change these estimates.
We will include these preprocessing experiments and amortization analyses in the appendix. A systematic investigation of decomposition algorithms, parallelization strategies, and their impact on end-to-end efficiency represents an interesting direction for future work.